Reviewing the status of USPTO’s crashed systems

Today I review several aspects of USPTO’s massive system crash that began two days ago on December 22, 2015.  (In other blog articles I discuss the “deeming” remedy that USPTO has proposed, and redundancy measures that USPTO needs to bring about urgently, as well as the administrative steps the USPTO needs to do (and has not yet done) to remediate the massive system crash.)

In this article I discuss the present and recent status of USPTO’s crashed systems.

The crash began on the afternoon of Tuesday, December 22, 2015.  We first noticed it when EFS-Web started to get flaky.  I was in the middle of trying to e-file a design patent application, and EFS-Web crashed shortly before I was going to click “submit”.  Others in our office noticed that PAIR was broken and that they were unable to e-file trademark applications and trademark responses in TEAS.

I posted a query on the EFS-Web listserv, wondering if it was “just me”.  No, it was not just me, I learned.  And soon it became clear that every external-facing e-commerce server at the USPTO was broken.  This included EFS-Web (normal and contingency), PAIR (public and private), TEAS (for e-filing trademark documents), EPAS and ETAS (for recording assignments), RAM (for paying maintenance fees), Financial Profile, AOTW (assignments on the web), ESTTA (e-filing of TTAB papers), OEMS (for ordering physical certified copies), PATFT and APPFT (patent full-text searching), TESS (trademark searching), the ID Manual, TSDR (trademark file inspection and status), and TTABVUE (TTAB proceeding file inspection), and the interference web portal.

A crash this massive, affecting every external-facing USPTO system except the main USPTO web site, has never been seen before.  It took hours and hours for USPTO to make any public comment, but eventually USPTO attributed this massive failure to a “power outage” that had “damaged” USPTO servers.

In a separate post I express skepticism as to how a well designed system could lead to “damage” to servers due to nothing more than a mere “power outage”.  In a well designed system, any server that has been switched to battery power rather than normal electrical power will shut itself down gracefully.  When power is restored, the server can be restarted as it normally would.  (In our office, after such a shutdown due to loss of electrical power, the servers actually restart automatically when power is restored.)

But it sort of does not matter what common sense might suggest about how long it ought to take to get things working after the power is turned back on.  What matters, on a practical level, is when the various servers actually returned to normal.

It is now 48 hours after the massive system crash began.  Which systems are working now?

This morning, alert list member Chico Gholz reported that the interference web portal is working again.  (He also reports that “Judge McKelvey put a lump of coal in someone’s stocking in 106023.”)

List member Mike Ervin reported that EPAS is working again.  Given that EPAS and ETAS are essentially the same thing, I’d guess this means that ETAS is also working again.

List member Ken Boone reports that TESS is working again, but as of earlier today it was not completely up to date.

List member Kevin Grierson reported that he was able to run Feathers today.  Meaning that at least some of the functionality of TSDR is back to normal.  But as Greg William pointed out, the “documents” function of TSDR was still broken as of earlier today.  Terry Carroll reported that the TSDR API was still broken.

Earlier today Private PAIR was still broken.  But just now (about 48 hours after the start of the outage) Private PAIR seems to be working again.  It has correspondence that was mailed early in the morning of Wednesday, December 23, but it does not contain any correspondence that was mailed early in the morning of Thursday, December 24.  (Maybe there was no correspondence mailed early in the morning of December 24, because maybe no work got done at the USPTO on December 23.)

EFS-Web is still broken — both the normal server and the contingency server.

TEAS is still broken.

RAM (for paying maintenance fees) is working again.

ESTTA (e-filing of TTAB papers) is still broken.

AOTW (assignments on the web) seems to be working normally.

PATFT and APPFT (patent full-text searching) are working normally, except that APPFT is a week behind, as it is missing the publications for December 24.

TTABVUE (TTAB proceeding file inspection) is working normally.

OEMS (for ordering physical certified copies) is working.

Financial Profile has a message saying that it is working intermittently.

The ID Manual seems to be working, albeit slowly.