Ant-Like Persistence

Today I will post several blog posts relating to the USPTO’s system crash.  In a first article I will review the status over the past few days of some of USPTO’s systems.  In a second article I will discuss USPTO’s proposed remedy of “deeming”several days to be federal holidays.   In a third article I will review the administrative remedies the USPTO needs to carry out. And in this article I will discuss the importance of redundancy. Continue reading “The need for redundancy”

Today I review several aspects of USPTO’s massive system crash that began two days ago on December 22, 2015.  (In other blog articles I discuss the “deeming” remedy that USPTO has proposed, and redundancy measures that USPTO needs to bring about urgently, as well as the administrative steps the USPTO needs to do (and has not yet done) to remediate the massive system crash.)

In this article I discuss the present and recent status of USPTO’s crashed systems.

The crash began on the afternoon of Tuesday, December 22, 2015.  We first noticed it when EFS-Web started to get flaky.  I was in the middle of trying to e-file a design patent application, and EFS-Web crashed shortly before I was going to click “submit”.  Others in our office noticed that PAIR was broken and that they were unable to e-file trademark applications and trademark responses in TEAS.

I posted a query on the EFS-Web listserv, wondering if it was “just me”.  No, it was not just me, I learned.  And soon it became clear that every external-facing e-commerce server at the USPTO was broken.  This included EFS-Web (normal and contingency), PAIR (public and private), TEAS (for e-filing trademark documents), EPAS and ETAS (for recording assignments), RAM (for paying maintenance fees), Financial Profile, AOTW (assignments on the web), ESTTA (e-filing of TTAB papers), OEMS (for ordering physical certified copies), PATFT and APPFT (patent full-text searching), TESS (trademark searching), the ID Manual, TSDR (trademark file inspection and status), and TTABVUE (TTAB proceeding file inspection), and the interference web portal.

A crash this massive, affecting every external-facing USPTO system except the main USPTO web site, has never been seen before.  It took hours and hours for USPTO to make any public comment, but eventually USPTO attributed this massive failure to a “power outage” that had “damaged” USPTO servers.

In a separate post I express skepticism as to how a well designed system could lead to “damage” to servers due to nothing more than a mere “power outage”.  In a well designed system, any server that has been switched to battery power rather than normal electrical power will shut itself down gracefully.  When power is restored, the server can be restarted as it normally would.  (In our office, after such a shutdown due to loss of electrical power, the servers actually restart automatically when power is restored.)

But it sort of does not matter what common sense might suggest about how long it ought to take to get things working after the power is turned back on.  What matters, on a practical level, is when the various servers actually returned to normal.

It is now 48 hours after the massive system crash began.  Which systems are working now?

This morning, alert list member Chico Gholz reported that the interference web portal is working again.  (He also reports that “Judge McKelvey put a lump of coal in someone’s stocking in 106023.”)

List member Mike Ervin reported that EPAS is working again.  Given that EPAS and ETAS are essentially the same thing, I’d guess this means that ETAS is also working again.

List member Ken Boone reports that TESS is working again, but as of earlier today it was not completely up to date.

List member Kevin Grierson reported that he was able to run Feathers today.  Meaning that at least some of the functionality of TSDR is back to normal.  But as Greg William pointed out, the “documents” function of TSDR was still broken as of earlier today.  Terry Carroll reported that the TSDR API was still broken.

Earlier today Private PAIR was still broken.  But just now (about 48 hours after the start of the outage) Private PAIR seems to be working again.  It has correspondence that was mailed early in the morning of Wednesday, December 23, but it does not contain any correspondence that was mailed early in the morning of Thursday, December 24.  (Maybe there was no correspondence mailed early in the morning of December 24, because maybe no work got done at the USPTO on December 23.)

EFS-Web is still broken — both the normal server and the contingency server.

TEAS is still broken.

RAM (for paying maintenance fees) is working again.

ESTTA (e-filing of TTAB papers) is still broken.

AOTW (assignments on the web) seems to be working normally.

PATFT and APPFT (patent full-text searching) are working normally, except that APPFT is a week behind, as it is missing the publications for December 24.

TTABVUE (TTAB proceeding file inspection) is working normally.

OEMS (for ordering physical certified copies) is working.

Financial Profile has a message saying that it is working intermittently.

The ID Manual seems to be working, albeit slowly.

As you may know (see my previous blog posts here and here) the USPTO somehow ended up on December 22 with all of its e-commerce systems broken.  The system failures are said to be due to a “power failure”.  Oddly, it is apparently not enough to restore the power, because supposedly this power failure “damaged equipment” at the USPTO.  USPTO has posted this notice:

In light of this emergency situation, the USPTO will consider each day from Tuesday, December 22, 2015, through Thursday, December 24, 2015, to be a “Federal holiday within the District of Columbia” in accordance with the description and regulations in this official notice posted here: https://www.uspto.gov/blog/ebiz/.

It had already been determined some years ago that Friday, December 25 would be a federal holiday.  And December 26 and 27 are weekend days.  In practical terms this means that any filing that might have been due in the USPTO on December 22, 23, 24, 25, 26 or 27 will be considered by the USPTO to be timely filed if it is filed by Monday, December 28.  This applies to the filing of new patent applications, entry into the US national phase from a PCT application, or response to an Office Action.

Every USPTO e-commerce system has been broken since about 5PM Eastern Time yesterday.   Many hours passed during which USPTO was silent about the cause of the outage, and it was left to customers to compare notes and to try to guess what was going on through discussions on the Trademark Practitioner’s Listserv and the Patent Practitioner’s Listserv.  (What?  You have not joined those listservs?  Click on the links to see how to join.)

About fifteen hours into the outage, USPTO finally revealed the cause of the outage — the electric power had gone out at the Madison Building in Alexandria.  USPTO does not, apparently, have backup generators to keep things going during such a power outage.  Nor does it have redundant electrical feeds from two different sources or feeds.

This power outage is like watching a bad movie again and again.  We all watched this bad movie on May 14, 2014 when all of the patent e-filing systems were broken for some eighteen hours.  The backup patent e-filing system was broken just like the main system.  The only way to file a patent application was to go to the Post Office or hand-carry it to the Patent Office.  Either of which would incur a $400 penalty for failing to e-file.

For some years now I have been urging USPTO (blog) to relocate its backup e-filing server to a geographically diverse location.  This has fallen on deaf ears at the USPTO.

Well today every USPTO e-commerce system has crashed.  Users are unable to reach EFS-Web (patent filing) or PAIR or patent assignments on the web.  EPAS is broken.  The system for paying maintenance fees is broken.

Oh and EFS-Web Contingency, the “backup” patent e-filing system that is “guaranteed” to be working even if the main EFS-Web system is broken, is broken.

On the trademark side, TEAS and TESS are both broken.  TSDR is down.  TTABVue is down.

Oddly the main USPTO web site is not broken.  But then the main USPTO web site does not actually do anything for USPTO customers.  To actually get anything done one must migrate away from the main USPTO web site to one of the other e-commerce sites.

It is clear that even now there is some Single Point of Failure that is able to knock out both the main EFS-Web server and the backup EFS-Web server.  An alert twelve-year old, hearing about such a problem, would have no trouble figuring out that the correct protective step would be to move one of the two servers to some geographically diverse location.  No single errant backhoe would rip up both fiber optical cable feeds at the two locations.  No power line downed because of ice would shut down both locations.

Which brings me to the Big Problem.  The Big Problem, as I have blogged about again and again, is that USPTO continues to fail to place its backup patent e-filing system in a geographically diverse location such as one of the other Patent Offices.

Oh and of course USPTO’s answer would be “you can always use 37 CFR 1.10” meaning “you can go to the Post Office”.  Except then you would get a slap in the face, as our firm did, with a notice of having to pay a $400 penalty for failing to e-file.

As the new year approaches, a blogger’s thoughts reflect back on the past year.  Which were the most popular Ant-Like Persistence blog articles of 2015?   Continue reading “Most popular Ant-Like Persistence blog articles of 2015”

Every December there is a sort of game of chicken, with federal employees (and customers of the USPTO) wondering until the last minute whether the President will sign an executive order giving federal employees an extra day off from work around Christmas.  This affects customers of the USPTO because they need to know whether patent applications and responses can be postponed until the next working day.

Slightly over half of the time over the last twenty years or so, what has happened in December is that the President signs such an order.  There has never been more than two or three weeks’ advance notice, and sometimes the Presidential order has been signed just a few days before the would-be holiday.

In 2014 Christmas fell on a Thursday (and thus was a federal holiday), and on December 5 of that year the President signed an order giving everybody an extra day off on Friday the 26^th.  This meant that any patent application or response that would normally be due on the 25^th or 26^th or 27^th or 28^th could be postponed until the 29^th.

In 2013 Christmas fell on a Wednesday and the president did not give any extra time off.

So how will it be this Christmas season?  Will federal employees get the day off on Thursday the 24^th?  Will customers of the USPTO be able to put off filings that would have been due on the 24^th until Monday the 28^th? Continue reading “Will the USPTO be closed this Thursday?”