The need for redundancy

Today I will post several blog posts relating to the USPTO’s system crash.  In a first article I will review the status over the past few days of some of USPTO’s systems.  In a second article I will discuss USPTO’s proposed remedy of “deeming”several days to be federal holidays.   In a third article I will review the administrative remedies the USPTO needs to carry out. And in this article I will discuss the importance of redundancy.

When USPTO first launched its EFS-Web server for the e-filing of US patent applications (and entries into the US national phase from PCT applications), it took a little while for USPTO to realize that the legacy mode for the filing of patent applications (namely a trip to the post office to send a package to the USPTO via Express Mail) was not really a reasonable redundancy approach that could serve as a substitute for a crashed EFS-Web server.  So the USPTO set up what it termed a “contingency” EFS-Web server.  The promise to customers was that whenever the main EFS-Web server might crash, the “contingency” EFS-Web server would still be available for the filing of urgent patent applications and entries into the US national phase from PCT applications.

Soon after the release of the “contingency” EFS-Web server, I pointed out to the USPTO that it was inadequate to the task.  I performed traceroutes to the two EFS-Web servers which revealed that they were connected to the Internet in exactly the same way, through the same Internet feed.  I was able to identify several miles of connection line along which an errant backhoe could knock out both servers.  I was pretty confident the two EFS-Web servers were actually in the same physical building, protected by the same single firewall, and were powered by the same single power line from the electric company.  This meant that there were several Single Points of Failure that could knock out both EFS-Web servers.

When I saw this, I immediately started pestering the USPTO that one of the two servers needed to be relocated to a geographically diverse location.  This was some six years ago.  My suggestions about this to the USPTO fell on deaf ears.

Then came the massive crash on May 14, 2014 in which both EFS-Web servers crashed for some eighteen hours, extending well past midnight on the evening of May 14, 2014.  Some thousands of filers were forced (as we were at Oppedahl Patent Law Firm LLC) to run to the post office to file their patent applications by Priority Mail Express (née Express Mail).  The USPTO then added insult to injury by making all of those filers (including our firm) pay the $400 penalty for having failed to e-file the patent applications.

The USPTO did not refund or waive that penalty.

The USPTO informally revealed that the cause of the May 14, 2014 crash of both EFS-Web servers was due to a failure in the single firewall that protected both servers.

After this crash in May of 2014, I wrote a letter to a very high-up person at the USPTO asking her to promise that all of the $400 penalty fees collected from applicants harmed by the crash of the two servers would be specifically directed toward a move of the supposedly redundant server to a geographically diverse location.  I never heard back in response to that letter.

I also blogged about this crash here and here and here and here.

In any event, the USPTO did not move its “contingency” patent e-filing server to a geographically diverse location.

A shorter-duration crash of EFS-Web happened on Saturday, August 9, 2014.  This crash likewise did not prompt USPTO to establish the much-needed redundancy.

Now comes the massive EFS-Web crash that began on December 22, 2015 and continues unabated to this day (December 24, 2015).  Both the main EFS-Web server and the “contingency” EFS-Web server crashed, and neither has been available at any time in the past 48 hours.  USPTO blames the massive crash on a “power failure” that somehow “damaged” the servers.

USPTO needs to stop the foot-dragging and move one of the EFS-Web servers to a geographically diverse location.  (My personal favorite for the location is the Denver patent office!)  This will reduce or maybe even eliminate the Single Points of Failure that presently exist.

There are also some simple protective measures that USPTO apparently failed to take, that it needs to take now urgently.  For example, how is it that a “power failure” caused “damage” to any servers?  This makes no sense.  As listserv member Mike Dryja said:

I guess what I don’t understand is this.  [If you are prudent,] you put the servers on UPSs.  The UPSs tell the servers that they’re running on battery power. The servers therefore shut down gracefully.  Then, when power comes back on, you turn the servers back on.

This would have limited the outage to essentially the duration of the power outage.

Why don’t they have UPSs?!?

Here at OPLF we do exactly as Mike Dryja described.  Our important servers each have a special control cable that runs to the associated UPS.  If the power goes out, the UPS sends a signal to the server telling it that the power will fail when the battery runs out.  The server then shuts itself down gracefully.  Later when the power comes back on, the UPS sends an “all clear” signal to the server and it restarts itself.  We don’t even need to manually restart it.

Why, why, why didn’t USPTO do this?

Years have passed during which my suggestions to the USPTO to set up genuine redundancy have fallen on deaf ears.  Maybe the embarrassment of this massive crash, now going into its fourth day, will prompt USPTO to do what my suggestions have not.