Ant-Like Persistence

Here is a statement dated December 24.  You can see the original here.

Statement of USPTO Acting Chief Communications Officer Patrick Ross

On Tuesday night, December 22, a major power disruption to the USPTO’s data center resulted in the shutdown of our public filing, searching, and payment systems, as well as the core systems our patent and trademark examiners use. Since then, dedicated teams have been working around the clock with our service providers to assess the situation and safely stabilize and restore those systems. Repair estimates remain the same—that the USPTO will be impacted at least through December 25.

Power that comes into the USPTO’s main building feeds two power filtration systems that provide steady, “filtered” power so systems don’t suffer from damaging surges or drops in power supply. A malfunction in the power supply lines feeding these two systems caused significant damage to both systems. This is what we believe caused our systems to go down on Tuesday night.

Because of their size, these large and highly complex power filtration systems cannot be easily replaced. We are working with service providers to obtain a source of uninterrupted conditioned power to the data center as soon as possible.

The USPTO will continue to provide updates, such as yesterday’s announcement of our filing deadline flexibility, through our systems status webpage (www.uspto.gov/blog/ebiz/) and Facebook account (www.facebook.com/uspto.gov (link is external)). With that information, users can make informed decisions about how best to allocate their own time and efforts while the problem is being addressed.

Our IP system is vital to our 21st century knowledge economy. Therefore, having timely and efficient public access to all of our agency’s filing, searching, and payment systems is also vital. The USPTO is mindful of our customer’s needs and appreciates the continuing patience.

So let’s try to put this into plain language.  Some years ago, USPTO spent oodles of money installing redundant “filtration systems” that are intended to protect USPTO’s e-commerce servers from problems with the electricity provided by the electric company.  On December 22, a problem happened with the electricity provided by the telephone company.  Every one of USPTO’s e-commerce servers promptly crashed.  I can’t quite put my finger on what sounds wrong about that.

 

It is now December 25.  About ¾ of USPTO’s e-commerce servers (the least mission-critical servers) are now back online.  So clearly somebody has figured out how to get power reconnected to servers.  Yet, though somebody has figured out how to get power reconnected to ¾ of the servers, still the EFS-Web and TEAS and IFW servers are broken.

The statement is not as clear as it might be, but I guess the situation is that the power that has been connected to the non-mission-critical servers is power that cannot be said to be “uninterrupted conditioned power”.  And until USPTO is able to restore “uninterrupted conditioned power”, USPTO is nervous about flipping the power switches to turn the mission-critical servers (such as EFS-Web and TEAS and IFW) back on.

Too bad that USPTO had not followed suggestions from years ago to move the “contingency” EFS-Web server to a geographically diverse location.

Anyway, “uninterrupted conditioned power” is actually quite easy to get.  USPTO could go on Amazon and place orders for two or three dozen of the biggest UPSs and they would be delivered the next day.  Or USPTO could dispatch employees to stop by all of the nearby Best Buy stores and buy all of the big UPSs.

Two or three dozen big UPSs would be more than enough to power the USPTO mission-critical systems.

Clicking around in various USPTO e-commerce systems, I see no progress from yesterday.

• IFW is still broken.
• EFS-Web is still broken, both the main server and the contingency server.
• TEAS is still broken.
• ESTTA is still broken.

If you see progress with any of the systems, please post a comment here.  Thanks.

Every external-facing system at the USPTO crashed two days ago on December 22, 2015.  As of today, December 24, most of the systems are still broken.  (In another blog post I review the status of the various systems up to the present time.  I also describe the administrative steps the USPTO still needs to do to remediate the crash.  I also discuss the redundancy measures that USPTO should have taken years ago but did not do, and now needs urgently to do.)

In the past, when a USPTO e-filing system failed, the rather brusque advice to the would-be filer of a patent application was that the customer could go to the post office instead of e-filing.  The customer was further told that the $400 penalty for failing to e-file would not be waived.  Translated into plain language, the would-be filer of a patent application was being told to pound sand.

In the aftermath of the May 14, 2014 massive crash, in which both EFS-Web servers crashed for some eighteen hours, thousands of customers who went to the post office were dinged with the $400 penalty.  We at Oppedahl Patent Law Firm LLC were dinged for that penalty for the one urgent patent application that we filed at the post office on the evening of May 14.  Pound sand indeed.

The most recent system crash that knocked out both of the EFS-Web servers (the main one and the “contingency” server that USPTO promises will always be working even if the main one crashes) happened on the afternoon of Tuesday, December 22, 2015.  Just over twenty-four hours later, after the close of business on Wednesday, December 23, 2015, the USPTO made an announcement that was (I expect) intended to be much more customer-friendly than the previous “pound sand” policy.  Here is what the USPTO announced yesterday evening:

In light of this emergency situation, the USPTO will consider each day from Tuesday, December 22, 2015, through Thursday, December 24, 2015, to be a “Federal holiday within the District of Columbia” in accordance with the description and regulations in this official notice posted here: https://www.uspto.gov/blog/ebiz/.

At first blush, this looks like a really smart and customer-friendly way to address the massive server crash.  The practical consequence, as I blogged yesterday, was that filers would get a free pass on anything that needed to be filed on the 22^nd or 23^rd or 24^th, and anything that was due to be filed that day (or on the 25^th or 26^th or 27^th) could be postponed until the 28^th.  Nobody would have to scramble around trying to find a post office that was open late.  Nobody would have to pay the $400 penalty for failing to e-file.  The problem had been remediated.

But several alert members of the patent practitioner listserv clued me in that I was wrong to think that this action by the USPTO would remediate the problem.  Mere minutes after USPTO’s announcement that it would “deem” these three days to be federal holidays in the District of Columbia, alert list member David Boundy wrote:

Does the statute give the PTO authority to “consider” a day to be a holiday?  No.   That declaration has to come from either the President or Congress.

35 U.S.C. § 21(a) gives the Director the authority to “consider” a paper to be timely filed if it is timely mailed, but § 21(b) on holidays has no such grant of authority.

Maybe the PTO can “consider” papers that are actually filed late to be timely, but I don’t know where a court would find similar authority when an issued patent is tested.  The PTO has the authority to waive regulations (and thus can allow extension fees to be tolled for two days), but they don’t have the authority to define “holiday” for purposes of statute.

This sure looks to me like the PTO taking a bad situation and making it worse — give a promise that they can’t deliver on, and invite detrimental reliance.

(What?  You don’t belong to the the patent practitioner listserv?  Time to join!)

USPTO is going to have to go to the Hill to get a special bill passed, deeming December 22-24 to have been a federal holiday in the District of Columbia at least for the purposes of the USPTO.  If USPTO fails to do this, there will be later litigations in which infringers argue (and might well successfully argue) that the USPTO only had the power to waive rules, and did not have the power to waive the statute.

While the USPTO is visiting at the Hill asking for a special bill to fix this December 22-24 problem retroactively, USPTO might as well ask that Congress undo the $400 penalty for failing to e-file that came into effect with the America Invents Act.  Or at least ask that Congress give the USPTO the power to waive that penalty when needed.

(I am delighted to observe (see blog) that USPTO has addressed the TEAS RF and TEAS Plus problem mentioned below.)

Today I review several aspects of USPTO’s massive system crash that began two days ago on December 22, 2015.  In other blog articles I discuss the status of various systems and the “deeming” remedy that USPTO has proposed and redundancy measures that USPTO needs to bring about urgently.  In this blog article I will talk about the administrative steps the USPTO needs to do (and has not yet done) to remediate the massive system crash.

When the USPTO’s patent e-filing servers (including the supposedly redundant server) crashed for some eighteen hours on May 14, 2014 (a crash that carried over to May 15), applicants were forced to file on paper.  Some hand-carried their patent applications to the USPTO.  Others (our firm included) used Priority Mail Express (née “Express Mail) at the post office to get a same-day filing date at the USPTO.

USPTO then slapped everyone who made use of either of these options with a penalty of $400 for failing to e-file.

Fairness would have suggested that USPTO should have waived the $400 penalty for failing to e-file, given that during the relevant time period USPTO’s e-filing systems (including the supposedly redundant system) were broken and it was impossible to e-file.

But fairness was not in the cards.  I personally asked a very high-up person at the USPTO to waive this fee.  USPTO’s official position in response was that the $400 penalty is statutory, not rule-based, and so it was not within the power of the Director to waive that penalty.

When I was first in practice as a patent attorney, nearly every big city had a 24-hour post office where you could go to send an Express Mail package to the USPTO at 11:59 PM and still obtain a same-day filing date.  (Many big cities, including my home towns of New York and Denver, had two 24-hour post offices, one downtown and another near the airport.)  But that was years ago, and nowadays the post office closes long before midnight.  In Denver, for example, the latest any post office is open is until 10 PM.  We managed to get our one drop-dead patent application on May 14, 2014 filed by reaching that post office by about 9:30 PM.

Applicants rely on USPTO’s EFS-Web server for filing patent applications.  And they rely on USPTO’s promise that even if the main EFS-Web server were to crash, USPTO’s “contingency” EFS-Web server would nonetheless be functioning.  But on May 14, 2014, both of these servers, including the “contingency” server, crashed.  It lasted for some eighteen hours, running well past midnight on May 14, 2014.

Many applicants simply lost the filing date.  Such an applicant, planning to e-file in the last few hours of May 14, 2014, and finding the main EFS-Web server to be broken, and then finding the “contingency” EFS-Web server to be broken, would likely find to his or her horror that there was no post office at all that was open at that hour of the evening.

Now comes this crash that began December 22, 2015 and has continued to today, December 24.

One thing that USPTO will need to do, as I detail here, is go to Capitol Hill to get a special bill passed deeming December 22-24 to have been a federal holiday in the District of Columbia for purposes of the USPTO.

In addition, there will doubtless be some customers who hand-carried or post-office-filed patent applications and US national-phase entries during the past couple of days, and who dutifully paid the $400 penalty for failing to e-file.  For each such customer, USPTO needs to refund the $400 penalty (after having gotten power from Congress to do so, as I discuss here).

After having gotten that power from Congress to refund the $400 penalty, USPTO should go back and refund it to everyone who was dinged with that fee during the massive EFS-Web outage of May 14-15, 2014.

For those customers who hand-carried or post-office-filed patent applications and US national-phase entries during the past couple of days, USPTO needs to not impose the $400 penalty for failing to e-file.

It gets worse.  The TEAS-Plus system imposes a $50 per trademark class penalty on any filer of a “Plus” or “RF” (reduced fee) application who files a response by means other than e-filing.  USPTO needs to not impose that penalty on those who responded to a Plus or RF office action by means other than e-filing during this system crash time.

Today I will post several blog posts relating to the USPTO’s system crash.  In a first article I will review the status over the past few days of some of USPTO’s systems.  In a second article I will discuss USPTO’s proposed remedy of “deeming”several days to be federal holidays.   In a third article I will review the administrative remedies the USPTO needs to carry out. And in this article I will discuss the importance of redundancy. Continue reading “The need for redundancy”

Today I review several aspects of USPTO’s massive system crash that began two days ago on December 22, 2015.  (In other blog articles I discuss the “deeming” remedy that USPTO has proposed, and redundancy measures that USPTO needs to bring about urgently, as well as the administrative steps the USPTO needs to do (and has not yet done) to remediate the massive system crash.)

In this article I discuss the present and recent status of USPTO’s crashed systems.

The crash began on the afternoon of Tuesday, December 22, 2015.  We first noticed it when EFS-Web started to get flaky.  I was in the middle of trying to e-file a design patent application, and EFS-Web crashed shortly before I was going to click “submit”.  Others in our office noticed that PAIR was broken and that they were unable to e-file trademark applications and trademark responses in TEAS.

I posted a query on the EFS-Web listserv, wondering if it was “just me”.  No, it was not just me, I learned.  And soon it became clear that every external-facing e-commerce server at the USPTO was broken.  This included EFS-Web (normal and contingency), PAIR (public and private), TEAS (for e-filing trademark documents), EPAS and ETAS (for recording assignments), RAM (for paying maintenance fees), Financial Profile, AOTW (assignments on the web), ESTTA (e-filing of TTAB papers), OEMS (for ordering physical certified copies), PATFT and APPFT (patent full-text searching), TESS (trademark searching), the ID Manual, TSDR (trademark file inspection and status), and TTABVUE (TTAB proceeding file inspection), and the interference web portal.

A crash this massive, affecting every external-facing USPTO system except the main USPTO web site, has never been seen before.  It took hours and hours for USPTO to make any public comment, but eventually USPTO attributed this massive failure to a “power outage” that had “damaged” USPTO servers.

In a separate post I express skepticism as to how a well designed system could lead to “damage” to servers due to nothing more than a mere “power outage”.  In a well designed system, any server that has been switched to battery power rather than normal electrical power will shut itself down gracefully.  When power is restored, the server can be restarted as it normally would.  (In our office, after such a shutdown due to loss of electrical power, the servers actually restart automatically when power is restored.)

But it sort of does not matter what common sense might suggest about how long it ought to take to get things working after the power is turned back on.  What matters, on a practical level, is when the various servers actually returned to normal.

It is now 48 hours after the massive system crash began.  Which systems are working now?

This morning, alert list member Chico Gholz reported that the interference web portal is working again.  (He also reports that “Judge McKelvey put a lump of coal in someone’s stocking in 106023.”)

List member Mike Ervin reported that EPAS is working again.  Given that EPAS and ETAS are essentially the same thing, I’d guess this means that ETAS is also working again.

List member Ken Boone reports that TESS is working again, but as of earlier today it was not completely up to date.

List member Kevin Grierson reported that he was able to run Feathers today.  Meaning that at least some of the functionality of TSDR is back to normal.  But as Greg William pointed out, the “documents” function of TSDR was still broken as of earlier today.  Terry Carroll reported that the TSDR API was still broken.

Earlier today Private PAIR was still broken.  But just now (about 48 hours after the start of the outage) Private PAIR seems to be working again.  It has correspondence that was mailed early in the morning of Wednesday, December 23, but it does not contain any correspondence that was mailed early in the morning of Thursday, December 24.  (Maybe there was no correspondence mailed early in the morning of December 24, because maybe no work got done at the USPTO on December 23.)

EFS-Web is still broken — both the normal server and the contingency server.

TEAS is still broken.

RAM (for paying maintenance fees) is working again.

ESTTA (e-filing of TTAB papers) is still broken.

AOTW (assignments on the web) seems to be working normally.

PATFT and APPFT (patent full-text searching) are working normally, except that APPFT is a week behind, as it is missing the publications for December 24.

TTABVUE (TTAB proceeding file inspection) is working normally.

OEMS (for ordering physical certified copies) is working.

Financial Profile has a message saying that it is working intermittently.

The ID Manual seems to be working, albeit slowly.

As you may know (see my previous blog posts here and here) the USPTO somehow ended up on December 22 with all of its e-commerce systems broken.  The system failures are said to be due to a “power failure”.  Oddly, it is apparently not enough to restore the power, because supposedly this power failure “damaged equipment” at the USPTO.  USPTO has posted this notice:

In light of this emergency situation, the USPTO will consider each day from Tuesday, December 22, 2015, through Thursday, December 24, 2015, to be a “Federal holiday within the District of Columbia” in accordance with the description and regulations in this official notice posted here: https://www.uspto.gov/blog/ebiz/.

It had already been determined some years ago that Friday, December 25 would be a federal holiday.  And December 26 and 27 are weekend days.  In practical terms this means that any filing that might have been due in the USPTO on December 22, 23, 24, 25, 26 or 27 will be considered by the USPTO to be timely filed if it is filed by Monday, December 28.  This applies to the filing of new patent applications, entry into the US national phase from a PCT application, or response to an Office Action.

Every USPTO e-commerce system has been broken since about 5PM Eastern Time yesterday.   Many hours passed during which USPTO was silent about the cause of the outage, and it was left to customers to compare notes and to try to guess what was going on through discussions on the Trademark Practitioner’s Listserv and the Patent Practitioner’s Listserv.  (What?  You have not joined those listservs?  Click on the links to see how to join.)

About fifteen hours into the outage, USPTO finally revealed the cause of the outage — the electric power had gone out at the Madison Building in Alexandria.  USPTO does not, apparently, have backup generators to keep things going during such a power outage.  Nor does it have redundant electrical feeds from two different sources or feeds.

This power outage is like watching a bad movie again and again.  We all watched this bad movie on May 14, 2014 when all of the patent e-filing systems were broken for some eighteen hours.  The backup patent e-filing system was broken just like the main system.  The only way to file a patent application was to go to the Post Office or hand-carry it to the Patent Office.  Either of which would incur a $400 penalty for failing to e-file.

For some years now I have been urging USPTO (blog) to relocate its backup e-filing server to a geographically diverse location.  This has fallen on deaf ears at the USPTO.

Well today every USPTO e-commerce system has crashed.  Users are unable to reach EFS-Web (patent filing) or PAIR or patent assignments on the web.  EPAS is broken.  The system for paying maintenance fees is broken.

Oh and EFS-Web Contingency, the “backup” patent e-filing system that is “guaranteed” to be working even if the main EFS-Web system is broken, is broken.

On the trademark side, TEAS and TESS are both broken.  TSDR is down.  TTABVue is down.

Oddly the main USPTO web site is not broken.  But then the main USPTO web site does not actually do anything for USPTO customers.  To actually get anything done one must migrate away from the main USPTO web site to one of the other e-commerce sites.

It is clear that even now there is some Single Point of Failure that is able to knock out both the main EFS-Web server and the backup EFS-Web server.  An alert twelve-year old, hearing about such a problem, would have no trouble figuring out that the correct protective step would be to move one of the two servers to some geographically diverse location.  No single errant backhoe would rip up both fiber optical cable feeds at the two locations.  No power line downed because of ice would shut down both locations.

Which brings me to the Big Problem.  The Big Problem, as I have blogged about again and again, is that USPTO continues to fail to place its backup patent e-filing system in a geographically diverse location such as one of the other Patent Offices.

Oh and of course USPTO’s answer would be “you can always use 37 CFR 1.10” meaning “you can go to the Post Office”.  Except then you would get a slap in the face, as our firm did, with a notice of having to pay a $400 penalty for failing to e-file.

As the new year approaches, a blogger’s thoughts reflect back on the past year.  Which were the most popular Ant-Like Persistence blog articles of 2015?   Continue reading “Most popular Ant-Like Persistence blog articles of 2015”