Here is why you suddenly cannot log in at the USPTO since Saturday.
Two people at the USPTO screwed up. One of the people who screwed up did it yesterday, Saturday, March 6, 2021. The other person who screwed up did it a couple of years ago, and that screwup only came into prominent view yesterday. The screwups relate to what USPTO calls “authenticator app” two-factor authentication. The screwups affect most trademark practitioners who practice before the USPTO, and they affect most patent practitioners who practice before the USPTO, and they affect most paralegals and administrative assistants who work with those patent practitioners. Briefly, you need to delete your old “authenticator app” setup and you need to create a new “authenticator app” setup. Here are the details.
USPTO requires nearly all of its users to use two-factor authentication (“2FA”). USPTO offers three options for 2FA:
- SMS (text message) sending of a second factor to the user
- email sending of a second factor to the user, and
- what USPTO calls “authenticator app” 2FA.
Most high-volume filers at the USPTO use the third option — “authenticator app” two-factor authentication. It is much faster and more efficient then the SMS approach or the email approach. (For the SMS to work, you have to be in range of a cell tower, meaning it is no good if you are in the basement of a building and no good if you are in an airplane, and it is no good if your cell phone’s battery ran down. For the email approach to work, you have to wait and wait until the email message eventually works its way through USPTO’s outbound email server and then works its way through your inbound email server.)
The “authenticator app” approach uses what is called a Time-based One-Time Password (Wikipedia article). For this to work, the user (client) and the server set up a “shared secret” which in this context is a Secret Code Number. This Secret Code Number gets stored in a database at the USPTO and gets stored at the user’s end as well. For each USPTO customer who might log in to a USPTO system, the database has a place where the USPTO stores the Secret Code Number for that USPTO customer.
When the time comes for a user to log in at the USPTO, the user’s authenticator app calculates a one-time password by using a cleverly designed math function. The function receives two inputs:
- the user’s Secret Code Number and
- what time it is right now.
These two things get fed into the cleverly designed math function and what pops out is a six-digit numerical one-time password.
At that same time, the USPTO server pays attention to who exactly is trying to log in, and the USPTO server looks up in its database to see what the Secret Code Number is for that particular user that is trying to log in. The USPTO system uses the same cleverly designed math function, and it feeds two inputs into that function:
- the user’s Secret Code Number (which the server just now retrieved from its database of Secret Code Numbers) and
- what time it is right now.
And what pops out is the exact same six-digit numerical one-time password.
The user enters the six-digit number that popped out from the user’s authenticator app into the USPTO system. The USPTO system then looks to see whether that six-digit number from the user matches the six-digit number that the USPTO system obtained from its own math function. Of course generally speaking the two numbers will match, at which point the USPTO system thenceforth considers that user to be logged in fully with 2FA.
Yesterday morning, USPTO did scheduled maintenance on its servers. When the scheduled maintenance finished, what USPTO’s users found is that they were no longer able to log in on any USPTO system. The “authenticator app” approach for 2FA had stopped working. Users were told, falsely, that “the verification code you entered is incorrect or expired.”
When this happened yesterday, USPTO users started comparing notes in the PAIR listserv and in the EFS-Web listserv. For some users there was a fallback position, namely that for some users what the user could do is make use of one of the inferior types of 2FA such as SMS or email. Or, for example, the user could simply forgo any login and file the paper by faxing it to the Central Fax Number at the USPTO.
The only thing that I was able to think of that might explain why all of a sudden nobody was able to log in at the USPTO using “authenticator app” 2FA was that maybe the USPTO server had gotten mixed up about what time zone it was in, something like that. This would bollix up one of the two inputs to the TOTP math function, making it so that the six-digit one-time password generated at the USPTO would be wrong and would thus fail to match the user’s own (correct) six-digit one-time password.
But no, as it turns out, my imagination was not fertile enough to conceive of what had really gone wrong. Here is USPTO’s announcement of earlier today about this situation:
Sunday Mar 07, 2021
During our system maintenance on Saturday, March 6, we discovered an issue that may affect your 2FA second factor authentication. As such, here are the detailed steps for any user that prefers to use an authenticator app as the primary delivery method.
The steps to reconfigure are as follows. It takes about 2 minutes to execute:
- Log in to MyUSPTO using the account to be reconfigured, entering user id and password when prompted.
- For Two-step authentication, choose to receive code via an option other than “Code generator (Authenticator app)”
- Retrieve and transcribe the second factor into the “authentication code” box
- Expand the user’s display name (upper right-hand corner) and select “Account”
- On the account page, find the section labeled “Code generator (Authenticator app)” and click the “Reconfigure” button
- Enter prompted values, following the prompts, and replace the prior configuration
Posted at 10:33AM Mar 07, 2021 in Current Status
I invite you to read this, maybe more than once if necessary, and see if you can figure out what is unsaid in this posting on the USPTO web site. See if you can figure out what exactly went wrong here.
Yes, the answer is that somebody at the USPTO yesterday morning had an “oops” moment. Somebody deleted the USPTO database of all of the Secret Code Numbers of every user of every USPTO system. Which now means that every USPTO user needs to start all over again with setting up the “authenticator app” two-factor authentication. Which you can only do after you (hopefully) successfully log in at the USPTO using some other kind of two-factor authentication other than the “authenticator app” two-factor authentication.
Oh to have been a fly on the wall to listen to what must have happened at the USPTO during some conference call yesterday. What happened, I am pretty sure, is that yesterday afternoon, after the trouble tickets started piling up at the USPTO, a bunch of IT people at the USPTO were on a conference call and they all suddenly looked at each other and each one pointed to somebody else and said “I thought you were the person who was doing daily backups of the TOTP shared secret database!” And I am not making this up, it must have turned out that nobody was backing up that database. So a second person at the USPTO also screwed up, namely whoever it is whose job it is to make sure that the USPTO makes periodic backups of all of the databases that needs to be periodically backed up.
So really what we have is two people made mistakes at the USPTO.
The first person I mentioned, the person who accidentally deleted the entire database of Secret Code Numbers for all USPTO users, actually should not be scolded very much. In a competently managed system, accidental deletion of any single file or any single database is never a big problem because some other person backed it up some time in the past 24 hours. You just go find the backed-up copy and click “restore”, something like that. The second person I mentioned, the person whose job it is to make sure that the USPTO makes periodic backups of all of the databases that needs to be periodically backed up, that is the person who needs to be scolded quite a lot.
So let’s see what the consequence is? The consequence is that every user of MyUSPTO, every trademark practitioner, every user of Private PAIR, every user of EFS-Web, every use of Patentcenter, must now go though the twenty or so mouse clicks. Each user must do the mouse clicks that bring up the existing secret code number, and the clicks that delete it. Each user must then go through the steps to generate a new secret code number, and get it loaded into the USPTO database, and get it loaded into the user’s own system.
It’s not only patent practitioners. It’s also trademark practitioners. And it is also the paralegals and administrative assistants who work with those patent practitioners. I am guessing maybe fifty thousand people who all of a sudden find that they cannot log in on any USPTO system.
The USPTO implementation of TOTP setup is more burdensome than most implementations for setup. With most implementations, you generate the Secret Code Number and you store it locally and the USPTO stores it at the USPTO, and then you test things by doing one match of one-time passwords. You and the server each generate one six-digit one-time password and you compare them, and if they match, then you count it as good and everybody moves on. This takes ten or fifteen mouse clicks and consumes three or four minutes of your valuable time.
But the USPTO system is more burdensome. With the USPTO system, you start by doing what all servers do which is, you and the server each generate one six-digit one-time password and you compare them, and if they match, great, but then the USPTO system says “we require that you do at all a second time”. The USPTO system then demands that you generate a second six-digit time-based password a few seconds later, and that one also has to match, and only then will the USPTO count it as good and people will be able to move on. This adds an unnecessary couple more minutes and an unnecessary several more mouse clicks to the process of setting up the “authenticator app” 2FA.
Most USPTO users who use the authenticator-app type of 2FA probably only store their Secret Code Number in one place, namely inside their authenticator app. Such a USPTO user is now stuck having to go through the entire TOTP setup process all over again, burning up maybe six minutes of their valuable time. This is maybe twenty mouse clicks.
But if you are a power user of TOTP, you will likely have stored your secret code number in several places. That is the smart thing to do. I carry a fob for example that has all of my TOTP shared secrets in it, in an encrypted memory.
So USPTO’s incompetence does not merely make me do the basic twenty or so mouse clicks to generate a new secret code number to replace the one that the USPTO accidentally deleted and had failed to back up. USPTO’s incompetence also forces me to do something like one hundred extra mouse clicks to get the new secret code number stored into all of the places where it needs to go, including in the secure fob. It will waste well over a quarter of an hour of my time.
The nameless person at the USPTO who crafted that announcement is being disingenuous in the extreme to estimate the time cost at a mere “two minutes”. The time cost to the customer is not only the time cost of cleaning up the mess made by the USPTO. The time cost to the customer previously included the lost ten or fifteen minutes during which the customer tries to log in, gets the error message, tries again, fails a second time, and then checks with co-workers to try to figure out “is it just me?” And only after that ten or fifteen minutes of “is it just me?” leads to the conclusion that it is not just me. And eventually the customer stumbles upon the fix which yes, if you have done it several times recently might only take you two minutes. More likely, since the last time you did it was two years ago, it will take you five or ten minutes.
We are talking about something like fifty thousand USPTO customers who must now spend five or ten unnecessary minutes deleting their “authenticator app” setups and setting up new “authenticator app” setups. I guess the total social cost of this will be something like ten thousand hours of wasted time. Conservatively the blended hourly rate for those USPTO customers might be $100 per hour, so the social cost of the USPTO screwups will be well in excess of one million dollars.
Back to who it is that screwed up at the USPTO. My great disappointment is in the person whose job it was to make sure that every important database in the USPTO systems gets backed up regularly. That person (whose name we customers probably will never learn) screwed up here. The screwup probably happened a couple of years ago, when USPTO just got started using the authenticator-app type of 2FA.
If the problem had been that something had caught fire and then it turned out the fire extinguisher in the room was no help because it had been empty for a couple of months now, you can imagine somebody would suddenly be given the task of going around and looking at all of the other fire extinguishers in the building to see if any of the other fire extinguishers are also empty. Sort of like closing the barn door after the horse has already escaped.
Back to being the fly on the wall. If it has not already happened, I am quite sure it will happen very soon that somebody somewhere at the USPTO will be given the rather thankless task of hunting around to make a list of all of the databases that USPTO ought to be backing up regularly, and then running down the list item by item to see which ones are not in fact getting backed up regularly. And then set it up so that finally now the things will get backed up that should have been being backed up.
Likely as not, it is that person’s direct supervisor, or the direct supervisor of that supervisor, who deserves the scolding for having failed to attend to this a couple of years ago. That person has cost the IP community well in excess of one million dollars.
Did you have to set up your “authenticator app” two-factor authentication all over again because of this? How long did it take you? How many people in your firm or corporation had to do the same thing? Please post a comment below.
The blog post speculates that the database of TOTP secrets was lost accidentally and that “nobody was backing up that database”. But the database might have been intentionally cleared by the PTO, e.g., as a response to a security incident in which some or all of the TOTP secrets, or a derivative thereof, were exposed to outsiders, such that outsiders might be able to generate valid TOTP codes. I have no reason to believe such a security incident occurred (nor any inside knowledge), but, arguably, either speculated fault–failure to back up or failure to secure the codes or derivatives–would seem to be within the same order-of-magnitude of failure to properly operate IT systems and likelihood of occurrance.
And, of course, we will never be told what the problem actually was that “may affect your 2FA second factor authentication.” One wonders if there is a single person in the world whose 2FA configuration still worked. But heaven forbid the PTO would admit, “because of an error on our part, we destroyed our 2FA system, so everyone has to start over.”
You are correct of course that a comprehensive list of possible explanations includes the USPTO intentionally clearing on Saturday at 4AM, because of some known penetration of USPTO’s systems or the like. Mere coincidence that this happened at the exact same time as a big scheduled maintenance task. No connection between those two things other than that the known penetration happened get discovered at about the same time as the scheduled maintenance.
But think about it. If that had been the impetus, then USPTO would have simultaneously sent around an email blast to all users, saying something like “to protect your security, we just now deep-sixed your user login information on purpose, and so now you need to set up your login information all over again.” And would have plastered it all of the initial login pages for all affected systems, including TEAS, Private PAIR, Patent Center, EFS-Web, Financial Manager, and MyUSPTO.
But that’s not what happened. Instead, 24 hours came and went before anybody at USPTO posted anything about this. This means the deep-sixing of the user login information was not deliberate at all, but was some kind of screwup. Only after the trouble tickets started piling up did some of the IT people start trying to figure out what had gone so badly wrong.
I suspect what we will find is that the screwup that deleted all of the TOTP login information may also have led to USPTO employees being unable to log in. The EBC that serves you and me does not answer the phone on weekends, but I bet there is something analogous to the EBC that does answer the phone for USPTO employees on weekends. And I bet that is the place where trouble tickets piled up and eventually led to today’s belated acknowledgment by the USPTO of the problem.
I’m generally willing to defer to your superior knowlege about the operation of the PTO. But I would note that if a security incident occurred, the security personnel may have delegated authority immediately immediately to take whatever emergency action is necessary to eliminate the vulnerability created by the exposure of the TOTP secrets (or whatever), whereas the email blast you’ve proposed might require several levels of sign-offs, which are not immediately available on the weekend. On the other hand, after giving this a bit more thought, the coincidence of the authentication problem and a maintenance window seems to tip the scales of likelihood much more toward your explanation.
Allow me to posit a different theory, one which only requires the screwup of a single “person.” The database of TOTP secrets is almost certainly encrypted at rest. But they are probably loaded into RAM by an authentication server. When that authentication server was rebooted, it tried to load in the secrets from persistent storage, only to discover that the encryption key was no good.
They may have had a backup, which likely couldn’t be decrypted for the same reason. Sysadmins say that a backup you haven’t tested may be worse than no backup. And here’s why: you get all the false sense of security, and none of the actual security.
It’s certainly possible that the USPTO discovered a security breach while doing maintenance and disabled TOTP immediately to mitigate the harm. I agree with Carl that this seems less likely.
I can think of several other plausible scenarios that could explain what happened and while all of these scenarios are not equally likely, their likelihoods are all within the same order-of-magnitude range. I have yet to see clues that reliably rule in or rule out those posted here. Unless someone has some inside information, it appars we’re all speculating.
And since the PTO does not have a history of candor, I’ll speculate further that we won’t learn the real mechanism unless there’s an IG report, or someone sues and it comes out in litigation, or some insider spills the beans in an unofficial channel of communication. I would be delighted to be proven wrong.
I estimate it took me 12-15 minutes on the activities needed to fix what the PTO broke, including logging in with a PTO-generated 2FA code transmitted some other way, generating and capturing the new TOTP secret, installing the TOTP secret in my authenticator software, entering more codes into the PTO web site, and making records and backups.
It was only two minutes for me, but I also did it at a slow time. Woe be Monday morning when everyone is trying to figure out what’s going on
Do not try (even a few times) to log in by re-entering the (should-be) correct 2FA / TOTP number from Google Authenticator. I did so and got a “20-minute” lockout for “unusual activity” that remains in effect hours later. I cannot now even get to the 2FA page to reset this. And, naturally, USPTO Help Desk for this is not open unti l tomorrow….
Unfortunately, before I knew this was a bigger issue, I ended up locking myself out of my account (yesterday). I was told I would be locked out for 20 minutes, only to discover that I am still locked out today. I have tried resetting my password only to receive further error messages, which leaves me with only one option: calling the USPTO Customer Support Center. Monday morning is not going to be fun.
Any chance that if we wait a day or two the USPTO may restore the lost database?
(A third person I as a user would fault even if they shrewdly maneuvered the politics of the situation from the USPTO management perspective is the one who wrote the notice and didn’t level with us so that we know what actually happened: “we discovered an issue that may affect your 2FA second factor authentication” gets my nomination for “captain-understatement-of-the-year”.)
Particularly disingenuous is the “may affect” part. Like they hope the reader might never catch on that it does indeed affect all USPTO customers. Like they hope the user might think “oh maybe I am the only person who was actually affected by this ‘issue’.”
I hope not! If they now restore the lost database, that will trash all the new codes that you-all have been creating today, yes?
Henry, good point. Ideally they would only merge in the TOTP secrets that had not been regenerated since the reset. But things don’t appear to be operating ideally.
I mucked through the app procedure (which I had not used before) and successfully logged in, and then successfully repeated the procedure to reset it. Now I can log in by email confirmation code (as before) or using the authentication app. HOWEVER when I actually try to invoke a form, I get a proxy error — a blank screen. Very frustrating. Hopefully the fax number works as an alternative filing method. Any suggestions will be welcome. charles.b.kramer@gmail.com
Another possible scenario, I suppose, is that they in fact had a current backup, but discovered that it was corrupted when they tried to restore it; and ditto with one or more earlier backups. It’s not enough to make regular backups; one must periodically test those backups to verify that they can actually be restored if necessary.
I believe that in the DBA (database administrator) world, responsibility for making backups tends to devolve upon the most junior DBA in the shop. Why? Because it’s a relentless, tedious, boring job that rarely matters–until one day when it actually does matter, e.g. in the situation Carl has outlined. So, junior DBAs tend to focus a lot of their time & effort on moving onward and upward so they can hand the [generally thankless] task of making backups down to some even more junior person.
That depends on how they do it, of course.
Some things fix themselves (i.e., from the user’s viewpoint, that is … theoretically IT staff should be frantically trying to mitigate their error … or is my faith misplaced and no such recovery is in the cards?).
I think it was Suzannah who makes the excellent points that: (a) the notice needs to plainly state that all previous 2FA codes are gone forever if that is the case; and (b) the right place for the notice is at the login screen.
The lockout policy also merits rethinking and/or more clarity (supposedly 20 minutes but actually apparently until you call EFS staff and get them to remove the lockout).
The head of IT at the USPTO, Jamie something or other, got an award last year for his great efforts in automating USPTO systems. It must be us, not him.
Why is there a PPAC, and why is contact information for its individual members not provided?
About 5 minutes, including a 1st attempt to log in using Google Authenticator rather than e-mail verification to see whether it worked.
BUT, that short time (150% longer than the estimated 2 minutes) was only because this blog post told me exactly what the problem was and how to remedy it. There is still no notice on the 2FA sign-in screen, so anyone not privy to this information is still misled into thinking there’s some problem with their app. And, there is still no big, catchy warning alert on the MyUSPTO page to inform one what the problem is (there is the same “we discovered an issue that may affect your 2FA second factor authentication” message visible IF one happens to note the “Systems Status” notices inconspicuously lurking in the lower left).
USPTO locked me out of my account after entering the wrong/right 2FA authenticator code several times on Saturday. Message said account was locked “for 20 minutes” but never came back. Automatic password reset did not work either. Spent about 1 hour this morning on hold with EBC to get my account unlocked. Setting authenticator app back up was quick in comparison.
FWIW, I’ve never bothered the using 2FA. Early on, the email authentication method was indeed slow to send the the email containing the code. But for years now the system spits over an email almost instantly — within seconds. Copy/paste the code and I’m on in under 10 secs. Though maybe not for the next few days, when the system will be loaded up with additional users!
Relying on an e-mail from the PTO adds many possible points of failure, almost all of which are outside your control. Even if you find it most convenient to receive the codes via e-mail, it would be prudent to have the other methods (phone, authenticator-app code generator) set up just in case the e-mail route happens to be broken one day.
Yes, USPTO systems can have multiple points of failure, almost all out of your control, a good cautionary point. Like 2FA right now …
All the cases that I filed at the beginning of 2020 are siting in art unit 3600 and of course no Examiner assigned. Anyone else?
Yes, we have a bunch sitting in art until 3600. Contacted someone in the art unit and received an unsatisfactory response. Will be trying someone else to try to get the applications assigned to an actual examiner. Part of the response was on one of the applications it had been assigned to an examiner but the examiner decided it shouldn’t be, so it got kicked to 3600 and is just sitting there.
On the subject of being locked out of logging in — EBC’s automated message today said they lost access to EFW-Web and private PAIR and are aware of the issue and are working on resolving the issue.
Similar thing occurring since yesterday (Friday 2022-11-04), continuing today (Saturday 2022-11-15). After all my part is complete, the USPTO server causes that my monitor displays a message that the authentication code was sent (to my E-mail address) but nothing is sent. I tried with both Google Chrome and with FireFox. Repeated requests for help, to several USPTO Email addresses, including ebc@uspto.gov are ignored.