A few thoughts about Uber's breach

- 36 min read - Text Only

Allegedly, an eighteen-year-old spammed an employee with two-factor authentication via push notifications on an employee with a known password. They got into the VPN and scanned for servers, found a file share without any access controls, and a script that could access break-the-glass credentials. With the highest level of credentials available, they then got effective root access to Slack, AWS, Google Suite, and active directory at Uber.

🍿for big tech company tonight Someone in IT did not take security awareness training. This is not developers fault at all.

Uber logo with text: "2016 was not enough" above

Hey there, if a certain phrase confuses you, check out the glossary at the bottom. Several things are explained there for those outside the tech community.
If you're here to learn how to improve your IT and engineering security in light of the risks Uber experienced, you will find several recommendations near the end.


We are currently responding to a cybersecurity incident. We are in touch with law enforcement and will post additional updates here as they become available.


Apparently there was an internal network share that contained powershell scripts... "One of the powershell scripts contained the username and password for a admin user in Thycotic (PAM) Using this i was able to extract secrets for all services, DA, DUO, Onelogin, AWS, GSuite"
Photo included with tweet
Seriously though, ya'll should review the risk you take with the zero trust flat network dream you've been sold. Definitely enable number matching/verified push for all logins regardless. These kind of attacks have been happening all year, at very large orgs.
Photo included with tweet
Update: A Threat Actor claims to have completely compromised Uber - they have posted screenshots of their AWS instance, HackerOne administration panel, and more. They are openly taunting and mocking @Uber.
Photo included with tweetPhoto included with tweetPhoto included with tweetPhoto included with tweet

Who's not at fault?

Is the employee who got tricked at fault?

We're talking about a human here, who was intentionally harassed late at night; the easy way out was to follow the attacker's prompt.

Our security needs to be resilient to humans that make mistakes, and our security needs to be supportive of humans who need compassion.

If phishing a single employee can lead to everything in your infrastructure being compromised that easily, that employee is not to blame

The fact that this keeps happening and is enabled by the biggest companies out there needs to be recognized and changed.

Literally how to exclude accessibility needs people, old people, fatigued people, someone who just wants a burger but can't figure out why their card is rejected people, you know: a human?

This issue is not new, it is not a surprise; the abuse that opened the perimeter at Uber has been perpetuated through big companies selling insecure security.

Seeing an increasing amount of abuse of MFA prompt "push" notifications. Attackers are simply spamming it until the users approve. Suggest disabling push in favor of pin, or something like @Yubico for simplicity. In the meantime, alert on volume of push attempts per account.
Whether or not Duo Security was specifically a gateway in the Uber incident is irrelevant. The technique of push-based approval is demonstratively unsafe.
1. After @jack was infamously hacked through a SIM-swap attack 2019, much of the world moved away from SMS-based one-time passwords (OTPs) The apparent @uber hack may lead to a similar decline in Magic Links... here's what you need to know 👇
@ClerkDev @jack @Uber Surely the real answer is to require FIDO2? That would've stopped this entirely.
I mention WebAuthn, others mention FIDO2. For this discussion these mean the same thing: using a security key or built-in chip to authenticate.
Check out Push notification two-factor auth considered harmful where Xe Iaso and I propose that WebAuthn would have cut out the weakness that compromised Uber's security perimeter.
Everyone, even you, talks about improving security on the perimeter. That is not what made this incident so bad. Where's the beef?

Break the glass

Allegedly, the attacker got break-the-glass credentials, which gave them access to literally everything their most senior IT personnel should have access to on a limited basis.

What are "break-the-glass" credentials for?

"Break the glass" credentials and accounts are used in rare circumstances outside the normal day-to-day operations.

Here's an example of an emergency where a break-the-glass activity is fine. As an IT administrator, you were just locked outside your account, and something requires immediate attention that cannot wait. All other IT administrators of equal or greater authorization are unavailable and you are unable to reset your account's status. You reach to break-the-glass to restore your own access, and then you return to your account to continue performing your duties.

Here's an example of a non-emergency where a break-the-glass activity is fine. As an IT or DevOps administrator, you need to provision a new AWS account for a new engineering team product. The company only creates a new product once a year and master payer account / management account is otherwise limited and off limits. You break-the-glass to access the management account to create a new account for the product, you document the new account's root user in a secured place, and set up SSO access to this account for the new team. Afterwards, you log out of the management account and continue performing your duties.

That makes a lot of sense! But who should have this level of access?
These credentials should be restricted to seniors that have a justified reason to have these credentials, who have trust in the organization, and who have consistently demonstrated attention to detail in stressful circumstances.
A new hire in IT should not be able to access break-the-glass credentials.
The CEO is the biggest target and should not have access to break-the-glass credentials.
A random person of any rank or position or lack of position should not have access to break-the-glass credentials.

Credentials to break-the-glass accounts are arguably the most important secret material to secure for an organization. It allows an organization to respond to an emergency and to respond to infrequent requirements that require risky access to complete.

How do you protect emergency credentials?

If you want a product, the key words to search are: Privileged Account Management.

Uber specifically relied upon Thycotic. There are others too, but in short these technology solutions have at least these features:

  • Unified repository for shared account credentials with excessive permissions.
  • Independent (or SSO) authentication of employee access.
  • Independent authorization to credentials.
  • Auditing of all access to each credential.

A less technological solution would be to store a static password on a Yubikey and throw it in a safe. Yes, people do this, and I think it is far better than leaving break-the-glass credentials in a public location like Uber.

Yubikey static password screen

Speaking of throwing credentials in a safe, check out how DNSSEC started in the video below! The ceremony itself is pretty boring, but the planning to make a system resilient to nuclear warfare, collusion, and individual death is neat to consider.
For your privacy, this youtube video was not automatically loaded.
Click this area to load an embedded youtube video.
This is an issue for crypto-currency too. People have died or disappeared while holding some perceived value that will never be recovered. Distributed autonomous organizations have had all perceived assets transferred because key material and authorization is not properly distributed among different people with different access controls in different physical locations. Alas:
Crypto is bound to repeat every mistake learned over the past hundred years in finance

Should you go with a safe, have some sort of immutable log strategy in place where you document the date, the reason you accessed the key material, and which key material you accessed. Maybe even go old school and fax evidence of your activity to a third party who files it away in case of forensic analysis.

Speaking of forensic analysis, here's a paranoid tip that may save you after a breach occurs. If you're ever doing something funky or unusual, like running an SSM command to disable MFA on a host, then create a ticket and assign it to yourself.
If it takes longer than a few minutes to do your task, then document the steps you took and for what purpose. When the changes are completed or reversed, then close the ticket. You are covering your rear. In the event something happens months down the line, an auditor or forensic analyst will see your ticket and not blame you for the breach.

Lastly, and out of good practice, whoever has access to the keys on an ongoing basis should be separate from the person who momentarily uses the keys.

If you need help in person, it is usually up to the Chief Information Security Officer (CISO) to provide privileged access to you for a short amount of time. In my opinion, a CISO should not be using the credentials. Their authority should be to delegate access, not to act with access.

So, in a way, Thycotic is a virtual extension to CISO. As a Privileged Account Management product, Thycotic is fulfilling the role of delegating authorized access to an authenticated person and auditing the action.

So is Thycotic at fault? I think not.

I'm not shilling them either. I have no experience with Thycotic or alternatives. The use case here seems interesting.

How did Uber protect emergency credentials?

Again, this is based on alleged information.

I do not work for Uber, never have worked for Uber, I do not know anyone inside Uber and have contacted no one who works at Uber to verify.

Here's what I would expect a sane CISO to implement with a privileged account management (PAM) product:

  • Users authenticate with the PAM through an SSO provider.
  • All staff besides the CISO and their immediate deputies have limited authorization to resources within the PAM.
  • Any disaster recovery credential for the PAM is stored securely in a place that can only be accessed by the CISO and their immediate deputies.
Instead, Uber had a potato-for-brains PAM admin put their credentials in a script for Thycotic on a file share that any employee with legitimate VPN access could have accessed and then abused.

A "fun story"

I got into security because my employer was breached. I used to be just another backend-focused developer. While on vacation at a furry convention of all places, I get a ping from my boss.
Boss: "Hey, did you spin up big GPU servers and call them analytics?"
I replied, "No." But I knew what it meant. I took a deep breath and continued to enjoy my vacation.
The rest of the team worked until 2 AM rotating every credential. Credentials for every service were in plain text in the database, bank numbers for partners were in there too. At least we did not store credit card numbers, just tokens.
Except for the credit card processor credentials. Those were hard coded.
Ultimately it came to me to figure out how to encrypt necessary data at rest, to secure our credentials for future use and rotation, and to promote a secure development and cloud environment.
Boss: How did this happen!?
The CEO was targeted. He re-used his Slack password somewhere else where their passwords were breached and recovered. The attacker got into his Slack account and found that marketing shared the root AWS credentials in the clear to each other to upload to S3. The attacker got into AWS, cloned the database from a snapshot and changed its root password. The attacker dumped the data out of our account with an EC2 box. And then months later they spun up Ethereum miners because why not.
Boss: Coincidentally, I was reviewing our AWS spend a few days later and freaked out at the estimated AWS bill.

There are similarities. My breach and Uber's 2022 breach had unrestricted credentials in the clear where any employee could have found them.

Now I am the one who delegates authority to our AWS account. The rest of the company's IT infra is outside of my reach under the CIO. I may not be a CISO, but I think my experience and responsibilities are getting close. I am also interested in this, but I'm not sure yet on if I should jump into it.

Uber's security history

RM @FrankPallone's statement on Uber's failure to protect the data of 57 million consumers 📱🚗 💳 👎
Photo included with tweet
.@Uber: "Oh hey, Mark had to testify at Congress because he didn't address their data breach... Maybe we should send our users a quick email?"
Photo included with tweet
@TPioreck So, you mean the same as is going on right now? I mean, I literally was called to testify before Congress last year regarding a breach cover-up done by Uber, with the extortion/hush money paid via a complicit bug bounty platform. Corps already underreport, it's why the regs exist
Uber has been hacked, and it looks bad. The hacker got in through social engineering and allegedly found a network share full of Microsoft PowerShell scripts that included Uber admin usernames and passwords to let them breach AWS, G Suite, and more 🥲 www.theverge.com/2022/9/16/23356213/uber-hack-teen-slack-google-cloud-credentials-powershell
This AM, I’m listening to the trial of former Uber exec who’s facing criminal obstruction charges for his role in paying hackers who breached the company in 2016. Uber’s CEO is expected to testify today. ICYMI, Uber is now responding to another breach: www.wsj.com/articles/uber-says-responding-to-cybersecurity-incident-11663299504

In short, Uber had a breach, the CISO at the time covered it up and paid a ransom, and then got thrown under the bus as a criminal.

Now Uber is having another breach because the CISO or someone with equivalent access left their credentials in the clear.

Lots of screenshots going around about Uber but this one shows how wide the hack is. "Security Response Break Glass Service Account" password 🔥
Photo included with tweet
In the above screenshot, the credential passwords are human-generated. That to me is a big no no and reveals that they are set once and forgotten for years.

What Uber could have done better

The following is completely my opinion, based on my experience and what I witness in the industry. I have not executed these recommendations myself and I do not know the difficulty or burden that these recommendations impose. However, I hope that the baseline for security in our industry improves by sharing my opinion.

Perimeter security would be improved by using WebAuthn and hardware tokens for all employees and contractors. The use of push-based two-factor authentication is dangerous because the human can be harassed.

Device profiles should be a requirement on the perimeter. A certificate specific to the device (and hopefully bound to its trusted platform module) will reduce the likelihood of perimeter compromise.

Account lockouts should be the norm when a two-factor fails several times in a row. I recommend five failures. Yubikey FIPS security tokens lock out WebAuthn after three failures. Allegedly the failure was performed over a hundred times to harass the employee who gave in and approved the attacker's access.

Hard-coded credentials must be avoided. IT and engineering should be educated that a better alternative is to prompt for credentials in the script or to perform a brief SSO handshake. Ongoing education for this situation should be regularly covered and reference examples available to use.

IT should adopt source control and use tools like GitGuardian to scan and detect credentials in scripts so the credential is invalidated as soon as possible.

Human made credentials must be avoided, especially for disaster recovery and emergency use. It is not unheard of that these credentials might even be single use. A culture of using generated passwords should be adopted at all depths. Secret scanners can detect and alert high entropy generated passwords. Use this to your advantage.

Credential rotation should be regularly performed for shared accounts, including disaster recovery accounts (and a backup account in case that is destroyed). This should involve a documented process with verification steps to ensure it is smoothly executed and meets expectations. Documentation of credential rotations should be performed and preserved.

I do not believe that credential rotation should be a requirement for individual accounts. Current unauthorized or former employees may abuse long lived credentials for shared accounts.

Privileged Account Management break-the-glass credentials must be protected and upon use queued for immediate rotation. The use of these credentials should set off alarms and be very visible.

Also, any break-the-glass credentials to a privileged account management product should produce an on-call page / call to several parties. Consider this a fire drill for a very real fire that happened at Uber.

Penetration tests should be regularly performed by multiple vendors with access to the internal and application network. Uber should place multiple flags (capture the flag style) in their infrastructure, services, source code, application containers, logs, chat rooms, etc. to measure coverage that each vendor achieved as a simulation of what an attacker could have found.

Honeypot credentials should be included in the privileged account management product, their retrieval should cause an on-call system to page / call the CISO, the on-call, and a situation manager.


The industry needs something like OWASP that provides ongoing risk assessment and guidance for CISOs and CIOs in informing their security policies. Turns out, we have MITRE ATT&CK! Unfortunately, I personally find MITRE ATT&CK to be overwhelming. Its incredible trove of techniques, detection strategies, and mitigations could be made more accessible to senior officers.

As an example, T1611 - Escape to Host eloquently describes the issue, examples of tools or malware that causes an issue, mitigations like read-only containers and pod security policies, and detection strategies like process spawn monitoring on container hosts.

I can understand this because of my background as a developer, DevOps administrator, and AWS infrastructure subject matter expert. Not every CISO has that technical knowledge. Likewise, I could not gauge anything in this repository that involves Active Directory. Some CISOs do have that experience.

I have heard from G Mark Hardy on CISO Tradecraft that CISOs often lack staff. Given the spectrum of issues on MITRE ATT&CK, I believe that we need guidance and recommendations on what specialties should report to a CISO to discover, formulate, enact, and verify security policies from information and application security.

Just as a CEO has reports for marketing, finance, sales, human resources, and so on, there are too many possible issues for one person as CISO to successfully grapple with security. This needs to be recognized and supported by chief officers, investors, and executive boards if we are to see an improvement.

Hey! I've since given recommendations to my organization. Check out how to convince leadership to improve security, where I share my personal experience with leadership. I found that my approach needed some refinement and it reminded me how important trust is with the chief officers.

Uber's refreshed job postings

Security is expensive. It is seen as a cost center. Until something like this happens. Unlike Patreon, maybe Uber learned that they need to hire more security people.

Whoa @Patreon laid off their ENTIRE security team. Wouldn’t trust my data there. Also there’s some amazing talent to scoop up.
Firing your security team is dumb. You cannot outsource your risk. Any sufficiently large business should have security staff to reduce risk. "Oops we got busted and leaked all our customer data" cannot be followed by "Because so and so at another company didn't do their job"
Fact: an organization's budget for cyber security increases only after an incident 🥹 #cybersecurity #hacking
Photo included with tweet
This may be a complete coincidence. Job postings are re-posted regularly to appear fresh.

You cannot delegate or outsource responsibility for risk. It is on you to protect the value of your organization and the privacy of your employees and customers. Today, it is on you to convince investors and the board that security teams are worth the expense. Tomorrow, maybe investors and the board will see things differently. Who knows.

2022-09-20 Uber's follow up

Uber provided a Security Update which describes their perspective on what happened.

  • A password was compromised and likely bought on the darkweb.
  • The target relented and accepted the push notification.
  • The attacker got access to other accounts within the perimiter.
  • The attacker defaced internal resources.

Uber says they rapidly responded to the event with existing monitoring and evicted the attacker from their network and resources. They followed several good practices in a breach such as rotating shared secrets, voiding all sessions, etc..

Uber currently believes the threat did not get customer data.

Uber has requested help from authorities to investigate.

Another recommendation

For Active Directory and Duo users, number matching is a feature in their respective applications.

A reminder to orgs using Azure MFA (incl. O365) to implement Number Matching, if using another MFA solution consider disabling push requests. docs.microsoft.com/en-us/azure/active-directory/authentication/how-to-mfa-number-match
Duo now offers number matching, it’s called Duo Verified Push - I would strongly recommend orgs who use Duo go back and enable this. duo.com/docs/policy
Photo included with tweet

Number matching makes the second factor authentication two-way. A push notification with an accept button is one way. What makes this two-way is the user enters the server's presented random code into their app which has already been trusted by the multi-authentication provider.

But, I have to wonder, can that code be scraped and presented through a well made phishing experience?
It is an improvement, but it looks like it can be man-in-the-middled to me.


To assist those who are interested in this event but lack some of the vocabulary, here's a glossary of select terms and phrases that mostly cover what you need to understand.

Multi Factor Authentication MFA
Authentication that involves a second or third factor, such as a one time pass code from a device
A technology by microsoft to automate behavior on microsoft platforms with a script (like python but their own language).
A link is sent to you through email to open, access to the email is a pseudo second factor.
Fast Identity Online (2) FIDO2
FIDO a specification that involves a secure device or chip to attest unique device identity.
SIM Swap Attack
Someone just walks up to a store or calls over the phone to a phone company claiming to be you and they get your number switched over to their phone. That way they can receive your text messages. Sometimes One-Time-Passcodes are sent over text messages to be a second factor, except the underlying device that is tied to you can be swapped to someone else.
DNS Security Extensions DNSSEC
DNSSEC is an active standard, set of infrastructure, and a group of people who implement security behind the scenes to ensure that websites you connect to are the websites you intended to connect to. For example, if someone swapped out discovery dot com and you went there to log in, you would want to be sure you were communicating to the correct website with your sensitive data. (This is in addition to HTTPS.)
Yubikey is a branded secure hardware token by Yubico. It attests user identity with FIDO2. It also has a button that if you push will spew a code that acts like a one-time-passcode, however there are issues with one time passcodes like this.
Simple Systems Manager SSM
An amazon technology which integrates with servers to provide reliable and reproducible administrative access and changes to servers at scale. A command that runs on SSM is something that changes the settings on a server through amazon.
Privileged Access Management PAM
A methodology and set of technologies that deal with securing access to resources which are powerful.
MITRE ATT&CK is a repository by MITRE that documents known threats and how those threats compromise systems.
Tactics Techniques and Procedures TTP
MITRE ATT&CK publishes TTPs. They include discovery and mitigation recommendations.