A few thoughts about Uber's breach
- 36 min read - Text OnlyAllegedly, an eighteen-year-old spammed an employee with two-factor authentication via push notifications on an employee with a known password. They got into the VPN and scanned for servers, found a file share without any access controls, and a script that could access break-the-glass credentials. With the highest level of credentials available, they then got effective root access to Slack, AWS, Google Suite, and active directory at Uber.
Officially:
Allegedly:
Who's not at fault?
Is the employee who got tricked at fault?
We're talking about a human here, who was intentionally harassed late at night; the easy way out was to follow the attacker's prompt.
Our security needs to be resilient to humans that make mistakes, and our security needs to be supportive of humans who need compassion.
The fact that this keeps happening and is enabled by the biggest companies out there needs to be recognized and changed.
This issue is not new, it is not a surprise; the abuse that opened the perimeter at Uber has been perpetuated through big companies selling insecure security.
Break the glass
Allegedly, the attacker got break-the-glass credentials, which gave them access to literally everything their most senior IT personnel should have access to on a limited basis.
"Break the glass" credentials and accounts are used in rare circumstances outside the normal day-to-day operations.
Here's an example of an emergency where a break-the-glass activity is fine. As an IT administrator, you were just locked outside your account, and something requires immediate attention that cannot wait. All other IT administrators of equal or greater authorization are unavailable and you are unable to reset your account's status. You reach to break-the-glass to restore your own access, and then you return to your account to continue performing your duties.
Here's an example of a non-emergency where a break-the-glass activity is fine. As an IT or DevOps administrator, you need to provision a new AWS account for a new engineering team product. The company only creates a new product once a year and master payer account / management account is otherwise limited and off limits. You break-the-glass to access the management account to create a new account for the product, you document the new account's root user in a secured place, and set up SSO access to this account for the new team. Afterwards, you log out of the management account and continue performing your duties.
Credentials to break-the-glass accounts are arguably the most important secret material to secure for an organization. It allows an organization to respond to an emergency and to respond to infrequent requirements that require risky access to complete.
How do you protect emergency credentials?
If you want a product, the key words to search are: Privileged Account Management.
Uber specifically relied upon Thycotic. There are others too, but in short these technology solutions have at least these features:
- Unified repository for shared account credentials with excessive permissions.
- Independent (or SSO) authentication of employee access.
- Independent authorization to credentials.
- Auditing of all access to each credential.
A less technological solution would be to store a static password on a Yubikey and throw it in a safe. Yes, people do this, and I think it is far better than leaving break-the-glass credentials in a public location like Uber.
Should you go with a safe, have some sort of immutable log strategy in place where you document the date, the reason you accessed the key material, and which key material you accessed. Maybe even go old school and fax evidence of your activity to a third party who files it away in case of forensic analysis.
Lastly, and out of good practice, whoever has access to the keys on an ongoing basis should be separate from the person who momentarily uses the keys.
If you need help in person, it is usually up to the Chief Information Security Officer (CISO) to provide privileged access to you for a short amount of time. In my opinion, a CISO should not be using the credentials. Their authority should be to delegate access, not to act with access.
So, in a way, Thycotic is a virtual extension to CISO. As a Privileged Account Management product, Thycotic is fulfilling the role of delegating authorized access to an authenticated person and auditing the action.
So is Thycotic at fault? I think not.
How did Uber protect emergency credentials?
Again, this is based on alleged information.
Here's what I would expect a sane CISO to implement with a privileged account management (PAM) product:
- Users authenticate with the PAM through an SSO provider.
- All staff besides the CISO and their immediate deputies have limited authorization to resources within the PAM.
- Any disaster recovery credential for the PAM is stored securely in a place that can only be accessed by the CISO and their immediate deputies.
A "fun story"
There are similarities. My breach and Uber's 2022 breach had unrestricted credentials in the clear where any employee could have found them.
Now I am the one who delegates authority to our AWS account. The rest of the company's IT infra is outside of my reach under the CIO. I may not be a CISO, but I think my experience and responsibilities are getting close. I am also interested in this, but I'm not sure yet on if I should jump into it.
Uber's security history
In short, Uber had a breach, the CISO at the time covered it up and paid a ransom, and then got thrown under the bus as a criminal.
Now Uber is having another breach because the CISO or someone with equivalent access left their credentials in the clear.
What Uber could have done better
Perimeter security would be improved by using WebAuthn and hardware tokens for all employees and contractors. The use of push-based two-factor authentication is dangerous because the human can be harassed.
Device profiles should be a requirement on the perimeter. A certificate specific to the device (and hopefully bound to its trusted platform module) will reduce the likelihood of perimeter compromise.
Account lockouts should be the norm when a two-factor fails several times in a row. I recommend five failures. Yubikey FIPS security tokens lock out WebAuthn after three failures. Allegedly the failure was performed over a hundred times to harass the employee who gave in and approved the attacker's access.
Hard-coded credentials must be avoided. IT and engineering should be educated that a better alternative is to prompt for credentials in the script or to perform a brief SSO handshake. Ongoing education for this situation should be regularly covered and reference examples available to use.
IT should adopt source control and use tools like GitGuardian to scan and detect credentials in scripts so the credential is invalidated as soon as possible.
Human made credentials must be avoided, especially for disaster recovery and emergency use. It is not unheard of that these credentials might even be single use. A culture of using generated passwords should be adopted at all depths. Secret scanners can detect and alert high entropy generated passwords. Use this to your advantage.
Credential rotation should be regularly performed for shared accounts, including disaster recovery accounts (and a backup account in case that is destroyed). This should involve a documented process with verification steps to ensure it is smoothly executed and meets expectations. Documentation of credential rotations should be performed and preserved.
Privileged Account Management break-the-glass credentials must be protected and upon use queued for immediate rotation. The use of these credentials should set off alarms and be very visible.
Penetration tests should be regularly performed by multiple vendors with access to the internal and application network. Uber should place multiple flags (capture the flag style) in their infrastructure, services, source code, application containers, logs, chat rooms, etc. to measure coverage that each vendor achieved as a simulation of what an attacker could have found.
Honeypot credentials should be included in the privileged account management product, their retrieval should cause an on-call system to page / call the CISO, the on-call, and a situation manager.
Conclusion
The industry needs something like OWASP that provides ongoing risk assessment and guidance for CISOs and CIOs in informing their security policies. Turns out, we have MITRE ATT&CK! Unfortunately, I personally find MITRE ATT&CK to be overwhelming. Its incredible trove of techniques, detection strategies, and mitigations could be made more accessible to senior officers.
As an example, T1611 - Escape to Host eloquently describes the issue, examples of tools or malware that causes an issue, mitigations like read-only containers and pod security policies, and detection strategies like process spawn monitoring on container hosts.
I can understand this because of my background as a developer, DevOps administrator, and AWS infrastructure subject matter expert. Not every CISO has that technical knowledge. Likewise, I could not gauge anything in this repository that involves Active Directory. Some CISOs do have that experience.
I have heard from G Mark Hardy on CISO Tradecraft that CISOs often lack staff. Given the spectrum of issues on MITRE ATT&CK, I believe that we need guidance and recommendations on what specialties should report to a CISO to discover, formulate, enact, and verify security policies from information and application security.
Just as a CEO has reports for marketing, finance, sales, human resources, and so on, there are too many possible issues for one person as CISO to successfully grapple with security. This needs to be recognized and supported by chief officers, investors, and executive boards if we are to see an improvement.
Uber's refreshed job postings
Security is expensive. It is seen as a cost center. Until something like this happens. Unlike Patreon, maybe Uber learned that they need to hire more security people.
You cannot delegate or outsource responsibility for risk. It is on you to protect the value of your organization and the privacy of your employees and customers. Today, it is on you to convince investors and the board that security teams are worth the expense. Tomorrow, maybe investors and the board will see things differently. Who knows.
2022-09-20 Uber's follow up
Uber provided a Security Update which describes their perspective on what happened.
- A password was compromised and likely bought on the darkweb.
- The target relented and accepted the push notification.
- The attacker got access to other accounts within the perimiter.
- The attacker defaced internal resources.
Uber says they rapidly responded to the event with existing monitoring and evicted the attacker from their network and resources. They followed several good practices in a breach such as rotating shared secrets, voiding all sessions, etc..
Uber currently believes the threat did not get customer data.
Uber has requested help from authorities to investigate.
Another recommendation
For Active Directory and Duo users, number matching is a feature in their respective applications.
Number matching makes the second factor authentication two-way. A push notification with an accept button is one way. What makes this two-way is the user enters the server's presented random code into their app which has already been trusted by the multi-authentication provider.
Glossary
To assist those who are interested in this event but lack some of the vocabulary, here's a glossary of select terms and phrases that mostly cover what you need to understand.
- Multi Factor Authentication MFA
- Authentication that involves a second or third factor, such as a one time pass code from a device
- Powershell
- A technology by microsoft to automate behavior on microsoft platforms with a script (like python but their own language).
- Magic Links
- A link is sent to you through email to open, access to the email is a pseudo second factor.
- Fast Identity Online (2) FIDO2
- FIDO a specification that involves a secure device or chip to attest unique device identity.
- SIM Swap Attack
- Someone just walks up to a store or calls over the phone to a phone company claiming to be you and they get your number switched over to their phone. That way they can receive your text messages. Sometimes One-Time-Passcodes are sent over text messages to be a second factor, except the underlying device that is tied to you can be swapped to someone else.
- DNS Security Extensions DNSSEC
- DNSSEC is an active standard, set of infrastructure, and a group of people who implement security behind the scenes to ensure that websites you connect to are the websites you intended to connect to. For example, if someone swapped out discovery dot com and you went there to log in, you would want to be sure you were communicating to the correct website with your sensitive data. (This is in addition to HTTPS.)
- Yubikey
- Yubikey is a branded secure hardware token by Yubico. It attests user identity with FIDO2. It also has a button that if you push will spew a code that acts like a one-time-passcode, however there are issues with one time passcodes like this.
- Simple Systems Manager SSM
- An amazon technology which integrates with servers to provide reliable and reproducible administrative access and changes to servers at scale. A command that runs on SSM is something that changes the settings on a server through amazon.
- Privileged Access Management PAM
- A methodology and set of technologies that deal with securing access to resources which are powerful.
- MITRE ATT&CK
- MITRE ATT&CK is a repository by MITRE that documents known threats and how those threats compromise systems.
- Tactics Techniques and Procedures TTP
- MITRE ATT&CK publishes TTPs. They include discovery and mitigation recommendations.