A few thoughts about Uber's breach - 2022-09-19

Allegedly, an eighteen-year-old spammed an employee with two-factor authentication via push notifications on an employee with a known password. They got into the VPN and scanned for servers, found a file share without any access controls, and a script that could access break-the-glass credentials. With the highest level of credentials available, they then got effective root access to Slack, AWS, Google Suite, and active directory at Uber.

Cendyne

🍿for big tech company tonight
Someone in IT did not take security awareness training.
This is not developers fault at all.

Sep 16 2022 02:50:20 UTC

Uber logo with text: "2016 was not enough" above

Hey there, if a certain phrase confuses you, check out the glossary at the bottom. Several things are explained there for those outside the tech community.
hi
talking
If you're here to learn how to improve your IT and engineering security in light of the risks Uber experienced, you will find several recommendations near the end.

Officially:

Uber Comms

We are currently responding to a cybersecurity incident. We are in touch with law enforcement and will post additional updates here as they become available.

Sep 16 2022 01:25:58 UTC (archived)

Allegedly:

Corben Leo

Apparently there was an internal network share that contained powershell scripts...

"One of the powershell scripts contained the username and password for a admin user in Thycotic (PAM) Using this i was able to extract secrets for all services, DA, DUO, Onelogin, AWS, GSuite"

photo added to the tweet

Sep 16 2022 01:17:13 UTC (archived)

Kevin Beaumont

Uber, what you need to know, the thread.

1)

photo added to the tweet

Sep 16 2022 10:15:27 UTC (archived)

Kevin Beaumont

Seriously though, ya'll should review the risk you take with the zero trust flat network dream you've been sold.

Definitely enable number matching/verified push for all logins regardless. These kind of attacks have been happening all year, at very large orgs.

photo added to the tweet

Sep 16 2022 11:25:57 UTC

vx-underground

Update: A Threat Actor claims to have completely compromised Uber - they have posted screenshots of their AWS instance, HackerOne administration panel, and more.

They are openly taunting and mocking @Uber.

photo added to the tweet

photo added to the tweet

photo added to the tweet

photo added to the tweet

Sep 16 2022 02:16:58 UTC (archived)

vx-underground

@Uber They disclosed Uber's financial data

🧐

photo added to the tweet

Sep 16 2022 02:18:51 UTC

Who's not at fault?

Is the employee who got tricked at fault?

We're talking about a human here, who was intentionally harassed late at night; the easy way out was to follow the attacker's prompt.

Our security needs to be resilient to humans that make mistakes, and our security needs to be supportive of humans who need compassion.

Ian Coldwater 📦💥

If phishing a single employee can lead to everything in your infrastructure being compromised that easily, that employee is not to blame

Sep 16 2022 09:30:03 UTC

The fact that this keeps happening and is enabled by the biggest companies out there needs to be recognized and changed.

william light

YOU CANNOT MAKE THIS SHIT UP

photo added to the tweet

Sep 10 2022 05:40:07 UTC

Cendyne

Literally how to exclude accessibility needs people, old people, fatigued people, someone who just wants a burger but can't figure out why their card is rejected people, you know: a human?

Sep 10 2022 17:28:56 UTC

This issue is not new, it is not a surprise; the abuse that opened the perimeter at Uber has been perpetuated through big companies selling insecure security.

Steve Elovitz

Seeing an increasing amount of abuse of MFA prompt "push" notifications. Attackers are simply spamming it until the users approve. Suggest disabling push in favor of pin, or something like @Yubico for simplicity. In the meantime, alert on volume of push attempts per account.

Feb 26 2022 15:44:12 UTC

break-windows
Whether or not Duo Security was specifically a gateway in the Uber incident is irrelevant. The technique of push-based approval is demonstratively unsafe.

Clerk

1. After @jack was infamously hacked through a SIM-swap attack 2019, much of the world moved away from SMS-based one-time passwords (OTPs)

The apparent @uber hack may lead to a similar decline in Magic Links... here's what you need to know 👇

Sep 16 2022 22:30:10 UTC

Matt Kane

@ClerkDev @jack @Uber Surely the real answer is to require FIDO2? That would've stopped this entirely.

Sep 16 2022 23:02:28 UTC

ssssh
I mention WebAuthn, others mention FIDO2. For this discussion these mean the same thing: using a security key or built-in chip to authenticate.
ceiling
Check out Push notification two-factor auth considered harmful where Xe Iaso and I propose that WebAuthn would have cut out the weakness that compromised Uber's security perimeter.
Everyone, even you, talks about improving security on the perimeter. That is not what made this incident so bad. Where's the beef?
bullshit

Jiro Su

Where's the Beef? May 13 2006 00:00:00 UTC

Jiro Su

Break the glass

Allegedly, the attacker got break-the-glass credentials, which gave them access to literally everything their most senior IT personnel should have access to on a limited basis.

What are "break-the-glass" credentials for?
do-you-mean-me

"Break the glass" credentials and accounts are used in rare circumstances outside the normal day-to-day operations.

Here's an example of an emergency where a break-the-glass activity is fine. As an IT administrator, you were just locked outside your account, and something requires immediate attention that cannot wait. All other IT administrators of equal or greater authorization are unavailable and you are unable to reset your account's status. You reach to break-the-glass to restore your own access, and then you return to your account to continue performing your duties.

Here's an example of a non-emergency where a break-the-glass activity is fine. As an IT or DevOps administrator, you need to provision a new AWS account for a new engineering team product. The company only creates a new product once a year and master payer account / management account is otherwise limited and off limits. You break-the-glass to access the management account to create a new account for the product, you document the new account's root user in a secured place, and set up SSO access to this account for the new team. Afterwards, you log out of the management account and continue performing your duties.

That makes a lot of sense! But who should have this level of access?
excited
think
These credentials should be restricted to seniors that have a justified reason to have these credentials, who have trust in the organization, and who have consistently demonstrated attention to detail in stressful circumstances.
corporate-drone
A new hire in IT should not be able to access break-the-glass credentials.
macro
The CEO is the biggest target and should not have access to break-the-glass credentials.
watching-you
A random person of any rank or position or lack of position should not have access to break-the-glass credentials.

Credentials to break-the-glass accounts are arguably the most important secret material to secure for an organization. It allows an organization to respond to an emergency and to respond to infrequent requirements that require risky access to complete.

How do you protect emergency credentials?

If you want a product, the key words to search are: Privileged Account Management.

Uber specifically relied upon Thycotic. There are others too, but in short these technology solutions have at least these features:

A less technological solution would be to store a static password on a Yubikey and throw it in a safe. Yes, people do this, and I think it is far better than leaving break-the-glass credentials in a public location like Uber.

Yubikey static password screen

youtube
Speaking of throwing credentials in a safe, check out how DNSSEC started in the video below! The ceremony itself is pretty boring, but the planning to make a system resilient to nuclear warfare, collusion, and individual death is neat to consider.

KSK Key Signing Ceremony (16 Jun 10) by ICANN

laser-eyes
This is an issue for crypto-currency too. People have died or disappeared while holding some perceived value that will never be recovered. Distributed autonomous organizations have had all perceived assets transferred because key material and authorization is not properly distributed among different people with different access controls in different physical locations. Alas:

Robert Leshner

Crypto is bound to repeat every mistake learned over the past hundred years in finance

Jun 11 2022 19:46:07 UTC

Should you go with a safe, have some sort of immutable log strategy in place where you document the date, the reason you accessed the key material, and which key material you accessed. Maybe even go old school and fax evidence of your activity to a third party who files it away in case of forensic analysis.

vibrating
Speaking of forensic analysis, here's a paranoid tip that may save you after a breach occurs. If you're ever doing something funky or unusual, like running an SSM command to disable MFA on a host, then create a ticket and assign it to yourself.
blep
If it takes longer than a few minutes to do your task, then document the steps you took and for what purpose. When the changes are completed or reversed, then close the ticket. You are covering your rear. In the event something happens months down the line, an auditor or forensic analyst will see your ticket and not blame you for the breach.

Lastly, and out of good practice, whoever has access to the keys on an ongoing basis should be separate from the person who momentarily uses the keys.

If you need help in person, it is usually up to the Chief Information Security Officer (CISO) to provide privileged access to you for a short amount of time. In my opinion, a CISO should not be using the credentials. Their authority should be to delegate access, not to act with access.

So, in a way, Thycotic is a virtual extension to CISO. As a Privileged Account Management product, Thycotic is fulfilling the role of delegating authorized access to an authenticated person and auditing the action.

So is Thycotic at fault? I think not.

shrug
I'm not shilling them either. I have no experience with Thycotic or alternatives. The use case here seems interesting.

How did Uber protect emergency credentials?

Again, this is based on alleged information.

citation-needed
I do not work for Uber, never have worked for Uber, I do not know anyone inside Uber and have contacted no one who works at Uber to verify.

Here's what I would expect a sane CISO to implement with a privileged account management (PAM) product:

potato-for-brainsInstead, Uber had a potato-for-brains PAM admin put their credentials in a script for Thycotic on a file share that any employee with legitimate VPN access could have accessed and then abused.

Michael Koczwara

Simple graph mapped to MITRE ATT@CK and TA TTPs used to breach UBER

https://whimsical.com/uber-breach-7JNtVoq4Tu73kBXzoisuiQ

photo added to the tweet

Sep 18 2022 09:35:50 UTC

A "fun story"

shocked
I got into security because my employer was breached. I used to be just another backend-focused developer. While on vacation at a furry convention of all places, I get a ping from my boss.
Boss: "Hey, did you spin up big GPU servers and call them analytics?"
concern
muffled-screaming
I replied, "No." But I knew what it meant. I took a deep breath and continued to enjoy my vacation.
tired-desk
The rest of the team worked until 2 AM rotating every credential. Credentials for every service were in plain text in the database, bank numbers for partners were in there too. At least we did not store credit card numbers, just tokens.
cough
Except for the credit card processor credentials. Those were hard coded.
proud
Ultimately it came to me to figure out how to encrypt necessary data at rest, to secure our credentials for future use and rotation, and to promote a secure development and cloud environment.
Boss: How did this happen!?
scream
well-heck
The CEO was targeted. He re-used his Slack password somewhere else where their passwords were breached and recovered. The attacker got into his Slack account and found that marketing shared the root AWS credentials in the clear to each other to upload to S3. The attacker got into AWS, cloned the database from a snapshot and changed its root password. The attacker dumped the data out of our account with an EC2 box. And then months later they spun up Ethereum miners because why not.
Boss: Coincidentally, I was reviewing our AWS spend a few days later and freaked out at the estimated AWS bill.
tap-tap

There are similarities. My breach and Uber's 2022 breach had unrestricted credentials in the clear where any employee could have found them.

Now I am the one who delegates authority to our AWS account. The rest of the company's IT infra is outside of my reach under the CIO. I may not be a CISO, but I think my experience and responsibilities are getting close. I am also interested in this, but I'm not sure yet on if I should jump into it.

Uber's security history

Energy and Commerce Committee

RM @FrankPallone's statement on Uber's failure to protect the data of 57 million consumers 📱🚗 💳 👎

photo added to the tweet

Nov 22 2017 15:02:57 UTC

Evan Price

.@Uber: "Oh hey, Mark had to testify at Congress because he didn't address their data breach... Maybe we should send our users a quick email?"

photo added to the tweet

Apr 13 2018 19:46:54 UTC

Katie🌻Moussouris (she/her)

@TPioreck So, you mean the same as is going on right now?
I mean, I literally was called to testify before Congress last year regarding a breach cover-up done by Uber, with the extortion/hush money paid via a complicit bug bounty platform.
Corps already underreport, it's why the regs exist

Jul 14 2019 18:10:09 UTC

Catalin Cimpanu

BREAKING: Former Uber CSO charged for covering up the company's 2016 security breach

https://www.zdnet.com/article/former-uber-cso-charged-for-2016-hack-cover-up/

photo added to the tweet

Aug 20 2020 20:53:50 UTC

Patrick C Miller

Former Uber CSO Faces New Charge for 2016 Breach https://j.mp/33ZyEor

Dec 26 2021 14:15:05 UTC

Tom Warren

Uber has been hacked, and it looks bad. The hacker got in through social engineering and allegedly found a network share full of Microsoft PowerShell scripts that included Uber admin usernames and passwords to let them breach AWS, G Suite, and more 🥲 https://www.theverge.com/2022/9/16/23356213/uber-hack-teen-slack-google-cloud-credentials-powershell

Sep 16 2022 08:54:11 UTC

Meghan Bobrowsky

This AM, I’m listening to the trial of former Uber exec who’s facing criminal obstruction charges for his role in paying hackers who breached the company in 2016.

Uber’s CEO is expected to testify today.

ICYMI, Uber is now responding to another breach: https://www.wsj.com/articles/uber-says-responding-to-cybersecurity-incident-11663299504

Sep 16 2022 16:09:15 UTC

In short, Uber had a breach, the CISO at the time covered it up and paid a ransom, and then got thrown under the bus as a criminal.

Now Uber is having another breach because the CISO or someone with equivalent access left their credentials in the clear.

_MG_

Lots of screenshots going around about Uber but this one shows how wide the hack is.
"Security Response Break Glass Service Account" password 🔥

photo added to the tweet

Sep 16 2022 04:10:51 UTC (archived)

no-bad
In the above screenshot, the credential passwords are human-generated. That to me is a big no no and reveals that they are set once and forgotten for years.

What Uber could have done better

public-speaking
The following is completely my opinion, based on my experience and what I witness in the industry. I have not executed these recommendations myself and I do not know the difficulty or burden that these recommendations impose. However, I hope that the baseline for security in our industry improves by sharing my opinion.

Perimeter security would be improved by using WebAuthn and hardware tokens for all employees and contractors. The use of push-based two-factor authentication is dangerous because the human can be harassed.

Device profiles should be a requirement on the perimeter. A certificate specific to the device (and hopefully bound to its trusted platform module) will reduce the likelihood of perimeter compromise.

Account lockouts should be the norm when a two-factor fails several times in a row. I recommend five failures. Yubikey FIPS security tokens lock out WebAuthn after three failures. Allegedly the failure was performed over a hundred times to harass the employee who gave in and approved the attacker's access.

Hard-coded credentials must be avoided. IT and engineering should be educated that a better alternative is to prompt for credentials in the script or to perform a brief SSO handshake. Ongoing education for this situation should be regularly covered and reference examples available to use.

IT should adopt source control and use tools like GitGuardian to scan and detect credentials in scripts so the credential is invalidated as soon as possible.

Human made credentials must be avoided, especially for disaster recovery and emergency use. It is not unheard of that these credentials might even be single use. A culture of using generated passwords should be adopted at all depths. Secret scanners can detect and alert high entropy generated passwords. Use this to your advantage.

Credential rotation should be regularly performed for shared accounts, including disaster recovery accounts (and a backup account in case that is destroyed). This should involve a documented process with verification steps to ensure it is smoothly executed and meets expectations. Documentation of credential rotations should be performed and preserved.

ych-bite
I do not believe that credential rotation should be a requirement for individual accounts. Current unauthorized or former employees may abuse long lived credentials for shared accounts.

Privileged Account Management break-the-glass credentials must be protected and upon use queued for immediate rotation. The use of these credentials should set off alarms and be very visible.

heard-you-were-talking
Also, any break-the-glass credentials to a privileged account management product should produce an on-call page / call to several parties. Consider this a fire drill for a very real fire that happened at Uber.

Penetration tests should be regularly performed by multiple vendors with access to the internal and application network. Uber should place multiple flags (capture the flag style) in their infrastructure, services, source code, application containers, logs, chat rooms, etc. to measure coverage that each vendor achieved as a simulation of what an attacker could have found.

Honeypot credentials should be included in the privileged account management product, their retrieval should cause an on-call system to page / call the CISO, the on-call, and a situation manager.

Conclusion

The industry needs something like OWASP that provides ongoing risk assessment and guidance for CISOs and CIOs in informing their security policies. Turns out, we have MITRE ATT&CK! Unfortunately, I personally find MITRE ATT&CK to be overwhelming. Its incredible trove of techniques, detection strategies, and mitigations could be made more accessible to senior officers.

As an example, T1611 - Escape to Host eloquently describes the issue, examples of tools or malware that causes an issue, mitigations like read-only containers and pod security policies, and detection strategies like process spawn monitoring on container hosts.

I can understand this because of my background as a developer, DevOps administrator, and AWS infrastructure subject matter expert. Not every CISO has that technical knowledge. Likewise, I could not gauge anything in this repository that involves Active Directory. Some CISOs do have that experience.

I have heard from G Mark Hardy on CISO Tradecraft that CISOs often lack staff. Given the spectrum of issues on MITRE ATT&CK, I believe that we need guidance and recommendations on what specialties should report to a CISO to discover, formulate, enact, and verify security policies from information and application security.

Just as a CEO has reports for marketing, finance, sales, human resources, and so on, there are too many possible issues for one person as CISO to successfully grapple with security. This needs to be recognized and supported by chief officers, investors, and executive boards if we are to see an improvement.

angel
Hey! I've since given recommendations to my organization. Check out how to convince leadership to improve security, where I share my personal experience with leadership. I found that my approach needed some refinement and it reminded me how important trust is with the chief officers.

Uber's refreshed job postings

Security is expensive. It is seen as a cost center. Until something like this happens. Unlike Patreon, maybe Uber learned that they need to hire more security people.

Whitney Merrill

Whoa @Patreon laid off their ENTIRE security team.

Wouldn’t trust my data there. Also there’s some amazing talent to scoop up.

Sep 08 2022 20:32:02 UTC

Cendyne

Firing your security team is dumb.
You cannot outsource your risk.
Any sufficiently large business should have security staff to reduce risk.

"Oops we got busted and leaked all our customer data" cannot be followed by "Because so and so at another company didn't do their job"

Sep 08 2022 23:25:24 UTC

Malik Mesellem

Fact: an organization's budget for cyber security increases only after an incident 🥹

#cybersecurity #hacking

photo added to the tweet

Sep 19 2022 11:17:01 UTC

i-dunno-man-im-just-a
This may be a complete coincidence. Job postings are re-posted regularly to appear fresh.

You cannot delegate or outsource responsibility for risk. It is on you to protect the value of your organization and the privacy of your employees and customers. Today, it is on you to convince investors and the board that security teams are worth the expense. Tomorrow, maybe investors and the board will see things differently. Who knows.

2022-09-20 Uber's follow up

Uber provided a Security Update which describes their perspective on what happened.

Uber says they rapidly responded to the event with existing monitoring and evicted the attacker from their network and resources. They followed several good practices in a breach such as rotating shared secrets, voiding all sessions, etc..

Uber currently believes the threat did not get customer data.

Uber has requested help from authorities to investigate.

Another recommendation

For Active Directory and Duo users, number matching is a feature in their respective applications.

Kevin Beaumont

A reminder to orgs using Azure MFA (incl. O365) to implement Number Matching, if using another MFA solution consider disabling push requests. https://docs.microsoft.com/en-us/azure/active-directory/authentication/how-to-mfa-number-match

Jun 29 2022 11:15:34 UTC

Kevin Beaumont

Duo now offers number matching, it’s called Duo Verified Push - I would strongly recommend orgs who use Duo go back and enable this. https://duo.com/docs/policy

photo added to the tweet

Aug 17 2022 16:02:14 UTC

Number matching makes the second factor authentication two-way. A push notification with an accept button is one way. What makes this two-way is the user enters the server's presented random code into their app which has already been trusted by the multi-authentication provider.

But, I have to wonder, can that code be scraped and presented through a well made phishing experience?
thinking
concern
It is an improvement, but it looks like it can be man-in-the-middled to me.

Glossary

To assist those who are interested in this event but lack some of the vocabulary, here's a glossary of select terms and phrases that mostly cover what you need to understand.

Multi Factor Authentication MFA
Authentication that involves a second or third factor, such as a one time pass code from a device
Powershell
A technology by microsoft to automate behavior on microsoft platforms with a script (like python but their own language).
A link is sent to you through email to open, access to the email is a pseudo second factor.
Fast Identity Online (2) FIDO2
FIDO a specification that involves a secure device or chip to attest unique device identity.
SIM Swap Attack
Someone just walks up to a store or calls over the phone to a phone company claiming to be you and they get your number switched over to their phone. That way they can receive your text messages. Sometimes One-Time-Passcodes are sent over text messages to be a second factor, except the underlying device that is tied to you can be swapped to someone else.
DNS Security Extensions DNSSEC
DNSSEC is an active standard, set of infrastructure, and a group of people who implement security behind the scenes to ensure that websites you connect to are the websites you intended to connect to. For example, if someone swapped out discovery dot com and you went there to log in, you would want to be sure you were communicating to the correct website with your sensitive data. (This is in addition to HTTPS.)
Yubikey
Yubikey is a branded secure hardware token by Yubico. It attests user identity with FIDO2. It also has a button that if you push will spew a code that acts like a one-time-passcode, however there are issues with one time passcodes like this.
Simple Systems Manager SSM
An amazon technology which integrates with servers to provide reliable and reproducible administrative access and changes to servers at scale. A command that runs on SSM is something that changes the settings on a server through amazon.
Privileged Access Management PAM
A methodology and set of technologies that deal with securing access to resources which are powerful.
MITRE ATT&CK
MITRE ATT&CK is a repository by MITRE that documents known threats and how those threats compromise systems.
Tactics Techniques and Procedures TTP
MITRE ATT&CK publishes TTPs. They include discovery and mitigation recommendations.