How to convince leadership to improve security
- 25 min read - Text OnlyUber's breach began with a contractor's device being compromised. Their credentials were exposed to the threat actor and by social engineering the contractor approved the threat actor's access to their account on a threat controlled device. Within Uber's network, the threat used another employee's account to access privileged credentials to important business services.
If you do not want to experience a breach like Uber, you need buy in from leadership (the chief officers) to improve your organization's security. Pointing at Uber has been highly effective, but it is not enough to just say "we can't be like them!" – leadership already thinks that. Instead, you need to understand the risks that the example suffered and produce tailored recommendations to reduce those risks.
I cover what those risks are and then discuss my experience in motivating my employer to improve their security while Uber's breach was still fresh on their minds.
Note:"IOCs of last campaign" in the above image means Indicators of Compromise, in other words: what did the threat actor do last time? The joke is: improving security for its own sake is a hard sale, improving security to not look bad is an easy sale.
Perimeter Security
Over the weekend, I collaborated with Xe Iaso on improving perimeter security. We argue that push notification two-factor auth should be considered harmful. The superior alternative to push notification authorization is WebAuthn, but this requires trusted platform modules or roaming security keys. Thankfully, this will be more accessible soon in the form of "passkeys," where a phone acts as a roaming security key.
If you cannot motivate a migration towards WebAuthn, an improvement is to use "number matching". Vendor specific authenticators facilitate a number matching handshake where their device to authenticate is showing them a code and they must use their second factor application to enter the same code.
With repeated use, end users should hopefully learn that these codes are not for anyone else to see or receive.
Privileged account security
If the threat actor was limited to just the contractor's access, we may have never heard about this incident.
This incident blew up because the threat actor got access to privileged credentials and created a lot of internal and public noise.
Uber failed to implement proper privileged account management (PAM) – they allegedly had an administrator's credential accessible in a public location. It is likely that Uber failed to follow the principle of least privilege. If they had, the human account in Thycotic (Uber's PAM vendor) would have had less access and therefore there would be less damage from this incident.
The account credentials in Thycotic were break-the-glass types, which permitted the threat actor to access Google Suite, Slack, AWS, and more.
My general recommendations
I prepared a few thoughts about Uber's breach where I proposed general recommendations for improving the security of any organization. In short I recommended the following:
- Secure your perimeter with WebAuthn.
- Take advantage of device management to authenticate with the perimeter.
- Ensure accounts are locked out after several two-factor failures – I recommend five to ten attempts produces an account lockout. Something is very unusual when a user receives a two-factor challenge and fails to pass. Their password is likely compromised.
- Remove hard-coded credentials, educate, and detect.
- Credentials can be detected with GitHub Advanced Security or GitGuardian.
- Monitoring for credentials should be applied to Slack, Jira, Confluence, and other workforce shared tools.
- IT should use source control for their scripts or some automatically audited file system or object store.
- Use high entropy generated passwords which can be detected and removed.
- Rotate credentials on shared accounts (this is harmful for individual accounts).
- Establish Privileged Account Management processes, protect privileged account credentials, and rotate them upon use. Also alarm on use.
- Include regular penetration testing with access to business and application networks.
- Include honeypot credentials to verify penetration test coverage and alarm on true threats. Find out more from Canarytokens - Quick, Free, Detection for the Masses.
At ${EMPLOYER}
I published my thoughts on Uber and then got to work on Monday.
And if you've paid attention, you'll know that on Monday, Gandalf gets to lick a graham cracker.More on this internet cat at Indoor Outdoor Kat
In my organization, I have equivalent team size and responsibility as the official titled director. I am close to the CIO and have kept the CIO informed of the Uber breach details as I learned of them. Now, without the CIO or CTO asking, I and the security engineer on my team prepared tailored and actionable recommendations for our organization.
The document layout looked something like this:
- Summary - a self-contained overview of the Uber Incident, the primary risk, and its parallels with our prior data breach.
- Organizational Recommendations - a people-centric set of recommendations focusing on educating the workforce about their relationship with IT, a reminder of what phishing looks like, and a suggestion to instill that we should rely upon video calls for confirmation if something is unusual.
IT Recommendations - a set of business software and system recommendations to improve the perimeter security and break-the-glass credential security.
- Switch away from TOTP. Promote WebAuthn adoption among privileged employees.
- Investigate an alternative solution to a VPN for administrative tool access.
- Switch password management provider because they do not support WebAuthn.
- Onboard privileged accounts in Google with their Advanced Protection Program.
- Improve cross-organization communication on employee termination.
Engineering Operation Recommendations - A set of recommendations specific to tech / engineering, separate from what IT can do.
- Migrate away from an SSH jump host.
- Migrate AWS SSO access from TOTP to WebAuthn.
- Rotate long lived secrets (only useful inside application network). Some credentials have not been changed in years.
- Separate certain applications and resources from the primary AWS account to limit lateral access.
- Reduce engineer access in the cloud console. Migrate remaining needs for administrative console use towards infrastructure as code.
- Add additional alerting when administrative access is used.
Product Recommendations - add
security.txtfor any security researchers to access, including key material for encrypted mail.
The CIO's take
By the end of Monday, we had this proposal prepared and shared with leadership.
The CIO received it quite well!
Thanks for putting this together.
I would like to look for ways we can prioritize these recommendations, or specific high-risk systems/users. Phasing-in these changes would be preferable if possible.
He had other specific feedback and details, but it boiled down to we have limited resources, so what should we prioritize first?
We recommended enabling self-enrollment for WebAuthn in our single sign on provider. We also said that we would champion adoption of WebAuthn with the existing platform authenticator built into every engineer's MacBook.
He then let us know that he was obtaining a safe so we could secure hardware tokens for AWS in a locked closet, and that it would be bolted to the building. Turns out it wasn't too costly (just $90).
The CTO's take
Likewise, he took it well too!
These are some great recommendations and action items.
Please put a cost estimate together for any additional IT spend that is required to implement these recommendations.
We then discussed the proposal briefly in our next one-on-one on Tuesday.
Verbally, he said something like this:
I was going to ask you to prepare something, but I was surprised that you did this without me even asking for it.
He took my initiative well and I mentioned that I have found an interest in CISO-like experience.
His eyes went wide, he smiled, and he replied along the lines of:
Back at ${PREVIOUS EMPLOYER}, we'd say "work the position you'd like to have". Though it might be seen as the workplace not paying you for it until it is recognized.He reaffirmed his support for me taking this action proactively. Then for a third time requested some estimated costs.
What do the chief officers want?
Based on this experience, here's what I saw and felt. The chief officers do value security, but the CIO and CTO required a subject matter expert to propose recommendations tailored to our organization.
The CIO wanted a prioritized list, specifically what we should do first. A plan like this should include phases and how each action item impacts risk.
The CTO wanted to see cost estimates and resource requirements.
Given this feedback, I conclude the proposal I made was closer to a categorized laundry list. While it was received positively, it would have gone smoother if I had presented a proposal targeted for chief officers.
For this audience, I believe the proposal would have performed better if I did this instead:
- Label the risks (hereafter "risks of interest") that Uber was compromised with, namely perimeter security and break-the-glass credential compromise.
- For each risk of interest, describe the organization's current sensitivity and potential for impact if the organization were similarly compromised (hereafter "risk profile").
- Supply immediate actions that reduce the organization's current risk profile. With each action, include how much it reduces risk and its effort, cost, and resource needs (hereafter "estimate").
- Include a road map of future actions and estimates. Do not be too detailed – detail here is a distraction.
My experience presenting to the chief officers confirms what G Mark Hardy has said in the CISO Tradecraft podcast. Essentially, make it easy for the chief officer to sponsor your plan by providing impact, cost, effort, and resource requirements.
You need a high level of trust from your chief officers if you are going to replicate what I did.
Without that trust, you will struggle to get the attention needed to make an informed decision. Without that trust, a proposal like the above will not be taken seriously and fail. Without that trust, your changes will not last.
The chief officers have limited attention to give. Your shortcut to getting your proposals accepted and supported is to develop trust with them. In a way, you are making the decision for them. Obviously, you must be trusted if you're making their decisions, and this underpins the impact you can have, and the lifetime your efforts will be in effect.
Talking with the tech team
The next day, I met with the entire tech organization (minus the CTO). This was a weekly meeting which usually runs on the short side where we catch each other up on product updates and priorities. If I did not speak, this meeting would have been over in five minutes.
Several tech team members were concerned about Uber, and this was my chance to show that we are on top of the issue. By sharing that we (the security engineer on my team and myself) got support from leadership, and that we were already taking actions to improve our risk profile, I believe that I raised and reinforced team morale.
Then, one of my reports asked an interesting question. It went along these lines:
What technology can we use to protect ourselves from phishing?
I think my response was similar to this in the moment:
Phishing is an attack where one human is attacking another human. This is inherently a social problem, and like social problems, technology can only do so much. We can reduce the risk phishing poses by educating our people and reducing what people can do outside the normal circumstances of their job.
If ${CEO person} was in Singapore for the week and you got an urgent request to wire transfer two million dollars, would you stop and think this is unusual? As we do our jobs, we perform all sorts of unwritten protocols. If you're being asked to do something outside of the normal protocols you've done before, call them up!IT will never call you for your password or your two factor credentials. This is not normal and you should report it to your boss.
The CIO then confirmed what I said and briefly described some of the next actions we were taking.
Overall, I felt like my message was received well and my message directly helped those outside of management and leadership feel better about our future.
The latest on Uber
This is not confirmed, only alleged information.
Supposedly teapotuberhacker claims responsibility for both the Rockstar Games hack and the Uber hack.
So what's next?
If teapotuberhacker and pot are the same person, this conversation suggests they also have access to DoorDash.
Whether they've released, sold, or used the DoorDash customer data is unknown.
If teapotuberhacker and pot was the 17-year-old arrested in Oxfordshire, then we may not get anything more while the investigation is underway with the suspect arrested.
The source for the above image is vx-underground. They have consistently provided accurate intelligence linking pot to these attacks.
This threat's legacy is not limited to Uber or Rockstar. They are also linked to Revolut (see Revolut data breach: 50,000+ users affected). Another image (not shown) also suggests they have access to Banco do Brasil. The following image is also attributed to vx-underground.
Uber Conclusion
When a single youth demonstrates equivalent impact to a nation state on a variety of businesses: you know we need to take security more seriously than we have.
This individual appears to have committed a series of crimes and has been caught. But they were young, noisy, and clearly not cautious. How will you protect yourself and your people from cautious, quiet threats?
Conclusion in leading security
Gain trust with those that can sponsor your security policies. Inform them of what they need to know to support you when the opportunity strikes (e.g., a market competitor fails in their security). Prioritize business specific information like immediate next actions, resource requirements, and expected expenses. Support individual contributors on the team and do not throw any individual under the bus.
We are all humans, we get tired, we make mistakes. Phishing is successful against tired humans. Phishing is a human to human threat that we seriously need to address. Our security should keep us safe when we are at our weakest, and phishing resistance should be prioritized going forward.
If you do not take security seriously, you are at risk of customer data exposure, business data exposure, a threat to business continuity, and ransomware. These threats target big and small. Being a small target will not protect you. Security is everyone's responsibility, convince your organization that everyone needs to engage and improve their security to stay open on the free market.
Final words
Security keeps our businesses, infrastructure, government institutions, and social connections alive and functioning in this world. Threat actors make a living by disrupting our way of life. Promote security and risk awareness around you. Hold your business relationships accountable. Hold your leadership accountable. Challenge them to say what they're doing to reduce their risk today.
In the future I will be writing about a ransomware attack. A small municipality was breached and they could not pay up. Now they are rebuilding their entire IT deployment from scratch and cannot serve their community. I know someone inside. This threat has turned their life upside down and their community's way of life has been impacted. This does not have to happen to you.