Evading Logging in the Cloud: Bypassing CloudTrail- 6 min read - Text Only
At DEF CON 31, I attended a talk at Cloud Village about how malicious actors could explore what they could do with an obtained AWS credential. The presentation covered how switching protocols can test for access without leaving a trace. It is desirable for a malicious actor to not leave traces as they explore and measure, because these events identify their intrusion and can lead to their automatic or timely removal. They can also do this by calling non-production endpoints and undocumented endpoints. AWS has fixed some of these issues and feels the other issues are not bugs.
This talk summary is part of my DEF CON 31 series. The talks this year have sufficient depth to be shared independently and are separated for easier consumption.
Nick Frichette is a senior security researcher at Datadog specializing in AWS offensive security. He regularly discovers and publishes zero-day vulnerabilities in AWS and is the creator of Hacking The Cloud and has been featured in podcasts like "Last Week in AWS": Hacking AWS in Good Faith with Nick Frichette. He presented at Cloud Village on Friday, August 11, 10:50 AM Pacific time.AWS CloudTrail vulnerability: Undocumented API allows CloudTrail bypass
To abuse a stolen or leaked AWS credential, threat actors must discover what services, resources, and permissions are attached to the credential. Nick called this permission enumeration.
First and third party tooling watch CloudTrail for permission enumeration in real time, as permission enumeration is a high quality signal of compromise. Alerts by these tools may go to the resource owner's Security Operations Center (SOC), a low-noise Slack channel, or be ignored entirely 🙂.
Nick introduced how he discovered multiple ways to do permission enumeration without leaving actionable logs in CloudTrail. The methods were:
- Changing the protocol
- Calling non-production endpoints
- Calling undocumented endpoints
Changing the protocol
He showed how changing the AWS protocol header from JSON
1.0 could enumerate access to an API without creating any CloudTrail logs. More on this can be seen on his blog at A look at AWS API Protocols and Enumerate AWS API Permissions Without Logging to CloudTrail.
My analysis is that AWS services have authorization services separate from the business logic, and that a common mistake in AWS services is that they do not sanity check inbound requests prior to authorization.
Calling non-production endpoints
Nick then shared how there are alternate endpoints, such as
ssm-gamma.us-east-1.amazonaws.com. These services may be able to read and write resources, only read resources, or not be able to read at all. However, they have the same authorization middleware in place as production and this makes for another silent privilege enumeration oracle.
Then Nick shares a twist to using non-production endpoints. Sometimes, they do log to CloudTrail. Except that the event source, being the service that was called, is the non-production endpoint. This can obscure forensic discovery and thwart realtime scanning which matches on exact matches to event sources.
Calling undocumented endpoints
iamadmin, the interface that enables AWS users to configure access roles, users, and policies, did not log any events in CloudTrail.
It is now resolved and CloudTrail logs events from IAM when the
iamadmin service is used.
Nick finished by recommending other researchers use similar tactics to find security issues:
- use certificate transparency logs to find interesting endpoints
- and experiment with cross protocol requests
At the beginning of the presentation, Nick said: "this is a defect in AWS's shared responsibility model." After seeing his work, I agree.