Evading Logging in the Cloud: Bypassing CloudTrail

At DEF CON 31, I attended a talk at Cloud Village about how malicious actors could explore what they could do with an obtained AWS credential. The presentation covered how switching protocols can test for access without leaving a trace. It is desirable for a malicious actor to not leave traces as they explore and measure, because these events identify their intrusion and can lead to their automatic or timely removal. They can also do this by calling non-production endpoints and undocumented endpoints. AWS has fixed some of these issues and feels the other issues are not bugs.

This talk summary is part of my DEF CON 31 series. The talks this year have sufficient depth to be shared independently and are separated for easier consumption.

Nick Frichette is a senior security researcher at Datadog specializing in AWS offensive security. He regularly discovers and publishes zero-day vulnerabilities in AWS and is the creator of Hacking The Cloud and has been featured in podcasts like "Last Week in AWS": Hacking AWS in Good Faith with Nick Frichette. He presented at Cloud Village on Friday, August 11, 10:50 AM Pacific time.

The presentation

To abuse a stolen or leaked AWS credential, threat actors must discover what services, resources, and permissions are attached to the credential. Nick called this permission enumeration.

First and third party tooling watch CloudTrail for permission enumeration in real time, as permission enumeration is a high quality signal of compromise. Alerts by these tools may go to the resource owner's Security Operations Center (SOC), a low-noise Slack channel, or be ignored entirely 🙂.

Nick introduced how he discovered multiple ways to do permission enumeration without leaving actionable logs in CloudTrail. The methods were:

• Changing the protocol
• Calling non-production endpoints
• Calling undocumented endpoints

Changing the protocol

He showed how changing the AWS protocol header from JSON 1.1 to 1.0 could enumerate access to an API without creating any CloudTrail logs. More on this can be seen on his blog at A look at AWS API Protocols and Enumerate AWS API Permissions Without Logging to CloudTrail.

My analysis is that AWS services have authorization services separate from the business logic, and that a common mistake in AWS services is that they do not sanity check inbound requests prior to authorization.

Calling non-production endpoints

Nick then shared how there are alternate endpoints, such as ssm-gamma.us-east-1.amazonaws.com. These services may be able to read and write resources, only read resources, or not be able to read at all. However, they have the same authorization middleware in place as production and this makes for another silent privilege enumeration oracle.

Then Nick shares a twist to using non-production endpoints. Sometimes, they do log to CloudTrail. Except that the event source, being the service that was called, is the non-production endpoint. This can obscure forensic discovery and thwart realtime scanning which matches on exact matches to event sources.

Calling undocumented endpoints

And lastly, Nick shares how he ripped the models from the AWS Console JavaScript and found undocumented APIs. Many AWS Console services use the same APIs as your application would to access services. When those APIs make for an awkward developer experience, they may instead go through an undocumented service like iamadmin.

Somehow, iamadmin, the interface that enables AWS users to configure access roles, users, and policies, did not log any events in CloudTrail.

It is now resolved and CloudTrail logs events from IAM when the iamadmin service is used.

Last remarks

Nick finished by recommending other researchers use similar tactics to find security issues:

• use certificate transparency logs to find interesting endpoints
• extract models from boto and the console JavaScript
• and experiment with cross protocol requests

At the beginning of the presentation, Nick said: "this is a defect in AWS's shared responsibility model." After seeing his work, I agree.