AWS audit logging

With a focus on Kubernetes and infrastructure orchestration at Darumatic, we often have developers testing out new tools, creating new clusters and spinning up different pieces of cloud infrastructure, this can lead to unnecessarily high sandbox cloud costs caused by forgotten and unused infrastructure. What makes shutting down or reducing unused cloud infrastructure tricky in multi-user environments is often ascertaining:

1. Who created this infrastructure?
2. What is it being used for?
3. Has it been forgotten?

There are many ways we can approach maintaining control over cloud service usage, some of these methods include access controls, methodical tagging and scheduling but today we're going to be focusing on an underutilised but an easily accessible tool - AWS CloudTrail to help us answer our first question.

Put short CloudTrail provides event history of your AWS account activity - it can tell you things from Who gave X those permissions to Who spun up those 8 rouge m4.16xlarge's. It's rarely advertised and in my experience underutilised. AWS on CloudTrail:

AWS CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account. With CloudTrail, you can log, continuously monitor, and retain account activity related to actions across your AWS infrastructure. CloudTrail provides event history of your AWS account activity, including actions taken through the AWS Management Console, AWS SDKs, command-line tools, and other AWS services. This event history simplifies security analysis, resource change tracking, and troubleshooting. In addition, you can use CloudTrail to detect unusual activity in your AWS accounts. These capabilities help simplify operational analysis and troubleshooting.

So, the first question. How much does it cost? The good news is that getting visibility into user and resource activity is free. By default AWS stores 90 days worth of management events. If you want to set up a custom trail and store this data for longer - you simply pay for the s3 storage consumed by the CloudTrail. More advanced use cases such as using this data with CloudTrail Insights incurs additional costs.

So how do we put it to work? Let's look are determining who created an ec2 instance. To do this all we need to do is find the ec2 instance ID by jumping into the ec2 dashboard and taking a look at our running instances.

Once we have the ec2 instance ID we need to head over to the CloudTrail Dashboard [console.aws.amazon.com/cloudtrail/]. Once there we can jump into Event History.

Event History

At the top of the Event History table, there is a filter bar. We're going to use this to find our ec2 instance by using the ID we found earlier by selecting Resource Name in the dropdown and then inserting our ID in the search box (ours is i-064bd95a9b9b0a4e6 in the screenshot above).

We should now be able to see the Events associated with this ec2 identifier. In this case, we can see a RunInstances event. If we click this event we can drill down into some of the details about this event including who, when and what, other resources involved in this event and also a raw Event record which includes other more specific data about this particular event (in this case extra details about our EC2 instance: network interfaces, cpu options etc).

Event Details

Event Resources

Event Record

In the Event Record (and in Details) we can see who created the instance under userIdentity, In this case Adam. We can also infer that it's creation was orchestrated by Terraform using the data in the userAgent field.

This same workflow works for all other AWS services including lambda and RDS. Generally once we've ascertained who's responsible for the creation of a particular service answer questions 2 and 3 are pretty straight forward.

CloudTrail is a very powerful tool. In this post we are only using it for a very modest task but in this case it certainly takes the leg work out of trying to figure out who created which resources.

Summary

With a focus on Kubernetes and infrastructure orchestration at Darumatic, we often have developers testing out new tools, creating new clusters and spinning up different pieces of cloud infrastructure, this can lead to unnecessarily high sandbox cloud costs caused by forgotten and unused infrastructure. What makes shutting down or reducing unused cloud infrastructure tricky in multi-user environments is often ascertaining.