S3 bucket logging

One aspect of cloud compliance I often see overlooked is audit readiness. If you work in any regulated industry or host data for your customers, you will eventually have your cloud environments audited.

When addressing an audit finding it is critical that you can answer the key questions of who, what, and when data was accessed or changed. Fail to answer these questions and your processes will be considered "out of control". Availability of logs to answer these questions can be the key difference between a minor audit finding (e.g. improved training) and having to revamp your whole change control process.

This post looks at how to automate server access logging for Amazon S3 buckets.

Server access logging provides detailed records for requests that are made to an Amazon S3 bucket. With logging enabled, the access log record contains the request type, the resources that are specified in the request, and the time and date that the request was processed. Access logging allows security teams and auditers to understand how authenticated users are interacting with your S3 buckets.

Traditional Workflow

The AWS console or APIs can be used to enable and disable access logging. When you enable this feature, Amazon periodically collects access log records, consolidates the records in log files, and then uploads log files to a target bucket as S3 objects.

Configuration considerations can increase complexity:

• The target bucket for logging must be in the same region as the source bucket.
• The target bucket must be owned by the same account as the source bucket.
• Multiple source buckets logging into the same target bucket may need different prefixes.
• Setting appropriate permissions on the logging bucket for separation of duties.
• Setting lifecycle policies for the logs based on the organization's approved log retention period.

This means that we need to create a logging destination bucket in each region we operate in, setup lifecycle policies on it, and configure it so the application team can read, but not delete from the bucket. One it is created we need to update every bucket in the account to point to that location with a designated prefix, and make sure they stay configured that way over time.

While it is possible to configure this manually for a few buckets across a couple regions, automation will be needed to support larger environments.

Get it done with Turbot Guardrails

In Turbot Guardrails, access logging guardrails control your cloud resource configurations. The key policy setting for access logging can be found directly under the resource in the policy hierarchy: AWS > S3 > Bucket > Access Logging. Policy sub-settings determine the target bucket and how to prefix the logs inside the bucket:

• AWS > S3 > Bucket > Access Logging > Bucket
• AWS > S3 > Bucket > Access Logging > Key Prefix

Using calculated policies, you can implement dynamic naming conventions to handle unique buckets per account, region and service as applicable. By using our prebuilt default logging settings for the destination and prefix, we can enable Turbot Guardrails automation with just a few clicks:

Setting the policy via Turbot Guardrails Terraform Provider is just as easy:

After setting these policies, Turbot Guardrails will identify all S3 buckets that do not have access logging enabled, and remediate them (i.e. enable the logging configuration).

If you are not yet ready to enforce remediation, you can still assess what buckets don't have access logging enabled by setting the policy value to Check: Enabled. In 'Check' mode Turbot Guardrails will alarm on buckets which do not have access logging in place. It's also possible to check that a specific logging configuration is in place by using the alternate setting: Check: Enabled to Access Logging > Bucket. After review of the alarms, you can selectively apply the enforcement settings or create exceptions as desired.

Given that access logging may not be appropriate for all buckets (e.g. development), make use of Turbot Guardrails policy exceptions as necessary to achieve your desired compliance outcome across all environments.

Make it happen!

See for yourself how easy it is to manage your access logging configurations across your cloud resources. A ready-to-run Terraform template is available to enable this configuration from the Turbot Development Kit (TDK). If you need any assistance, let us know in our Slack community #guardrails channel. If you are new to Turbot, connect with us to learn more!