One simple trick that can save 99.8% on AWS S3 KMS charges

In our last post we showed how one of our customers would save 93% on their EBS volume costsby shifting from gp2 to gp3 volume types. It was hard to top that, but we found another great opportunity... This post will show how a single configuration change to a bucket will lower our S3 KMS charges by 99.8%!

The latest version of Turbot Guardrails governance platform uses AWS serverless technologies to run guardrails at massive scale across AWS, GCP and Azure cloud services. In one of our development environments, we have 2,000+ event driven lambda functions that generate over 38 million PUT calls to S3 every month.

One of the features quietly rolled out this week at re:Invent AWS KMS S3 Bucket Keys will result in extrodinary savings for anyone using KMS keys to encrypt their S3 objects.

This is important for us because we recently saw a large jump in KMS key usage that began when we moved our process logs out of RDS to S3. Every time a guardrail runs, Turbot Guardrails writes several process logs to S3. Because we follow the best practice of encrypting everything, all objects in those buckets are encrypted with a KMS key. This results in usage charges for the AWS Key Management Service of $0.03 per 10,000 PUTs. In our dev environment with 38 million monthly events this results in charges of $114 per month or $1,368 per year.

The inital announcement intrigued us, but AWS was vague on specifics: "Bucket Keys reduce the request costs of Amazon S3 server-side encryption (SSE) with AWS Key Management Service (KMS) by up to 99%". Up to 99% could mean 1% or 99% (quite the range) so we set out to do a real world test and measure the actual cost savings for a specific use case.

The test methodology we used was to create two buckets, one with standard KMS key encryption, and one configured with the same key but with the new bucket key option enabled. (All we need now is 10,000 files to upload to each.) To create the files, we created a single large 10GiB file of random data using dd and then split that into ten thousand 100 byte files using split. Once the files were created, we used the AWS CLI to push the files into our two test buckets:

Test Results

As expected, uploading 10,001 files to the standard kms bucket resulted in 10,001 KMS requests with $0.03 showing on our bill the following day. Amazingly, the same number of files uploaded to the bucket with the 'bucket key' feature enabled resulted in only 23 key requests. That is one request per every 435 S3 PUTs. For our development account that would reduce the number of key requests from 38,000,000 to 87,356. At a billed rate of $0.03 per 10,000 requests our new KMS bill will be just $0.26, a savings of 99.8%, virtually eliminating that cost as a consideration moving forward.

We will be following up soon with a guide for using Turbot Guardrails to check if your buckets have KMS Bucket Keys enabled. If you need any assistance, let us know in our Slack community #guardrails channel. If you are new to Turbot, connect with us to learn more!