Automated AWS EMR OS controls

Last year we announced Turbot AWS EMR Guardrails that automate, manage, and secure our customer's EMR workloads that help them accelerate using AWS EMR across one or many AWS accounts.

Although the AWS EMR Service tier is well automated between out of the box AWS EMR features + Turbot Guardrails, the Operating System (OS) tier was left for the customer to manage their own controls, user management, patching, and environment variables.

In most cases, customers will run AWS EMR as a transient compute layer that solely runs for specific jobs and then shuts down. This makes assurance on OS controls less of a risk as the environment does not reside on the customer network for more than a few hours. However there are many workloads that require persistent clusters to last for days or months depending on if users are working directly in Hadoop for development or data analytics, the size of Hadoop jobs require many hours or days to run, or there is a consistent flow of jobs that keep the EMR cluster running for longer periods of time.

Up until now, most of our customers were left to manage users locally (outside of Active Directory), AWS EMR OS was not hardened to internal standards or best practices, patching was not enforced, and managing environment variables were handled manually.

Turbot AWS EMR OS Guardrails

To further accelerate our customer's use of AWS EMR and automate additional security and compliance controls, Turbot has extended our Linux hardening guardrails to also support AWS EMR!

Turbot AWS EMR OS Guardrails in Action

To enable all EMR OS Guardrails, the following Turbot Guardrails options must be enabled across the applicable account(s) using the feature:

• EC2 > EC2 App Enabled = Enabled (Whitelists EC2 for use in the account)
• EC2 > Linux CIS Hardening = Enabled (Allows Turbot Guardrails to apply customer defined CIS Level Benchmarks)
• EC2 > Linux User Management = Enabled (Allows Turbot Guardrails to sync applicable users to the cluster)
• EC2 > Linux Environment Management = Enabled (Allows Turbot Guardrails to apply various customer defined Linux environment variables)
• EC2 > Linux Patch Management = Enabled (Allows Turbot Guardrails to apply customer defined patch scheduling configurations)
• EMR > EMR App Enabled = Enabled (Whitelists EMR for use in the account)
• Linux > Linux App Enabled = Enabled (Allows Turbot Guardrails to run customer defined Linux configurations)

Your existing users in your account(s) will need to have some level of Linux permission assigned in Turbot Guardrails for user management feature to sync the user to the Cluster. The example below shows both Bob and Nathan have Linux/Admin permissions in this example:

For Turbot Guardrails to automate these OS guardrails, the Turbot Guardrails Key Pair must be associated to the cluster during provisioning:

Once the EMR Cluster provisions, Turbot Guardrails detects the new instance, and then checks to see if it adheres to CIS, User Management, Patching, and Environment policies. Turbot Guardrails continuously applies these configurations every 10 minutes across applicable EMR Clusters; if the configurations meet policy then Turbot Guardrails skips applying configurations.

For the EMR Cluster below, Turbot Guardrails detected it was out of policy and automatically applied the following controls:

EMR Master Node: i-0cf07c8a

Turbot Guardrails detects i-0cf07c8a, applies the following OS guardrails:

Turbot Guardrails applies CIS Level Benchmarks, changing 77 items to become CIS compliant (summary output of job run):

Turbot Guardrails applies User Management, syncing both Bob and Nathan who have Linux permissions in this account. Bob can then use his same SSH credentials to login to EMR as he would for any other Linux instance in any of his Linux instances across any account he has access to:

Turbot Guardrails also applies the patching configuration for scheduling and environment variables for time zone, DNS, and various Linux utilities.

Available Now

Turbot AWS EMR OS guardrails are available now in all Turbot Guardrails accounts that have the correct options enabled.