AWS CloudTrail logs give you a record of who did what, when. From a security perspective, analyzing this behavior helps identify patterns of excessive privilege, suspicious access, or unintended activity.

There are countless ways to explore your CloudTrail data, but if you're just trying to ask a few pointed questions without setting up your entire SIEM pipeline first, [Tailpipe](https://tailpipe.io) is a fast way to start. With a few SQL queries, you can quickly dig into your environment and surface potential risks, right from your terminal.

*We will assume you have Tailpipe installed and configured to collect your AWS CloudTrail logs. If not, you can [download Tailpipe](https://tailpipe.io/downloads) and install the [AWS plugin](https://hub.tailpipe.io/plugins/turbot/aws) to get started!*

>For all of these queries, you can paste them right into the Tailpipe interactive query shell by running `tailpipe query` or you can save them as `.sql` files and run them using `tailpipe query <filename.sql>`.

## Who is accessing my AWS account the most?

A good place to start is to list the top 5 actors by frequency, excluding AWS services and service roles (to focus on human users). Seeing high amounts of activity from unexpected users can be a signal for further investigation.

```sql
select
  user_identity.arn as "Actor",
  count(*) as "Logs"
from
  aws_cloudtrail_log
where
  user_identity.type != 'AWSService'
  and user_identity.arn not like '%AWSServiceRole%'
group by
  user_identity.arn
order by
  count(*) desc
limit
  5;
```

Using Tailpipe + sample data from the [flAWS challenge](http://flaws.cloud), we see an output like this:


```
+-----------------------------------------------------------------------------+--------+
| Actor                                                                       | Logs   |
+-----------------------------------------------------------------------------+--------+
| arn:aws:iam::811596193553:user/backup                                       | 915834 |
| arn:aws:iam::811596193553:user/Level6                                       | 905082 |
| arn:aws:sts::811596193553:assumed-role/flaws/i-aa2d3b42e5c6e801a            | 17208  |
| arn:aws:sts::811596193553:assumed-role/SecurityMonkey/secmonkey             | 12354  |
| arn:aws:iam::811596193553:root                                              | 10997  |
+-----------------------------------------------------------------------------+--------+
```

You can review this list to determine if any of these users are performing an unusual number of actions or if any of them are unauthorized.

## Is anyone logging in as the IAM root user?

Logging on as the root user is generally discouraged unless absolutely necessary, as it can lead to security vulnerabilities. Monitoring root user activity is crucial for maintaining a secure AWS environment. Use this query to identify any console logins by the root user:

```sql
select
  tp_timestamp as timestamp,
  tp_source_ip as source_ip
from
  aws_cloudtrail_log
where
  event_source = 'signin.amazonaws.com'
  and event_name = 'ConsoleLogin'
  and user_identity.type = 'Root'
  and error_code is null
order by
  event_time desc;
```

The output from this query looks like this:

```
+---------------------+-----------------+
| timestamp           | source_ip       |
+---------------------+-----------------+
| 2020-10-07 15:52:02 | 3.1.254.37      |
| 2020-10-06 21:10:45 | 3.1.254.37      |
| 2020-08-06 15:57:23 | 252.1.22.60     |
| 2020-05-22 18:09:17 | 252.1.22.60     |
| 2019-08-27 14:34:27 | 228.139.46.252  |
| 2019-08-23 15:47:30 | 228.139.46.252  |
+---------------------+-----------------+
```

We can review the timestamps and source IP addresses to determine if any of these logins are unauthorized or suspicious.

## Has anyone been granted the AdministratorAccess policy?

Assigning the `AdministratorAccess` policy grants **full access** to AWS resources, increasing the risk of unauthorized privilege escalation.  Monitoring for when this policy was attached to an IAM user helps enforce the principle of least privilege and ensures that administrative permissions are only assigned when necessary.

```sql
select
  tp_timestamp as timestamp,
  string_split(event_source, '.') [ 1 ] || ':' || event_name as operation,
  request_parameters ->> 'userName' as resource,
  user_identity.arn as actor,
  tp_source_ip as source_ip
from
  aws_cloudtrail_log
where
  event_source = 'iam.amazonaws.com'
  and event_name = 'AttachUserPolicy'
  and (request_parameters ->> 'policyArn') like 'arn:%:iam::aws:policy/AdministratorAccess'
  and error_code is null
order by
  event_time desc;
```

Let's review the result:

```
+---------------------+----------------------+----------+--------------------------------+-----------------+
| timestamp           | operation            | resource | actor                          | source_ip       |
+---------------------+----------------------+----------+--------------------------------+-----------------+
| 2017-05-16 03:03:56 | iam:AttachUserPolicy | piper    | arn:aws:iam::811596193553:root | 255.253.125.115 |
+---------------------+----------------------+----------+--------------------------------+-----------------+
```

We can see that the user `piper` has been granted very high permissions. If we don't recognize this user as requiring these permissions, this is a definite red flag!

## Has the master password of an RDS DB instance been modified?

Monitoring changes to the master password of Amazon RDS DB instances is essential for maintaining database security. Use this query to track when that password was modified:

```sql
select
  tp_timestamp as timestamp,
  request_parameters ->> 'dBInstanceIdentifier' as db_instance,
  user_identity.arn as actor,
  tp_source_ip as source_ip,
  aws_region as region,
from
  aws_cloudtrail_log
where
  event_source = 'rds.amazonaws.com'
  and event_name = 'ModifyDBInstance'
  and (request_parameters -> 'masterUserPassword') is not null
  and error_code is null
order by
  event_time desc;
```

The output will look similar to the following:

```
+---------------------+-------------+--------------------------------+------------+------------+
| timestamp           | db_instance | actor                          | source_ip  | region     | 
+---------------------+-------------+--------------------------------+------------+------------+
| 2020-10-06 21:10:45 | finance-db  | arn:aws:iam::811596193553:root | 3.1.254.37 | us-east-1  |
+---------------------+-------------+--------------------------------+------------+------------+
```

We can review the timestamps, database instance, and actors to determine if any of these password modifications are unauthorized or suspicious.

## Are any of my S3 buckets publicly accessible?

Public access to S3 buckets can expose sensitive data to unauthorized users, increasing the risk of data breaches. This query helps you detect when public access was granted to an S3 bucket by modifying its policy. Based upon the results, you can take immediate action to restrict public access if necessary.

```sql
select
  tp_timestamp as timestamp,
  string_split(event_source, '.') [ 1 ] || ':' || event_name as operation,
  request_parameters ->> 'bucketName' as bucket_name,
  user_identity.arn as actor,
  tp_source_ip as source_ip
from
  aws_cloudtrail_log
where
  event_source = 's3.amazonaws.com'
  and event_name = 'PutBucketPolicy'
  and (
    json_contains(
      request_parameters -> 'bucketPolicy',
      '{"Principal": "*"}'
    )
    or json_contains(
      request_parameters -> 'bucketPolicy',
      '{"Principal": {"AWS": "*"}}'
    )
  )
  and json_contains(
    request_parameters -> 'bucketPolicy',
    '{"Effect": "Allow"}'
  )
  and error_code is null
order by
  event_time desc;
```

This will result in output like this:

```
+---------------------+--------------------+------------------------------------------------------+--------------------------------+-----------------+
| timestamp           | operation          | bucket_name                                          | actor                          | source_ip       |
+---------------------+--------------------+------------------------------------------------------+--------------------------------+-----------------+
| 2017-02-26 20:07:12 | s3:PutBucketPolicy | theend-c1aad500c62e2a57cf12cebf93b282cf.flaws.cloud  | arn:aws:iam::811596193553:root | 255.253.125.115 |
| 2017-02-26 19:49:02 | s3:PutBucketPolicy | level5-8dbc21dbbd6129208d054061ef063f04.flaws.cloud  | arn:aws:iam::811596193553:root | 255.253.125.115 |
| 2017-02-12 22:19:13 | s3:PutBucketPolicy | b5677c799b465420d8e7b0a6689a0bb0c4afbc9e.flaws.cloud | arn:aws:iam::811596193553:root | 255.253.125.115 |
+---------------------+--------------------+------------------------------------------------------+--------------------------------+-----------------+
```

Review the bucket names, timestamps, and actors to determine if any of these public access grants are unexpected.

## Conclusion

These questions are just the beginning of what you can ask about your AWS CloudTrail logs using Tailpipe. Since they are all just SQL, it's easy to modify these queries for your own specific needs.

As a bonus, these queries are all built into the [Powerpipe AWS CloudTrail Detections mod](https://hub.powerpipe.io/mods/turbot/tailpipe-mod-aws-cloudtrail-log-detections), which you can use to visualize the results in interactive dashboards.

Download [Tailpipe](https://tailpipe.io/downloads) to give this a try! What other questions do you like to ask about your AWS security environment? [Join the conversation](http://turbot.com/community/join) and let us know!



**TL;DR?** We've got you covered.

Our announcements this week span the breadth of Turbot offerings:

**Guardrails**
- GitHub organization and repository governance
- OpenTofu support for cloud landing zones
- Native cloud organization sync for AWS, Azure & GCP 
- Cloud activity tracking with new dashboards
- Control muting for alert suppression

**Tailpipe**
- Introducing Tailpipe: select * from logs;
- The next generation of SIEM, powered by DuckDB and Parquet

**Powerpipe**
- MITRE ATT&CK-aligned detection mods for AWS, Azure, GCP
- Enhanced filtering and visualization

## Monday January 27: Securing GitHub with Guardrails

**[GitHub Organization Governance](https://turbot.com/guardrails/blog/2025/01/guardrails-for-github)**

Guardrails extends governance capabilities to GitHub organizations and repositories. This integration enables continuous monitoring of GitHub configurations, real-time detection of misconfigurations, and automated remediation. Key features include inventory tracking, audit trails, SecOps alerting for issues like unprotected branches and exposed secrets, and continuous compliance with automated corrections.

## Tuesday January 28: Cloud Landing Zones with OpenTofu

**[OpenTofu Support](https://turbot.com/guardrails/blog/2025/01/guardrails-opentofu)**

OpenTofu is now supported in new and improved Guardrails Stack [Native] controls. This enables cloud teams to deploy and manage standardized patterns across their enterprise cloud environment while leveraging the community-driven, Linux Foundation-backed OpenTofu project. The integration provides unified governance, policy-driven control, automated drift correction, and context-aware deployments.

## Wednesday January 29: Cloud Organization Integration

**Native Cloud Organization Support**
- [AWS Organizations](https://turbot.com/guardrails/blog/2025/01/aws-org-sync)
- [Azure Tenants](https://turbot.com/guardrails/blog/2025/01/azure-tenant-sync)
- [GCP Organizations](https://turbot.com/guardrails/blog/2025/01/gcp-org-sync)

AWS Organizations, Azure Tenants, and GCP Organizations can now be imported into Guardrails with a few clicks and keystrokes. This enables teams to automatically sync and maintain their entire cloud organization structure in Guardrails. The feature provides instant discovery of accounts/subscriptions/projects, continuous synchronization as structures evolve, and flexible exclusion policies.

**[Control Muting](https://turbot.com/guardrails/blog/2025/01/control-muting)**

Guardrails controls can now be muted, enabling teams to focus on priorities by selectively suppressing controls based on context without compromising their security posture. It's particularly useful during scheduled maintenance, when actively addressing an issue, and when tuning detection logic.


**[Activity Tracking](https://turbot.com/guardrails/blog/2025/01/cloud-activity-tracking)**

New activity tracking dashboards for Guardrails enhance visibility into cloud environments. These dashboards transform comprehensive audit data into daily snapshots and trends, providing insights into controls, resources, and actions over time.

## Thursday January 30: Introducing Tailpipe

**[Tailpipe Launch](https://tailpipe.io/blog/introducing-tailpipe)**

Tailpipe, the first SIEM for builders, is launched! This lightweight and open source tool brings unified log collection and SQL-based analysis to your terminal, enabling you to analyze millions of events in seconds. It includes plugins for major cloud providers and integrates seamlessly with Powerpipe for advanced visualization and interactive threat hunting.

**[Next-generation SIEM](https://tailpipe.io/blog/new-siem-era)**

Lightweight, fully open source, and powered by DuckDB and Parquet, Tailpipe stretches the boundaries of SIEM. Fast, flexible, and developer-first, it's a natural fit for DevOps teams constrained by heavy, vendor-locked tools.

**[Detection Mods](https://powerpipe.io/blog/powerpipe-detection-mods)**

Powerpipe adds new detection mods powered by Tailpipe. Aligned with the MITRE ATT&CK framework, these mods provide pre-built dashboards and detections to monitor and analyze activity across AWS CloudTrail, Azure Activity, and GCP Audit logs.

**[Enhanced Visualization](https://powerpipe.io/blog/row-filters-charts-and-more)**

Powerpipe gains powerful new visualization features including row-level filters, column filters, row views, and heatmap charts. These enhancements make benchmarks and dashboards more interactive and expressive than ever.

## The B-sides

These were the Launch Week 7 hits, but plenty of other good things happened during this cycle. Check the [B-sides](https://turbot.com/blog/2025/01/launch-week-7-b-sides) for details!

## And that's a wrap!

Launch Week 7 represents a significant expansion of our cloud governance and security capabilities. From securing GitHub organizations to native cloud structure integration, from OpenTofu landing zones to advanced log analysis with Tailpipe and Powerpipe, these releases strengthen our commitment to comprehensive cloud intelligence and security. And there's more to come. Enjoy the current batch of new products and features, [let us know](https://turbot.com/community/join) how it goes, and stay tuned for our next Launch Week in a few months!

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/IDcrIRl8G2Y"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Launch Week 7 is a wrap!"
>
</iframe>
</div>




As [Launch Week 7](https://turbot.com/launch-week/07) draws to a close, we wanted to take a moment to highlight some of the exciting updates and announcements that slipped under the radar this week across our Turbot products and open-source projects.

## Guardrails: New AWS, Azure and Kubernetes controls

We continue to listen to customer feedback to add new cloud resource types and controls for security, compliance, and operational guardrails. Since last Launch Week we added new capabilities in our AWS, Azure, and Kubernetes mods:

- **AWS**:
  - Configure the maximum aggregation interval in the `AWS > VPC > VPC > Flow Logging` control for Cloud Watch and/or S3.
  - Check or enforce SQS queues for SSE encryption in the `AWS > SQS > Queue > Encryption at Rest` control.
  - New budget state added: `On Target`. For example, you can set the `AWS > Account > Budget > Enabled` policy to `Check: Budget > State is On Target`.
  - Check or enforce deletion for any RDS DB Clusters that lack encryption at rest in the `AWS > RDS > DB Cluster > Approved > Encryption at Rest` control.
- **Azure**:
  - New resource type added to support `Azure SQL Managed Instances` in the CMDB, along with Active, Approved and Tagging controls.
  - Refreshed the list of supported Azure regions to the latest available.
  - Additional filtering capabilities added to the `Azure > Turbot > Event Poller` options to exclude events from the control.
- **Kubernetes**: 
  - Added additional ServiceNow Import Set sync controls for Cron Jobs, Daemon Sets, Ingress, Jobs, Persistent Volumes, Replication Controllers, and Stateful Sets.
  - Check if Kubernetes clusters are approved for use in the `Kubernetes > Cluster > Approved` control.

## Steampipe: New tables and columns supported

[Steampipe](https://steampipe.io) has expanded its capabilities with 15 new tables and columns across major cloud providers to query more of your cloud data with SQL:

**[AWS Steampipe plugin](https://hub.steampipe.io/plugins/turbot/aws)** added six new tables including `aws_keyspaces_table`, `aws_costoptimizationhub_recommendation`, and `aws_config_delivery_channel`. The `aws_kms_key` table includes `multi-region` columns, and there are new qualification options to the `aws_accessanalyzer_finding` table.

**[Azure Steampipe plugin](https://hub.steampipe.io/plugins/turbot/azure)** added a new `azure_compute_virtual_machine_size` table, and a new column `firewall_rules` added to the `azure_postgresql_flexible_server` table.

**[GCP Steampipe plugin](https://hub.steampipe.io/plugins/turbot/gcp)** added a new `gcp_compute_instance_group_manager` table, and additional columns for `labels` and `tags` to the `gcp_compute_global_forwarding_rule` table, along with many new columns for the `gcp_sql_database_instance` table.

**[Alibaba Cloud Steampipe plugin](https://hub.steampipe.io/plugins/turbot/alicloud)** added a new `alicloud_alidns_domain` table to query their domain record metadata.

## Powerpipe: Install mods from GitLab projects

Extending Powerpipe's support for [local mod](https://powerpipe.io/docs/reference/cli/mod#powerpipe-mod-install) and remote mod installations from GitHub, Powerpipe now supports mod installs from GitLab projects. Set your [POWERPIPE_GIT_TOKEN](https://powerpipe.io/docs/reference/env-vars/powerpipe_git_token) and run `powerpipe mod install gitlab.com/path/to/some/project` to install.


## Pipes: New security benchmarks and controls supported
[Powerpipe](https://powerpipe.io) has added 3 new security benchmarks and additional controls to our AWS & Azure compliance mods. These new benchmarks are available in the [Powerpipe Hub](https://hub.powerpipe.io) and [Turbot Pipes](https://turbot.com/pipes) to assess your security posture and share status with your team. The latest benchmarks include:

- [Azure CIS v3.0.0](https://hub.powerpipe.io/mods/turbot/steampipe-mod-azure-compliance/benchmarks/benchmark.cis_v300)
- [AWS CIS v4.0.0](https://hub.powerpipe.io/mods/turbot/steampipe-mod-aws-compliance/benchmarks/benchmark.cis_v400)
- [AWS New York State Department of Financial Services (NYDFS) Cybersecurity Regulation](https://hub.powerpipe.io/mods/turbot/steampipe-mod-aws-compliance/benchmarks/benchmark.nydfs_23)


<div style={{width:"100%", textAlign:"center"}}>
  <img alt="AWS New York State Department of Financial Services (NYDFS) Cybersecurity Regulation Powerpipe benchmark" src="/images/blog/2025-01-launch-week-7-b-sides/aws-nydfs23-benchmark.png" />
</div>


## Pipes: 100+ new Azure 'detect and correct' pipelines 
[Flowpipe](https://flowpipe.io) has added 100+ new 'detect and correct' pipelines to identify Azure resources that are non-compliant with common security and compliance checks. These pipelines can also remediate non-compliant findings automatically or with approval steps. You can run them individually through the [Azure Compliance Mod](https://hub.flowpipe.io/mods/turbot/azure_compliance) or compiled as a [Azure CIS v3.0.0](https://hub.flowpipe.io/mods/turbot/azure_cis/pipelines/azure_cis.pipeline.cis_v300) baseline. These new pipelines are available in the [Flowpipe Hub](https://hub.flowpipe.io) and [Turbot Pipes](https://turbot.com/pipes) to ensure your security posture and share status with your team.


## Pipes: Enterprise tenant snapshot visibility

Enhance control over your workspace data sharing with new snapshot visibility options in Pipes Enterprise. Set permissions to restrict snapshot sharing across your workspaces to only those with authenticated access, or optionally allow link sharing visible to anyone with a link.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Pipes Tenant Snapshot Visibility configuration" src="/images/blog/2025-01-launch-week-7-b-sides/snapshot-visibility.png" />
</div>

## Pipes: Last activity tracking

You can now track the `last activity` timestamp of when a user last performed some activity in a Pipes Tenant, Organization or Workspace. This helps Pipes administrators identify when users are no longer active to determine whether they should reach out to the user to re-engage or remove them from Pipes.


<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Pipes User Last Activity Tracking" src="/images/blog/2025-01-launch-week-7-b-sides/last-activity.png" />
</div>


## Hacktoberfest 2024

[Hacktoberfest 2024](https://turbot.com/blog/2024/09/hacktoberfest-2024) was a big success, with worldwide community contributions across our open-source projects, including Steampipe, Powerpipe, and Flowpipe, along with our samples and docs repos for Turbot Pipes and Guardrails. 

We've seen a variety of improvements, ranging from bug fixes to feature enhancements across our repositories. It's inspiring to see the passion and creativity from our contributors. Whether it's fixing a bug, adding new features, or creating insightful content, every effort helps make our open-source community stronger.

Contributors who participated in this year's event earned some exclusive [Turbot swag](https://turbot.com/blog/2024/09/hacktoberfest-2024#any-special-swag-from-turbot)!

<div style={{ display: "flex", justifyContent: "center" }}>
  <blockquote class="twitter-tweet docs"><p lang="en" dir="ltr">Thanks <a href="https://twitter.com/turbothq?ref_src=twsrc%5Etfw">@turbothq</a> for the swag 😬 happy to contribute 🖖 <a href="https://t.co/lArjA4y3ME">pic.twitter.com/lArjA4y3ME</a></p>&mdash; Andoni A. (@andoni013) <a href="https://twitter.com/andoni013/status/1879421525687955963?ref_src=twsrc%5Etfw">January 15, 2025</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
</div>

We accept contributions year-round and will continue to send out swag as a small token of appreciation. We look forward to seeing what you build next!

## Flip over to A-sides for the Wrap Up
Thank you for joining us for another exciting Launch Week! Check out the week's daily announcements summary in our [Launch Week 7 Wrap Up](https://turbot.com/blog/2025/01/launch-week-7-wrap) post. Stay connected with us in our [Slack community](https://turbot.com/community/join) for our next Launch Week in a few months! 


Tailpipe brings easy, powerful, and consistent log collection and analysis right into your terminal. It provides:

- **Unified log collection**. Tailpipe [plugins](https://hub.tailpipe.io/plugins) — for AWS, GCP, Azure, and more — all work the same way. No more juggling a hodgepodge of collectors.

- **SQL-based analysis**: Query diverse log sources using SQL, the data access standard that's been rocking it for decades.

Because Tailpipe runs locally, you keep control of your data without needing to provision complex infrastructure. The collection process maps log records to schemas with common fields like IP address, so you can correlate log events from sources within one cloud platform (e.g. AWS CloudTrail and ALB logs) and across other supported sources. 

Tailpipe's engine, DuckDB, slices through hundreds of millions of log records in seconds. Tailpipe also works with Powerpipe, enabling a [new breed of mods](https://hub.powerpipe.io?engines=tailpipe) that run MITRE ATT&CK-aligned detections and provide interactive visualization of your log data.

Let's dive in and see how it works!

## Collect AWS CloudTrail logs

Start by [downloading and installing](https://tailpipe.io/downloads) Tailpipe:

```
brew install turbot/tap/tailpipe
```

or

```
sudo /bin/sh -c "$(curl -fsSL https://tailpipe.io/install/tailpipe.sh)"
```

Then install the [AWS plugin](https://hub.tailpipe.io/plugins/turbot/aws/):

```
tailpipe plugin install aws
```

Now you can collect your first batch of log records. This command collects logs for the most recent 7 days:

```
tailpipe collect aws_cloudtrail_log
```

Want more? Specify a start date:

```
tailpipe collect aws_cloudtrail_log --from 2024-01-1
```

`aws_cloudtrail_log` is the name of a [table](https://hub.tailpipe.io/plugins/turbot/aws/tables/aws_cloudtrail_log) defined and populated by the AWS plugin. Of course there's configuration behind the scenes, but not much! It can be as simple as:

```hcl
connection "aws" "prod" {
  profile = "log-admin"
}

partition "aws_cloudtrail_log" "prod" {
  source "aws_s3_bucket" {
    connection = connection.aws.prod
    bucket     = "aws-cloudtrail-logs-12345"
  }
```
The [connection](/docs/manage/connection) authenticates Tailpipe using an AWS profile. The `partition` defines a [source](/docs/manage/source) which names the S3 bucket where the logs accumulate. 

And that's it! With this minimal setup you can collect days, months, or years of log data, right up to today. If you run the same command tomorrow, Tailpipe will pick up where it left off and collect one more day of log data. 

## Query CloudTrail logs

The first thing you'll want to know is how many records you collected. 

```sql
tailpipe query "select count(*) from aws_cloudtrail_log"
```

After that, the sky's the limit. The [schema](https://tailpipe.io/plugins/turbot/aws/tables/aws_cloudtrail_log#schema) for the table will give you a sense of what's possible, but there's no need to start from a blank slate. The plugin documentation provides common examples:

```sql
-- List the top 10 events and how many times they were called.

 select
  event_source,
  event_name,
  count(*) as event_count
from
  aws_cloudtrail_log
group by
  event_source,
  event_name,
order by
  event_count desc
limit 10;
```

```bash
+-------------------+---------------------------+-------------+
| event_source      | event_name                | event_count |
+-------------------+---------------------------+-------------+
| ec2.amazonaws.com | RunInstances              | 1225268     |
| ec2.amazonaws.com | DescribeSnapshots         | 101158      |
| sts.amazonaws.com | AssumeRole                | 78380       |
| s3.amazonaws.com  | GetBucketAcl              | 19095       |
| ec2.amazonaws.com | DescribeInstances         | 18366       |
| sts.amazonaws.com | GetCallerIdentity         | 16512       |
| iam.amazonaws.com | GetPolicyVersion          | 14737       |
| s3.amazonaws.com  | ListBuckets               | 13206       |
| ec2.amazonaws.com | DescribeSpotPriceHistory  | 10714       |
| ec2.amazonaws.com | DescribeSnapshotAttribute | 9107        |
+-------------------+---------------------------+-------------+
```

Using [basic SQL idioms](https://tailpipe.io/docs/sql) you can explore your logs, find patterns, and extract insights. 

## From ad-hoc analysis to formal detection

While ad-hoc SQL queries are a handy way to explore and investigate, you'd also like a more systematic way to identify patterns that indicate security threats, compliance violations, or operational issues. A **detection** is a formalized query that looks for such patterns. The new Tailpipe-powered [Powerpipe mods](https://hub.powerpipe.io?engines=tailpipe) provide libraries of detections packaged as benchmarks aligned with the MITRE ATT&CK framework.

To see the [AWS CloudTrail Log Detections](https://hub-powerpipe-io/mods/turbot/aws_cloudtrail_log_detections) mod in action, first [install Powerpipe](https://powerpipe.io/downloads) if you haven't already. Then:

```
mkdir cloudtrail_logs
cd cloudtrail_log
powerpipe mod install github.com/turbot/tailpipe-mod-aws-detections
powerpipe server
```

Visit http://localhost:9033 in a browser and select *MITRE ATT&CK v16.1 for CloudTrail Logs*. The view defaults to the most recent 7 days, if you've collected more you can use the **Custom** date range widget to expand the view. 

![](/images/blog/cloudtrail-detections-by-mitre.png)

The detections are powered by a set of [well-documented queries](https://hub.powerpipe.io/mods/turbot/tailpipe-mod-aws-cloudtrail-log-detections/queries) that report one or more log rows when they detect named patterns. Overwhelmed by too many rows? Use Powerpipe's new [inline filters](https://powerpipe.io/blog/row-filters-date-inputs-and-more) to refine the view by iteratively discarding benign items in order to focus on a potentially malicious subset.

## Roll your own detections

Like [Steampipe](https://steampipe.io), Tailpipe is an open source engine that powers an ecosystem of mods, also open source, that share a common style: HCL to configure [controls](https://powerpipe.io/docs/powerpipe-hcl/control) and [detections](https://powerpipe.io/docs/powerpipe-hcl/detection), SQL to flow data into them. The patterns encoded in Tailpipe mods are simple and easy to emulate. Here's the query for the `IAM Root User Console Login` detection shown in the screenshot.

```sql
select
  *
from
  aws_cloudtrail_log
where
  event_source = 'signin.amazonaws.com'
  and event_name = 'ConsoleLogin'
  and user_identity.type = 'Root'
  and json_extract_string(response_elements, '$.ConsoleLogin') = 'Success'
```

Hunting for patterns like this requires no special SQL expertise. If you know what you're looking for, it's trivial to write a query to find it. You can package such queries as detections and combine those into [benchmarks](https://powerpipe.io/docs/powerpipe-hcl/benchmark) that extend existing mods or implement new ones.

## See it in Action

Watch this demo of both the Tailpipe CLI and Powerpipe Detection Mods:

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/IR9MK1DMvW4"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Introducing Tailpipe: select * from logs;"
>
</iframe>
</div>

## Getting started

To get started with Tailpipe: [download](/downloads) the tool, follow the [tutorial](/docs), and explore the [plugins](https://hub.tailpipe.io#plugins) and [mods](https://hub.powerpipe.io?engines=tailpipe). Then start collecting, querying, and visualizing your logs and [let us know](https://turbot.com/community/join) how it goes!





SIEM (Security Information and Event Management) is evolving. From legacy platforms to cloud-native systems, and now to open source, local-first solutions, this progression  mirrors the broader story of big data: more flexibility, more accessibility, and more control.

[Tailpipe](/) is the next step in this journey. Lightweight, fully open source, and powered by [DuckDB](https://duckdb.org) and [Parquet](https://parquet.apache.org/), it stretches the boundaries of SIEM. Tailpipe is fast, flexible, and developer-first: a natural fit for DevOps teams constrained by heavy, vendor-locked tools.

## Three eras of SIEM

SIEM has evolved through three distinct phases, each bringing new approaches to security data management.

### Legacy

Centralized systems like Splunk, ArcSight, and QRadar pioneered log collection and analysis. But they require heavy  infrastructure, use specialized query languages, incur high costs, and lock users into proprietary ecosystems.

### Cloud database

Cloud-native platforms like Panther introduce familiar standards (SQL, Python) and use the cloud (Snowflake) for large-scale storage. They are more accessible, but still require heavy infrastructure and impose hard trade-offs on cost vs. scale.

### Open, local-first

Tailpipe represents a new approach: local-first, open source, standards-based. Logs are collected, enriched, standardized, and filtered using open source plugins. Data is stored in optimized Parquet format with a simple hive structure that's easy to manage locally or in low-cost cloud object storage. DuckDB and SQL put lightning-fast query right at your fingertips.

## Why is Tailpipe different?

Tailpipe overcomes the limitations of traditional systems while empowering developers.

### Open source and transparent

Everything about Tailpipe — from its [CLI](https://github.com/turbot/tailpipe) to its [plugins](https://github.com/topics/tailpipe-plugin) and [mods](https://github.com/topics/tailpipe-mod) — is open source. This transparency enables collaboration, spurs innovation, and nurtures a thriving community in which developers contribute and build together.

With ready-to-go plugins for [AWS](https://hub.tailpipe.io/plugins/turbot/aws), [GCP](https://hub.tailpipe.io/plugins/turbot/gcp), [Azure](https://hub.tailpipe.io/plugins/turbot/azure), and more, Tailpipe makes log collection simple and extensible.

### Logs at your fingertips with DuckDB + Parquet

Tailpipe brings the power of modern analytics directly to your laptop.

**Blazing-fast queries**: DuckDB ("SQLite for analytics") delivers in-memory performance: slice gigabytes of data in seconds.

**Optimized storage**: Parquet’s columnar format minimizes I/O and works seamlessly with Tailpipe’s hive-style directory structure, enabling rapid query across specific time ranges or datasets.

This "practical scale" — analyzing hundreds of millions of log entries locally —
is a game changer for day-to-day work and lays the foundation for large-scale
programs.

### Actionable insights with "Detections as Code"

When combined with [Detection mods in Powerpipe](https://powerpipe.io/blog/powerpipe-detection-mods), Tailpipe queries become actionable insights that help teams move from reactive troubleshooting to proactive security and operational excellence.

Its open source detection mods align with frameworks like [MITRE ATT&CK](https://attack.mitre.org/), providing a structured approach to identifying anomalies, unauthorized access, or misconfigurations. Pre-built benchmarks offer a starting point for quick wins while remaining fully customizable to meet unique organizational needs.

With Tailpipe, the transition from exploratory analysis to continuous monitoring is seamless, from SQL queries to reusable benchmarks that drive automation. Teams gain insights they can act on to stay ahead of risks and optimize cloud operations.

## Join the next generation

Tailpipe isn’t just another SIEM, it’s an open source platform that redefines how we approach log collection, detection, and analysis. With its local-first design, SQL-driven workflows, and the unmatched power of DuckDB + Parquet, Tailpipe sets a new standard for flexibility, speed, and cost-efficiency. Ready to experience the future of SIEM? [Download Tailpipe](/downloads) today and [join our community](https://turbot.com/community/join).



With the introduction of [Tailpipe](https://tailpipe.io/blog/introducing-tailpipe), Powerpipe gains a new family of mods that specialize in analysis of log data. Until now, benchmarks were bundles of [controls](/docs/powerpipe-hcl/control) that use [Steampipe](https://steampipe.io) to check cloud infrastructure against frameworks like CIS, HIPAA, and SOC 2. The new mods we're announcing today use a different query engine, Tailpipe, they introduce a new benchmark component — the [detection](/docs/powerpipe-hcl/detection) — and they bundle detections to align with [MITRE ATT&CK](https://attack.mitre.org/), a comprehensive knowledge base of adversary tactics. 

![](/images/blog/2025-01-detection-mods/mitre-01.png)

## Get started with a detections mod

Let's look at [AWS CloudTrail Log Detections](https://hub.powerpipe.io/mods/turbot/aws_cloudtrail_log_detections) which provides pre-built dashboards and detections to monitor and analyze activity across your AWS accounts.

This mod requires AWS CloudTrail logs to be collected using [Tailpipe](https://tailpipe.io) with the [AWS plugin](https://hub.tailpipe.io/plugins/turbot/aws):
- [Get started with the AWS plugin for Tailpipe →](https://hub.tailpipe.io/plugins/turbot/aws#getting-started)

Install the mod:

```sh
mkdir dashboards
cd dashboards
powerpipe mod install github.com/turbot/tailpipe-mod-aws-cloudtrail-log-detections
```

Start the dashboard server:

```sh
powerpipe server
```

Browse and view your dashboards at **http://localhost:9033**.

## The new suite of detection mods

The AWS CloudTrail Log Detections is joined by companions for Azure and GCP. Collectively these new mods provide a wealth of detections and queries for those working with individual clouds or across all of them.

| Mod | Dashboards | Benchmarks | Queries |
| - | - | - | - |
| [AWS CloudTrail Log Detections Mod](https://hub.steampipe.io/mods/turbot/tailpipe-mod-aws-cloudtrail-log-detections) | 6 | 86 | 108
| [Azure Activity Log Detections](https://hub.steampipe.io/mods/turbot/tailpipe-mod-azure-activity-log-detections) | 4 | 36 | 39
| [GCP Audit Log Detections](https://hub.steampipe.io/mods/turbot/tailpipe-mod-azure-activity-log-detections) | 2 | 55 | 55

## Run a suite of detections

The `AWS CloudTrail Log Detections` mod provides two views of detections: `CloudTrail Log Detections` which organizes them by AWS service, and `MITRE ATT&CK v16.1 for CloudTrail Logs` which uses the MITRE taxonomy. These are views over the same set of log entries that you've collected, using a common set of [queries](https://powerpipe.io/mods/turbot/tailpipe-mod-aws-cloudtrail-log-detections/queries). Let's start with `CloudTrail Log Detections`. The row of buttons at the top governs the number of days included in the view. The default is `7d`: the most recent 7 days. In this example we've used the `Custom` date picker to include 1.8 million log entries spanning several years. That may sound like a lot but it really isn't, this is just the sample [flaws.cloud](https:/flaws.cloud) data available [here](https://summitroute.com/downloads/flaws_cloudtrail_logs.tar). We've used Tailpipe on a six-month sample of an internal log that's 100 times larger. Tailpipe's engine, DuckDB, can slice through hundreds of millions of log records in seconds!

![](/images/blog/2025-01-detection-mods/detections-01.png)

If you've collected a meaningful amount of log data for the most recent 7 days, you can start exploring right away. If not, expand the date range to include more days, weeks, or even years.

As with a compliance mod, a detection mod displays a row of [infocards](/docs/powerpipe-hcl/card). For a detections view, these cards summarize the number and severity of log entries matching each detection's query. And as with a compliance mod, the body of the display presents a hierarchy of benchmarks, like `CloudWatch Detections` as shown above. For a detections view, these benchmarks bundle detections instead of controls, like `CloudWatch Log Group Created with Encryption Disabled`. You can click the `?` button on any detection for an overview of its nature and purpose.

![](/images/blog/2025-01-detection-mods/detections-02.png)

A control on a compliance dashboard reports a single row with the [standard columns](/docs/powerpipe-hcl/control#required-control-columns): **status**, **resource**, **reason**. A detection on a detections dashboard works differently, it reports the set of log entries that match that detection's underlying query. And that tabular view is richly interactive!

## Hunt for bad actors

You're hunting for needles in haystacks. To help you narrow down and focus on the log entries found by detections, Powerpipe offers powerful new tools.  Hover over any cell to reveal a row of four icons:

- Copy value

- Filter by this value

- Exclude value from results

- View row

![](/images/blog/2025-01-detection-mods/detections-03-annotated.png)

If you want to keep only rows with the */aws/lambda/Level6* resource, select the **Filter** icon. Conversely if you want to toss that set of rows to focus on everything else, select the **Exclude** icon. Using these tools you can refine the set of detected rows to ignore those that are benign and focus on the ones that might be malicious.

When you find an interesting row, use the **View** icon to explore it. 

![](/images/blog/2025-01-detection-mods/detections-04.png)

## Apply the MITRE ATT&CK lens

To view results through the lens of the MITRE ATT&CK framework, select the Powerpipe logo to return to the mod's home page and then select `MITRE ATT&CK v16.1 for CloudTrail Logs`. Now detections are organized by MITRE tactics and techniques. For example, in the opening screenshot above:

**TA0001: Initial Access** is a high-level tactic, representing an adversary's goal to gain an initial foothold in a system. 

**T1078: Valid Accounts** is a technique related to use of valid credentials (stolen, guessed, or default) to access systems. 

**T1078.001: Valid Accounts: Default Accounts** narrows that down to focus on root accounts or accounts with default passwords. That's the context for the **IAM Root User Console Login** detection.

It's not the only context, though. The same detection also appears under tactic **TA0004: Privilege Escalation**. The framework enables defenders to consider a detection from different perspectives, in order to evaluate how an action fits into the overall attack chain.

However you arrive at a detection, Powerpipe's filtering helps you quickly separate the wheat from the chaff. If the actor performing the login is known and trusted, exclude all console logins from that actor with a single click in order to focus on others that may warrant scrutiny.

## See it in Action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/kyjnHp7FyEA"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Tailpipe detection mods for Powerpipe"
>
</iframe>
</div>


## Level up your log analysis

With Tailpipe for log collection and query, and Powerpipe for interactive visualization, you gain a powerful new way to monitor cloud infrastructure for signs of compromise. You can quickly filter massive log datasets, and connect individual actions to adversary tactics, in order to identify and investigate potential security incidents. Try one of the new [detection mods](https://hub.powerpipe.io/mods?engines=tailpipe) and [let us know](https://turbot.com/community/join) how it goes!



The new [Tailpipe mods](https://hub.powerpipe.io?engines=tailpipe) prompted a set of Powerpipe features that enhance the interactivity of benchmarks and dashboards. But the rising tide lifts all boats! Even if you're not yet using [Tailpipe](https://tailpipe.io), you'll still reap the benefits:

- **Row-level filters**: Include or exclude table rows based on column values.

- **Column filters**: Adjust which columns are displayed in a table.

- **Row view**: Explore an individual row.

- **Heatmap**: Chart values aggregated by ranges of values.

Let's take a look.

## Toss or keep rows in tables

When spelunking logs, a [detection](/blog/powerpipe-detection-mods) often surfaces a lot more tabular data than you care about. But that can be true for any table. Here's the **AWS S3 Bucket Lifecycle Report** in the [AWS Insights](https://hub.powerpipe.io/mods/turbot/aws_insights), displaying buckets for multiple accounts:

![](/images/blog/2025-01-filters-and-dates/row-filter-before-annotated.png)

If you'd like to focus on just one account, use the new *Filter* icon to include only buckets in that account. Need even more granularity? You can adjust based on *Region* column. And for any column you can either include or exclude based on its value. Here's a view that includes buckets in the silverwater account, except for those in European regions.

![](/images/blog/2025-01-filters-and-dates/row-filter-with-exclusion.png)

## Select visible columns

Another new option gives you control over the columns displayed in a table. Since we've pinned this view to an account, there's no need to display that column:

![](/images/blog/2025-01-filters-and-dates/visible-columns.png)

When a table presents many columns, this new feature enables you to see just what you want to see, without having to scroll horizontally.

## View a row

The *View Row* icon opens a pane that displays all of the rows columns vertically. Even if you've hidden some columns in the table view, you'll see all of them in the row view:

![](/images/blog/2025-01-filters-and-dates/view-row.png)

## View a heatmap

The new [heatmap](/docs/powerpipe-hcl/chart#heatmap-chart) chart type displays ranges of values aggregated by color intensity.

![](/images/blog/2025-01-filters-and-dates/heatmap_chart.png)


## Filter, focus, and explore

These new filters, and charts help you shape your data exactly how you want to see it. Give them a try, then [tell us](https://turbot.com/community/join) about it!


Turbot Guardrails now provides enhanced visibility into your cloud environment with cloud activity tracking dashboards. These dashboards transform our comprehensive audit data into daily snapshots and trends over time.

Guardrails already tracks resource inventory, configuration changes, and governance control posture across AWS, Azure, GCP, Kubernetes, GitHub, and ServiceNow in real-time. This data is available through audit trails in the console activity tab, queries, and streaming notifications. Now available in the **Metrics** tab, these new cloud activity dashboards visualize this data through easy-to-read graphs showing how your environment changes every day. Each view provides a 30-day history with daily stats to track trends in your cloud FinOps and security posture, resource changes, and operational patterns.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails Cloud Activity Tracking" src="/images/blog/2025-01-cloud-activity-tracking/guardrails-cloud-activity-tracking-dashboards.png" />
</div>

Let's explore how each dashboard helps you monitor and understand your cloud environment's daily progression through a few success cases from early adopters of the feature.

## Controls over time

Track your governance posture through an intuitive stacked bar chart showing daily control state changes across active controls, alerts, errors, and muted states. Quick-reference cards compare current states against last week to help identify significant changes in your security and compliance posture.

Use Case: A financial tech enterprise cloud team who has been using the feature already updated their S3 bucket encryption-in-transit policy across 1000 AWS accounts. The Controls dashboard shows a spike in alerts as non-compliant buckets are detected, some accounts in "Enforce" mode saw an immediate drop in alerts, while other accounts in "Check" mode saw a daily progression as teams address these encryption requirements after receiving Guardrail notifications. This helped the cloud team validate that remediation efforts are making progress and identify any accounts falling behind.

## Resources over time

Monitor your cloud infrastructure footprint through daily resource counts, creations, updates, and deletions. Info cards highlight prior week trends, making it easy to spot unusual activity or validate cleanup efforts.

Use Case: A life science enterprise cloud team noticed an unexpected 40% spike in cloud resources across their development accounts. The Resources dashboard highlighted this unusual growth, prompting an investigation that revealed a misconfigured auto-scaling group in their cloud account. The team was able to quickly correct the configuration and terminate the unnecessary instances, avoiding significant cost overrun.

## Actions over time

Understand how often Guardrails actions are making changes in your environment through a daily activity timeline. Identify change frequencies, understand typical baselines, and spot unusual spikes in cloud operations. This visibility helps teams understand the value and impact of automated governance.

Use Case: A SaaS company is meticulous with their infrastructure as code reviews and typically catches misconfigurations before they are deployed. They use Guardrails as a safety net to catch any issues that may arise in runtime. The Actions dashboard revealed an unexpected daily spike in activity, leading to the discovery that a new application feature was dynamically creating S3 buckets without versioning enabled. Guardrails monitoring the runtime identified the misconfiguration and automatically enabled versioning on each new bucket per policy. Visibility from the dashboard into the activity allowed the Cloud team to work with the App team to fix the problem upstream.

## Getting Started
For existing Turbot Guardrails Enterprise customers, these dashboards are available in [TE v5.48](https://turbot.com/guardrails/changelog/te-v5-48-0) or later. Once enabled, metrics will begin populating moving forward. View Alerts over time on the homepage and all other dashboards in the **Metrics** tab at the cloud account, folder, or global level.

For teams new to Guardrails, [start your 14-day free trial](https://turbot.com/start) for a consolidated multi-cloud activity trail and powerful visualizations to govern your cloud environment.


Control muting in Guardrails is a powerful new feature that enables cloud teams to focus on priorities by selectively suppressing controls based on context without compromising their security posture.

## What is control muting?

[Control muting](https://turbot.com/guardrails/docs/concepts/controls#muting-controls) allows cloud teams to temporarily or indefinitely suppress specific controls while retaining their underlying security policies. 

Teams can mute controls via the Guardrails console or [API](https://turbot.com/guardrails/docs/reference/graphql/mutation/muteControl) using annotations. Muting can be applied for a set duration (e.g. 1 day, 1 month, or custom periods) or based on control state (e.g. until the control status changes from ERROR to OK).

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails Control Muting" src="/images/blog/2025-01-control-muting/guardrails-control-muting-options.png" />
</div>

## When to use control muting

Control muting helps streamline operations without compromising security policies. It’s a lightweight alternative to adjusting policy settings or creating exceptions, and it’s particularly useful in these scenarios:

### Planned changes and maintenance

During scheduled maintenance or infrastructure updates, muting specific controls reduces unnecessary alerts while preserving visibility into other issues. For example, muting high availability controls during a planned 4-hour failover test, or replication controls during a 2-week regional migration.

### Known issues under resolution

When actively addressing an issue, mute controls to focus efforts:
- Suppress alerts until a fix is deployed (e.g., mute until the control status changes to OK).
- Temporarily silence noisy controls while fixing errors (e.g., mute ERROR states).
- Mute controls for a specific timeframe during fix deployment (e.g., mute for X days).
- Reduce noise from multiple related controls, focusing on the key issue.

### False Positive Management
While tuning detection logic, mute controls generating known false positives. This allows teams to adjust underlying rules without being overwhelmed by irrelevant alerts.

## Muting vs. Exceptions: 

Both muting and [policy exceptions](https://turbot.com/guardrails/docs/concepts/policies/values-settings#exceptions) help manage governance controls, but they serve distinct purposes. The primary difference lies in when they are applied:

**Control Muting**: Suppresses alerts after the control has already evaluated its state. Muting is an operational action that doesn’t affect how the control state is determined.

**Policy Exceptions**: Modify governance requirements and directly influence how the control state is evaluated. Exceptions adjust the rules that dictate a resource's compliance posture.

| **Control Muting**                     | **Policy Exceptions**                       
|----------------------------------------|---------------------------------------------
| Applied **after** the control evaluates its state. | Applied **before** the control evaluates its state.  
| Suppresses alerts.         | Modifies posture evaluation rules.  
| Operational flexibility (no impact on posture). | Adjusts security posture requirements.  
| Visibility-only (alerts suppressed, but monitoring continues). | Changes control evaluation logic for compliance.  
| Example: Muting a control in ERROR while resolving an issue. | Example: Adding an exception to allow a non-compliant configuration.  

### Using muting and exceptions together

You can combine muting and exceptions for even greater flexibility:

- **During exception rollout**: Mute controls while implementing an approved exception.
- **Gradual policy changes**: Use muting to suppress alerts temporarily while the exception is under review or approval.
- **Complex deployments**: Combine long-term exceptions with short-term muting during deployments to maintain operational flexibility.

## Get started with control muting

Control muting is now available in [Guardrails TE v5.48 release](https://turbot.com/guardrails/changelog/te-v5-48-0). To start using it, navigate to any control in the Guardrails console and click the "Mute" icon in the upper-right corner.

New "Muted" state filters are also available in reports, allowing you to maintain visibility of suppressed controls while prioritizing active issues.

If you’re new to Guardrails, [start your 14-day free trial](https://turbot.com/start) and experience automated cloud governance controls in your environment.




We're excited to announce native AWS Organizations support in Turbot Guardrails, making it easier than ever to bring your existing AWS organizational structure into your cloud governance framework.

Historically, cloud teams have structured their AWS accounts primarily around billing and cost center considerations. This structure, while effective for financial management, may not always align perfectly with governance requirements. Guardrails has always supported connecting individual [AWS accounts](https://turbot.com/guardrails/docs/guides/aws/import-aws-account) through our UI, [API](https://turbot.com/guardrails/docs/reference/graphql), and [Terraform provider](https://turbot.com/guardrails/docs/reference/terraform). This flexibility allowed customers to either maintain their existing cloud organization structure or transform it into a governance-focused hierarchy, often integrating accounts across multiple cloud providers.

As AWS Organizations has matured and become the standard for AWS account management, our customers have increasingly requested a simpler way to import and maintain their existing AWS Organizations structure within Guardrails.

## Guardrails for AWS Organizations

With this new integration, you can now import your entire AWS Organizations structure into Guardrails with just a few clicks. This feature automatically:
- Discovers and imports your complete AWS Organizations hierarchy instantly
- Detects new accounts as they're added to your AWS organization
- Maintains your organization units (OUs) and accounts structure in sync as changes occur
- Supports exclusion policies to filter specific OUs and accounts
- Enables controls to be applied at any layer of your organization hierarchy

This feature also enables enterprises to consolidate governance across multiple AWS organizations by grouping and applying policies across their entire AWS footprint.


## How to connect your AWS organization
The connect page now offers an option to [connect your AWS organization(s)](https://turbot.com/guardrails/docs/guides/aws) to Turbot Guardrails.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails cloud connect options" src="/images/blog/2025-01-aws-org-sync/guardrails-cloud-connections.png" />
</div>

You can choose between adding individual AWS accounts or connecting an entire organization.

To connect your AWS organization, Guardrails requires two key components: an AWS IAM role to query your organization structure, and a role in each member account to discover and govern AWS resources. We've simplified this setup by providing ready-to-use CloudFormation templates - one CFN template for the organization role and a CFN StackSet for your member accounts.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails AWS organization cloud connect options" src="/images/blog/2025-01-aws-org-sync/guardrails-aws-org-cloud-connections.png" />
</div>

Before connecting, you can specify [exclusions](https://hub.guardrails.turbot.com/mods/aws/policies/aws/organizationCmdbExclude) using Account IDs, aliases, or OU names. Flexible regex patterns are supported to ensure your exclusions remain effective as your organization evolves over time.

## Syncing to the Guardrails CMDB
After clicking the "Connect" button, Guardrails immediately starts discovering your organization configurations, OU structure and AWS accounts. It then continues to discover all resources in each account to build your complete asset inventory in the Guardrails CMDB.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails AWS organization org discovery" src="/images/blog/2025-01-aws-org-sync/guardrails-aws-org-discovery.png" />
</div>

With your AWS organization now integrated into Guardrails, you gain immediate access to powerful capabilities: comprehensive resource exploration, full-text inventory search, and the ability to implement governance controls that ensure consistent FinOps and Security posture.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails AWS organization in CMDB Search" src="/images/blog/2025-01-aws-org-sync/guardrails-aws-org-in-cmdb-search.png" />
</div>

Guardrails maintains continuous synchronization with your organization, automatically reflecting changes as accounts move between OUs, new accounts are added, or organizational configurations evolve.

Managing multiple AWS Organizations? Simply repeat these steps - you can either segregate organizations in separate Guardrails folders or unify them under a single [hierarchy](https://turbot.com/guardrails/docs/concepts/resources/hierarchy).

## See it in Action

Watch this demo to see how Turbot Guardrails can integrate your entire AWS organization(s) in just a few clicks:

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/tTUfLxUEzQI"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Connect your AWS Organizations with Guardrails"
>
</iframe>
</div>

## Elevate your AWS organization accounts with Guardrails

This new AWS organization feature addresses a common challenge faced by our users with large AWS footprints who want to bring in their entire organization structure as-is. It would be time-consuming to do this by hand or with Terraform scripting. Now you can bring your whole AWS organization into Guardrails quickly, easily, and automatically. Best of all, Guardrails stays up-to-date with your organization as it evolves, automatically reflecting any changes to your OUs and accounts.

[Get started](https://turbot.com/start) with a 14-day free trial of [Guardrails for AWS Organizations](https://turbot.com/guardrails/docs/integrations/aws) today.




We're excited to announce native GCP organization support in Turbot Guardrails, making it easier than ever to bring your existing GCP organizational structure into your cloud governance framework.

Historically, cloud teams have structured their GCP projects primarily around billing and cost center considerations. This structure, while effective for financial management, may not always align perfectly with governance requirements. Guardrails has always supported connecting individual [GCP projects](https://turbot.com/guardrails/docs/guides/gcp/import-gcp-project) through our UI, [API](https://turbot.com/guardrails/docs/reference/graphql), and [Terraform provider](https://turbot.com/guardrails/docs/reference/terraform). This flexibility allowed customers to either maintain their existing cloud organization structure or transform it into a governance-focused hierarchy, often integrating accounts across multiple cloud providers.

As enterprises have matured their use of GCP organization and folders, our customers have increasingly requested a simpler way to import and maintain their existing GCP organization structure within Guardrails.

## Guardrails for GCP Organizations

With this new integration, you can now import your entire GCP Organizations structure into Guardrails with just a few clicks. This feature automatically:
- Discovers and imports your complete GCP Organizations hierarchy instantly
- Detects new projects as they're added to your GCP organization
- Maintains your GCP folders and projects structure in sync as changes occur
- Supports exclusion policies to filter specific folders and projects
- Enables controls to be applied at any layer of your organization hierarchy

Beyond single organization support, this feature enables enterprises to consolidate governance across multiple GCP organizations by grouping and applying policies across their entire GCP footprint.

## How to connect your GCP Organization
The connect page now offers an option to [connect your GCP organization(s)](https://turbot.com/guardrails/docs/guides/gcp) to Turbot Guardrails.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails cloud connect options" src="/images/blog/2025-01-gcp-org-sync/guardrails-cloud-connections.png" />
</div>

You can choose between adding individual GCP projects or connecting an entire organization.

To connect your GCP organization, Guardrails supports Service Impersonation mode or System Account credentials file.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails GCP organization cloud connect options" src="/images/blog/2025-01-gcp-org-sync/guardrails-gcp-org-cloud-connections.png" />
</div>

Before connecting, you can specify [exclusions](https://hub.guardrails.turbot.com/mods/gcp/policies/gcp/organizationCmdbExclude) using project IDs or folder IDs. Flexible regex patterns are supported to ensure your exclusions remain effective as your organization evolves over time.

## Syncing to the Guardrails CMDB
After clicking the "Connect" button, Guardrails immediately starts discovering your organization configurations, folder structure and GCP projects. It then continues to discover all resources in each project to build your complete asset inventory in the Guardrails CMDB.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails GCP organization org discovery" src="/images/blog/2025-01-gcp-org-sync/guardrails-gcp-org-discovery.png" />
</div>

With your GCP organization now integrated into Guardrails, you gain immediate access to powerful capabilities: comprehensive resource exploration, full-text inventory search, and the ability to implement governance controls that ensure consistent FinOps and Security posture.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails GCP organization in CMDB Search" src="/images/blog/2025-01-gcp-org-sync/guardrails-gcp-org-in-cmdb-search.png" />
</div>

Guardrails maintains continuous synchronization with your organization, automatically reflecting changes as projects move between folders, new projects are added, or organizational configurations evolve.

Managing multiple GCP organizations? Simply repeat these steps - you can either segregate organizations in separate Guardrails folders or unify them under a single [hierarchy](https://turbot.com/guardrails/docs/concepts/resources/hierarchy).

## See it in Action

Watch this demo to see how Turbot Guardrails can integrate your entire GCP organization(s) in just a few clicks:

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/9kkVr17mHHM"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Connect your GCP Organizations with Guardrails"
>
</iframe>
</div>

## Elevate your GCP organization projects with Guardrails

This new GCP organization feature addresses a common challenge faced by our users with large GCP footprints who want to bring in their entire organization structure as-is. It would be time-consuming to do this by hand or with Terraform scripting. Now you can bring your whole GCP organization into Guardrails quickly, easily, and automatically. Best of all, Guardrails stays up-to-date with your organization as it evolves, automatically reflecting any changes to your folders and projects.

[Get started](https://turbot.com/start) with a 14-day free trial of [Guardrails for GCP Organizations](https://turbot.com/guardrails/docs/integrations/gcp) today.




We're excited to announce native Azure tenant support in Turbot Guardrails, making it easier than ever to bring your existing Azure tenant structure into your cloud governance framework.

Historically, cloud teams have structured their Azure subscriptions primarily around billing and cost center considerations. This structure, while effective for financial management, may not always align perfectly with governance requirements. Guardrails has always supported connecting individual [Azure subscriptions](https://turbot.com/guardrails/docs/guides/azure/import) through our UI, [API](https://turbot.com/guardrails/docs/reference/graphql), and [Terraform provider](https://turbot.com/guardrails/docs/reference/terraform). This flexibility allowed customers to either maintain their existing cloud tenant structure or transform it into a governance-focused hierarchy, often integrating accounts across multiple cloud providers.

As enterprises have matured their use of Azure tenants and management groups, our customers have increasingly requested a simpler way to import and maintain their existing Azure tenant structure within Guardrails.

## Guardrails for Azure tenants

With this new integration, you can now import your entire Azure tenant structure into Guardrails with just a few clicks. This feature automatically:
- Discovers and imports your complete Azure Tenant hierarchy instantly
- Detects new subscriptions as they're added to your Azure Tenant
- Maintains your management groups and subscriptions structure in sync as changes occur
- Supports exclusion policies to filter specific management groups and subscriptions
- Enables controls to be applied at any layer of your Tenant hierarchy

This feature also enables enterprises to consolidate governance across multiple Azure tenants by grouping and applying policies across their entire Azure footprint.

## How to connect your Azure Tenant
The connect page now offers an option to [connect your Azure tenant(s)](https://turbot.com/guardrails/docs/guides/azure) to Turbot Guardrails.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails cloud connect options" src="/images/blog/2025-01-azure-tenant-sync/guardrails-cloud-connections.png" />
</div>

You can choose between adding individual Azure subscription or connecting an entire tenant.

To connect your Azure tenant, Guardrails requires the Tenant ID, Client ID, Client Secret, and which Azure cloud environment (Global Cloud, US Government or China Cloud).

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails Azure tenant cloud connect options" src="/images/blog/2025-01-azure-tenant-sync/guardrails-azure-tenant-cloud-connections.png" />
</div>

Before connecting, you can specify [exclusions](https://hub.guardrails.turbot.com/mods/azure/policies/azure/tenantCmdbExclude) using Subscription IDs, or management group names. Flexible regex patterns are supported to ensure your exclusions remain effective as your tenant evolves over time.

## Syncing to the Guardrails CMDB
After clicking the "Connect" button, Guardrails immediately starts discovering your tenant configurations, management group structure and Azure subscriptions. It then continues to discover all resources in each subscription and resource group to build your complete asset inventory in the Guardrails CMDB.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails Azure tenant discovery" src="/images/blog/2025-01-azure-tenant-sync/guardrails-azure-tenant-discovery.png" />
</div>

With your Azure tenant now integrated into Guardrails, you gain immediate access to powerful capabilities: comprehensive resource exploration, full-text inventory search, and the ability to implement governance controls that ensure consistent FinOps and Security posture.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails Azure tenant in CMDB Search" src="/images/blog/2025-01-azure-tenant-sync/guardrails-azure-tenant-in-cmdb-search.png" />
</div>

Guardrails maintains continuous synchronization with your tenant, automatically reflecting changes as subscriptions move between management groups, new subscriptions are added, or tenant configurations evolve.

Managing multiple Azure tenants? Simply repeat these steps - you can either segregate tenants in separate Guardrails folders or unify them under a single [hierarchy](https://turbot.com/guardrails/docs/concepts/resources/hierarchy).

## See it in Action

Watch this demo to see how Turbot Guardrails can integrate your entire Azure tenant(s) in just a few clicks:

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/qyKj3QyTAdM"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Connect your Azure Tenants with Guardrails"
>
</iframe>
</div>

## Elevate your Azure tenant & subscriptions with Guardrails

This new Azure tenant feature addresses a common challenge faced by our users with large Azure footprints who want to bring in their entire tenant structure as-is. It would be time-consuming to do this by hand or with Terraform scripting. Now you can bring your whole Azure tenant into Guardrails quickly, easily, and automatically. Best of all, Guardrails stays up-to-date with your tenant as it evolves, automatically reflecting any changes to your management groups and subscriptions.

[Get started](https://turbot.com/start) with a 14-day free trial of [Guardrails for Azure Tenants](https://turbot.com/guardrails/docs/integrations/azure) today.




We're excited to announce that Turbot Guardrails now supports OpenTofu for cloud landing zones and governance. This enhancement enables cloud teams to deploy and manage standardized patterns across their enterprise cloud environment while leveraging the community-driven, [Linux Foundation](https://www.linuxfoundation.org/)-backed OpenTofu project.

## Why use Guardrails for deploying stacks?

While teams can use OpenTofu in their own deployment pipelines, using Guardrails' Stacks to deploy OpenTofu is specifically designed for cloud teams managing enterprise-wide patterns and standards across a multi-cloud environment. Common use cases would be to deploy Landing Zone configurations, consistent networking patterns, IAM resources, resource policies, etc.

Guardrails Stacks gives cloud teams additional benefits:

- **Unified Governance:** Manage infrastructure standards alongside existing cloud security and FinOps guardrails.
- **Policy-Driven Control:** Apply consistent patterns while supporting exceptions through Guardrails' policy engine.
- **Automated Drift Correction:** Ensure configurations remain compliant with automated drift detection and correction **without managing state files**.
- **Context-aware:** Dynamically bring in context from the Guardrails CMDB, audit trail, and outside data sources. Wrap the context with conditional logic using [calculated policies](https://turbot.com/guardrails/docs/concepts/policies/calculated-faq) to handle a wide variety of enterprise use cases.

## Guardrails Stack [Native]

Our new governance stacks, labeled [Stack \[Native\]](https://turbot.com/guardrails/docs/concepts/guardrails/stacks), bring expanded capabilities and flexibility to cloud teams. Powered by OpenTofu, these stacks offer significant improvements over our original [Stack/Configured controls](https://turbot.com/guardrails/docs/concepts/guardrails/configured):

| Feature                 | Stack/Configured          | Stack [Native]
|-------------------------|---------------------------|----------------------------
| **Version**             | Terraform 0.15 and earlier | OpenTofu 1.8.3 and later
| **Triggering**          | Resource update           | Resource update, interval
| **CMDB Required?**      | Yes                       | No
| **Resources Supported** | Only resources supported in Guardrails | Any resource supported by the [provider](https://opentofu.org/docs/language/providers/)
| **Support for `count`, `for_each`**| No             | Yes


## How to use Guardrails OpenTofu Stacks

### Update your AWS mod
[Update](https://turbot.com/guardrails/docs/guides/configuring-guardrails/update-mod) the [@turbot/aws](https://hub.guardrails.turbot.com/mods/aws/mods/aws) mod to version v5.33 or later.

Once the mod is installed, you will see new [Stack \[Native\] policies](https://turbot.com/guardrails/docs/concepts/guardrails/stacks#stack-policies) added to your environment. For example, `AWS > Account > Stack [Native] > *` policies:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails OpenTofu Stack Policies" src="/images/blog/2025-01-guardrails-opentofu/guardrails-opentofu-stack-policies.png" />
</div>

### Set your first stack policy

Let's start with an existing policy pack sample from the [Guardrails Hub](https://hub.guardrails.turbot.com) to [Deploy AWS IAM Stack](https://hub.guardrails.turbot.com/policy-packs/aws_iam_deploy_aws_iam_stack).

This policy pack creates an IAM Role with `ReadOnlyAccess` access in each applicable AWS account.

The policy pack sets the following OpenTofu HCL in the `AWS > IAM > Stack [Native] > Source` policy:

```hcl
variable "trusted_principals" {
  description = "A list of principal ARNs that can assume the role. These wil be added to the trust policy."
  type        = set(string)
}

resource "aws_iam_role" "read_only_role" {
  name = "read_only_role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          AWS = var.trusted_principals
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "read_only" {
  role       = aws_iam_role.read_only_role.name
  policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}
```

### Setting conditional logic to your stack

The policy pack also uses the `AWS > IAM > Stack [Native] > Variables` policy to define `.tfvar`-style variable overrides. While we can simply place a static variable(s) in the configuration, using calculated policies, we can dynamically look up values.

The variable we set in the HCL source is for trusted principals. Using calculated policies we can look up a list of trusted AWS Account IDs. In this case, we will lookup the account ID the stack will apply to, essentially allowing the role to be assumed from within the same account.

A calculated policy consists of a GraphQL input to pull in the account tags from the Guardrails CMDB, and a Nunjucks template in the policy to define the conditional logic:

```hcl
trusted_principals = ["arn:aws:iam::{{$.account.Id}}:root"]
```

To test how the stack will apply, we can [attach the policy pack](https://turbot.com/guardrails/docs/guides/configuring-guardrails/policy-packs#attach-a-policy-pack-to-a-resource) to our `Acme` organization to see the calculated values per account:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Turbot Guardrails AWS IAM Role Policy Pack Policy Values" src="/images/blog/2025-01-guardrails-opentofu/guardrails-iam-role-policy-pack-policy-values.png" />
</div>

When ready, we can officially deploy the policy pack by changing the `AWS > IAM > Stack [Native]` policy setting from `Check: Configured` to `Enforce: Configured`.

## Automated drift correction

Guardrails can also detect and correct drift of your applied stacks. By default, [drift detection](https://turbot.com/guardrails/docs/concepts/guardrails/stacks#drift-detection) policies will run automatically when a resource changes.

When any changes occur in runtime affecting the stack, Guardrails will alert and auto-correct the misconfiguration. 

For example, if a developer modifies the AWS Principal in the policy, Guardrails will immediately identify the misconfiguration, alert, then take automated action to fix the policy. Here we can see Bob updated the resource, then within a minute, Guardrails detected the drift and updated the resource.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails Stack drift detect and correct" src="/images/blog/2025-01-guardrails-opentofu/guardrails-drift-detection.png" />
</div>

Exploring the control detail, we can visualize the configuration correction back to the desired state of the stack:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails Stack drift detect and correct details" src="/images/blog/2025-01-guardrails-opentofu/guardrails-drift-detection-detail.png" />
</div>

## See it in Action

Watch this demo to see how Turbot Guardrails manages OpenTofu stacks with automatic drift correction in real time:

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/yD85oxAfD3k"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Cloud Governance OpenTofu Stacks with drift detection"
>
</iframe>
</div>

## Get started with Guardrails Stacks for cloud landing zones

Turbot Guardrails transforms how you manage OpenTofu deployments across AWS, Azure, and GCP. By combining OpenTofu with Guardrails' policy engine, cloud teams can ensure their cloud landing zones stay compliant and secure through context-aware deployments, automated drift detection and correction.

[Start](https://turbot.com/start) your 14-day free trial to experience automated cloud landing zones with Guardrails.



Cloud teams prioritize securing infrastructure ecosystems like AWS, Azure, GCP, and Kubernetes. These are foundational layers of the cloud, where misconfigurations often arise and can have severe consequences if left unaddressed. However, infrastructure security is only part of the story.

Every change to your cloud infrastructure flows through your development pipeline in GitHub. A compromised repository, workflow, or credential provides attackers direct access to modify your cloud infrastructure. Bringing your cloud and GitHub configurations under Guardrails delivers complete governance from pipeline to cloud.

## Securing the Software Supply Chain with Guardrails

Turbot Guardrails provides cloud teams with real-time detection, compliance, and remediation capabilities across [AWS](https://hub.guardrails.turbot.com/mods/aws/mods), [Azure](https://hub.guardrails.turbot.com/mods/azure/mods), [GCP](https://hub.guardrails.turbot.com/mods/gcp/mods), [Kubernetes](https://hub.guardrails.turbot.com/mods/kubernetes/mods/kubernetes), and [ServiceNow](https://hub.guardrails.turbot.com/mods/servicenow/mods). By continuously monitoring configurations, Guardrails ensures cloud services remain secure, compliant, and operationally optimized.

Adding GitHub to its scope, Guardrails extends these same governance capabilities to your development environment:
* **Inventory:** Continuously update asset inventory of your GitHub organizations
* **Audit Trail:** Track changes to organization and repository configurations.
* **SecOps Alerting:** Identify and alert on unprotected branches, exposed secrets, etc.
* **Continuous Compliance:** Automatically correct misconfigurations in real-time.

By securing both layers—cloud infrastructure and development pipelines, Guardrails ensures your organization’s entire ecosystem remains secure from pipeline to runtime configurations.

## How to Enable GitHub Guardrails

### Install the GitHub Mod
To get started, [install](https://turbot.com/guardrails/docs/mods/guide/install) the [@turbot/github](https://hub.guardrails.turbot.com/mods/github/mods/github) mod.

### Connect Your First GitHub Organization
Once the mod is installed, your Connect page will now have an option to [connect your GitHub organization(s)](https://turbot.com/guardrails/docs/guides/github) to Turbot Guardrails.

Guardrails will immediately discover all GitHub organization configurations and repositories. This instantly updates the Guardrails Configuration Management Database (CMDB) with real-time configurations. Guardrails also sets up [event handlers](https://turbot.com/guardrails/docs/guides/github/real-time-events) to automatically receive real-time events as changes occur, allowing instant updates to inventory, audit trail, and control execution based on your governance posture.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GitHub organization configurations in Guardrails CMDB" src="/images/blog/2025-01-guardrails-for-github/github-organization-in-guardrails-cmdb.png" />
</div>

## GitHub Guardrail Policy Pack Examples

The GitHub mod includes several pre-built [organization and repository guardrail policies](https://hub.guardrails.turbot.com/mods/github/mods/github) ready to use.

**At the Organization level**, you can control member access and permissions, manage Deploy Keys, configure security settings, and enforce consistent policies across all repositories.

**At the Repository level**, you can standardize security configurations, manage Dependabot settings, enforce pull request requirements, and ensure consistent repository settings.

Here are some of the most commonly used GitHub guardrails our customers implement through policy packs available on the [Guardrails Hub](https://hub.guardrails.turbot.com):

### Repository Visibility

Ensuring that [GitHub repositories are private](https://hub.guardrails.turbot.com/policy-packs/github_enforce_repositories_are_private) protects your codebases from accidental exposure, protecting sensitive information from unauthorized access.

In most use cases, you would consider all your repositories are set to `Private`. Where by exception you may want to ensure some intended repositories are made `Public` to be accessible where applicable.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GitHub Repository Visibility Guardrail Policy" src="/images/blog/2025-01-guardrails-for-github/github-repo-visibility-guardrail.png" />
</div>

The inverse could be true for your organization; at Turbot, we [manage hundreds of open-source repositories](https://turbot.com/blog/2023/10/repo-management) intended to be public for our community. In our use case, we have a few repos intended to be `Private`, while the majority are `Public` supporting our [Steampipe](https://steampipe.io), [Powerpipe](https://powerpipe.io), and [Flowpipe](https://flowpipe.io) open-source projects. Ensuring our intended `Public` repos remain available is essential to our user community.

### Enable Dependabot Alerts

[Enabling Dependabot alerts](https://hub.guardrails.turbot.com/policy-packs/github_enforce_dependabot_alerts_are_enabled_for_repositories) for GitHub repositories is essential for identifying and addressing vulnerabilities in dependencies. This helps maintain the security and integrity of the codebase by providing timely notifications about potential risks and ensuring compliance with best practices for code dependency management.

While the alerts can be noisy, it's the nature of maintaining an up-to-date code base. Having Guardrails ensure Dependabot alerts remain enabled across all repositories helps teams stay ahead of potential vulnerabilities and makes it easier to identify, prioritize and action.

### Enforce Secret Scanning

Enforcing GitHub repositories having [secret scanning enabled](https://hub.guardrails.turbot.com/policy-packs/github_enforce_secret_scanning_is_enabled_for_repositories) detects sensitive information, such as API keys, tokens, passwords, and other credentials, that may accidentally be committed to the repository. 

There are countless examples over time, even just recently [Toyota Suffered a Data Breach by Accidentally Exposing A Secret Key Publicly On GitHub](https://blog.gitguardian.com/toyota-accidently-exposed-a-secret-key-publicly-on-github-for-five-years), highlighting the critical importance of automated secret detection and remediation.

## Auto-remediation of a GitHub misconfiguration

One of the unique aspects of using Guardrails for GitHub is its ability to not only instantly alert when there is a misconfiguration in GitHub, but also its ability to auto-correct the misconfiguration as soon as it occurs.  

After you have installed the [Secret Scanning Enabled](https://hub.guardrails.turbot.com/policy-packs/github_enforce_secret_scanning_is_enabled_for_repositories) policy pack from the Guardrails Hub, you can [attach](https://turbot.com/guardrails/docs/guides/configuring-guardrails/policy-packs#attach-a-policy-pack-to-a-resource) it to your GitHub Organization in Guardrails.

The policy pack defaults to `Check: Enabled`, which means all your repositories will alarm if the repository does not have secret scanning enabled.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GitHub Enable Secret Scanning Guardrails Controls" src="/images/blog/2025-01-guardrails-for-github/github-repo-secret-scanning-guardrail-control.png" />
</div>

To enforce all public repositories have secret scanning enabled, you can flip the policy to `Enforce: Enabled`. This will instantly enable secret scanning on all applicable repositories. Ongoing, Guardrails will continuously monitor any changes to this configuration. 

If an administrator disables secret scanning, Guardrails will immediately alert and auto-correct the misconfiguration in real-time.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GitHub Enable Secret Scanning Guardrails Control Enforced" src="/images/blog/2025-01-guardrails-for-github/github-repo-secret-scanning-guardrail-control-enforced.png" />
</div>


## See it in Action

Watch this demo to see how Turbot Guardrails manages GitHub organization and repository guardrails in real time:

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/hFqY-1HC4xc"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Guardrails for GitHub"
>
</iframe>
</div>

## Elevate Your GitHub Security with Guardrails

Guardrails for GitHub brings enterprise-grade governance to your repositories, helping you maintain secure GitHub configurations at all times.

[Get started](https://turbot.com/start) with a 14-day free trial of [Guardrails for GitHub](https://turbot.com/guardrails/docs/integrations/github) today.




Our announcements this week span the breadth of Turbot offerings: 

- Guardrails
   - Enhanced ServiceNow CMDB [relationship sync](https://turbot.com/guardrails/blog/2024/10/servicenow-cmdb-relationships)
   - 33 new [step-by-step guides](https://turbot.com/guardrails/blog/2024/10/guardrails-guides) for easier operations
   - Major runtime [cost reductions](https://turbot.com/guardrails/blog/2024/10/guardrails-runtime-cost-reductions) through optimization

- Open Source
   - [v1.0](https://turbot.com/blog/2024/10/open-source-v1) for Steampipe, Powerpipe, Flowpipe, and 116 plugins and 44 mods

- Flowpipe
   - Interactive wizard for AWS [CIS v3.0 and v4.0 remediation](https://flowpipe.io/blog/compliance-workflows-for-aws)

- Pipes
   - Run Flowpipe workflows [in Turbot Pipes](https://turbot.com/pipes/blog/2024/10/flowpipe-in-pipes)
   - Pipes [platform strategy](https://turbot.com/pipes/blog/2024/10/pipes-assemble)

## Monday October 21: Making Guardrails Better

### Enhanced ServiceNow CMDB Integration
We [extended](https://turbot.com/guardrails/blog/2024/10/servicenow-cmdb-relationships) our ServiceNow CMDB integration with automatic relationship management for cloud resources. Now Guardrails automatically creates and maintains relationships among cloud resources in ServiceNow's CMDB, providing critical visibility into how AWS, Azure, GCP, and Kubernetes resources are interconnected. This builds on our existing ServiceNow integration, requiring no additional ServiceNow modules or cloud services.

### 33 New Step-by-Step Guides
We [released](https://turbot.com/guardrails/blog/2024/10/guardrails-guides) 33 new step-by-step guides focused on simplifying the installation, configuration, and monitoring of your Guardrails environment. These guides cover everything from importing cloud accounts to troubleshooting event handlers, making it easier than ever to get started with Guardrails and maintain it over time. The guides are now publicly available on GitHub, enabling community contributions that improve and expand the documentation.

### Runtime Cost Reductions
We [announced](https://turbot.com/guardrails/blog/2024/10/guardrails-runtime-cost-reductions) a suite of technical improvements that have dramatically reduced Guardrails' runtime costs. Highlights include 
migration to ARM64 for AWS ECS and Lambda workloads, Redis cache optimization, new activity log retention policies, support for PostgreSQL 16, and upgraded Node.js runtime in Lambda functions. Each of these improvements delivers solid cost and/or performance benefits. Collectively they add up to a big win for Guardrails customers.

## Tuesday October 22: Version 1.0 Milestones

### Major Version 1.0 Releases

We [announced](https://turbot.com/blog/2024/10/open-source-v1) a coordinated v1.0 release that represents a major milestone in our open source journey. Each component has been battle-tested in production environments and has evolved based on real-world feedback from our active community. Individually they enable you to query cloud resources, report on them, and act on them. They work even better together as when, for example, a Steampipe query feeds a Powerpipe dashboard or a Flowpipe pipeline. 

Flipping the v1.0.0 switch is always an emotional moment. We're just getting started, but it was time to recognize the maturity of this ecosystem and celebrate with everyone who has traveled with us on this journey.


## Wednesday October 23: Interactive Compliance Workflows in Flowpipe

### AWS CIS Remediation

  We [introduced](https://flowpipe.io/blog/compliance-workflows-for-aws) Flowpipe's interactive wizard for remediating AWS CIS benchmark findings. Along with the new [AWS CIS](https://hub.flowpipe.io/mods/turbot/aws_cis) mod, which supports both CIS v3.0 and v4.0, we released the foundational [AWS Compliance](https://hub.flowpipe.io/mods/turbot/aws_compliance) mod — a suite of detect-and-correct pipelines that support `AWS CIS` but are available for separate use in other contexts.

  With these new mods you can run Flowpipe in an interactive terminal to march through a checklist of CIS findings and fix misconfigurations with human oversight. You can also schedule individual controls to run autonomously. 

## Thursday October 24: Turbot Pipes Integration

### Flowpipe in Pipes

We [announced](https://turbot.com/pipes/blog/2024/10/flowpipe-in-pipes) the integration of Flowpipe into Turbot Pipes. Now you can run your workflows in the Pipes web UI, with vastly easier configuration of connections, integrations, notifiers, and mod settings. Flowpipe is available in every new Pipes workspace, and so is a Steampipe instance ready to be used by Flowpipe. With the new Slack integration it's trivial to push approval steps into a Slack channel. And the GitHub integration you've enjoyed for installation of Powerpipe mods works just the same way for Flowpipe mods.

### Pipes Assemble!

We [discuss](https://turbot.com/pipes/blog/2024/10/pipes-assemble) how Steampipe, Powerpipe, and Flowpipe work together in Pipes as a comprehensive platform for cloud intelligence and security. Now all three core components are first-class citizens of Pipes. As with AWS services, you choose your tool based on the job to be done. You can quickly navigate among them, and each taps into common Pipes infrastructure. When you create an AWS connection in a Pipes organization or tenant, for example, you can make it available to some or all workspaces. In those workspaces, all three Pipes components can use the connection: Steampipe for queries, Powerpipe for benchmarks and dashboards, and Flowpipe to enforce security compliance.

## The B-sides

These were the Launch Week 6 hits, but plenty of other good things happened during this cycle. Check the [B-sides](https://turbot.com/blog/2024/10/launch-week-6-b-sides) for details!

## And that's a wrap!

The announcements this week are individually exciting, but the Turbot story is bigger than the sum of its parts. Our products work together to deliver cloud security and intelligence with incomparable breadth, depth, power, speed, and versatility. And there's more to come. Enjoy the current batch of new products and features, [let us know](https://turbot.com/community/join) how it goes and stay tuned for our next Launch Week in a few months!

## Watch the Wrap Party

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/ox3mVj4N63w"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Launch Week 6 is a wrap!"
>
</iframe>
</div>



As [Launch Week 6](https://turbot.com/launch-week/06) draws to a close, we wanted to take a moment to highlight some of the exciting updates and announcements that slipped under the radar this week across our Turbot products and open-source projects.

## Guardrails: 70+ controls added

We continue to listen to customer feedback to add new cloud resource types and controls for security, compliance, and operational guardrails. Recently, we added 10 new resource types:

- **AWS**: RDS DB Cluster Parameter Group
- **Azure**: Container Registry, User Assigned Managed Identity, Metric Alerts, Private Link Service
- **GCP**: Vertex AI (Endpoint, Notebook Runtime Template), Dataplex (Lake, Task, Zone)

Accompanying them are over 70 controls from the common suite of Guardrails controls, including [CMDB](https://turbot.com/guardrails/docs/concepts/guardrails/cmdb) / [Discovery](https://turbot.com/guardrails/docs/concepts/guardrails/discovery), [Tags](https://turbot.com/guardrails/docs/concepts/guardrails/tagging), [Stacks / Configured](https://turbot.com/guardrails/docs/concepts/guardrails/configured), [Usage](https://turbot.com/guardrails/docs/concepts/guardrails/usage), and [ServiceNow Cloud discovery sync](https://turbot.com/guardrails/docs/guides/servicenow/guardrails-to-servicenow-sync#configuration-items-syncing). 

Additional controls were added for auto-remediation of:
- AWS Manage Fargate FIPS Mode for GovCloud accounts
- AWS Support permission delegation to which roles
- Azure MySQL flexible services minimum TLS version
- Azure Storage Account Table logging policies
- GCP Service Account disable based on inactive status
- GCP Subnetwork Flow Log policy configuration
- and 65+ more

## Guardrails: Improved Multi-Query in Calculated Policies
When using Guardrails [calculated policies](https://turbot.com/guardrails/docs/concepts/policies/calculated-faq) to retrieve data from multiple sources you previously had to write two queries. Common context fields such as the parent resource’s ID and URI, and the current resource’s ID, tags, and URI are now readily available. For example, previously for the parent resource's ID, then use that result in a second query to filter data. Now you can simply call the data field as `$.resource.parent.turbot.id` without the GraphQL query.

```graphql
budgetData: resources(filter: "resourceTypeId:'tmod:@turbot/aws#/resource/types/budget' resourceId:{{ $.resource.parent.turbot.id }}")
```

## Guardrails: Large scale improvements behind the scenes 
In line with this Launch Week’s theme of making Guardrails [easier](https://turbot.com/guardrails/blog/2024/10/guardrails-guides) and more [efficient](https://turbot.com/guardrails/blog/2024/10/guardrails-runtime-cost-reductions), we've introduced a number of mass improvements across Guardrails mods to boost performance, add flexibility, and simplify default policy behavior:

- **Better performance**: 40 Azure mods have been updated to support the latest Azure SDK versions, enhancing the discovery and management of 163 Azure resource types. The new SDK provides performance improvements and new data fields, which are included in the Guardrails CMDB.
- **More flexibility**: All supported AWS, Azure, GCP, and Kubernetes mods now include a new ServiceNow Import Set mode policy, allowing records to be shipped in either a single or multiple record format, offering more flexible integration options.
- **Improved default behavior**:
  - 36 GCP mods have been updated so the CMDB control for 142 GCP service resource types no longer depends on the API Enabled policy being set to Enforce: Enabled for the service.
  - 172 AWS mods have been updated for 431 AWS service resource types, adjusting the `Approved > Usage` policy for resource types to default to `Approved` instead of `Approved if AWS > {service} > Enabled`.

## Steampipe: 20+ new tables and columns supported

Steampipe has added support for over 20 new tables and columns across several cloud providers and services. These updates allow for deeper insights and more comprehensive queries for your cloud resources. 

New tables have been added for AWS, Azure, GCP, Linode, and Okta, including:
- **AWS**: New tables like `aws_app_runner_service` and `aws_timestreamwrite_database`.
- **Azure**: Support for `azure_key_vault_certificate` and `azure_web_application_firewall_policy`.
- **GCP**: Expanded support with tables for `gcp_dataplex_lake`, `gcp_composer_environment`, and `gcp_vpc_access_connector`.
- **Okta**: `okta_authentication_policy`
- **Linode**: `linode_firewall` and `linode_node_balancer`

Additionally, new columns have been added to existing tables, such as:
- **AWS**: Columns added to `aws_health_event` and `aws_ec2_instance_type`.
- **Azure**: Expanded data for `azure_compute_virtual_machine` and `azure_postgresql_flexible_server`.
- **GCP**: New columns for `gcp_cloudfunctions_function` and `gcp_sql_database_instance`.

## Steampipe: Improved query JSON output format
Steampipe has updated its query JSON output format to include a `columns` property along with the `rows` property, enabling better handling of duplicate column names by appending a unique suffix. This update aligns all cases around a single output format, making queries more consistent across Steampipe, Powerpipe, and Pipes.

Example of the new JSON format:
```json
{
  "columns": [
    { "name": "account_id", "data_type": "text" },
    { "name": "arn", "data_type": "text" }
  ],
  "rows": [
    { "account_id": "123456789012", "arn": "arn:aws:::123456789012" }
  ]
}
```

This standardized format now applies to:
- `steampipe query --output json`
- `powerpipe query run --output json`
- Snapshot data format
- Pipes query API

## Powerpipe: New GitHub Actions

Powerpipe has introduced new GitHub Actions to automate cloud compliance and Infrastructure as Code (IaC) checks within your CI/CD pipelines.

- **[Powerpipe Action Setup](https://github.com/turbot/powerpipe-action-setup)**: Simplifies the installation of Powerpipe and Steampipe, and sets up connections to your cloud accounts or IaC files. This action gets everything ready to run benchmarks and controls in subsequent workflow steps.
- **[Powerpipe Action Check](https://github.com/turbot/powerpipe-action-check)**: Executes benchmarks and controls from Powerpipe mods to scan for compliance issues across your cloud environments or IaC configurations. When used with [IaC mods](https://hub.powerpipe.io/?categories=iac), this action can create GitHub annotations for any controls in an alarm state, helping you identify specific lines in your configuration files that are non-compliant.


  Examples of what you can do:
  - Run specific AWS compliance controls to check for unused EBS volumes or unencrypted S3 buckets.
  - Execute cloud-wide security benchmarks (e.g., CIS or NIST) across multiple mods, including AWS, Azure, GCP, and Kubernetes.
  - Automate Terraform compliance checks, ensuring your infrastructure configurations meet security and best practice standards before deployment.
  - Check Kubernetes clusters for security best practices, such as ensuring encryption on secrets or restricting pod security policies.

By integrating these actions into your GitHub workflows, you can continuously monitor and enforce compliance for both cloud services, Kubernetes clusters, and IaC files, helping to catch issues early in the development lifecycle.

Note: the prior `Steampipe Action Check v1.1.0` is now deprecated and will no longer be maintained.

## Flowpipe: On-demand trigger execution

Flowpipe now supports [on-demand](https://flowpipe.io/docs/flowpipe-hcl/trigger/query) trigger execution, enabling SQL queries to be scheduled, and capturing data changes for pipelines. This powerful feature enables you to run a query on a defined schedule and pass the results—such as inserted, updated, or deleted rows—to the relevant pipeline for further processing.

For example, a query trigger can check for expired AWS IAM access keys and pass the data to a pipeline for deletion:

```hcl
trigger "query" "expired_access_keys" {
  database    = "postgres://steampipe@localhost:9193/steampipe"
  primary_key = "access_key_id"
  schedule    = "daily"

  sql = <<EOQ
      select
          access_key_id,
          user_name,
          create_date
      from
          aws_iam_access_key
      where
          create_date < now() - interval '90 days';
  EOQ

  capture "insert" {
    pipeline = pipeline.delete_expired_access_keys
    args = {
        access_keys = self.inserted_rows
    }
  }
}
```

Flowpipe keeps track of data in a SQLite database to detect new, updated, or deleted rows, and automatically triggers the pipeline based on these changes. This enables you to automate workflows such as handling access keys, DNS changes, or instance lifecycle management, all from a simple query.

## Pipes: New connection - Prisma Cloud
Pipes now supports Prisma Cloud, allowing you to centrally query, report, and collaborate on over 26 resource types. These include:
- Alerts and Rules
- Compliance stats
- IAM permissions, roles, and users
- Inventory assets and workloads
- Vulnerability assets
- and more

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Prisma Cloud Powerpipe plugin in Turbot Pipes" src="/images/blog/2024-10-launch-week-6-b-sides/prisma-cloud-in-pipes.png" />
</div>

This expands the growing list of security tools you can query and join data across, such as [CrowdStrike](https://hub.steampipe.io/plugins/turbot/crowdstrike), [Splunk](https://hub.steampipe.io/plugins/turbot/splunk), [Turbot Guardrails](https://hub.steampipe.io/plugins/turbot/guardrails), [Vanta](https://hub.steampipe.io/plugins/turbot/vanta), and more.

## Pipes: GCP temporary credential support added
To avoid the use of long-term service account credentials, Pipes now supports temporary credentials through service account impersonation using the `Service Account Token Creator` role.

With this update, [GCP integrations](https://turbot.com/pipes/docs/integrations/gcp) can use temporary credentials to securely access project data. To enable this in Pipes, choose **Service Account Impersonation** from the **Access Mode** dropdown, and grant the `Service Account Token Creator` role to the Pipes service. Additionally, label your project with the key `turbot-pipes` and follow the steps to automate role assignment using the provided Terraform plan or gcloud command. Click the **Test Discovery** button to verify the configuration.

Alternatively, you can still use a service key file by selecting **Credentials** from the **Access Mode** dropdown and uploading a JSON key pair for an existing GCP service account.

## Pipes: 10+ new security benchmarks and controls supported
Powerpipe has added 4 new security benchmarks and 7 new controls to our AWS, Azure, and GCP compliance mods. These new benchmarks are available in the Powerpipe Hub and Turbot Pipes to assess your security posture and share status with your team. The latest benchmarks include:

- [AWS Australian Cyber Security Center (ACSC) Essential Eight](https://hub.powerpipe.io/mods/turbot/aws_compliance/controls/benchmark.acsc_essential_eight)
- [AWS CIS Compute Services v1.0.0](https://hub.powerpipe.io/mods/turbot/aws_compliance/controls/benchmark.cis_compute_service_v100)
- [AWS NIST 800-172](https://hub.powerpipe.io/mods/turbot/aws_compliance/controls/benchmark.nist_800_172)
- [GCP SOC2 2017](https://hub.powerpipe.io/mods/turbot/gcp_compliance/controls/benchmark.soc_2_2017)

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="AWS Australian Cyber Security Center (ACSC) Essential Eight Powerpipe benchmark" src="/images/blog/2024-10-launch-week-6-b-sides/aws-acsc-benchmark.png" />
</div>

New controls span services like Azure Application Gateway, Azure Application Insights, Azure Log Analytics Workspace, and GCP AlloyDB Instance Logs, offering more detailed and comprehensive compliance coverage.

## Hacktoberfest 2024: Mid-event update

[Hacktoberfest 2024](https://turbot.com/blog/2024/09/hacktoberfest-2024) is in full swing, and we’re excited to see incredible contributions from the community! This year, Turbot is participating in the global event, encouraging developers and enthusiasts to contribute to our open-source projects, including Steampipe, Powerpipe, and Flowpipe, along with our samples and docs repos for Turbot Pipes and Guardrails.

So far, we’ve seen a variety of contributions, ranging from bug fixes to feature enhancements across our repositories. A few highlights include:

* @dbermuehler - Adding [Shield Tables](https://github.com/turbot/steampipe-plugin-aws/pull/2315) and [EPSS Score](https://github.com/turbot/steampipe-plugin-aws/pull/2321) to the AWS plugin.
* @pdecat - New `gcp_organization_project` and `gcp_compute_instance_group_manager` [tables](https://github.com/turbot/steampipe-plugin-gcp/pull/663) in the GCP plugin.
* @oguzhan-yilmaz - creating a Steampipe + Powerpipe [helm chart](https://github.com/oguzhan-yilmaz/steampipe-powerpipe-kubernetes)
* @brunorossi - leveraging AWS Steampipe FDWs in a custom [AWS query tool](https://github.com/brunorossi/aws_sql_tool) via REST APIs.
* @l-teles - creating a new [Steampipe plugin for Detectify](https://github.com/l-teles/steampipe-plugin-detectify)
* and countless other cool features, improvements, doc fixes, blog posts, etc

It's inspiring to see the passion and creativity in these contributions. Whether it’s fixing a bug, adding new features, or creating insightful content, every effort helps make our open-source community stronger.

We’re looking forward to more great contributions in the final stretch of Hacktoberfest. If you haven’t joined yet, there’s still time to get involved and earn some exclusive [Turbot swag](https://turbot.com/blog/2024/09/hacktoberfest-2024#any-special-swag-from-turbot)!


## Flip over to A-sides for the Wrap Up
Thank you for joining us for another exciting Launch Week! Check out the week's daily announcements summary in our [Launch Week 6 Wrap Up](https://turbot.com/blog/2024/10/launch-week-6-wrap) post. Stay connected with us in our [Slack community](https://turbot.com/community/join) for our next Launch Week in a few months! 



### Pipes, Assemble!

Just like a certain heroic team coming together to defend the world, [Steampipe](https://steampipe.io), [Powerpipe](https://powerpipe.io), and [Flowpipe](https://flowpipe.io) are now united under the banner of **Pipes**. With Flowpipe officially becoming a [first-class citizen](/blog/2024/10/flowpipe-in-pipes) in the Pipes platform, your cloud tools have never been more powerful. Ready to join forces in your Pipes workspace, Flowpipe now works seamlessly alongside Steampipe and Powerpipe. Assemble your dream team of tools and get started with a single click - whether it’s your first Steampipe [query](https://turbot.com/pipes/docs/using/steampipe/query), Powerpipe [dashboard](https://turbot.com/pipes/docs/using/powerpipe/dashboards), or Flowpipe [pipeline](https://turbot.com/pipes/docs/using/flowpipe/pipelines).

### The Superpowers of Pipes

In the battle for cloud intelligence and security, Pipes delivers more than just tools - it gives you superpowers. Each component in Pipes has its own strengths, but they’re even more powerful together. While platforms like Splunk fight the good fight as monolithic titans, Pipes takes a different approach: empowering you to work with standalone tools or as a fully integrated, cloud-based system. Pipes is available in the cloud or right in your [terminal](https://turbot.com/pipes/docs/connect) - giving developers the flexibility to fight their cloud challenges on their terms. Assemble your team, build your solutions, and level up from individual developer efforts to full-blown organizational strength. 

### Seamless Collaboration Between Heroes

The key to any superhero team is seamless collaboration, and Pipes is no exception. When you use Steampipe, Powerpipe, or Flowpipe, it’s like watching your favorite heroes in action. Take, for example, a shared AWS [connection](https://turbot.com/pipes/docs/using/steampipe/connections) in Pipes. Once it’s set up, every tool in the Pipes suite can use it effortlessly. Steampipe can harness that connection for its queries, Powerpipe for its benchmarks and dashboards, and Flowpipe for enforcing security compliance. It’s all about using the right hero for the right job - no more setting up connections multiple times!

![](/images/blog/2024-10-pipes-assemble/pipes-assemble.png)

### Custom Mods: Your Secret Weapon

Every hero has a secret weapon, and in Pipes, custom mods are yours. For [Launch Week 5](https://turbot.com/launch-week/05), we introduced a [GitHub integration](https://turbot.com/pipes/blog/2024/05/github-custom-mods) that deploys [Powerpipe mods](https://turbot.com/pipes/docs/using/powerpipe/mods) from a branch or tag in a private repo. Now, with Flowpipe integrated, it works the same way for your custom [Flowpipe mods](https://turbot.com/pipes/docs/using/flowpipe/mods). Pipes automatically refreshes mods when changes are detected, ensuring your tools are always powered up and ready for action.

### One Unified Team, One Mission

Pipes isn’t just about powerful tools - it’s about teamwork. Every piece of Pipes speaks the same language: open source, as-code, and developer-friendly. Whether you’re writing SQL queries, building dashboards, or designing pipelines in HCL, it’s all part of the same mission. Together, Steampipe, Powerpipe, and Flowpipe give you everything you need to tackle cloud governance, security, and more. 

### What’s Next for the Team?

The Pipes team is just getting started. With Flowpipe now part of the action, the Pipes universe has room to grow, and we’re excited to see what’s next. More components? New powers? Who knows! But one thing’s for sure — Pipes is here to help you assemble the ultimate team for cloud security and intelligence. Ready to suit up? Take Pipes for a spin and [let us know](https://turbot.com/community/join) how it goes!



[Flowpipe](https://flowpipe.io) is the open-source solution for DevOps and security pros who use workflows-as-code to automate cloud operations. Today we're thrilled to announce that Flowpipe joins its open-source siblings [Steampipe](https://steampipe.io) and [Powerpipe](https://powerpipe.io) as a [component](/pipes/blog/2024/10/pipes-assemble) of the Pipes platform. Flowpipe features will gradually be rolling out across all Pipes users in the coming days.

To see it in action, we'll use an [AWS Thrifty](https://hub.flowpipe.io/mods/turbot/aws_thrifty/pipelines/aws_thrifty.pipeline.detect_and_correct_s3_buckets_without_lifecycle_policy) pipeline to remediate buckets that don't have a lifecycle policy. Let's see how that works in Pipes.

## An AWS-like collection of services

The first thing you'll notice, in a new Pipes workspace, is that all three core components — Steampipe, Powerpipe, and Flowpipe — are right there ready to use. In this case our job requires Flowpipe so that's the tool of choice. 

![new workspace with 3 components](/images/blog/2024-10-flowpipe-in-pipes/three-components.png)

## Easy mod installation

The Thrifty mod is available on the hub, but Pipes brings that interface into your workspace and enables point-and-click installation. 

![mod installation](/images/blog/2024-10-flowpipe-in-pipes/mod-install.png)

## Form-based configuration

You configure Flowpipe mods with variables which, when running the standalone tool, you edit by hand. Pipes automagically turns those variables into forms. In this case, we won't need to change any of the variables. The pipeline will ask for a decision about whether to remediate each non-compliant bucket, and we want that decision to happen in Slack, but that's the default notifier for this workspace so we are ready to go. The pipeline also needs a Steampipe AWS connection, in order to query for non-compliant buckets, but that comes for free as well. The organization that owns this workspace has set up the connection and permitted this workspace to use it. So we can just select the **Run Pipeline** button.

![pipeline ready to run](/images/blog/2024-10-flowpipe-in-pipes/run-pipeline.png)

## Slack-based approval

Here's a non-compliant bucket found by the pipeline. We've routed the approval decision to Slack, but could have used email instead, or MSTeams. We want this bucket to have a lifecycle policy so we choose `Apply lifecycle configuration`. What if the approver doesn't see the message right away? No problem, Pipes suspends the workflow so you won't be billed for the wait, and restarts it when approver replies.

![slack  approval](/images/blog/2024-10-flowpipe-in-pipes/slack-approval.png)

## Hands-free remediation

Flowpipe receives the reply, takes the action, and reports the outcome. Of course it can get old making decisions like this on a per-bucket basis. If you want all existing and new buckets to block public access, you can run the pipeline in a hands-free way, on a schedule. To do that you configure a couple of variables. For this Flowpipe needs to run in server mode, but Pipes takes care of that for you.


## Easy Slack integration

The default [notifier](https://flowpipe.io/docs/reference/config-files/notifier#notifiers) for this workspace is Slack. The notifier in turn refers to an [integration](https://flowpipe.io/docs/reference/config-files/integration#integration) that was already set up in the organization and available to this workspace. How did that happen? Let's create a new Slack integration to see how easy that is. Here are the types of integrations we can create.

![integrations screen](/images/blog/2024-10-flowpipe-in-pipes/integrations-available.png)

We've already seen the [AWS](/pipes/blog/2024/07/aws-org-sync), [Azure](/pipes/blog/2024/07/azure-tenant-sync), [GCP](/pipes/blog/2024/07/gcp-org-sync) integrations, which populate your Pipes organization or tenant with accounts, subscriptions, and projects from those platforms. And we've also seen the [GitHub](/pipes/blog/2024/05/github-custom-mods) integration, which uses Pipes' 3rd-party GitHub app to install mods into a workspace and automatically refresh them when changes are pushed to a branch or tag. The [Slack integration](https://turbot.com/pipes/docs/integrations/slack) works like the [GitHub integration](https://turbot.com/pipes/docs/integrations/github). You name it, install into your Slack instance as a 3rd-party Slack app, and pick the Slack channel where you want notifications sent. 

![slack app](/images/blog/2024-10-flowpipe-in-pipes/slack-app.png)

## Custom mods

The [Flowpipe Hub](https://hub.flowpipe.io) provides a growing collection of libraries and samples that capture a variety of common DevOps patterns, but everyone's situation is unique. It's often useful to write your own mods to suit your particular needs. How to install them? You may already have used the [GitHub](/pipes/blog/2024/05/github-custom-mods) to install Powerpipe mods into a workspace, and automatically refresh them when changes are pushed to a branch or tag. The same integration works exactly the same way for Flowpipe mods.

## Next-level workflows as code

As noted in [Pipes Assemble!](/pipes/blog/2024/10/pipes-assemble), Pipes is a platform on which Steampipe, Powerpipe, and now Flowpipe share common patterns and infrastructure. You can build powerful solutions with the standalone components, but Pipes brings them together smoothly, in a way that's easy to manage across your organization or tenant. If you're a Pipes user who hasn't yet tried Flowpipe, fire it up in a workspace and [let us know](https://turbot.com/community/join) how it goes.


## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/lwATElOgJ_4"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Run Flowpipe Workflows in Turbot Pipes"
>
</iframe>
</div>




The Center for Internet Security (CIS) benchmarks for AWS are widely used by cloud teams as guidance to configure their AWS accounts securely. With Flowpipe's new [AWS CIS](https://hub.flowpipe.io/mods/turbot/aws_cis) mod you can detect misconfigurations and apply fixes according to the v3.0.0 or v4.0.0 recommendations. Let's run v4.0.0's section 2 which focuses on storage.

```bash
$ flowpipe pipeline run cis_v400_2
```

Flowpipe walks you through each subsection, finds resources that don't comply with the recommendations, and offers you the choice to skip each resource or apply the recommendation. Here's how that looks in progress. We've arrived at section 2.1.4, and are running a control that checks whether an S3 bucket blocks public access as the recommendation says it should. Flowpipe has found two buckets that don't comply. We opted to skip the first one, and are ready to remediate the second.


![cis run in progess](/images/blog/2024-10-compliance-workflows-for-aws/cis-run-in-progress-1.png)

Here we've chosen `Block public access`. Flowpipe takes the action, reports the outcome, and proceeds. 

![aws console before fix](/images/blog/2024-10-compliance-workflows-for-aws/cis-run-in-progress-2.png)

You can step through all of section 2 in this way. You could even step through the entire CIS benchmark, section by section and control by control. 

## Under the hood: AWS Compliance

The [AWS CIS](https://hub.flowpipe.io/mods/turbot/aws_cis) mod is actually a thin wrapper around another new mod, [AWS Compliance](https://hub.flowpipe.io/mods/turbot/aws_compliance), which provides a suite of pipelines that do the work of detecting misconfigurations and enabling human-supervised correction. `AWS Compliance` in turn relies on two helpers: Flowpipe's [library mod for AWS](https://hub.flowpipe.io/mods/turbot/aws) and a [helper mod](https://github.com/turbot/flowpipe-mod-detect-correct) that implements the detect-and-correct pattern.

Here's how things look from the perspective of the installed `AWS CIS` mod.

```bash
~/flowpipe-mod-aws-cis$ flowpipe mod list
aws_cis
└── github.com/turbot/flowpipe-mod-aws-compliance@v1.0.0
    ├── github.com/turbot/flowpipe-mod-aws@v1.0.0
    └── github.com/turbot/flowpipe-mod-detect-correct@v1.0.0
```

You can list the top-level CIS pipelines like this.

```bash
~/flowpipe-mod-aws-cis$ flowpipe pipeline list | grep aws_cis
aws_cis           cis_v300            The CIS Amazon Web Services Foundations Benchmark provides...
aws_cis           cis_v300_1
aws_cis           cis_v300_1_1
aws_cis           cis_v300_1_10
```

You can also list the supporting detect-and-correct pipelines like this.

```bash
~/flowpipe-mod-aws-cis$ flowpipe pipeline list | grep detect_and_correct
aws_compliance    detect_and_correct_accounts_without_alternate_contact_security_registered                             Detects account alternate ...
aws_compliance    detect_and_correct_apigateway_rest_api_stage_if_xray_tracing_disabled                                 Detects unattached EIPs ...
...
```

One of those, `detect_and_correct_s3_buckets_if_publicly_accessible`, is the pipeline that's used by CIS 2.1.4. You can also install `AWS Compliance`  separately and use that pipeline directly in a standalone context.

```bash
~/flowpipe-mod-aws-compliance$ flowpipe pipeline run detect_and_correct_s3_buckets_with_block_public_access_disabled
```

By default this works just as we've already seen: you're prompted with each bucket that doesn't block access, and in each case you choose to skip or apply the action. But making all those decisions can get old. What if you want to apply the fix to all non-compliant buckets? You can do that by setting a mod variable, `approvers`, to the empty list. Flowpipe will then automatically apply the fix across all your buckets.

Now suppose you want to apply the same rule to new buckets as they're created? To do that, configure the pipeline to run on a schedule. Again it's just a matter of setting variables. To activate a scheduled run of the pipeline, set the variable `s3_bucket_access_enabled_trigger_enabled` to `true`. And if you want to run on a schedule other than the default, which is every 15 minutes, also set `s3_bucket_access_enabled_trigger_schedule` to use either a [named interval](https://flowpipe.io/docs/flowpipe-hcl/trigger/schedule#arguments) or a custom cron expression.

## Degrees of freedom: benchmarks, controls, and pipelines

  If your priority is CIS, we've got you covered with the [AWS CIS](https://hub.flowpipe.io/mods/turbot/aws_cis) mod. It provides the flexibility to run the whole suite, or just one section, or even individual controls, with interactive supervision on a per-resource basis. If you need even more flexibility, the supporting [AWS Compliance](https://hub.flowpipe.io/mods/turbot/aws_compliance) pipelines are available to be used in other ways, with or without human approval, on a one-time or scheduled basis. But there's even more flexibility than that. If you need to act on AWS resources in ways other than `AWS Compliance` does, the [AWS](https://hub.flowpipe.io/mods/turbot/aws) library mod gives you the raw ingredients to run different kinds of queries and take different kinds of actions. Use these mods in whatever ways make sense to you, and [let us know](https://turbot.com/community/join) how it goes.


## See it in action

<div className="flex justify-center">

<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/I3RtrxQM6T4" 
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Remediate AWS CIS v3.0 and v4.0 with Flowpipe's interactive wizard"
>
</iframe>
</div>



Today we're promoting all three of Turbot's open source components — Steampipe, Powerpipe, and Flowpipe — to v1.0.0. It's a major milestone on a journey that began with the launch of Steampipe v0.1.0 on January 21, 2021, along with an initial suite of 13 plugins featured on the then-new Steampipe Hub. 

<object style={{ float:'right', width:'30%', 'marginLeft': '1em'}} data="/images/steampipe_wordmark.svg" type="image/svg+xml"></object>

The first version of Steampipe's most popular plugin, [AWS](https://hub.steampipe.io), provided 68 tables, already a formidable collection that has since grown to nearly [500](https://hub.steampipe.io/plugins/turbot/aws). In the [inaugural blog post](https://steampipe.io/blog/introducing-steampipe) we wrote:

> Steampipe, a new open source project from Turbot, enables cloud pros (e.g. software developers, operations engineers and security teams) to query their favorite cloud services with SQL. It has quickly become one of our favorite tools in-house and we hope it finds a way into your toolbox as well.

Nearly four years later, that hope has become a reality. Here are a few top-line Steampipe stats:

  <table style={{borderCollapse: 'separate', borderSpacing: 0, width: '300px'}}>
    <tr>
      <td style={{padding: '8px', textAlign: 'right', borderRight: '1px solid lightgrey', borderBottom: '1px solid lightgrey'}}>6,876</td>
      <td style={{padding: '8px', borderBottom: '1px solid lightgrey'}}><a href="https://github.com/turbot/steampipe">GitHub stars</a></td>
    </tr>
    <tr>
      <td style={{padding: '8px', textAlign: 'right', borderRight: '1px solid lightgrey', borderBottom: '1px solid lightgrey'}}>145</td>
      <td style={{padding: '8px', borderBottom: '1px solid lightgrey'}}><a href="https://hub.steampipe.io/plugins">plugins</a></td>
    </tr>
    <tr>
      <td style={{padding: '8px', textAlign: 'right', borderRight: '1px solid lightgrey'}}>125</td>
      <td style={{padding: '8px'}}><a href="https://steampipe.io/blog/news-and-reviews">news and reviews</a></td>
    </tr>
  </table>

  The core query capability provided by Steampipe plugins was soon augmented with a suite of [mods](https://hub.steampipe.io/mods) that focused, initially, on compliance checks for AWS, Azure, GCP, and a few others. These mods wrapped Steampipe queries in HCL-defined controls and benchmarks tailored to the [Center for Internet Security](https://www.cisecurity.org/cis-benchmarks) recommendations. Today we offer compliance mods for Audit Manager Control Tower, AWS Foundational Security Best Practices, CIS, FedRAMP, FFIEC, GDPR, HIPAA, NIST, PCI, SOC 2, and many more, along with mods for asset inventory/insights, shift-left scanning, and cost management.

<table style={{borderCollapse: 'separate', borderSpacing: 0, width: '300px'}}>
    <tr>
      <td style={{padding: '8px', textAlign: 'right', borderRight: '1px solid lightgrey', borderBottom: '1px solid lightgrey'}}>17</td>
      <td style={{padding: '8px', borderBottom: '1px solid lightgrey'}}><a href="https://hub.powerpipe.io/?objectives=compliance">compliance</a></td>
    </tr>
    <tr>
      <td style={{padding: '8px', textAlign: 'right', borderRight: '1px solid lightgrey', borderBottom: '1px solid lightgrey'}}>21</td>
      <td style={{padding: '8px', borderBottom: '1px solid lightgrey'}}><a href="https://hub.powerpipe.io/?objectives=dashboard">asset inventory and insights</a></td>
    </tr>
    <tr>
      <td style={{padding: '8px', textAlign: 'right', borderRight: '1px solid lightgrey', borderBottom: '1px solid lightgrey'}}>6</td>
      <td style={{padding: '8px', borderBottom: '1px solid lightgrey'}}><a href="https://hub.powerpipe.io/?objectives=iac">shift-left scanning</a></td>
    </tr>
    <tr>
      <td style={{padding: '8px', textAlign: 'right', borderRight: '1px solid lightgrey', borderBottom: '1px solid lightgrey'}}>6</td>
      <td style={{padding: '8px', borderBottom: '1px solid lightgrey'}}><a href="https://hub.powerpipe.io/?objectives=cost">cost control</a></td>
    </tr>
  </table>

These continue to evolve rapidly. [AWS Compliance](https://hub.powerpipe.io/), for example, now provides 1,229 controls and 30 dashboards, and adds support for security and compliance benchmarks on a [regular basis](https://github.com/turbot/steampipe-mod-aws-compliance/blob/main/CHANGELOG.md).

Today all these mods join the v1.0.0 release party.

## From benchmarks to dashboards

From the beginning we've standardized on two declarative languages: SQL to acquire data from clouds, HCL to report and visualize. That core pattern has carried through the suite of open source components, while evolving along the way. Initially, for example, the benchmarks we provided (and that you could [compose or create](https://steampipe.io/blog/release-0-11-0) yourself) used HCL to declare and combine controls. Then, with [Dashboards as Code](https://steampipe.io/blog/dashboards-as-code), we added new HCL syntax for [tables](https://powerpipe.io/docs/powerpipe-hcl/table), [chart](https://powerpipe.io/docs/powerpipe-hcl/chart), [infocards](https://powerpipe.io/docs/powerpipe-hcl/card), [inputs](https://powerpipe.io/docs/powerpipe-hcl/input), [relationship graphs](https://powerpipe.io/docs/powerpipe-hcl/graph), and more. What had been an engine for running benchmarks could now also power dashboards, all in the as-code, GitHub-friendly style of modern DevOps.

The arrival of [relationship graphs](https://steampipe.io/blog/release-0-18-0) showcased the value of this approach. Use SQL to query AWS for sets of resources related to, for example, a VPC. Then use HCL to visualize its full context.

![relationship graph](/images/blog/2024-10-open-source-v1/relgraph.png)

## Database-agnostic plugins

Originally, Steampipe plugins ran in a captive instance of Postgres equipped with a plugin manager and [foreign data wrapper](https://github.com/turbot/steampipe-postgres-fdw) extension. People often asked: <em>"Can we also run those plugins in our own databases?"</em> We made it so, with "zero-ETL" distributions of plugins for [your own instance of Postgres](https://steampipe.io/blog/2023-12-postgres-extensions), or for [SQLite](https://steampipe.io/blog/2023-12-sqlite-extensions). While we were at it, we also made them available as [standalone exporters](https://steampipe.io/blog/2023-12-steampipe-export).

Now a bigger picture started to come into focus. The Steampipe ecosystem, as it grew and evolved, was becoming a suite of components that worked well together in their original contexts, but could be recombined to serve other purposes in other contexts. 

## Powerpipe emerges from Steampipe

<object style={{ float:'right', width:'30%', 'marginLeft': '1em'}} data="/images/powerpipe_wordmark.svg" type="image/svg+xml"></object> 

Next, we realized that the benchmarks-and-dashboards subsystem included with Steampipe ought to be [unbundled](https://steampipe.io/blog/steampipe-unbundled) from the core query engine. These were logically two different components that worked well together, but needn't be joined at the hip. Once decoupled from Steampipe, for example, the benchmarks-and-dashboards subsystem could become database-agnostic just as plugins had. Thus was born [Powerpipe](https://powerpipe.io), a standalone tool that runs all your [benchmarks and dashboards](https://powerpipe.io/blog/migrating-from-steampipe) but can also connect to [any Postgres](https://powerpipe.io/docs/powerpipe-hcl/query#postgres), or [SQLite](https://powerpipe.io/docs/powerpipe-hcl/query#sqlite), or [MySQL](https://powerpipe.io/docs/powerpipe-hcl/query#mysql), or [DuckDB](https://powerpipe.io/docs/powerpipe-hcl/query#duckdb). 

Now you could build dashboards-as-code to visualize data in your own instance of Postgres, or even data living in [Parquet files](https://thenewstack.io/llm-assisted-translation-from-postgres-to-sqlite-and-duckdb/) using DuckDB. Decoupled from Steampipe, Powerpipe could still serve its original purpose while also enabling wider ambitions.

## Flowpipe delivers HCL-powered workflows for DevOps

<object style={{ float:'right', width:'30%', 'marginLeft': '1em'}} data="/images/flowpipe_wordmark.svg" type="image/svg+xml"></object> 

All this visibility into resources and their configuration prompted users to ask: <em>"How can we act on everything we can now see?"</em> To build DevOps workflows that create, update, and delete you have to orchestrate scripting, CLI commands, Lambda functions, containers, and human approvals across multiple providers and identity frameworks. Bash and Python have served us well, but there had to be a better way.

Enter [Flowpipe](https://flowpipe.io), Turbot's third major open source component, which debuted in [December 2023](https://flowpipe.io/blog/introducing-flowpipe). It's <em>"like an HCL-powered alternative to Step Functions"</em>, said [Aidan Steele](https://x.com/__steele/status/1735411461399453996). Noting Flowpipe's DAG-powered implicit parallelisation, he added: <em>"I wish SFN had that!"</em>

Like Steampipe and Powerpipe, Flowpipe is a self-contained binary that you can easily deploy locally, or in any CI/CD enviroment. Like its siblings, it's supported by a [hub](https://hub.flowpipe.io) where you can find prebuilt workflows for common DevOps chores, supported by libraries that you can use to build your own workflows, and samples that capture common patterns.

<table style={{borderCollapse: 'separate', borderSpacing: 0, width: '300px'}}>
    <tr>
      <td style={{padding: '8px', textAlign: 'right', borderRight: '1px solid lightgrey', borderBottom: '1px solid lightgrey'}}>35</td>
      <td style={{padding: '8px', borderBottom: '1px solid lightgrey'}}><a href="https://hub.flowpipe.io/?type=library">library mods</a></td>
    </tr>
    <tr>
      <td style={{padding: '8px', textAlign: 'right', borderRight: '1px solid lightgrey', borderBottom: '1px solid lightgrey'}}>41</td>
      <td style={{padding: '8px', borderBottom: '1px solid lightgrey'}}><a href="https://hub.flowpipe.io/?type=sample">sample mods</a></td>
    </tr>
  </table>

And yes, these mods also join today's v1.0.0 release party.

## Semantic versioning

We've held onto v0.x for a long time, allowing the tools to evolve and mature. Now that we see their shapes clearly we'll continue to move them forward, and we'll do that with regular releases that follow semantic versioning. And with a solid pattern in place, we're in a great position to grow the family of components.

## Many components, one community

In Turbot's [Slack community](https://turbot.com/community/join), nearly 7,000 users and contributors gather to discuss Steampipe, Powerpipe, and Flowpipe, along with the growing suites of mods they enable. You can use these core components individually to query cloud resources, report on them, and act on them. And they work even better together as when, for example, a Steampipe query feeds a Powerpipe dashboard or a Flowpipe pipeline.

Flipping the v1.0.0 switch is always an emotional moment. We're just getting started, but it was time to recognize the maturity of this ecosystem and celebrate with everyone who has traveled with us on this journey. Thanks, and stay tuned!



In previous posts, we’ve highlighted how Turbot Guardrails simplifies cloud resource discovery for [AWS](https://turbot.com/guardrails/blog/2024/01/servicenow-aws-discovery), [Azure](https://turbot.com/guardrails/blog/2024/03/servicenow-azure-discovery), [GCP](https://turbot.com/guardrails/blog/2024/03/servicenow-gcp-discovery), and [Kubernetes](https://turbot.com/guardrails/blog/2024/05/servicenow-kubernetes-discovery), syncing these resources directly to ServiceNow's CMDB CI tables or through [Import Sets](https://turbot.com/guardrails/blog/2024/07/servicenow-import-set-cloud-discovery). Now, we’ve extended our resource discovery capabilities to manage relationships automatically, offering a more dynamic and flexible approach to handling CMDB CI relationships for both existing and new CMDB CI tables.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="ServiceNow relationship dependency relationship graph extended" src="/images/blog/2024-10-servicenow-cmdb-relationships/servicenow-relationship-graph-extended.png" />
</div>

This integration requires no additional ServiceNow modules or cloud services from AWS, Azure, GCP, or Kubernetes. Turbot Guardrails provides a direct, cost-effective solution for automating real-time cloud discovery and relationship management within your CMDB

Guardrails' ServiceNow features are ideal for organizations new to cloud discovery or those looking to enhance existing ServiceNow cloud integrations.

## Cloud resource relationship examples

Turbot Guardrails syncs hundreds of different cloud resource relationships using standard ServiceNow relationship types, providing a clear view of how resources interact. These relationships are critical for understanding dependencies, ensuring proper resource configuration, and maintaining real-time insights into cloud infrastructure. Below are four examples spanning AWS, Azure, GCP, and Kubernetes:

| **Parent**       | **Child**       | **Example**                                                               |
|------------------|-----------------|---------------------------------------------------------------------------|
| **Uses**         | Used by         | **AWS EC2 Instances** use **Key pairs** for access                        | 
| **Contains**     | Contained by    | **Azure Storage Accounts** contain **Queues** for messaging               |
| **Provides**     | Provided by     | **GCP Forwarding Rules** are provided by **Target Pools** for assignments |
| **Instantiates** | Instantiates by | **Kubernetes Replicasets** instantiate **Pods** in the cluster            |

## Why are relationships critical for your CMDB?

Managing relationships between cloud resources in ServiceNow provides essential insights into how cloud services are interconnected, along with critical business context. These relationships benefit multiple teams across the organization:

- **Operations Teams**: Understanding resource dependencies enables teams to quickly assess the impact of outages or changes. For example, knowing that an AWS EC2 instance is tied to specific VPCs, security groups, and storage volumes allows for faster incident resolution and reduced downtime.
- **Security Teams**: Visibility into resource relationships helps identify potential vulnerabilities and ensures consistent security policies across cloud environments. By tracking dependencies among virtual machines, databases, and network components, security teams can respond more effectively to risks.
- **SACM Teams (Service Asset & Configuration Management)**: For SACM professionals managing assets, relationships between cloud resources provide a detailed view of how assets align with business services and cost centers. Automatic management of these relationships reduces overhead and provides more accurate tracking of business service ownership, cost allocation, and operational oversight.

## Turbot Guardrails approach to cloud discovery

The Turbot Guardrails ServiceNow integration not only extends but also enhances native discovery capabilities by offering broader coverage, real-time updates, and flexible management of relationships—all without the need for additional licensing or services. By combining Turbot Guardrails' automated discovery with ServiceNow, you can accurately sync cloud resources and relationships into your CMDB in real-time, without the added costs or complexity of native tools.

| Feature                  | Native ServiceNow Discovery          | Turbot Guardrails                                 |
|--------------------------|--------------------------------------|---------------------------------------------------|
| **Licensing & Services** | Requires additional licensing        | No extra licensing required                       |
| **Resource Coverage**    | Limited to core IaaS services        | 750+ cloud services across AWS, Azure, GCP, & K8s |
| **Relationship Data**    | Minimal relationship coverage        | Relationship associations for all resource types  |
| **Updates**              | Scheduled batch jobs                 | Real-time discovery and updates                   |
| **Customization**        | Requires professional services       | Flexible, point-and-click setup & management      |
| **Business Context**     | Manual mappings to non-infra context | Automatic mappings to apps, cost centers, etc    |
| **Management**           | Managed separately for each platform | Centralized multi-cloud discovery management      |


## Configuring ServiceNow cloud resource relationships

### Enable Relationship Sync Policies
Relationships between cloud resources are defined and managed through Guardrails policies, configured per resource type such as Azure CosmosDB Account, GCP Firebase Project, Kubernetes ReplicaSet, or AWS S3 Bucket. For instance, to set a policy on an AWS S3 Bucket, you would navigate to the `AWS > S3 > Bucket > ServiceNow > Relationships` policy and set it to `Enforce: Enabled`:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails ServiceNow relationship enabled for AWS S3 Bucket" src="/images/blog/2024-10-servicenow-cmdb-relationships/enable-relationship-s3-bucket.png" />
</div>

The `AWS > S3 > Bucket > ServiceNow > Relationships > Template` already includes built-in logic to relate the AWS Region with a relationship type of `Contains::Contained by`. Additionally, it includes a `Feeds::Fed By` relationship from VPC Flow Logs as a log destination for the logs.

Once enabled, Guardrails will automatically add all AWS S3 buckets to the ServiceNow CI Relationships table (`cmdb_rel_ci_list`), with the appropriate parent, child, and relationship type details between the buckets and regions.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails ServiceNow relationship CMDB Relationships updated" src="/images/blog/2024-10-servicenow-cmdb-relationships/s3-bucket-relationships.png" />
</div>

Now the CI Relationships Table has those CI records synced, all native relationship graphs, visualizations, and views are updated with the latest relationships:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="ServiceNow relationship dependency relationship graph" src="/images/blog/2024-10-servicenow-cmdb-relationships/servicenow-relationship-graph.png" />
</div>

Guardrails will continuously discover and manage relationships if there are changes to the existing resources or when new resources are created. All changes are discovered and synced to ServiceNow in real-time.

### Managing custom relationships

Turbot Guardrails provides hundreds of out-of-the-box relationships in templates that are automatically populated when resources are created, updated, or deleted. These templates can be customized and extended to suit your organization’s specific needs.

For example, an AWS S3 Bucket contained by an AWS Region is defined by the following template:

```
- type: "Contains::Contained by"
  parent:
    name: {{ $.region.turbot.title }}
    sysId: {{ $.region.metadata.serviceNow.sysId }}
    tableName: {{ $.region.metadata.serviceNow.tableName }}
```

The variables can be adjusted to reference any datapoint discovered by the cloud provider, Turbot, or ServiceNow. In this template, for the `acme-demo-turbot-9` S3 Bucket and its related region, Guardrails automatically identifies the metadata that will update the `cmdb_rel_ci_list`  table with the appropriate region `name`, ServiceNow `sysId`, and the ServiceNow `tableName` for the regions table.

```
- type: "Contains::Contained by"
  parent:
    name: "us-east-2"
    sysId: "5c9eff1f83095210b30766d0deaad331"
    tableName: "cmdb_ci_cloud_region"
```

You can modify existing policies by adjusting the details above, or extend the relationship by adding another type. For instance, AWS VPC Flow logs are used by VPCs:

```
- type: "Uses::Used by"
  parent:
    name: {{ $.vpc.turbot.title }}
    sysId: {{ $.vpc.metadata.serviceNow.sysId }}
    tableName: {{ $.vpc.metadata.serviceNow.tableName }}
```

You can further extend this by adding relationships for log destination details (e.g., S3 Bucket, AWS CloudWatch log group):

```
- type: "Uses::Used by"
  parent:
    name: {{ $.vpc.turbot.title }}
    sysId: {{ $.vpc.metadata.serviceNow.sysId }}
    tableName: {{ $.vpc.metadata.serviceNow.tableName }}
- type: "Feeds::Fed By"
  child:
    name: {{ $.logDestination.turbot.title }}
    sysId: {{ $.logDestination.metadata.serviceNow.sysId }}
    tableName: {{ $.logDestination.metadata.serviceNow.tableName }}
```

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="ServiceNow relationship dependency relationship graph extended" src="/images/blog/2024-10-servicenow-cmdb-relationships/servicenow-relationship-graph-extended.png" />
</div>

With this level of flexibility, Guardrails allows you to customize and extend relationships to reflect your cloud architecture and business requirements.

## Add business context to your ServiceNow relationships

In addition to syncing cloud infrastructure relationships, Guardrails allows you to relate cloud resources to any other CMDB CI table. Many organizations find it useful to associate their cloud resources with cost centers, applications, and business services. For example, a cloud account is often related to a Business Service.

Using the `AWS > Account > ServiceNow > Relationships > Template` Guardrails policy, you can relate the business service name and sysId to the cloud account, either directly inputting the information or dynamically deriving it through lookups:

```
- type: "Owns::Owned by"
  parent:
    name: "Supply Chain Orchestrator"
    sysId: "7d62102b83895210b30766d0deaad398"
    tableName: "cmdb_ci_business_app"
```

Once applied, Guardrails automatically updates ServiceNow and continuously manages these relationship mappings, ensuring that business context—such as ownership of cloud resources—is always accurate. This automatic management is critical for tracking ownership, financial governance, cost allocation, and operational oversight.


<div style={{width:"100%", textAlign:"center"}}>
  <img alt="ServiceNow relationship dependency relationship graph" src="/images/blog/2024-10-servicenow-cmdb-relationships/servicenow-relationship-graph-with-business-app.png" />
</div>

## See it in action

Watch this demo to see how Turbot Guardrails manages cloud resource relationships in ServiceNow CMDB, automating discovery, syncing, and maintaining relationships in real time:

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/_qyUJu-7YZI"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="ServiceNow CMDB CI relationship sync: faster, more complete"
>
</iframe>
</div>

## Ready your ServiceNow CMDB with automated resource relationships

Turbot Guardrails provides ultimate flexibility in managing cloud CMDB CI relationships within ServiceNow. With real-time updates, customizable relationship policies, and continuous syncing, your CMDB will always have the most accurate and comprehensive view of your cloud resources—without manual intervention.

[Get started](https://turbot.com/start) with a 14-day free trial of Turbot Guardrails to experience [automated discovery](https://turbot.com/guardrails/docs/integrations/servicenow) and sync your cloud resource relationships into your ServiceNow instance. Enjoy real-time, comprehensive visibility starting at just $0.05 per resource per month.


It's never been easier to operationalize and optimize Guardrails. During [Launch Week 5](https://turbot.com/launch-week/05) we [introduced the Guardrails Hub](https://turbot.com/guardrails/blog/2024/07/guardrails-hub) with over [140 policy packs](https://hub.guardrails.turbot.com/policy-packs). Over the last few months we have simplified and reorganized our [documentation](https://turbot.com/guardrails/docs) and published it into a public [GitHub repo](https://github.com/turbot/guardrails-docs). We hope this encourages the community to improve and expand the documentation.

Today, we announce the release of 33 new step-by-step guides for Guardrails admins, focused on simplifying the installation, configuration, and monitoring of your Guardrails environment.

## New and improved Guardrails Guides

Just as [Guardrails Policy Packs](https://hub.guardrails.turbot.com/policy-packs) simplified policy configurations for compliance and security, Guides simplify the day-to-day operations of managing Guardrails to govern your cloud environments. The Guides provide detailed steps, actionable instructions, and real-world examples to help you succeed in your cloud governance journey. Each Guide walks you through a single task (e.g. [importing an AWS account](https://turbot.com/guardrails/docs/getting-started/getting-started-aws/connect-an-account), [installing a mod](https://turbot.com/guardrails/docs/guides/configuring-guardrails/install-mod), or [troubleshooting event handlers](https://turbot.com/guardrails/docs/guides/troubleshooting/troubleshooting-event-handlers-guardrails-console)).

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guides listed in the Guardrails Docs" src="/images/blog/2024-10-guardrails-guides/docs_guides.png" />
</div>

## Check out the new Guides

### Getting started guides
- [Prepare an account for import into Guardrails](https://turbot.com/guardrails/docs/getting-started/getting-started-aws/prepare-account)
- [Import an AWS account](https://turbot.com/guardrails/docs/getting-started/getting-started-aws/connect-an-account)
- [Monitor AWS resource activities](https://turbot.com/guardrails/docs/getting-started/getting-started-aws/observe-aws-activity)
- [Attach a Policy Pack](https://turbot.com/guardrails/docs/getting-started/getting-started-aws/enable-policy-pack)
- [Account wide governance controls](https://turbot.com/guardrails/docs/getting-started/getting-started-aws/review-account-wide)
- [Create a policy exception](https://turbot.com/guardrails/docs/getting-started/getting-started-aws/create-static-exception)
- [Create a calculated policy](https://turbot.com/guardrails/docs/getting-started/getting-started-aws/create-calculated-exception)
- [Setup Guardrails notifications](https://turbot.com/guardrails/docs/getting-started/getting-started-aws/send-alert-to-email)
- [Enable Quick Actions](https://turbot.com/guardrails/docs/getting-started/getting-started-aws/apply-quick-action)
- [Enable an automated enforcement](https://turbot.com/guardrails/docs/getting-started/getting-started-aws/enable-enforcement)

### Guides for Managing Guardrails
- [Install a mod](https://turbot.com/guardrails/docs/guides/configuring-guardrails/install-mod)
- [Update a mod](https://turbot.com/guardrails/docs/guides/configuring-guardrails/update-mod)
- [Auto mod updates](https://turbot.com/guardrails/docs/guides/configuring-guardrails/auto-mod-update)
- [Uninstall a mod](https://turbot.com/guardrails/docs/guides/configuring-guardrails/uninstall-mod)

### Guides for installing and updating the hosted version of Guardrails
- [Installing TEF into a Guardrails-managed VPC.](https://turbot.com/guardrails/docs/guides/hosting-guardrails/installation/install-ted)
- [Installing TEF into an existing VPC.](https://turbot.com/guardrails/docs/guides/hosting-guardrails/installation/install-tef/install-tef-into-existing-vpc)
- [Installing Turbot Guardrails Enterprise Database (TED)](https://turbot.com/guardrails/docs/guides/hosting-guardrails/installation/install-ted)
- [Installing Turbot Guardrails Enterprise (TE)](https://turbot.com/guardrails/docs/guides/hosting-guardrails/installation/install-te)
- [Create workspace](https://turbot.com/guardrails/docs/guides/hosting-guardrails/installation/workspace-manager)
- [First time workspace login](https://turbot.com/guardrails/docs/guides/hosting-guardrails/installation/workspace-login)
- [Delete TE Version](https://turbot.com/guardrails/docs/guides/hosting-guardrails/installation/delete-te)
- [Update the TEF stack](https://turbot.com/guardrails/docs/guides/hosting-guardrails/updating-stacks/update-tef)
- [Update the TED stack](https://turbot.com/guardrails/docs/guides/hosting-guardrails/updating-stacks/update-ted)
- [Update a workspace version](https://turbot.com/guardrails/docs/guides/hosting-guardrails/updating-stacks/update-workspace)

### Guides for monitoring the hosted version of Guardrails
- [Database instance not in available state](https://turbot.com/guardrails/docs/guides/hosting-guardrails/troubleshooting/database-instance-not-in-available-state)
- [Enable verbose logging](https://turbot.com/guardrails/docs/guides/hosting-guardrails/troubleshooting/enable-verbose-logging)
- [Service catalog out of sync](https://turbot.com/guardrails/docs/guides/hosting-guardrails/troubleshooting/service-catalog-out-of-sync)
- [Troubleshooting peer mod dependency error](https://turbot.com/guardrails/docs/guides/hosting-guardrails/troubleshooting/peer-mod-dependency-error)
- [Diagnose control errors](https://turbot.com/guardrails/docs/guides/hosting-guardrails/monitoring/diagnose-control-error)
- [Investigate event floods](https://turbot.com/guardrails/docs/guides/hosting-guardrails/monitoring/investigate-event-flood)
- [Perform a workspace health check](https://turbot.com/guardrails/docs/guides/hosting-guardrails/monitoring/workspace-health-check)
- [Setup database critical alarms](https://turbot.com/guardrails/docs/guides/hosting-guardrails/installation/post-installation/create-te-critical-alarms)
- [Setup workspace critical alarms](https://turbot.com/guardrails/docs/guides/hosting-guardrails/installation/post-installation/create-ted-critical-alarms)

This is just the initial batch of the latest guides. We will continue to grow the collection with contributions from the community and from our Customer Success team.

## How to contribute to the docs

For those who want to contribute, all Guardrails docs are publicly available on [GitHub](https://github.com/turbot/guardrails-docs). We encourage contributions to help expand and improve these resources and make Guardrails more effective for teams worldwide.

## See it in action
Watch our Customer Success team discuss how these new Guides will help you get the most out of Turbot Guardrails, whether you’re configuring new cloud environments, updating existing infrastructure, or troubleshooting issues in real-time.

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/nI6zkI1UtbE"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Making Guardrails easier: 37 new step-by-step Guides"
>
</iframe>
</div>

## Control your cloud with Guardrails
Ready to streamline your Turbot Guardrails operations? Start exploring our growing library of Guides today and take full control of your Guardrails environment.

If you are new to Guardrails, try it now with a [14-day free trial](https://turbot.com/start) and see how easy it is to keep your cloud environment automatically governed at all times.



We continually strive to optimize the performance and cost efficiency of Guardrails, ensuring our customers take advantage of smarter, more innovative solutions to reduce these costs without sacrificing performance. Starting with the release of [Turbot Guardrails Enterprise v5.45.0](https://turbot.com/guardrails/changelog/te-v5-45-0), we’ve focused on improving runtime efficiency by migrating to ARM64 and optimizing Redis memory usage. In following releases we also introduced new improvements from upgraded Node.js in Lambda functions, a new Activity Retention policy to better handle record versions in the database, and a guide to resize the PostgreSQL database without downtime. These changes have resulted in significant cost savings, all while maintaining the high standards of performance and reliability that our customers depend on.

## ARM64: Reducing ECS and Lambda costs by 30%
A major part of our strategy has been the transition from traditional x86_64 architecture to ARM64 for AWS ECS and Lambda workloads. ARM instances offer the same performance as x86_64 but at a much lower cost, making them an ideal solution for reducing operational expenses without sacrificing efficiency.

By migrating ECS and Lambda workloads to ARM64, we’ve realized **30% savings in infrastructure costs**. This shift allows us to continue offering the same high level of service while driving down the cost of running Guardrails. While the transition is ongoing, we’re already seeing significant benefits, and similar savings are projected in the months ahead.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="x86 to ARM64 improvements" src="/images/blog/2024-10-guardrails-runtime-cost-reductions/awsx86-v-arm64.png" />
  <span style={{fontSize:"10px", padding:"4px 4px 0px 0px", fontStyle: "italic" }}>Cost savings transitioning to ARM64</span>
</div>

## Cache optimization: Reducing Redis memory usage by 50%
Another area of focus has been improving how we use Redis as part of our AWS Lambda processing. Previously, high message backlogs during processing caused Redis memory usage to spike due to the creation of numerous keys—such as those for processes, locks, and logs. When Redis reached its memory limit, it began deleting these keys randomly, leading to failures and inefficiencies.

To solve this, we restructured the key creation process. Now, during polling, only a minimal lock key is created with a 24-hour time-to-live (TTL), and full process keys are generated only when a message is actively handled by a worker node.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Redis memory usage after optimizations" src="/images/blog/2024-10-guardrails-runtime-cost-reductions/memory-usage-after-optimization.png" />
  <span style={{fontSize:"10px", padding:"4px 4px 0px 0px", fontStyle: "italic" }}>Memory usage after optimizations</span>
</div>

This change has dramatically reduced memory usage, allowing us to **cut our Amazon ElastiCache Redis cluster size by half and achieve 50% savings on our runtime costs**. By optimizing how and when keys are created, we’ve not only reduced costs but also made the system more efficient and reliable under high load.

## Activity log retention: New log lifecycle policies to manage activity records
New `Turbot > Workspace > Retention > Activity Retention` and `Turbot > Workspace > Retention > Activity Purge Limit` policies were introduced in the [@turbot/turbot-5.46.0](https://turbot.com/guardrails/changelog/turbot-v5-46-0) mod. These policies enable administrators to configure how long activity records such as actions, events, and notifications are retained before being purged from the database. With Smart Retention logic built in, important record versions are preserved while outdated records are automatically cleaned up, preventing excessive table growth and reducing storage overhead. 

By default when you update the mod, the value is set to 'None' to ensure your data isn't deleted. You can adjust this value based on your needs, and activity will be deleted accordingly. We recommend setting it to 90 days. Customers who have implemented this policy already have seen significant improvements, with **an average 60% reduction in activity records and a 40% decrease in database storage size**. And with millions of unnecessary records gone, queries run faster.

## PostgreSQL optimization: Reclaiming space with zero downtime
Turbot Enterprise Administrators can resize PostgreSQL databases and reclaim unused storage with minimal manual intervention and downtime. This new [guide](https://turbot.com/guardrails/docs/guides/hosting-guardrails/disaster-recovery/database-upgrade-storage-optimization) outlines the process from setting up logical replication to sync the original database with a new instance, followed by a pg_dump and pg_restore of all schemas. Once both databases are synchronized, the new instance seamlessly takes over.

While the entire process can take hours depending on the size of the environment, the bulk of the operation runs in the background, reducing the impact on normal operations. The end result is a more efficient database. Some customers have already **reclaimed up to 50% of unused storage with no downtime**.

## Node.js upgrade: Enhancing Lambda performance
We’ve upgraded the Node.js runtime in our AWS Lambda functions from Node.js v18 to v20 to improve performance and efficiency. This upgrade provides better memory management, faster startup times, and more responsive functions. All of this is handled behind the scenes for administrators as part of the latest TE version, delivering the benefits of quicker execution times for Guardrails controls and more efficient handling of workloads.

## PostgreSQL 16 support: Unlocking new database performance gains
Turbot Guardrails Enterprise now supports PostgreSQL 16. This latest version of Postgres includes significant performance enhancements:
* 40% faster query execution through improved parallelism
* 20% more efficient indexing with B-tree deduplication
* 30% better write performance during large data syncs

Guardrails Enterprise customers can update their TED stack to PostgreSQL 16 to take advantage of these latest performance improvements.

## A collaborative effort for greater impact
The success of these optimizations—whether it's the transition to ARM64, Redis memory improvements, PostgreSQL upgrades, or the introduction of the Activity Retention Policy—is the result of close collaboration between Turbot’s product, operations, and customer success teams. By identifying areas where performance and cost can be improved, we ensure that both our SaaS infrastructure and self-hosted Guardrails environments benefit from these changes.

For our Turbot Guardrails Enterprise customers, these enhancements mean not only cost savings but also improved performance and operational efficiency. Whether it's running faster queries, reclaiming storage, or reducing the overhead of managing millions of records, we continue to focus on delivering a smarter, more cost-effective cloud governance solution.

We encourage all Guardrails Enterprise customers to take advantage of these optimizations. Reach out to your Customer Success team lead to help you get started. 



Turbot is participating in [Hacktoberfest](https://hacktoberfest.com/) this year!

## Who is Turbot?

Turbot empowers Cloud and DevOps teams to conquer cloud complexity with:
- [Steampipe](https://steampipe.io) - select * from cloud;
- [Powerpipe](https://powerpipe.io) - Dashboards for DevOps.
- [Flowpipe](https://flowpipe.io) - Workflow for DevOps.
- [Pipes](https://turbot.com/pipes) - Cloud intelligence and security.
- [Guardrails](https://turbot.com/guardrails) - Automated cloud governance.

We manage over 200 public open source repositories where we maintain open source samples and developer tools for our [Guardrails](https://github.com/search?q=topic%3Aturbot-guardrails+org%3Aturbot&type=Repositories) and [Pipes](https://github.com/search?q=topic%3Aturbot-pipes+org%3Aturbot&type=Repositories) products. As well as our open source projects [Steampipe](https://github.com/search?q=topic%3Asteampipe+org%3Aturbot&type=repositories), [Powerpipe](https://github.com/search?q=topic%3Apowerpipe+org%3Aturbot&type=repositories), and [Flowpipe](https://github.com/search?q=topic%3Aflowpipe+org%3Aturbot&type=repositories) and their related plugins and mod repositories.

## Why contribute? 

Be part of a growing community. Last year we merged over 108 commits from 57 community members. We have since grown to over 2,600 community members who are passionate engineers that love our open source projects and products as much as we do:

<blockquote class="docs">
"I recently started using Steampipe and it has made my life about 30x better." - @bearjaws
</blockquote>

<blockquote class="docs">
"Steampipe is without a doubt one of the best open source tools for security compliance, operations and API orchestration." - @ashishsecdev
</blockquote>

<blockquote class="docs">
"The Steampipe is the best tool I have seen since Terraform." - @forty
</blockquote>

## What is Hacktoberfest?

DigitalOcean is in its 11th year organizing the overall [Hacktoberfest](https://hacktoberfest.com) event to celebrate the open source community. Hacktoberfest invites you to submit pull requests to open source projects. Those who submit multiple PRs will qualify for digital badges from DigitalOcean.

Turbot has marked all of our applicable public GitHub repos with the [hacktoberfest](https://github.com/search?q=org%3Aturbot+topic%3Ahacktoberfest) topic. If we accept one or more of your PRs, they'll be labeled `hacktoberfest-accepted` and will count towards the DigitalOcean Hacktoberfest campaign.

## Any special swag from Turbot?

Yes! In addition to the DigitalOcean event, Turbot is adding our own swag for contributions to our projects. We will offer unique Turbot stickers and t-shirts for no-code, low-code and high-code contributions. 

![](/images/blog/2024-swag-preview.jpg)

Here are examples of contributions to earn Turbot swag. Each swag item is limited to one per person.

**No-code** for a Turbot Hacktoberfest sticker:

* Publish a public facing blog post featuring Steampipe, Powerpipe, Flowpipe, Turbot Pipes, or Turbot Guardrails ([examples](https://steampipe.io/blog/news-and-reviews)).

* Record a publicly-accessible presentation, podcast, or talk about Steampipe, Powerpipe, Flowpipe Turbot Pipes, or Turbot Guardrails.

* Submit an update to improve our Steampipe, Powerpipe, Flowpipe Turbot Pipes, or Turbot Guardrails [docs](https://github.com/turbot?q=docs).

* File a bug issue, with a detailed explanation of the testing you performed, to one of our public [Turbot repos](https://github.com/search?q=org%3Aturbot+topic%3Ahacktoberfest).

**Low-code** for a Turbot Hacktoberfest sticker:

* Raise a pull request for a bug fix or code improvement.

* Add a new column to an existing Steampipe table.

* Add a [new control](https://powerpipe.io/docs/build/writing-controls) to an existing Powerpipe benchmark.

* Add a [new pipeline](https://flowpipe.io/docs/build/write-pipelines) to an existing Flowpipe mod.

* Add an example to one of our [samples](https://github.com/turbot?q=docs) repos.

**High-code** for a Turbot Hacktoberfest t-shirt:

* Add a [new Steampipe table](https://steampipe.io/docs/develop/writing-your-first-table) to an existing [plugin](https://hub.steampipe.io).

* Add a new [Powerpipe dashboard](https://powerpipe.io/docs/mods/writing-dashboards) to an existing [Insights mod](https://hub.powerpipe.io/?q=insights).

* Add a new Powerpipe benchmark to an existing [compliance mod](https://hub.powerpipe.io/?objectives=compliance,security).

* Develop a [new Steampipe plugin](https://steampipe.io/docs/develop/writing-plugins) and publish it to [Steampipe Hub - Plugins](https://hub.steampipe.io).

* Develop a [new Powerpipe mod](https://powerpipe.io/docs/mods/overview) and publish it to [Powerpipe Hub - Mods](https://hub.powerpipe.io).

* Develop a [new Flowpipe mod](https://flowpipe.io/docs/build) and publish it to [Flowpipe Hub - Mods](https://hub.flowpipe.io).

* Participate in a company case study published on our [Turbot](https://turbot.com/blog?category=Case%20Study) blog.

Inspire us! We'll also award prizes for innovative contributions that help move the community forward.

## When do you get your Turbot swag?

In the week following the event we will share links to redeem your swag. Most items will arrive in November, but depending on your location and t-shirt order time may take longer.

## Ready to contribute?

We can’t wait to see what incredible contributions you’ll make! If you have any questions, or need assistance, join us in our [Slack community](https://turbot.com/community/join) **#contributors** channel.



Our announcements this week span the breadth of Turbot offerings: 

- Guardrails

   - New ServiceNow integration
   
   - Remediation for Azure and GCP CIS benchmarks
   
   - The new Guardrails Hub with Policy Packs

- Flowpipe
 
  - Detect-and-correct mods to enforce standard AWS/Azure tags and GCP labels

  - Thrifty mods to cut costs on Azure and GCP

  - Interactive workflows in your terminal

- Pipes

  - Import and sync an AWS organization

  - Import and sync an Azure tenant

  - Import and sync a GCP organization

## Monday July 22: Guardrails + ServiceNow; CIS remediation for Azure, GCP

You could already populate the ServiceNow CMDB with Guardrails' discovery of AWS, Azure, and GCP resources. In [Automated cloud discovery into ServiceNow import sets](https://turbot.com/guardrails/blog/2024/07/servicenow-import-set-cloud-discovery) we provide another approach for enterprises whose preferred integration pattern is import sets.

And you could already get to green with real-time remediation for [AWS CIS v3.0.0](https://turbot.com/guardrails/blog/2024/05/instant-aws-cis-v3-compliance), now you also can do that for [Azure CIS v2.0.0](https://turbot.com/guardrails/blog/2024/07/instant-azure-cis-v2-compliance) and [GCP CIS v2.0.0](https://turbot.com/guardrails/blog/2024/07/instant-gcp-cis-v2-compliance) too. 

## Tuesday July 23: The Guardrails Hub and Policy Packs

The new [Guardrails Hub](https://turbot.com/guardrails/blog/2024/07/guardrails-hub) gathers detailed reference data for thousands of mods, resources, policies, and controls across all supported cloud service providers: [AWS](https://hub.guardrails.turbot.com/mods/aws/mods), [Azure](https://hub.guardrails.turbot.com/mods/azure/mods), [GCP](https://hub.guardrails.turbot.com/mods/gcp/mods), [ServiceNow](https://hub.guardrails.turbot.com/mods/servicenow/mods), [Kubernetes](https://hub.guardrails.turbot.com/mods/kubernetes/mods) and [Turbot](https://hub.guardrails.turbot.com/mods/turbot/mods).

Policies are the primary configuration language of Guardrails. [Policy Packs](https://turbot.com/guardrails/blog/2024/07/guardrails-policy-packs) are the heart of the Guardrails Hub. They are targeted collections of these policies that work together to achieve specific control objectives.

## Wednesday July 24: New Flowpipe mods for tagging and FinOps, interaction in the terminal

It's good to detect when resources aren't tagged according to your standard, and even better to fix those errors — interactively, or automatically. Enter three new Flowpipe detect-and-correct mods for [AWS](https://flowpipe.io/blog/tag-workflows-for-aws), [Azure](https://flowpipe.io/blog/tag-workflows-for-azure), and [GCP](https://flowpipe.io/blog/label-workflows-for-gcp). Define and enforce tagging standards for all the clouds.

You could already detect and correct costly configuration errors with [29 FinOps workflows for AWS](https://flowpipe.io/blog/finops-workflows-for-aws). Now double down with [24 for Azure ](https://flowpipe.io/blog/finops-workflows-for-azure) and [22 for GCP](https://flowpipe.io/blog/finops-workflows-for-gcp).

The demos for these new detect-and-correct mods showcase Flowpipe's new ability to run interactive workflows [in your terminal](https://flowpipe.io/blog/interactive-workflows-pipelines), as well as in Slack, MSTeams, or email. 

## Thursday July 25: Pipes import and sync for AWS / Azure / GCP

You asked, we delivered. Now it's easy to import [all the accounts](https://turbot.com/pipes/blog/2024/07/aws-org-sync) from an AWS organization, [all the subscriptions](https://turbot.com/pipes/blog/2024/07/azure-tenant-sync) from an Azure tenant, and [all the projects](https://turbot.com/pipes/blog/2024/07/gcp-org-sync) from a GCP organization. The structures — of OUs, management groups, and folders — map to  [connection folders](https://turbot.com/pipes/docs/connections#creating-connections--folders) so Pipes administrators can delegate permissions accordingly. Pipes also creates aggregators for the mapped connection folders and, crucially, syncs changes from the clouds.

## The B-sides

These were the Launch Week 5 hits, but plenty of other good things happened during this cycle. Check the [B-sides](https://turbot.com/blog/2024/07/launch-week-5-b-sides) for details!

## And that's a wrap!

The announcements this week are individually exciting, but the Turbot story is bigger than the sum of its parts. Our products work together to deliver cloud security and intelligence with incomparable breadth, depth, power, speed, and versatility. And there's more to come. Enjoy the current batch of new products and features, [let us know](https://turbot.com/community/join) how it goes and stay tuned for our next Launch Week in a few months!


## Watch the Wrap Party

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/vVHz5ECM930"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Launch Week 5 is a wrap!"
>
</iframe>
</div>



As [Launch Week 5](https://turbot.com/launch-week/05) draws to a close, we wanted to take a moment to highlight some of the exciting updates and announcements that slipped under the radar this week across our Turbot products and open-source projects.

## Pipes and Guardrails docs now available for contributions on GitHub
Similar to [Steampipe](https://github.com/turbot/steampipe-docs), [Powerpipe](https://github.com/turbot/powerpipe-docs), and [Flowpipe](https://github.com/turbot/flowpipe-docs) docs, Turbot Pipes and Turbot Guardrails docs are now publicly available on GitHub, and we are accepting contributions! Moving our docs to a public repository invites our community to participate in improving them. Also, our [Turbot Guardrails Samples](https://github.com/turbot/guardrails-samples) repo now includes a section for contributing to [Policy Packs](https://turbot.com/guardrails/blog/2024/07/guardrails-policy-packs) that can be published to the [Turbot Guardrails Hub](https://hub.guardrails.turbot.com).

Check out the docs and start contributing today:

* [Turbot Pipes Docs](https://github.com/turbot/pipes-docs)
* [Turbot Guardrails Docs](https://github.com/turbot/guardrails-docs)
* [Turbot Guardrails Samples](https://github.com/turbot/guardrails-samples)


## Guardrails: 75+ controls added
Guardrails now supports several new resource types, 100s of new policies, and over 75 new controls for security, compliance, and operational guardrails. Many of the new Guardrails features were in support of auto-remediation of CIS benchmarks for [Azure](https://turbot.com/guardrails/blog/2024/07/instant-azure-cis-v2-compliance) and [GCP](https://turbot.com/guardrails/blog/2024/07/instant-gcp-cis-v2-compliance), [ServiceNow import set](https://turbot.com/guardrails/blog/2024/07/servicenow-import-set-cloud-discovery) support, and ongoing customer feedback. They include:

* Azure Security Center auto-provisioning
* Azure Network Watcher Flow Log Retention policies
* Azure App Service Web Apps client certificate Mode
* GCP DNS Managed Zone DNSSEC configurations
* and 70+ more

## Guardrails: Static CIDR ranges
You can now configure Guardrails Mod Lambda functions to operate with static CIDR ranges. That makes it easier to whitelist Guardrails in your organization cloud ingress policies with a consistent set of IP ranges from Guardrails to your cloud providers. 

This feature is an optional setting for Turbot Guardrails Enterprise and is already enabled in Turbot Guardrails Cloud. Reach out to your Turbot Customer Success lead for more information.

## Guardrails: Increases performance at lower runtime costs

For Turbot Guardrails Enterprise customers, the latest [Turbot Guardrails Enterprise (TE) v5.45.0 release](https://turbot.com/guardrails/changelog/te-v5-45-0) massively Redis memory consumption that smooths cache operation. Also enabled in this release: conversion from x86 to ARM64 architecture for all AWS Lambda functions. This has increased performance and reduced runtime costs by 30%.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="x86 to ARM64 improvements" src="/images/blog/2024-07-launch-week-5-b-sides/awsx86-v-arm64.png" />
  <span style={{fontSize:"10px", padding:"4px 4px 0px 0px", fontStyle: "italic" }}>
    <a target="_blank" href="https://www.linkedin.com/pulse/aws-lambda-x86-vs-arm64-graviton-processors-raghunath-erumal-1yicc">Image Credit</a>
  </span>
</div>

## Pipes: Connection folders

Formerly, connections created at Tenant, Organization or workspace level were displayed and managed as flat lists. That was unwieldy both for administrators who create connections for use in workspaces, and workspace owners who create connections in their own workspaces. Now, at all levels, connections can be bundled into [connection folders](https://turbot.com/pipes/docs/connections#creating-connections--folders) that are much easier to display and manage.

<div>
  <img alt="Pipes Connection Sharing Permissions" src="/images/blog/2024-07-launch-week-5-b-sides/pipes-connection-sharing-permissions.png" />
</div>

You can create connection folders manually, or automatically with the new [AWS](https://turbot.com/pipes/docs/integrations/aws), [Azure](https://turbot.com/pipes/docs/integrations/azure) and [GCP](https://turbot.com/pipes/docs/integrations/gcp) integrations. These integrations import accounts, subscriptions, and project as Pipes connections organized into Pipes connection folders that mirror the administrative structure of the imported organizations (AWS, GCP) or tenants (Azure), so that delegation of permissions in Pipes matches delegation in the origin systems.

When a folder contains two connections of the same type (`aws1`, `aws2`), Pipes automatically creates an [aggregator](https://turbot.com/pipes/docs/aggregators) (`all_aws`) that bundles them for all-at-once query. Although connections and connection folders can exist at any level, aggregators are — as before — purely workspace-level constructs.

## Pipes: Workspace-level connections

Formerly, unprivileged workspace owners — including those with Developer accounts — needed to ask administrators to create connections for them. Now a workspace owner is empowered to create non-shareable connections scoped to the workspace. Less friction, faster development.

## Pipes: Enterprise-scale Datatank performance

The new integrations([AWS](https://turbot.com/pipes/docs/integrations/aws), [Azure](https://turbot.com/pipes/docs/integrations/azure), [GCP](https://turbot.com/pipes/docs/integrations/gcp)) make it fast and easy to create large numbers of connections that may want to leverage [Datatank](https://turbot.com/pipes/blog/2023/10/datatank-launch) to speed up queries. Datatank's new "fleet" mode will handle the larger workloads arising from the new integrations.


## Pipes: New Connections - CrowdStrike and Semgrep

Pipes now supports new connections with [CrowdStrike](https://hub.steampipe.io/plugins/turbot/crowdstrike) and [Semgrep](https://hub.steampipe.io/plugins/gabrielsoltz/semgrep) to centrally query, report and collaborate on:

* Semgrep deployments, findings, and projects. Special thanks to community member [@gabrielsoltz](https://github.com/gabrielsoltz) for publishing the community-built Semgrep Steampipe plugin.
* CrowdStrike detections, hosts, intel_actors, users, and more.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="CrowdStrike Connection for Turbot Pipes" src="/images/blog/2024-07-launch-week-5-b-sides/crowdstrike-connection.png" />
</div>


## Pipes: Quality of life improvements

Changes to make the user experience friendlier include:

* Set search path prefix in your workspace settings to override default aggregators with custom ones.
* Refreshed Azure and AzureAD (Entra) connection instructions.
* Prefix of `all_` in schema names now reserved to avoid conflicts.
* Process logs now have a copy button to easily extract logs from UI.
* Improved breadcrumb navigation among Pipes tenants, organizations and workspaces.

<div style={{width:"80%", textAlign:"center"}}>
  <img alt="Turbot Pipes Navigation" src="/images/blog/2024-07-launch-week-5-b-sides/pipes-navigation.png" />
</div>

## Pipes & Powerpipe: 8 new security benchmarks supported
Powerpipe has added eight new security benchmarks to our [AWS](https://hub.powerpipe.io/mods/turbot/aws_compliance), [Azure](https://hub.powerpipe.io/mods/turbot/azure_compliance), and [GCP](https://hub.powerpipe.io/mods/turbot/gcp_compliance) compliance mods. These new benchmarks are available in the [Powerpipe Hub](https://hub.powerpipe.io) and in [Turbot Pipes](https://pipes.turbot.com) to assess your security posture and share status with your team.

* AWS: 
  * [RBI ITF NBFC](https://hub.powerpipe.io/mods/turbot/aws_compliance/controls/benchmark.rbi_itf_nbfc)
* Azure:
  * [FedRAMP High](https://hub.powerpipe.io/mods/turbot/azure_compliance/controls/benchmark.fedramp_high)
  * [NIST 800-171 Rev 2](https://hub.powerpipe.io/mods/turbot/azure_compliance/controls/benchmark.nist_sp_800_171_rev_2)
  * [RBI ITF NBFC](https://hub.powerpipe.io/mods/turbot/azure_compliance/controls/benchmark.rbi_itf_nbfc_v2017)
* GCP:
  * [NIST Cybersecurity Framework (CSF) v1.0](https://hub.powerpipe.io/mods/turbot/gcp_compliance/controls/benchmark.nist_csf_v10)
  * [NIST 800-53 Revision 5](https://hub.powerpipe.io/mods/turbot/gcp_compliance/controls/benchmark.nist_800_53_rev_5)
  * [HIPAA](https://hub.powerpipe.io/mods/turbot/gcp_compliance/controls/benchmark.hipaa)
  * [PCI DSS v3.2.1](https://hub.powerpipe.io/mods/turbot/gcp_compliance/controls/benchmark.pci_dss_v321) 

## Pipes & Powerpipe: Automatically manage duplicate column names in snapshots

The latest version of [Powerpipe CLI](https://powerpipe.io/changelog/powerpipe-cli-v0-4-0), v0.4.0, now automatically handles duplicate column names in Powerpipe and Pipes snapshots. The rows property in the JSON and snapshot output will now have unique column names for duplicate column names, while the columns property will have the original column name as original_name. For example, when running a query with the same column names:

```sql
powerpipe query run "select arn as title, account_id as title, title as title from aws_account" --output pps
```

Here is the updated JSON output behind the scenes:

```json
{
 "columns": [
  {
   "name": "title",
   "data_type": "text"
  },
  {
   "name": "title_t5zj1",
   "data_type": "text",
   "original_name": "title"
  },
  {
   "name": "title_t5zj2",
   "data_type": "text",
   "original_name": "title"
  }
 ],
 "rows": [
  {
   "title": "arn:aws:::882789663776",
   "title_t5zj1": "882789663776",
   "title_t5zj2": "882789663776"
  }
 ],
 ...
}
```

## Flowpipe: Use files, branches, or tags for Flowpipe mod dependencies

[Flowpipe CLI v0.5.0](https://flowpipe.io/changelog/flowpipe-cli-v0-5-0) now supports installing mods from a branch or from the local file system, similar to our [Powerpipe CLI v0.3.0](https://powerpipe.io/changelog/powerpipe-cli-v0-3-0) release. This update makes it easier to manage and refactor mods, enabling you to extract shared components into dependent mods for rapid development and testing. 

You've also now got a lot more flexibility when installing or updating mods, using the `--pull` argument to specify various [update strategies](https://flowpipe.io/docs/reference/cli/mod#update-strategy).

If you haven't tried writing your own [Flowpipe mods](https://flowpipe.io/docs/build), now's a great time to get started. Build something useful and [tell us about it](https://turbot.com/community/join)!

## Flowpipe: 125 pipelines added
Over the past few months 125 pipelines have been added to the [AWS](https://hub.flowpipe.io/mods/turbot/aws/pipelines), [Azure](https://hub.flowpipe.io/mods/turbot/azure/pipelines) and [GCP](https://hub.flowpipe.io/mods/turbot/gcp/pipelines) mods. Many were in support of the new detect and correct [Tagging](https://hub.flowpipe.io/mods/turbot/aws_tags) and [Thrifty](https://hub.flowpipe.io/mods/turbot/aws_thrifty) mods, as well as pipelines added based on community feedback.  Examples of a few of the pipelines added:

* Detach AWS EBS Volumes
* Release AWS EIPs
* Update AWS Route53 records
* Delete Azure Kubernetes Cluster
* Remove GCP IAM Policy Binding from KMS Key
* and 120 more

## Flip over to A-sides for the Wrap Up
Thank you for joining us for another exciting Launch Week! Check out the week's daily announcements summary in our [Launch Week 5 Wrap Up](https://turbot.com/blog/2024/07/launch-week-5-wrap) post. Stay connected with us in our [Slack community](https://turbot.com/community/join) for our next Launch Week in a few months! 



You asked, we delivered. Now you can import your entire AWS organization, with just a few clicks, as a set of Pipes connections. This new integration dramatically simplifies connecting AWS resources to Pipes, and enables you to query and analyze your cloud infrastructure more thoroughly than ever before. 

## Get ready to import

Here's the AWS structure we'll import.

<div style={{width:"70%"}}>
  <img alt="aws org structure" src="/images/blog/2024-07-aws-org-sync/aws-org-structure.png" />
</div>


To create a new [AWS Integration](https://turbot.com/pipes/docs/integrations/aws), select `AWS` from the integration options.

<div>
  <img alt="select aws integration" src="/images/blog/2024-07-aws-org-sync/select-aws-integration.png" />
</div>

You can accept the default handle for the integration, `AWS`, or choose your own.

In `Discovery Settings`, you set up the authentication Pipes uses to discover your AWS OUs and accounts. You can use a cross-account role (recommended) or an access key pair; either way, test the connection to verify it works.

<div>
  <img alt="discovery settings" src="/images/blog/2024-07-aws-org-sync/aws-discovery-settings.png" />
</div>

In `Connection Settings`, you set up the trust relationship for discovered accounts. For this you'll need a cross-account role in each discovered account; Pipes can generate a CloudFormation StackSet to help automate the process. See [the docs](https://turbot.com/pipes/docs/integrations/aws) for details. Again, test to verify it works.

<div>
  <img alt="connection settings" src="/images/blog/2024-07-aws-org-sync/aws-connection-settings.png" />
</div>

## Set permissions

Now choose which workspaces — all, some, or none — are permitted to use the connections.

<div>
  <img alt="set permissions" src="/images/blog/2024-07-aws-org-sync/aws-set-permissions.png" />
</div>

Note the separation of concerns: Pipes administrators make connections available to workspaces, and workspace owners decide, on the workspace's `Advanced / Connections` page, which (if any) to activate for query.

## Import OUs and accounts

Now click `Create Integration` to import the discovered OUs and accounts. Pipes performs the import, and creates a tree that matches your AWS structure, so delegation of permissions works the same way in Pipes as it does in AWS.

<div>
  <img alt="imported tree" src="/images/blog/2024-07-aws-org-sync/aws-imported-tree.png" />
</div>

## Add connections to a workspace

Subject to the connections they are permitted to see, workspace owners can now use any OU that was imported into Pipes to add its corresponding set of connections to a workspace. In this example, we choose the `Pipeling Scale Testing` connection folder which contains 200 accounts. 

The workspace owner uses the `Add to schema` button to make the corresponding connections available for query.

<div>
  <img alt="add to schema" src="/images/blog/2024-07-aws-org-sync/aws-add-to-schema.png" />
</div>

When a Pipes connection folder contains two or more connections, Pipes automatically creates an aggregator for them. Here's `all_aws` which aggregates the 200 connections acquired from the `Pipeling Scale Testing` OU.

<div>
  <img alt="aws all" src="/images/blog/2024-07-aws-org-sync/aws-all.png" />
</div>

## Query the connections that were imported and added

Now, in the query pane, you can select and query any of the aggregated connections, or use the aggregator to query all of them. Here we query the aggregator.

<div>
  <img alt="query" src="/images/blog/2024-07-aws-org-sync/aws-query.png" />
</div>

And that's it! We've gone from a standing start to a queryable set of imported AWS connections in just minutes.

## Try it yourself!

This new AWS integration addresses a common challenge faced by our users with large AWS footprints. It would be time-consuming to do this by hand or with Terraform scripting. Now you can bring your whole AWS organization into Pipes quickly, easily, and automatically. And when OUs or accounts change, Pipes syncs the changes.

We're thrilled to offer this enhanced AWS support and can't wait to see how it accelerates your cloud operations and security workflows. As always, we welcome [your feedback](https://turbot.com/community/join) and we're to help as you explore this powerful new capability.


## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/923ssGdgIJ8"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Connect your AWS Organizations with Pipes"
>
</iframe>
</div>





You asked, we delivered. Now you can import your entire Azure tenant, with just a few clicks, as a set of Pipes connections. This new integration dramatically simplifies connecting Azure resources to Pipes, and enables you to query and analyze your cloud infrastructure more thoroughly than ever before. 

## Get ready to import

Here's the Azure structure we'll import.

<div>
  <img alt="azure tenant structure" src="/images/blog/2024-07-azure-tenant-sync/azure-tenant-structure.png" />
</div>


To create a new [Azure Integration](https://turbot.com/pipes/docs/integrations/azure), select `Azure` from the integration options.

<div>
  <img alt="select azure integration" src="/images/blog/2024-07-azure-tenant-sync/select-azure-integration.png" />
</div>

In `Setup Integration`, you can accept the default handle for the integration, `Azure`, or choose your own.

To authenticate, specify your environment (e.g. `Public Cloud`) and enter credentials for a role with at least `Global Reader` access. Then test to verify they work. 

<div>
  <img alt="setup integration" src="/images/blog/2024-07-azure-tenant-sync/setup-azure-integration.png" />
</div>

## Set permissions

Now choose which workspaces — all, some, or none — are permitted to use the connections.

<div>
  <img alt="set permissions" src="/images/blog/2024-07-aws-org-sync/aws-set-permissions.png" />
</div>

Note the separation of concerns: Pipes administrators make connections available to workspaces, and workspace owners decide, on the workspace's `Advanced / Connections` page, which (if any) to activate for query.

## Import management groups and subscriptions

Now click `Create Integration` to import the discovered management groups and subscriptions. Pipes performs the import, and creates a tree that matches your Azure structure, so delegation of permissions works the same way in Pipes as it does in Azure.

<div>
  <img alt="imported tree" src="/images/blog/2024-07-azure-tenant-sync/azure-imported-tree.png" />
</div>

## Add connections to a workspace

Subject to the connections they are permitted to see, workspace owners can now use any management group that was imported into Pipes to add its corresponding set of connections to a workspace. In this example, we choose the `Pipeling Scale Testing Small` connection folder which contains 60 subscriptions. 

The workspace owner uses the `Add to schema` button to make the corresponding connections available for query.

<div>
  <img alt="add to schema" src="/images/blog/2024-07-azure-tenant-sync/azure-add-to-schema.png" />
</div>

When a Pipes connection folder contains two or more connections, Pipes automatically creates an aggregator for them. Here's `all_azure` which aggregates the 60 subscriptions acquired from the `Pipeling Scale Testing` management group.

<div>
  <img alt="azure all" src="/images/blog/2024-07-azure-tenant-sync/all-azure.png" />
</div>

## Query the connections that were imported and added

Now, in the query pane, you can select and query any of the aggregated connections, or use the aggregator to query all of them. Here we query the aggregator.

<div>
  <img alt="query" src="/images/blog/2024-07-azure-tenant-sync/azure-query.png" />
</div>

And that's it! We've gone from a standing start to a queryable set of imported Azure connections in just minutes.

## Try it yourself!

This new Azure integration addresses a common challenge faced by our users with large Azure footprints. It would be time-consuming to do this by hand or with Terraform scripting. Now you can bring your whole Azure organization into Pipes quickly, easily, and automatically. And when management groups or subscriptions change, Pipes syncs the changes.

We're thrilled to offer this enhanced Azure support and can't wait to see how it accelerates your cloud operations and security workflows. As always, we welcome [your feedback](https://turbot.com/community/join) and we're to help as you explore this powerful new capability.



## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/K84Ow7wvK1s"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Connect your Azure Tenants with Pipes"
>
</iframe>
</div>




You asked, we delivered. Now you can import your entire GCP organization, with just a few clicks, as a set of Pipes connections. This new integration dramatically simplifies connecting GCP resources to Pipes, and enables you to query and analyze your cloud infrastructure more thoroughly than ever before. 

## Get ready to import

Here's the GCP structure we'll import.

<div>
  <img alt="gcp org structure" src="/images/blog/2024-07-gcp-org-sync/gcp-org-structure.png" />
</div>


To create a new [GCP Integration](https://turbot.com/pipes/docs/integrations/gcp), select `GCP` from the integration options.

<div>
  <img alt="select gcp integration" src="/images/blog/2024-07-gcp-org-sync/select-gcp-integration.png" />
</div>


In `Setup Integration`, you can accept the default handle for the integration, `GCP`, or choose your own.

To authenticate, drop the key file for a service account with `Viewer` permissions for the folder hierarchy you'll import. Then test to verify the credentials work. 

<div>
  <img alt="setup integration" src="/images/blog/2024-07-gcp-org-sync/setup-gcp-integration.png" />
</div>


## Set permissions

Now choose which workspaces — all, some, or none — are permitted to use the connections.

<div>
  <img alt="set permissions" src="/images/blog/2024-07-gcp-org-sync/gcp-set-permissions.png" />
</div>

Note the separation of concerns: Pipes administrators make connections available to workspaces, and workspace owners decide, on the workspace's `Advanced / Connections` page, which (if any) to activate for query.

## Import folders and projects

Now click `Create Integration` to import the discovered folders and projects. Pipes performs the import, and creates a tree that matches your GCP structure, so delegation of permissions works the same way in Pipes as it does in GCP.

<div>
  <img alt="imported tree" src="/images/blog/2024-07-gcp-org-sync/gcp-imported-tree.png" />
</div>

## Add connections to a workspace

Subject to the connections they are permitted to see, workspace owners can now use any folder that was imported into Pipes to add its corresponding set of connections to a workspace. In this example, we choose the `Pipeling Scale Testing Small` folder which contains 15 projects. 

The workspace owner uses the `Add to schema` button to make the corresponding connections available for query.

<div>
  <img alt="add to schema" src="/images/blog/2024-07-gcp-org-sync/gcp-add-to-schema.png" />
</div>

When a Pipes connection folder contains two or more connections, Pipes automatically creates an aggregator for them. Here's `all_gcp` which aggregates the 15 connections acquired from the `Pipeling Scale Testing Small` GCP folder.

<div>
  <img alt="gcp all" src="/images/blog/2024-07-gcp-org-sync/gcp-all.png" />
</div>

## Query the connections that were imported and added

Now, in the query pane, you can select and query any of the aggregated connections, or use the aggregator to query all of them. Here we query the aggregator.

<div>
  <img alt="query" src="/images/blog/2024-07-gcp-org-sync/gcp-query.png" />
</div>

And that's it! We've gone from a standing start to a queryable set of imported GCP connections in just minutes.

## Try it yourself!

This new GCP integration addresses a common challenge faced by our users with large GCP footprints. It would be time-consuming to do this by hand or with Terraform scripting. Now you can bring your whole GCP organization into Pipes quickly, easily, and automatically. And when folders or projects change, Pipes syncs the changes.

We're thrilled to offer this enhanced GCP support and can't wait to see how it accelerates your cloud operations and security workflows. As always, we welcome [your feedback](https://turbot.com/community/join) and we're to help as you explore this powerful new capability.


## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/mRk5iBe5MTA"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Connect your GCP Organizations with Pipes"
>
</iframe>
</div>





  Flowpipe supports workflows that require decisions based on human input. As detailed in our blog posts [Workflow for DevOps: Approvals and inputs to engage your team](https://flowpipe.io/blog/human-interaction) and [Control AWS costs with 29 FinOps workflows](https://flowpipe.io/blog/finops-workflows-for-aws), pipelines can push that decision-making to Slack, MSTeams, or email. Now, these interactions can happen directly in your terminal too.

  ## Decisions in your terminal

Here's an example: a pipeline called `check_old_branches` that asks whether to delete stale branches in a repository.

```
$ flowpipe pipeline run check_old_branches
┃ Do you want to delete branch add-tables?
┃ > Skip
┃   Delete
```

Flowpipe pauses for a decision. Let's skip this one, and proceed.

```
┃ Do you want to delete branch update-slack-community-link?
┃   Skip
┃ > Delete
```

Here we'll choose Delete.

If those are the only two branches matching a Steampipe query that finds stale branches, Flowpipe concludes with this report.

```
Skipped branch add-tables
Deleted branch update-slack-community-link
```

## How it works

To look for branches older than 30 days that also lack pull requests, the pipeline runs a [query step](https://flowpipe.io/docs/flowpipe-hcl/step/query) that joins two Steampipe tables: [github_branch](https://hub.steampipe.io/plugins/turbot/github/tables/github_branch) and [github_pull_request](https://hub.steampipe.io/plugins/turbot/github/tables/github_pull_request).

Here's the pipeline that `check_old_branches` calls to make a decision for each branch. It receives a branch name from the query step, prompts to `Skip` or `Delete`, then calls the `skip_branch` or `delete_branch` pipelines accordingly.

```hcl
pipeline "handle_one_branch" {
  param "name" {
    type        = string
    description = "The name of the branch."
  }

  step "input" "approve_deletion" {
    type     = "button"
    prompt   = "Do you want to delete branch ${param.name}?"
    notifier = notifier.default
    option "Skip" {}
    option "Delete" {}
  }

  step "pipeline" "delete_branch" {
    pipeline = pipeline.delete_branch
    args = {
      name = param.name
    }
    if = step.input.approve_deletion.value == "Delete"
  }

  step "pipeline" "skip_branch" {
    pipeline = pipeline.skip_branch
    args = {
      name = param.name
    }
    if = step.input.approve_deletion.value == "Skip"
  }
}
```

To run a pipeline like this, you'd first install the [GitHub mod for Flowpipe](https://hub.flowpipe.io/mods/turbot/github) to gain access to the [delete_branch](https://hub.flowpipe.io/mods/turbot/github/pipelines/github.pipeline.delete_branch) pipeline.

## Develop in the terminal, then interact with your team

The example uses the default [notifier](https://flowpipe.io/docs/reference/config-files/notifier) to interact in the terminal, versus alternate notifiers that you might configure for [Slack](https://flowpipe.io/docs/reference/config-files/integration/slack), [MSTeams](https://flowpipe.io/docs/reference/config-files/integration/msteams), or [email](https://flowpipe.io/docs/reference/config-files/integration/email). If you're the decision-maker, you can handle approval-based workflows directly. When other team members need to approve, you can route decisions to them using their preferred communication channels. 

And of course you can tap into two powerful ecosystems: Steampipe's [suite of API plugins](https://hub.steampipe.io), and Flowpipe's [collection of utility pipelines](https://hub.flowpipe.io).

Give this new feature a try, and [let us know](https://turbot.com/community/join) how it enhances your workflow automation!



In the world of cloud infrastructure management, consistent and accurate resource tagging is crucial. It's nice to be able to detect when tags deviate from your standards, but even better to correct those deviations automatically. Enter the [AWS Tags mod for Flowpipe](https://github.com/turbot/flowpipe-mod-aws-tags), a new tool that enables you to automate tag management using a declarative ruleset. Let's take a look.

## Add mandatory tags

Here's a Steampipe query that reveals incorrectly-tagged S3 buckets.

```sql
select name, tags from aws_s3_bucket where name ~ 'flowpipe'
+--------------------+------------------+
| name               | tags             |
+--------------------+------------------+
| flowpipe-bucket-01 | {"cc":"default"} |
| flowpipe-bucket-02 | <null>           |
+--------------------+------------------+
```

First, we'd like to apply the mandatory tag `env:dev` to all of them. To do that, we'll use the `detect_and_correct_s3_buckets_with_incorrect_tags` pipeline in the AWS Tags mod.

Per the README, the setup entails:

- Cloning the mod

- Visiting its directory and installing dependencies (`flowpipe mod install`)

- Starting Steampipe as a service (`steampipe service start`)

- Using `credential_import` to give Flowpipe access to the credentials in Steampipe `aws.spc` 

- Configuring tag rules

We'll use these definitions in `add-notify.fpvars`.

```hcl
approvers = []
base_tag_rules = {
  add = {
    env = "dev"
  }
}
```

With `approvers = []`, the mod will report tag violations to the default notifier, the terminal, but take no action. The only rule is to add our desired mandatory tag.

Now we'll run the pipeline to detect and correct S3 bucket tags, and point it at that package of variables.

```
$ flowpipe pipeline run detect_and_correct_s3_buckets_with_incorrect_tags --var-file=add-notify.fpvars
```

Flowpipe reports the changes it would make.

```
Detected flowpipe-bucket-01 (070715377127/us-east-1/arn:aws:s3:::flowpipe-bucket-01) with incorrect tags. Tags that will be added or updated: env=dev.

Detected flowpipe-bucket-02 (070715377127/us-east-1/arn:aws:s3:::flowpipe-bucket-02) with incorrect tags. Tags that will be added or updated: env=dev.
```

To make the changes, we'll use `add-wizard.fpvars`, with the same tag rules but with `approvers = ["default"]`. That activates an [input step](https://flowpipe.io/docs/flowpipe-hcl/step/input) that will (in this case) prompt to `Skip` or `Apply` the proposed change.

Flowpipe pauses on the first detected violation.

```
$ flowpipe pipeline run detect_and_correct_s3_buckets_with_incorrect_tags --var-file=add-wizard.fpvars
┃ Detected flowpipe-bucket-02 (070715377127/us-east-1/arn:aws:s3:::flowpipe-bucket-02) with incorrect tags. Tags that will be added or updated: env=dev.
┃ > Skip
┃   Apply
```

We'll down-arrow and choose `Apply`. Flowpipe reports:

```
Applied changes to tags on flowpipe-bucket-02.
```

Because we specified the default approver, this interaction happens in the terminal. You could shift the interaction to Slack, MSTeams, or email by naming — in `approvers` — one or more [notifiers](https://flowpipe.io/docs/reference/config-files/notifier) configured with alternate [integrations](https://flowpipe.io/docs/reference/config-files/integration).

We'll choose `Apply` for the second bucket and recheck the situation.

```
+--------------------+------------------------------------------+
| name               | tags                                     |
+--------------------+------------------------------------------+
| flowpipe-bucket-01 | {"cc":"default","env":"dev"}             |
| flowpipe-bucket-02 | {"cc":"default"}                         |
+--------------------+------------------------------------------+
```

That's a good start, but we now want to correct `cc`, that's a bogus tag that some people use for `cost_center`. Let's fix that.


## Update tag keys

We'll use this ruleset in `update-fix.fpvars`.

```hcl
approvers = []

base_tag_rules = {
  update_keys = {
    cost_center = ["~*:^cc|cost_centre$"]
  }
}

incorrect_tags_default_action = "apply"
```

With `approvers = []` the rules run without pausing for input. 

The `update_keys` rule uses pattern matching to converge a set of bogus tags (`cc` or `cost_centre`) onto `cost_center`. 

The default `incorrect_tags_default_action` is `notify`, which would be a dry run, but we want to apply the fix so we set it to `apply`.

We run the pipeline again, with this new package of variables.

```
$ flowpipe pipeline run detect_and_correct_s3_buckets_with_incorrect_tags --var-file=update-fix.fpvars
```

Flowpipe reports:

```
Applied changes to tags on flowpipe-bucket-01.
```

And here's the new situation.

```
+--------------------+---------------------------------------+
| name               | tags                                  |
+--------------------+---------------------------------------+
| flowpipe-bucket-01 | {"cost_center":"default","env":"dev"} |
| flowpipe-bucket-02 | {"env":"dev"}                         |
+--------------------+---------------------------------------+
```

## Tag operations

We've seen `add` and `update_keys`, here's the full set of operations documented in the [README](https://github.com/turbot/flowpipe-mod-aws-tags).

- **Add**. Ensure resources have mandatory tags.

- **Remove**. Remove specified tags.

- **Remove Except**. Inverse of remove: remove all but specified tags.

- **Update Keys**. Rewrite tag keys.

- **Update Values**. Rewrite tag values.

To match tags you can use literal strings, or Postgres regular expressions.

## Automate the fixes

It would be tedious to apply these kinds of changes with one-at-a-time approval. Let's automate with a new ruleset in `continuous-compliance.fpvars`.

```hcl
approvers = []

base_tag_rules = {
  update_keys = {
    cost_center = ["~*:^cc|cost_centre$"]
  }
  add = {
    env = "dev"
  }
}

s3_buckets_with_incorrect_tags_trigger_enabled   = true
s3_buckets_with_incorrect_tags_trigger_schedule  = "daily"
incorrect_tags_default_action                    = "apply"
```

Note that you can combine operations like `add` and `update_keys` to build up a complex ruleset. 

Here we're also using config variables to active the pipeline's [query trigger](https://flowpipe.io/docs/flowpipe-hcl/trigger/query), define its schedule, and define the default action (`apply`). With these settings, Flowpipe will run the pipeline daily and apply the changes we've seen to new buckets.

To run the pipeline on a schedule, use Flowpipe in [server](https://flowpipe.io/docs/run/server) mode.

```
flowpipe server --var-file=continuous-compliance.fpvars
```

Nothing will happen the first day because we've already fixed the buckets. But if we create `flowpipe-bucket-03` with no tags, it will — within a day — automagically be tagged with `env:dev`.

## Continuous compliance for tags

The AWS Tags mod for Flowpipe enables flexible and powerful management of AWS resource tags at scale. With declarative rulesets and automated workflows, you can ensure consistent tagging across your AWS infrastructure with or without manual intervention. Whether you're looking to standardize existing tags, enforce new tagging policies, or maintain ongoing compliance, this mod provides the tools you need to streamline your tagging discipline. Give it a try and please [let us know](https://turbot.com/community/join) how it goes.

## See it in action

<div className="flex justify-center">

<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/FNJt-a_nizY" 
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Define and enforce tagging standards in AWS"
>
</iframe>
</div>





In the world of cloud infrastructure management, consistent and accurate resource tagging is crucial. It's nice to be able to detect when tags deviate from your standards, but even better to correct those deviations automatically. Enter the [Azure Tags mod for Flowpipe](https://github.com/turbot/flowpipe-mod-azure-tags), a new tool that enables you to automate tag management using a declarative ruleset. Let's take a look.

## Add mandatory tags

Here's a Steampipe query that reveals incorrectly-tagged Azure storage accounts.

```sql
select name, tags from azure_storage_account where name ~ 'flowpipe'
+------------------+------------------+
| name             | tags             |
+------------------+------------------+
| flowpipestorage1 | {"cc":"default"} |
| flowpipestorage2 | {}               |
+------------------+------------------+
```

First, we'd like to apply the mandatory tag `env:dev` to all of them. To do that, we'll use the `detect_and_correct_storage_accounts_with_incorrect_tags` pipeline in the Azure Tags mod.

Per the README, the setup entails:

- Cloning the mod

- Visiting its directory and installing dependencies (`flowpipe mod install`)

- Starting Steampipe as a service (`steampipe service start`)

- Using `credential_import` to give Flowpipe access to the credentials in Steampipe `azure.spc` 

- Configuring tag rules

We'll use these definitions in `add-notify.fpvars`.

```hcl
approvers = []
base_tag_rules = {
  add = {
    env = "dev"
  }
}
```

With `approvers = []`, the mod will report tag violations to the default notifier, the terminal, but take no action. The only rule is to add our desired mandatory tag.

Now we'll run the pipeline to detect and correct tags on storage accounts, and point it at that package of variables.

```
$ flowpipe pipeline run detect_and_correct_storage_accounts_with_incorrect_tags --var-file=add-notify.fpvars
```

Flowpipe reports the changes it would make.

```
Detected flowpipestorage1 (288007e6-582c-478d-97cf-c27eea910d65/eastus) with incorrect tags. Existing tags: cc=default New tags: cc=default, env=dev

Detected flowpipestorage2 (288007e6-582c-478d-97cf-c27eea910d65/eastus) with incorrect tags. New tags: env=dev
```

To make the changes, we'll use `add-wizard.fpvars`, with the same tag rules but with `approvers = ["default"]`. That activates an [input step](https://flowpipe.io/docs/flowpipe-hcl/step/input) that will (in this case) prompt to `Skip` or `Apply` the proposed change.

Flowpipe pauses on the first detected violation.

```
$ flowpipe pipeline run detect_and_correct_storage_accounts_with_incorrect_tags --var-file=add-wizard.fpvars
┃ Detected flowpipestorage1 (288007e6-582c-478d-97cf-c27eea910d65/eastus) with incorrect tags. Tags that will be added or updated: env=dev.
┃ > Skip
┃   Apply
```

We'll down-arrow and choose `Apply`. Flowpipe reports:

```
Applied changes to tags on flowpipestorage1.
```

Because we specified the default approver, this interaction happens in the terminal. You could shift the interaction to Slack, MSTeams, or email by naming — in `approvers` — one or more [notifiers](https://flowpipe.io/docs/reference/config-files/notifier) configured with alternate [integrations](https://flowpipe.io/docs/reference/config-files/integration).

We'll choose `Apply` for the second storage account and recheck the situation.

```sql
+--------------------+------------------------------------------+
| name               | tags                                     |
+--------------------+------------------------------------------+
| flowpipestorage1   | {"cc":"default","env":"dev"}             |
| flowpipestorage2   | {"env":"dev"}                            |
+--------------------+------------------------------------------+
```

That's a good start, but we now want to correct `cc`, that's a bogus tag that some people use for `cost_center`. Let's fix that.


## Update tag keys

We'll use this ruleset in `update-fix.fpvars`.

```hcl
approvers = []

base_tag_rules = {
  update_keys = {
    cost_center = ["~*:^cc|cost_centre$"]
  }
}

incorrect_tags_default_action = "apply"
```

With `approvers = []` the rules run without pausing for input. 

The `update_keys` rule uses pattern matching to converge a set of bogus tags (`cc` or `cost_centre`) onto `cost_center`. 

The default `incorrect_tags_default_action` is `notify`, which would be a dry run, but we want to apply the fix so we set it to `apply`.

We run the pipeline again, with this new package of variables.

```
$ flowpipe pipeline run detect_and_correct_storage_accounts_with_incorrect_tags --var-file=update-fix.fpvars
```

Flowpipe reports:

```
Applied changes to tags on flowpipestorage1.
```

And here's the new situation.

```
+--------------------+---------------------------------------+
| name               | tags                                  |
+--------------------+---------------------------------------+
| flowpipestorage1   | {"cost_center":"default","env":"dev"} |
| flowpipestorage2   | {"env":"dev"}                         |
+--------------------+---------------------------------------+
```

## Tag operations

We've seen `add` and `update_keys`, here's the full set of operations documented in the [README](https://github.com/turbot/flowpipe-mod-azure-tags).

- **Add**. Ensure resources have mandatory tags.

- **Remove**. Remove specified tags.

- **Remove Except**. Inverse of remove: remove all but specified tags.

- **Update Keys**. Rewrite tag keys.

- **Update Values**. Rewrite tag values.

To match tags you can use literal strings, or Postgres regular expressions.

## Automate the fixes

It would be tedious to apply these kinds of changes with one-at-a-time approval. Let's automate with a new ruleset in `continuous-compliance.fpvars`.

```hcl
approvers = []

base_tag_rules = {
  update_keys = {
    cost_center = ["~*:^cc|cost_centre$"]
  }
  add = {
    env = "dev"
  }
}

storage_accounts_with_incorrect_tags_trigger_enabled   = true
storage_accounts_with_incorrect_tags_trigger_schedule  = "daily"
incorrect_tags_default_action                          = "apply"
```

Note that you can combine operations like `add` and `update_keys` to build up a complex ruleset. 

Here we're also using config variables to active the pipeline's [query trigger](https://flowpipe.io/docs/flowpipe-hcl/trigger/query), define its schedule, and define the default action (`apply`). With these settings, Flowpipe will run the pipeline daily and apply the changes we've seen to new storage accounts.

To run the pipeline on a schedule, use Flowpipe in [server](https://flowpipe.io/docs/run/server) mode.

```
flowpipe server --var-file=continuous-compliance.fpvars
```

Nothing will happen the first time because we've already fixed the storage accounts. But if we create `flowpipestorage3` with no tags, it will — within a day — automagically be tagged with `env:dev`.

## Continuous compliance for tags

The Azure Tags mod for Flowpipe enables flexible and powerful management of Azure resource tags at scale. With declarative rulesets and automated workflows, you can ensure consistent tagging across your Azure infrastructure with or without manual intervention. Whether you're looking to standardize existing tags, enforce new tagging policies, or maintain ongoing compliance, this mod provides the tools you need to streamline your tagging discipline. Give it a try and please [let us know](https://turbot.com/community/join) how it goes.

## See it in action

<div className="flex justify-center">

<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/ExoZNJPSHks" 
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Define and enforce tagging standards in Azure"
>
</iframe>
</div>






In the world of cloud infrastructure management, consistent and accurate resource tagging is crucial. It's nice to be able to detect when GCP labels deviate from your standards, but even better to correct those deviations automatically. Enter the [GCP Labels mod for Flowpipe](https://github.com/turbot/flowpipe-mod-gcp-labels), a new tool that enables you to automate label management using a declarative ruleset. Let's take a look.

## Add mandatory labels

Here's a Steampipe query that reveals incorrectly-labeled GCP buckets.

```sql
select name, labels from gcp_storage_bucket where name ~ 'flowpipe'
+--------------------+------------------+
| name               | labels           |
+--------------------+------------------+
| flowpipe-bucket-01 | {"cc":"default"} |
| flowpipe-bucket-02 | <null>           |
+--------------------+------------------+
```

First, we'd like to apply the mandatory label `env:dev` to all of them. To do that, we'll use the `detect_and_correct_storage_buckets_with_incorrect_labels` pipeline in the GCP Labels mod.

Per the README, the setup entails:

- Cloning the mod

- Visiting its directory and installing dependencies (`flowpipe mod install`)

- Starting Steampipe as a service (`steampipe service start`)

- Using `credential_import` to give Flowpipe access to the credentials in Steampipe `gcp.spc` 

- Configuring label rules

We'll use these definitions in `add-notify.fpvars`.

```hcl
approvers = []
base_label_rules = {
  add = {
    env = "dev"
  }
}
```

With `approvers = []`, the mod will report label violations to the default notifier, the terminal, but take no action. The only rule is to add our desired mandatory label.

Now we'll run the pipeline to detect and correct bucket labels, and point it at that package of variables.

```
$ flowpipe pipeline run detect_and_correct_storage_buckets_with_incorrect_labels --var-file=add-notify.fpvars
```

Flowpipe reports the changes it would make.

```
Detected flowpipe-bucket-01 (flowpipe-bucket-01/encoded-blend-333001/) with incorrect labels. Labels that will be added or updated: env=dev.

Detected flowpipe-bucket-02 (flowpipe-bucket-02/encoded-blend-333001/) with incorrect labels. Labels that will be added or updated: env=dev.
```

To make the changes, we'll use `add-wizard.fpvars`, with the same label rules but with `approvers = ["default"]`. That activates an [input step](https://flowpipe.io/docs/flowpipe-hcl/step/input) that will (in this case) prompt to `Skip` or `Apply` the proposed change.

Flowpipe pauses on the first detected violation.

```
$ flowpipe pipeline run detect_and_correct_storage_buckets_with_incorrect_labels --var-file=add-wizard.fpvars
┃ Detected flowpipe-bucket-01 (flowpipe-bucket-01/encoded-blend-333001/) with incorrect labels. Labels that will be added or updated: env=dev.
┃ > Skip
┃   Apply
```

We'll down-arrow and choose `Apply`. Flowpipe reports:

```
Applied changes to labels on flowpipe-bucket-01.
```

Because we specified the default approver, this interaction happens in the console. You could shift the interaction to Slack, MSTeams, or email by naming — in `approvers` — one or more [notifiers](https://flowpipe.io/docs/reference/config-files/notifier) configured with alternate [integrations](https://flowpipe.io/docs/reference/config-files/integration).

We'll choose `Apply` for the second bucket and recheck the situation.

```
+--------------------+------------------------------------------+
| name               | labels                                   |
+--------------------+------------------------------------------+
| flowpipe-bucket-01 | {"cc":"default","env":"dev"}             |
| flowpipe-bucket-02 | {"env":"dev"}                            |
+--------------------+------------------------------------------+
```

That's a good start, but we now want to correct `cc`, that's a bogus label that some people use for `cost_center`. Let's fix that.


## Update label keys

We'll use this ruleset in `update-fix.fpvars`.

```hcl
approvers = []

base_label_rules = {
  update_keys = {
    cost_center = ["~*:^cc|cost_centre$"]
  }
}

incorrect_labels_default_action = "apply"
```

With `approvers = []` the rules run without pausing for input. 

The `update_keys` rule uses pattern matching to converge a set of bogus labels (`cc` or `cost_centre`) onto `cost_center`. 

The default `incorrect_labels_default_action` is `notify`, which would be a dry run, but we want to apply the fix so we set it to `apply`.

We run the pipeline again, with this new package of variables.

```
$ flowpipe pipeline run detect_and_correct_storage_buckets_with_incorrect_labels --var-file=update-fix.fpvars
```

Flowpipe reports:

```
Applied changes to labels on flowpipe-bucket-01.
```

And here's the new situation.

```
+--------------------+---------------------------------------+
| name               | labels                                |
+--------------------+---------------------------------------+
| flowpipe-bucket-01 | {"cost_center":"default","env":"dev"} |
| flowpipe-bucket-02 | {"env":"dev"}                         |
+--------------------+---------------------------------------+
```

## Label operations

We've seen `add` and `update_keys`, here's the full set of operations documented in the [README](https://github.com/turbot/flowpipe-mod-gcp-labels).

- **Add**. Ensure resources have mandatory labels.

- **Remove**. Remove specified labels.

- **Remove Except**. Inverse of remove: remove all but specified labels.

- **Update Keys**. Rewrite label keys.

- **Update Values**. Rewrite label values.

To match labels you can use literal strings, or Postgres regular expressions.

## Automate the fixes

It would be tedious to apply these kinds of changes with one-at-a-time approval. Let's automate with a new ruleset in `continuous-compliance.fpvars`.

```hcl
approvers = []

base_label_rules = {
  update_keys = {
    cost_center = ["~*:^cc|cost_centre$"]
  }
  add = {
    env = "dev"
  }
}

storage_buckets_with_incorrect_labels_trigger_enabled  = true
storage_buckets_with_incorrect_labels_trigger_schedule = "daily"
incorrect_labels_default_action                        = "apply"
```

Note that you can combine operations like `add` and `update_keys` to build up a complex ruleset. 

Here we're also using config variables to active the pipeline's [query trigger](https://flowpipe.io/docs/flowpipe-hcl/trigger/query), define its schedule, and define the default action (`apply`). With these settings, Flowpipe will run the pipeline daily apply the changes we've seen to new buckets.

To run the pipeline on a schedule, use Flowpipe in [server](https://flowpipe.io/docs/run/server) mode.

```
flowpipe server --var-file=continuous-compliance.fpvars
```

Nothing will happen the first day because we've already fixed the buckets. But if we create `flowpipe-bucket-03` with no labels it will — within a day — automagically be labeled with `env:dev`.

## Continuous compliance for labels

The GCP Labels mod for Flowpipe enables flexible and powerful management of GCP labels at scale. With declarative rulesets and automated workflows, you can ensure consistent labeling across your GCP infrastructure with or without manual intervention. Whether you're looking to standardize existing labels, enforce new labeling policies, or maintain ongoing compliance, this mod provides the tools you need to streamline your labeling discipline. Give it a try and please [let us know](https://turbot.com/community/join) how it goes.

## See it in action

<div className="flex justify-center">

<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/8o11poEgf8c" 
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Define and enforce labeling standards in GCP"
>
</iframe>
</div>




Cloud infrastructure managers are always looking for ways to save costs. It's good to detect underused or unused resources, and even better to automatically streamline or remove them. Enter the [Azure Thrifty mod](https://github.com/turbot/flowpipe-mod-azure-thrifty) for Flowpipe, a new tool that enables you automate cost optimization. Let's see how it works.

## Detect and release unused external IPs

Here's a Steampipe query that finds unattached external IPs. 

```sql
> select name, region from azure_public_ip;
+---------+--------+
| name    | region |
+---------+--------+
| ipaddr1 | eastus |
| ipaddr2 | eastus |
+---------+--------+
```

We're paying for these, and if we don't need them we should release them. To do that, we'll run the `detect_and_correct_compute_addresses_if_unattached` pipeline in the Azure Thrifty mod.

Per the [README](https://github.com/turbot/flowpipe-mod-azure-thrifty?tab=readme-ov-file#azure-thrifty-mod-for-flowpipe), the setup entails:

- Cloning the mod

- Visiting its directory and installing dependencies (`flowpipe mod install`)

- Starting Steampipe as a service (`steampipe service start`)

- Using `credential_import` to give Flowpipe access to the credentials in Steampipe `azure.spc` 

- Optionally, configuring variables that govern the mod's behavior.

Let's run the pipeline.

```
$ flowpipe pipeline run detect_and_correct_compute_addresses_if_unattached --arg='approvers=[""]

Detected unattached Compute address ip-addr-1.
Detected unattached Compute address ip-addr-2.
```

Now let's take action on them. 

## Wizard mode for interactive decisions

In wizard mode, Flowpipe prompts you to decide whether to release each address. It's the default, so we can run the pipeline with no arguments.


```
flowpipe pipeline run detect_and_correct_compute_addresses_if_unattached
```

Now, Flowpipe pauses on each detected address, so you can decide whether to release it. We'll choose `Skip` and keep this one for now.

```
┃ Detected unattached Compute address ip-addr-1.
┃ > Skip
┃   Delete Compute Address
```

Note that this interaction can instead happen in Slack, MSTeams, or email by naming — in `approvers` — one or more notifiers configured with alternate [integrations](https://flowpipe.io/docs/reference/config-files/integration).

Flowpipe proceeds to the next detected address. This time we'll choose `Delete Compute Address`; Flowpipe takes that action and reports the outcome.

```
Deleted Compute address ip-addr-2.
```

Nice! 

## Delete old compute snapshots automatically

You won't want to automatically release IP addresses without checking each one, but you might want to automatically delete compute snapshots older than a threshold number of days. There's another pipeline for that, `detect_and_correct_compute_snapshots_exceeding_max_age`. 

To use that pipeline automatically, you'll activate a [query trigger](https://flowpipe.io/docs/flowpipe-hcl/trigger/query) to query for old snapshots. And you'll set some configuration variables to control how it works. We'll put them into a file, `continuous-compliance.fpvars`.

```hcl
approvers = []
compute_snapshots_exceeding_max_age_trigger_enabled  = true
compute_snapshots_exceeding_max_age_trigger_schedule = "weekly"
compute_snapshots_exceeding_max_age_default_action   = "delete_snapshot"
compute_snapshots_exceeding_max_age_days             = 90
```

Then we'll run Flowpipe in server mode and point it at that package of variables.

```
flowpipe server --var-file=continuous-compliance.fpvars
```

With `approvers = []` the pipeline handles each detected old snapshot without requiring approval. The default action is normally `notify` but here we override to `delete_snapshot`. Now, snapshots older than 90 days will be automatically deleted on a weekly basis.

## Configuration-driven workflow

A detect-and-correct mod like this one doesn't require you to write any SQL or HCL. It's a tool that you configure and then use to perform a range of tasks you'd like to automate, with or without approval. Give Azure Thrifty a try, and please [let us know](https://turbot.com/community/join) how it goes.

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/lTpHqQK02C0" 
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Control Azure costs with 24 workflows for FinOps"
>
</iframe>
</div>





Cloud infrastructure managers are always looking for ways to save costs. It's good to detect underused or unused resources, and even better to automatically streamline or remove them. Enter the [GCP Thrifty mod](https://github.com/turbot/flowpipe-mod-gcp-thrifty) for Flowpipe, a new tool that enables you automate cost optimization. Let's see how it works.

## Detect and release unused external IPs

Here's a Steampipe query that finds unattached external IPs. 

```sql
> select name, location from gcp_compute_address
+-----------+-------------+
| name      | location    |
+-----------+-------------+
| ip-addr-2 | us-central1 |
| ip-addr-1 | us-central1 |
+-----------+-------------+
```

We're paying for these, and if we don't need them we should release them. To do that, we'll run the `detect_and_correct_compute_addresses_if_unattached` pipeline in the GCP Thrifty mod.

Per the [README](https://github.com/turbot/flowpipe-mod-gcp-thrifty?tab=readme-ov-file#gcp-thrifty-mod-for-flowpipe), the setup entails:

- Cloning the mod

- Visiting its directory and installing dependencies (`flowpipe mod install`)

- Starting Steampipe as a service (`steampipe service start`)

- Using `credential_import` to give Flowpipe access to the credentials in Steampipe's `gcp.spc` 

- Optionally, configuring variables that govern the mod's behavior.

Let's run the pipeline. 

```
$ flowpipe pipeline run detect_and_correct_compute_addresses_if_unattached --arg='approvers=[""]'
```

With no approvers, the pipeline runs without interruptions and reports that both addresses are unattached.

```
Detected unattached Compute address ip-addr-1 [us-central1/encoded-blend-333001].
Detected unattached Compute address ip-addr-2 [us-central1/encoded-blend-333001].
```

Now let's take action on them. 

## Wizard mode for interactive decisions

In wizard mode, Flowpipe prompts you to decide whether to release each address. It's the default, so we can run the pipeline with no arguments.

```
flowpipe pipeline run detect_and_correct_compute_addresses_if_unattached
```

Now, Flowpipe pauses on each detected address, so you can decide whether to release it. We'll choose `Skip` and keep this one for now.

```
┃ Detected unattached Compute address ip1 [us-central1/encoded-blend-333001].
┃ > Skip
┃   Delete Compute Address
```

Note that this interaction can instead happen in Slack, MSTeams, or email by naming — in `approvers` — one or more notifiers configured with alternate [integrations](https://flowpipe.io/docs/reference/config-files/integration).

Flowpipe proceeds to the next detected address. This time we'll choose `Delete Compute Address`; Flowpipe takes that action and reports the outcome.

```
Deleted Compute address ip-addr-2 [us-central1/encoded-blend-333001].
```

Nice! 

## Delete old compute snapshots automatically

You won't want to automatically release IP addresses without checking each one, but you might want to automatically delete compute snapshots older than a threshold number of days. There's another pipeline for that, `detect_and_correct_compute_snapshots_exceeding_max_age`. 

To use that pipeline automatically, you'll activate a [query trigger](https://flowpipe.io/docs/flowpipe-hcl/trigger/query) to query for old snapshots. And you'll set some configuration variables to control how it works. We'll put them into a file, `continuous-compliance.fpvars`.

```hcl
approvers = []
compute_snapshots_exceeding_max_age_trigger_enabled  = true
compute_snapshots_exceeding_max_age_trigger_schedule = "weekly"
compute_snapshots_exceeding_max_age_default_action   = "delete_snapshot"
compute_snapshots_exceeding_max_age_days             = 90
```

Then we'll run Flowpipe in server mode and point it at that package of variables.

```
flowpipe server --var-file=continuous-compliance.fpvars
```

With `approvers = []` the pipeline handles each detected old snapshot without requiring approval. The default action is normally `notify` but here we override to `delete_snapshot`. Now, snapshots older than 90 days will be automatically deleted on a weekly basis.

## Configuration-driven workflow

A detect-and-correct mod like this one doesn't require you to write any SQL or HCL. It's a tool that you configure and then use to perform a range of tasks you'd like to automate, with or without approval. Give GCP Thrifty a try, and please [let us know](https://turbot.com/community/join) how it goes.

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/v0ODTKP3blQ" 
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Control GCP costs with 22 workflows for FinOps"
>
</iframe>
</div>


We are excited to announce the launch of the [Guardrails Hub](https://hub.guardrails.turbot.com), a comprehensive central repository for automating cloud operations, security, and FinOps control objectives. The Guardrails Hub is designed to help you find and deploy the top compliance use cases directly to your Guardrails workspace, making it easier than ever to manage and enforce policies across your cloud environments.

<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://hub.guardrails.turbot.com" target="_new"><img alt="Guardrails Hub homepage" src="/images/blog/2024-07-guardrails-hub/hub_homepage.png" /></a>
</div>


### Why we created the Guardrails Hub

The rapid evolution of cloud services creates an ever-expanding scope of threats and best practices that cloud operators must contend with, and Turbot Guardrails provides a robust governance platform that excels at meeting these diverse and complex requirements. We created the Guardrails Hub to address the need for a centralized, user-centric repository where organizations can quickly find and deploy solutions that accelerate their cloud governance, compliance, and operational efficiency goals. 

### Mod Directory

The Guardrails Hub includes a detailed reference data for thousands of mods, resources, policies, and controls across all supported cloud service providers: [AWS](https://hub.guardrails.turbot.com/mods/aws/mods), [Azure](https://hub.guardrails.turbot.com/mods/azure/mods), [GCP](https://hub.guardrails.turbot.com/mods/gcp/mods), [ServiceNow](https://hub.guardrails.turbot.com/mods/servicenow/mods), [Kubernetes](https://hub.guardrails.turbot.com/mods/kubernetes/mods) and [Turbot](https://hub.guardrails.turbot.com/mods/turbot/mods). Browsing the directory will help you understand and leverage the full potential of the Guardrails ecosystem.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails Hub homepage" src="/images/blog/2024-07-guardrails-hub/mod_detail.png" />
</div>

### Policy Packs

The heart of the hub is our growing [library of policy packs](https://hub.guardrails.turbot.com/policy-packs) examples. These packs provide full working configurations for automating common control objectives across our supported cloud providers. These fully functional examples serve as practical guides, demonstrating how to configure Guardrails to meet specific governance and operational goals:

- **[Access Management](https://hub.guardrails.turbot.com/policy-packs?categories=access-management)**
- **[Compliance](https://hub.guardrails.turbot.com/policy-packs?categories=compliance)**
- **[Cost Controls](https://hub.guardrails.turbot.com/policy-packs?categories=cost-controls)**
- **[Data Protection](https://hub.guardrails.turbot.com/policy-packs?categories=data-protection)**
- **[Logging](https://hub.guardrails.turbot.com/policy-packs?categories=logging)**
- **[Security](https://hub.guardrails.turbot.com/policy-packs?categories=security)**
- **[Tagging](https://hub.guardrails.turbot.com/policy-packs?categories=tagging)**

<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://hub.guardrails.turbot.com/policy-packs" target="_new"><img alt="Guardrails Hub homepage" src="/images/blog/2024-07-guardrails-hub/policy_packs.png" /></a>
</div>

### Open for contributions

We encourage community contributions to our policy pack library to help improve and expand our collection. All policy packs are defined using [Turbot Guardrails Terraform provider](https://registry.terraform.io/providers/turbot/turbot/latest/docs), and hosted in the [Guardrails samples public repository](https://github.com/turbot/guardrails-samples). We welcome submissions of new policy packs, updates to existing, and issue reporting from users. Here's how you can contribute:

1. Visit our public repository: [Guardrails Samples](https://github.com/turbot/guardrails-samples)
2. Fork the repository and create a new branch for your contributions.
3. Add your policy packs or make updates to existing ones.
4. Submit a pull request with a detailed description of your changes.
5. You can also raise issues to report bugs or suggest improvements.

By contributing to the policy pack library, you help the community benefit from a broader range of governance solutions and best practices.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/qL1Gr2kTh4M"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Introducing Guardrails Hub: Examples and accelerators for cloud governance"
>
</iframe>
</div>

## Go beyond reporting with Guardrails

Take your operational controls and compliance to the next level with Turbot Guardrails' auto-remediation policies. With continuous monitoring, instant remediation, and detailed audit trails, Guardrails makes it effortless to maintain a secure and compliant cloud environment.

Try it now with a [14-day free trial](https://turbot.com/start) and see how easy it is to keep your cloud environment aligned with best practices.


In the ever-evolving landscape of cloud services, ensuring that your organization meets its control objectives for security, FinOps, and cloud operations best practices can be a complex and time-consuming task. 

Turbot Guardrails provides a powerful governance platform to address these challenges across AWS, Azure, GCP and Kubernetes. However, with a vast [library of over 300 mods and 12,000 policies](https://hub.guardrails.turbot.com), knowing how to configure Guardrails to meet specific control objectives can be daunting for new users. This is where Policy Packs come into play.

## Policy Packs (feat. Smart Folders)

To simplify and streamline the process of meeting specific control objectives, we are introducing a new [core concept](/guardrails/docs/concepts) named [Policy Packs](/guardrails/docs/concepts/policy-packs). We have renamed the feature previously known as "Smart Folders" to "Policy Packs" and re-imagined how we can use this capability to accelerate the development and delivery of cloud governance automation.

Policies are the primary configuration language of Guardrails, they store metadata used to assess resources and define the automated actions Guardrails takes when responding to unapproved configuration of your cloud resources.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails Policy Settings" src="/images/blog/2024-07-guardrails-policy-packs/policy_settings.png" />
</div>

Policy Packs are targeted collections of these policies that work together to achieve specific control objectives.

### Key Benefits of Policy Packs:

- **Simplicity**: Pre-packaged policies mean you don’t have to start from scratch.
- **Efficiency**: Quickly find, test and deploy policies to meet specific control objectives.
- **Consistency**: Standardized configurations ensure uniformity across your cloud environment.
- **Flexibility**: Customize and extend Policy Packs to fit your unique needs.

## Howl at the Hub

An exciting part of this transformation is the addition of the [Guardrails Hub](https://hub.guardrails.turbot.com).  The Hub serves as a central library and reference tool for Policy Packs.  It is fed from [Guardrails Samples](https://github.com/turbot/guardrails-samples), an open source GitHub repo containing tons of example scripts, API tools, GraphQL queries and (now) policy packs.

We welcome contributions to this repo both in terms of new code/examples and in raising issues against the existing content.

### 100+ best practice examples

To launch the Guardrails Hub, we have reviewed, polished and [published 100+ common control objectives](https://hub.guardrails.turbot.com/policy-packs) requested by Guardrails customers, across a wide range of categories:

<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://hub.guardrails.turbot.com/policy-packs"><img alt="Guardrails Policy Settings" src="/images/blog/2024-07-guardrails-policy-packs/policy-pack-list.png" /></a>
</div>

### Naming

Each pack is named in a way that makes it possible to understand the underlying control objective. The opening action word is designed to communicate the type of action the control pack is designed to take:

- **Check**: Controls that do not have an enforcement option and can only notify on error.
- **Enforce**: Controls where Guardrails can take an enforcement action against misconfiguration.
- **Configure**: Change configuration of a resource.
- **Deny**: Guardrails IAM preventative controls.
- **Prevent**: Cloud native preventative controls.

The action verb is followed by a descriptive sentence indicating the resource types involved and the criteria they are measured against. The verbosity of the names is intentional, to make it easy to find control objectives using search and to quickly grok the intent without needed to read the underlying policy settings or code in detail.  Here are some examples:

- "Check MFA is Enabled for AWS IAM Root Accounts"
- "Enforce Approved Extensions Are Installed on Azure Compute Virtual Machines"
- "Configure Uniform Access is Enabled for GCP Storage Buckets"

## How to use

Existing guardrails users can get started using policy packs today:

1. Navigate to https://hub.guardrails.turbot.com
2. Search for an applicable policy pack.
3. Read and follow the installation instructions.

For a detailed walk-through of how to deploy and use policy packs in guardrails watch the video below.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/JcdlPK2xNyQ"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Policy Packs for Guardrails: The fast track to meet your control objectives"
>
</iframe>
</div>

## Control your cloud with Guardrails

Take your operational controls and compliance to the next level with Turbot Guardrails' auto-remediation policies. With continuous monitoring, instant remediation, and detailed audit trails, Guardrails makes it effortless to maintain a secure and compliant cloud environment.

Try it now with a [14-day free trial](https://turbot.com/start) and see how easy it is to keep your cloud environment aligned with best practices.



Turbot Guardrails already supports direct cloud discovery from [AWS](https://turbot.com/guardrails/blog/2024/01/servicenow-aws-discovery), [Azure](https://turbot.com/guardrails/blog/2024/03/servicenow-azure-discovery), [GCP](https://turbot.com/guardrails/blog/2024/03/servicenow-gcp-discovery), and [Kubernetes](https://turbot.com/guardrails/blog/2024/05/servicenow-kubernetes-discovery) to target CMDB CI tables; this ensures records are in sync at all times as changes in the cloud occur. As we have been working with large enterprises who leverage ServiceNow import sets as their preferred integration pattern, Guardrials now supports our cloud discovery features to push to ServiceNow import sets as well.

The integration requires no additional ServiceNow modules, AWS, Azure, GCP or Kubernetes services. Instead, Turbot Guardrails enables a direct, low-cost way to get real-time cloud discovery delivered to import sets to stage records for transformation to your target CMDB tables.

Whether you are using our direct discovery to CMDB tables features, or discovery to import sets, Guardrails provides broader coverage, in-depth configuration details, and more accurate ServiceNow CMDB data by surfacing comprehensive and timely cloud resource insights. With the breadth, depth, and speed of the Guardrails cloud discovery features, Guardrails is an ideal solution for those new to cloud discovery or looking to augment existing ServiceNow cloud integrations.

## Why an automated cloud discovery for ServiceNow?

Manual and scheduled discovery of cloud resources leads to inaccurate ServiceNow data. Cloud resources change too quickly for legacy discovery methods to keep up. As a result, CMDBs end up with blindspots and incorrect configurations.

When you automate continuous cloud discovery, and integrate resources into ServiceNow, you:

* Eliminate blindspots by covering more cloud services beyond the native integrations
* Prevent missing and stale data with instant updates when resources change
* Enable flexible control over how resource data syncs to CMDB CI tables
* Surface cloud resource tags and metadata to enrich CMDB context
* Archive historical records of provisioned resources that get deleted

## Augmenting native cloud discovery in ServiceNow

Native ServiceNow connectors and integrations are available to enable discovery of core cloud services such as Compute, Storage, and Networking. But these have limitations including:

* Extra licensing and services
* Limited resource coverage
* Minimal configuration data captured
* Reliance on scheduled batch jobs rather than real-time discovery
* Need for professional services and customization

The Turbot Guardrails ServiceNow integration augments native capabilities by:

* Point-and-click integration setup
* Discovering IaaS & PaaS cloud resource types out of the box
* Reducing licensing dependency and service overhead
* Automatically updating ServiceNow when resources changes directly to CMDB tables or via import sets
* Providing flexible control to configure synced data as needed
* Centralized management for all cloud discovery scopes: [AWS](https://turbot.com/guardrails/blog/2024/01/servicenow-aws-discovery), [Azure](https://turbot.com/guardrails/blog/2024/03/servicenow-azure-discovery), [GCP](https://turbot.com/guardrails/blog/2024/03/servicenow-gcp-discovery), and [Kubernetes](https://turbot.com/guardrails/blog/2024/05/servicenow-kubernetes-discovery)

By combining automated discovery from Turbot Guardrails with ServiceNow, you accurately sync cloud resource configs into ServiceNow's CMDB — or via import sets in real-time — without added cost and complexity.

## Configuring automated cloud discovery sync to ServiceNow import sets

### ServiceNow import set configuration 
While our direct-to-CMDB CI tables handle table creation, record transformation, and CI record syncs, when using import sets you will need to configure a staging table, choose a target table, and create the transform map to bring the data from staging to target.

The first step is to ensure you have an import set staging table already created in ServiceNow. Whether you have multiple staging tables, use unique fields per table, or keep it simple with a generic staging table and fields, Guardrails makes it easy to map to the data architecture you prefer.

In this example, `Guardrails Staging Table` exists with a `Guardrails Resource ID` coalesce field, and a `Cloud Resource Data` field for all the configuration data about the resource.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails ServiceNow cloud discovery import set table" src="/images/blog/2024-07-servicenow-import-set-cloud-discovery/import-set-table.png" />
</div>

Also ensure you have a ServiceNow CMDB CI target table in place. In our example, we will focus on GCP storage buckets. The target table is an extension table from `cmdb_ci` specific for cloud storage buckets.

Import sets require a transform map that will move the record from the staging table to the target table. In our script, we map the `Cloud Resource Data` field from the import set `cloud_resource_data` field to the columns in the target table.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails ServiceNow cloud discovery import set transform map" src="/images/blog/2024-07-servicenow-import-set-cloud-discovery/transform-map.png" />
</div>

Now that we have a ServiceNow import set in place, we can flip Guardrails configuration to discover and sync cloud data to the import set table.

### Enable Guardrails to push to import set tables

In this example we will show how to enable syncing GCP Storage Buckets directly to the import, so they can ultimately be added to the CMDB CI target table.

Set the Turbot Guardrails policy `GCP > Storage > Bucket > ServiceNow > Import Set` to “Enforce: Sync” and apply to all or specific GCP projects:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails ServiceNow cloud discovery import set sync policy for GCP storage buckets" src="/images/blog/2024-07-servicenow-import-set-cloud-discovery/gcp-storage-buckets-to-import-set-sync.png" />
</div>

For the integration-enabled GCP project, the following GCP resources will be in scope for the GCP discovery:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails ServiceNow GCP storage buckets import set examples" src="/images/blog/2024-07-servicenow-import-set-cloud-discovery/gcp-storage-buckets-import-set-example.png" />
</div>

Instantly the GCP resources are pushed to the import set table, then transformed to the target table:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails ServiceNow GCP storage buckets import set synced" src="/images/blog/2024-07-servicenow-import-set-cloud-discovery/gcp-storage-buckets-import-set-synced.png" />
</div>

As GCP resources are added, updated, or deleted, Guardrails handles the configuration drift and keeps the ServiceNow CMDB updated.

For example, when an GCP resource changes, Guardrails captures the drift and updates the ServiceNow CMDB:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails ServiceNow GCP storage buckets import set config drift" src="/images/blog/2024-07-servicenow-import-set-cloud-discovery/gcp-storage-buckets-import-set-config-drift.png" />
</div>

Cloud resource deletion can be set up to remove the resource or update an operational status column with an [archive status](https://turbot.com/guardrails/docs/integrations/servicenow/guardrails-to-servicenow-sync#archiving-records).

## Add cloud security & FinOps posture details to ServiceNow

Adjust what's included in the import set using Guardrails policy settings. You can add additional data from the cloud provider, audit trails, and context from Guardrails on the security & FinOps posture of the cloud resource.

In the `GCP > Storage > Bucket > ServiceNow > Import Set > Record` policy, we can add additional fields to the `cloud_resource_data` column in the import set:
* Include the versioning_enabled configuration: `versioning_enabled: {{ $.resource.data.versioning.enabled }}`
* Elevate a label (tag) into its own column: `environment: {{$.resource.turbot.tags.environment }}`
* Add the state of the Guardrails control to protect against public access: `public_access: {{ $.resource.public.items[0].state }}`

## See Turbot Guardrails cloud discovery to ServiceNow import sets in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/NW50Aeo7TAQ"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Automated cloud discovery to ServiceNow import sets"
>
</iframe>
</div>

## Ready your ServiceNow CMDB with automated cloud discovery

[Get started](https://turbot.com/start) with a 14-day free trial of Turbot Guardrails to experience [automated discovery](https://turbot.com/guardrails/docs/integrations/servicenow) and sync for cloud resources into your ServiceNow instance. Real-time, comprehensive visibility within minutes starting at just $0.05 per resource per month.



In our previous blog post, [Real-time Azure v2.0.0 CIS Benchmark assessments](https://turbot.com/guardrails/blog/2024/05/real-time-azure-cis-benchmark), we showed how Turbot Guardrails enables continuous monitoring of your Azure environment against the CIS v2.0.0 Benchmark. Guardrails discovers changes in real-time, instantly evaluates compliance, sends alerts when misconfigurations are detected, and takes quick actions to immediately apply fixes.

But what if you could automatically fix those misconfigurations and maintain a secure and compliant environment without manual intervention? With Turbot Guardrails' auto-remediation policies, you can.

## Effortless CIS compliance with auto-remediation
Traditional security incident management relies on manual processes to address misconfigurations identified in periodic assessments. This approach is time-consuming, error-prone, and leaves your environment vulnerable between scans.

Turbot Guardrails takes CIS assessments to the next level with auto-remediation policies. Guardrails detects configuration drift in real-time, evaluates the change for a misconfiguration, and can automatically take action to fix the issue and bring the resource back into compliance with the Azure CIS v2.0.0 Benchmark.

**Key benefits of Turbot Guardrails' auto-remediation policies:**
* **Continuous compliance:** As soon as a misconfiguration is detected, Guardrails automatically remediates the issue, ensuring your environment remains compliant with CIS recommendations at all times.
* **Reduced security operations overhead:** Auto-close incidents. Eliminate the need for manual intervention and free up your team to focus on higher-value tasks. Guardrails handles the heavy lifting of maintaining compliance.
* **Customizable policies:** Define auto-remediation policies that align with your organization's specific requirements and risk posture. Set different policies for different accounts, regions, resources, or per specific resource configurations. Set time-based exceptions to ignore recommendations per cloud account or per resource when not applicable.
* **Audit trail:** Guardrails keeps a detailed record of all auto-remediation actions taken, providing a complete audit trail for compliance reporting and troubleshooting.
* **Comprehensive coverage:** Guardrails provides over 10,000+ security and operational policies OOTB for AWS, Azure, GCP, Kubernetes and ServiceNow, so you can centralize your security posture management efforts across your entire cloud infrastructure.


## How to enable auto-remediation for Azure CIS v2.0.0 using Guardrails

### Prerequisite: Install the Azure CIS mods
If you haven't already, install the `@turbot/cis` and the `@turbot/azure-cisv2-0` mods, and enable the `Azure > CIS v2.0` policy to `Check: All CIS Benchmarks` as described in our previous [blog post]((https://turbot.com/guardrails/blog/2024/05/real-time-azure-cis-benchmark)). Installing the CIS mods will also ensure you have the appropriate `azure-*` dependent mods.

### Remediation for a control
You can focus on specific CIS Benchmark recommendations and set policies that map to them.  For example, `3.15 Ensure the 'Minimum TLS version' for storage accounts is set to 'Version 1.2'` has a number of storage accounts that are not meeting the related CIS recommendation.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS section Azure v2 3.15" src="/images/blog/2024-07-instant-azure-cis-v2-compliance/guardrails-cis-section-azure-cis-3-15.png" />
</div>

To remediate, use the following `Azure > Storage > Storage Account > Minimum TLS Version` policy to enforce the minimum TLS version setting:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS remediation policy settings for section Azure v2 3.1.5" src="/images/blog/2024-07-instant-azure-cis-v2-compliance/guardrails-cis-remediation-azure-cis-3-15-enforce.png" />
</div>

Once the policy is set to enforce, Guardrails will immediately remediate all applicable Azure storage accounts in scope:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS remediation activity for section Azure v2 3.15" src="/images/blog/2024-07-instant-azure-cis-v2-compliance/guardrails-cis-remediation-activity-azure-cis-3-15.png" />
</div>

All the buckets were remediated and now the CIS recommendation 3.15 is showing 'green' for the controls:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS now green for section Azure v2 3.15" src="/images/blog/2024-07-instant-azure-cis-v2-compliance/guardrails-cis-now-green-azure-cis-3-15.png" />
</div>

Now that enforcement policies are set, Guardrails will continuously monitor and take action whenever a new bucket is created or an existing bucket is misconfigured.

### Remediation for a CIS section
While you can set your own policies to remediate CIS recommendations, you can also leverage the collection of open-source policy samples available in the [Guardrails Samples repository](https://github.com/turbot/guardrails-samples). Included in the repo are [Policy Packs](https://github.com/turbot/guardrails-samples/tree/main/policy_packs) which are a collection of policies that can be applied to your cloud resources. Specifically for Azure CIS v2.0.0, there is a collection of [Azure CIS v2.0.0 remediation Policy Packs](https://github.com/turbot/guardrails-samples/tree/main/policy_packs/azure/cis_v200) that can be applied to your Guardrails workspace. 

<div style={{width:"80%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Policy Packs" src="/images/blog/2024-07-instant-azure-cis-v2-compliance/guardrails-cis-remediation-smart-folders.png" />
</div>

In the Acme environment, there are a number of Application Services resources which don't adhere to the broader set of Azure CIS v2.0.0 Section 9 recommendations.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Section 9 Alerts" src="/images/blog/2024-07-instant-azure-cis-v2-compliance/guardrails-cis-remediation-section-9-alerts.png" />
</div>

Using `Azure CIS v2.0.0 Section 9 - Application Services` Policy Pack as an example, we can apply a group of preset policies to cover the Section 9 recommendations. 

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Section 9 Policy Pack" src="/images/blog/2024-07-instant-azure-cis-v2-compliance/guardrails-cis-remediation-section-9-smart-folder.png" />
</div>

To apply the Policy Pack policies, you can attach it to any resource. In this case we attach it to the `Acme` folder which contains a few Azure subscriptions.

<div style={{width:"90%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Policy Pack Attachment" src="/images/blog/2024-07-instant-azure-cis-v2-compliance/guardrails-cis-remediation-smart-folder-attachment.png" />
</div>

Once the Policy Pack is attached, Guardrails will immediately remediate all applicable app services resources in scope:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS remediation activity for section 9 of Azure v2.0.0" src="/images/blog/2024-07-instant-azure-cis-v2-compliance/guardrails-cis-remediation-activity-azure-cis-section-9.png" />
</div>

All the storage account resources were remediated and now Section 3 of the CIS Benchmark is showing 'green' for the controls:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS now green for section 9 of Azure v2.0.0" src="/images/blog/2024-07-instant-azure-cis-v2-compliance/guardrails-cis-now-green-azure-cis-section-9.png" />
</div>


## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/UltiNc-X0Tk"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Get to green with real-time remediation for Azure CIS v2.0.0"
>
</iframe>
</div>

## Level up your compliance game with Guardrails

Take your Azure CIS v2.0.0 Benchmark compliance to the next level with Turbot Guardrails' auto-remediation policies. With continuous monitoring, instant remediation, and detailed audit trails, Guardrails makes it effortless to maintain a secure and compliant cloud environment.

Try it now with a [14-day free trial](https://turbot.com/start) and see how easy it is to keep your Azure environment aligned with CIS best practices.

Looking for AWS & GCP CIS guardrails? Guardrails also supports auto-remediation policies and covers the latest CIS benchmark assessments for [AWS](/guardrails/blog/2024/05/instant-aws-cis-v3-compliance) and [GCP](/guardrails/blog/2024/07/instant-gcp-cis-v2-compliance).



In our previous blog post, [Real-time GCP v2.0.0 CIS Benchmark assessments](https://turbot.com/guardrails/blog/2024/05/real-time-gcp-cis-benchmark), we showed how Turbot Guardrails enables continuous monitoring of your GCP environment against the CIS v2.0.0 Benchmark. Guardrails discovers changes in real-time, instantly evaluates compliance, sends alerts when misconfigurations are detected, and takes quick actions to immediately apply fixes.

But what if you could automatically fix those misconfigurations and maintain a secure and compliant environment without manual intervention? With Turbot Guardrails' auto-remediation policies, you can.

## Effortless CIS compliance with auto-remediation
Traditional security incident management relies on manual processes to address misconfigurations identified in periodic assessments. This approach is time-consuming, error-prone, and leaves your environment vulnerable between scans.

Turbot Guardrails takes CIS assessments to the next level with auto-remediation policies. Guardrails detects configuration drift in real-time, evaluates the change for a misconfiguration, and can automatically take action to fix the issue and bring the resource back into compliance with the GCP CIS v2.0.0 Benchmark.

**Key benefits of Turbot Guardrails' auto-remediation policies:**
* **Continuous compliance:** As soon as a misconfiguration is detected, Guardrails automatically remediates the issue, ensuring your environment remains compliant with CIS recommendations at all times.
* **Reduced security operations overhead:** Auto-close incidents. Eliminate the need for manual intervention and free up your team to focus on higher-value tasks. Guardrails handles the heavy lifting of maintaining compliance.
* **Customizable policies:** Define auto-remediation policies that align with your organization's specific requirements and risk posture. Set different policies for different accounts, regions, resources, or per specific resource configurations. Set time-based exceptions to ignore recommendations per cloud account or per resource when not applicable.
* **Audit trail:** Guardrails keeps a detailed record of all auto-remediation actions taken, providing a complete audit trail for compliance reporting and troubleshooting.
* **Comprehensive coverage:** Guardrails provides over 10,000+ security and operational policies OOTB for AWS, Azure, GCP, Kubernetes and ServiceNow, so you can centralize your security posture management efforts across your entire cloud infrastructure.


## How to enable auto-remediation for GCP CIS v2.0.0 using Guardrails

### Prerequisite: Install the GCP CIS mods
If you haven't already, install the `@turbot/cis` and the `@turbot/gcp-cisv2-0` mods, and enable the `GCP > CIS v2.0` policy to `Check: All CIS Benchmarks` as described in our previous [blog post]((https://turbot.com/guardrails/blog/2024/05/real-time-gcp-cis-benchmark)). Installing the CIS mods will also ensure you have the appropriate `gcp-*` dependent mods.

### Remediation for a control
You can focus on specific CIS Benchmark recommendations and set policies that map to them.  For example, `5.2 Ensure that Cloud Storage buckets have uniform bucket-level access enabled` has a number of storage buckets that are not meeting the related CIS recommendation.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS section GCP v2 5.2" src="/images/blog/2024-07-instant-gcp-cis-v2-compliance/guardrails-cis-section-gcp-cis-5-2.png" />
</div>

To remediate, use the following `GCP > Storage > Bucket > Access Control` policy to enforce uniform bucket-level access on a bucket:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS remediation policy settings for section GCP v2 5.2" src="/images/blog/2024-07-instant-gcp-cis-v2-compliance/guardrails-cis-remediation-gcp-cis-5-2-enforce.png" />
</div>

Once the policy is set to enforce, Guardrails will immediately remediate all applicable GCP storage buckets in scope:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS remediation activity for section GCP v2 5.2" src="/images/blog/2024-07-instant-gcp-cis-v2-compliance/guardrails-cis-remediation-activity-gcp-cis-5-2.png" />
</div>

All the buckets were remediated and now the CIS recommendation 5.2 is showing 'green' for the controls:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS now green for section GCP v2 5.2" src="/images/blog/2024-07-instant-gcp-cis-v2-compliance/guardrails-cis-now-green-gcp-cis-5-2.png" />
</div>

Now that enforcement policies are set, Guardrails will continuously monitor and take action whenever a new bucket is created or an existing bucket is misconfigured.

### Remediation for a CIS section
While you can set your own policies to remediate CIS recommendations, you can also leverage the collection of open-source policy samples available in the [Guardrails Samples repository](https://github.com/turbot/guardrails-samples). Included in the repo are [Policy Packs](https://github.com/turbot/guardrails-samples/tree/main/policy_packs) which are a collection of policies that can be applied to your cloud resources. Specifically for GCP CIS v2.0.0, there is a collection of [GCP CIS v2.0.0 remediation Policy Packs](https://github.com/turbot/guardrails-samples/tree/main/policy_packs/gcp/cis_v200) that can be applied to your Guardrails workspace. 

<div style={{width:"80%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Policy Packs" src="/images/blog/2024-07-instant-gcp-cis-v2-compliance/guardrails-cis-remediation-smart-folders.png" />
</div>

In the Acme environment, there are a number of storage buckets which don't adhere to the broader set of GCP CIS v2.0.0 Section 5 recommendations.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Section 5 Alerts" src="/images/blog/2024-07-instant-gcp-cis-v2-compliance/guardrails-cis-remediation-section-5-alerts.png" />
</div>

Using `GCP CIS v2.0.0 Section 5 - Storage` Policy Pack as an example, we can apply a group of preset policies to cover the Section 5 recommendations. 

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Section 5 Policy Pack" src="/images/blog/2024-07-instant-gcp-cis-v2-compliance/guardrails-cis-remediation-section-5-smart-folder.png" />
</div>

To apply the Policy Pack policies, you can attach it to any resource. In this case we attach it to the `Acme` folder which contains a few GCP projects.

<div style={{width:"90%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Policy Pack Attachment" src="/images/blog/2024-07-instant-gcp-cis-v2-compliance/guardrails-cis-remediation-smart-folder-attachment.png" />
</div>

Once the Policy Pack is attached, Guardrails will immediately remediate all applicable storage bucket resources in scope:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS remediation activity for section 5 of GCP v2.0.0" src="/images/blog/2024-07-instant-gcp-cis-v2-compliance/guardrails-cis-remediation-activity-gcp-cis-section-5.png" />
</div>

All the storage bucket resources were remediated and now Section 5 of the CIS Benchmark is showing 'green' for the controls:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS now green for section 5 of GCP v2.0.0" src="/images/blog/2024-07-instant-gcp-cis-v2-compliance/guardrails-cis-now-green-gcp-cis-section-5.png" />
</div>


## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/QmeZLPCIkvI"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Get to green with real-time remediation for GCP CIS v2.0.0"
>
</iframe>
</div>

## Level up your compliance game with Guardrails

Take your GCP CIS v2.0.0 Benchmark compliance to the next level with Turbot Guardrails' auto-remediation policies. With continuous monitoring, instant remediation, and detailed audit trails, Guardrails makes it effortless to maintain a secure and compliant cloud environment.

Try it now with a [14-day free trial](https://turbot.com/start) and see how easy it is to keep your GCP environment aligned with CIS best practices.

Looking for AWS & Azure CIS guardrails? Guardrails also supports auto-remediation policies and covers the latest CIS benchmark assessments for [AWS](/guardrails/blog/2024/05/instant-aws-cis-v3-compliance) and [Azure](/guardrails/blog/2024/07/instant-azure-cis-v2-compliance).



We are proud to announce that we have successfully completed our annual SOC 2 audit for Turbot Pipes and Turbot Guardrails. This achievement underscores our dedication to exceeding customer expectations in data security, privacy, and compliance.

At Turbot, maintaining and earning our customers' trust is paramount. Our operations, product, and engineering teams are committed to designing, architecting, and developing our products with an unwavering focus on security. This year, we've continued to comply with our SOC 2 requirements, maintained our CIS Benchmark Certification, passed multiple Well-Architected audits with our cloud partners, participated in our enterprise customer security audits, successfully completed third-party penetration tests, and upheld our privacy commitments under GDPR, CCPA, and the [Data Privacy Framework](https://www.dataprivacyframework.gov).

Maintaining our SOC 2 compliance over the years is crucial in supporting our customers' requirements and affirms our dedication to ensuring we are following industry-standard processes and procedures to maintain a secure and operationally managed service offering.

Our annual SOC audits are conducted each April, with audit reports available by late May. Customers and prospects can request our full SOC 2 Type II audit report under NDA or download our SOC 3 report from the [Turbot Security](https://turbot.com/security) page.

We encourage you to connect with us in our [Slack community](https://turbot.com/community/join), explore Turbot Pipes with a [free developer account](https://pipes.turbot.com), and start your Turbot Guardrails [14-day free trial](https://turbot.com/connect).



Our announcements this week span the breadth of Turbot offerings: 

- Steampipe's new connection-level qualifiers

- Detect-and-correct mods for Flowpipe

- KSPM and Kubernetes/ServiceNow discovery in Guardrails

- Updated Guardrails CIS AWS/Azure/GCP assessments

- Real-time CIS remediation for AWS in Guardrails

- GitHub integration for custom mods in Pipes

- Local and branch-based mod installation for Powerpipe


## Watch the Wrap Party

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/OdS_-MjECoA"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Turbot Launch Week 4 Wrap Party"
>
</iframe>
</div>
<div className="text-2xl font-bold pt-10">Full Week Recap</div>

## Monday May 13: Steampipe connection-level qualifiers, AWS Thrifty for Flowpipe

  **Steampipe** now [accelerates queries](https://steampipe.io/blog/connection-level-qualifiers) that involve only one of many aggregated connections. Previously a query like `select count(*) from aws_sqs_queue where account_id = '767398119043'`, if that account is one of 10, would have to list queues in all accounts before filtering down to just those in the account mentioned in the `where` clause. Now, that same `where` clause enables Steampipe to target that one account with surgical precision. The speedup is dramatic, as is the reduction in API load. In one test, the query ran 8 times faster and fetched 96.5% fewer rows! 

A new kind of [detect-and-correct](https://flowpipe.io/blog/finops-workflows-for-aws) mod for **Flowpipe**, [AWS Thrifty](https://hub.flowpipe.io/mods/turbot/aws_thrifty), not only finds costly mistakes like its [Powerpipe sibling](https://hub.powerpipe.io/mods/turbot/aws_thrifty) does, but also fixes them — automatically, or with human approval that routes through Slack, MS Teams, or email. Flowpipe's AWS Thrifty is the first of a new wave of these no-code, config-driven remediation workflows. 

## Tuesday May 14: Kubernetes in Guardrails

Are you familiar with CSPM (cloud security posture management)? The [new KSPM features](https://turbot.com/guardrails/blog/2024/05/kubernetes-security-posture-management) in **Guardrails** follow suit. You can now evaluate Kubernetes resources as changes occur, instantly assess compliance, and maintain a real-time view of your K8S security posture. 

**Guardrails** also just made it a lot easier to [discover Kubernetes resources in ServiceNow's CMDB](https://turbot.com/guardrails/blog/2024/05/servicenow-kubernetes-discovery). Each Kubernetes resource type can be configured to sync to the ServiceNow CMDB. As Kubernetes resources are added, updated, or deleted, Guardrails handles the configuration drift and keeps the ServiceNow CMDB updated.

## Wednesday May 15: Guardrails real-time remediation and CIS benchmark assessments

With [new auto-remediation policies](https://turbot.com/guardrails/blog/2024/05/instant-aws-cis-v3-compliance), **Guardrails** doesn't just watch your AWS environment for deviations from the CIS v3.0.0 Benchmark, it also automatically fixes misconfiguration in real-time, so you can get to green and then stay green.

We've also upgraded **Guardrails**' real-time CIS assessments across the board: for [AWS v3.0.0](https://turbot.com/guardrails/blog/2024/05/real-time-aws-cis-benchmark), [Azure v2.0.0](https://turbot.com/guardrails/blog/2024/05/real-time-azure-cis-benchmark), and [GCP v2.0.0](https://turbot.com/guardrails/blog/2024/05/real-time-gcp-cis-benchmark).

## Thursday May 16: Pipes/GitHub integration, local and branch-based installation of Powerpipe mods

Life just got way easier for developers of **Pipes** custom mods. With the new [GitHub integration](https://turbot.com/pipes/blog/2024/05/github-custom-mods) you can now install a mod from a private as well as a public repo, and now from branches as well as tags. Pipes then watches your repo and, when you push changes, automatically updates the mod and — magically! — refreshes live dashboards.

Mods destined for Pipes begin with local development in **Powerpipe**. You can now rapidly iterate on mod dependencies by [installing from the local filesystem](https://turbot.com/pipes/blog/2024/05/github-custom-mods). You can also now install mods from public or private repos, using branches as well as tags.

## And that's a wrap!

The announcements this week are individually exciting, but the Turbot story is bigger than the sum of its parts. Our products work together to deliver cloud security and intelligence with incomparable breadth, depth, power, speed, and versatility. And there's more to come. Enjoy the current batch of new products and features, [let us know](/community/join) how it goes, and stay tuned for our next Launch Week in a few months!



Suppose you're developing a Powerpipe mod, called `my-aws-insights`, to visualize AWS resources. It includes a dashboard for S3 buckets, with two charts and a table.

<div>
<img src="/images/blog/2024-05-flexible-mod-dependencies/initial-dashboard.png" />
</div>

The source code for the mod includes HCL widgets to define the charts and the table, and also includes the queries that fill those widgets with data.

Now suppose you'd like to source those queries from a separate mod. A couple of new features in Powerpipe — installing from the local filesystem, or from a remote branch — make it much easier to manage that separation of concerns. 

## Install and use a local dependency

Our mod lives in the local directory `~/my-aws-insights`. Now we'll create a new directory, `~/aws-queries`, initialize it with [powerpipe init](https://powerpipe.io/docs/reference/cli/mod#powerpipe-mod-init), copy the queries into it, and remove them from `my-aws-insights`. 

Then we'll adjust the charts and table in `my-aws-insights` to refer to the queries in the `aws-queries` mod.

<div>
<img src="/images/blog/2024-05-flexible-mod-dependencies/adjusted-query-refs.png" />
</div>

After completing the search-and-replace, query references like `query.s3_bucket_month_created` are fully-qualified with the name of the dependency (`aws_queries.query.s3_bucket_month_created`). Now we can install the mod from the local filesystem.

```
$ pwd
/home/jon/my-aws-insights
$ powerpipe mod install ~/aws-queries

Installed 1 mod:

local
└── /home/jon/aws-queries
```
The dashboard runs as before, but sources the queries from this new library mod.

## Install from a branch

We can now iterate rapidly on this local library mod but, at some point, we'll want to move it to GitHub so others can use it. So let's do that.

First we uninstall the local mod.

```
powerpipe mod uninstall `~/aws-queries`
```

Then we reinstall from GitHub. If the repo is tagged with a semver constraint, this command installs the latest version.

```
powerpipe mod install github.com/judell/aws-queries
```

And if it isn't tagged? No problem! You can now also install from a branch.

```
powerpipe mod install github.com/judell/aws-queries#main
```

Again the dashboard runs as before, sourcing queries from the mod that was installed from GitHub. Now others can depend on the library mod, perhaps even on alternate or experimental versions. And as we continue to evolve the library mod, we can switch back to a local install to rapidly iterate on it.

## Flexible update strategies

You've also now got a lot more flexibility when installing or updating mods, using the `--pull` argument to specify various update strategies.

<div>
<img src="/images/blog/2024-05-flexible-mod-dependencies/update-strategies.png" />
</div>

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/rxkp0Pj5t8c"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Powerpipe: Use files, branches, or tags for mod dependencies"
>
</iframe>
</div>

## Reuse, remix, refactor

You could always [reuse and remix](https://steampipe.io/blog/remixing-dashboards) Powerpipe mods. Now it's easy to refactor them too, so you can extract shared components into dependent mods that you can rapidly develop and test, and easily share. If you haven't tried writing your own mods, now's a great time to get started. Go build something useful, then [tell us](https://turbot.com/community/join) about it!




Pipes has long enabled you to install and use custom mods but you had to source them from public GitHub repositories, and it was a manual process to install and then update your mods. Now, with our new GitHub integration, you can install a custom mod from a public or private repo.

You can do this as an individual identity, as the owner of a Pipes [organization](https://turbot.com/pipes/docs/organizations) or — as a Pipes Enterprise user — in a Pipes [tenant](https://turbot.com/pipes/docs/tenants). In all these scenarios, you install your custom mod from a GitHub repo that you choose from a GitHub app. And when you push changes, Pipes automatically updates the mod.

To illustrate, we'll use a simple dashboard that counts S3 buckets with versioning enabled or disabled. As a  custom mod developer, you always want to test locally using Powerpipe to make sure the mod runs correctly. Then push to GitHub and create a GitHub integration in Pipes to gain access to it. 

## Create an integration

From the `Settings` page of your Pipes organization or tenant, click `New Integration`, choose `GitHub`, name the integration, and click `Create Integration`.

<div style={{width:"90%"}}>
  <img alt="new integration" src="/images/blog/2024-05-github-custom-mods/new-integration.png" />
</div>

## Choose your repo

You're redirected to GitHub, where you choose an identity and then pick the repos that Pipes can see. 

<div style={{width:"70%"}}>
  <img alt="choosing a repo for a custom mod" src="/images/blog/2024-05-github-custom-mods/authorize-the-repo.png" />
</div>


Typically you'll grant access to `All`, here we'll be more specific and just grant access to one  demo repo. 

## Install the mod

After installing the GitHub app with that configuration, you're redirected back to Pipes where you can now use the integration to install a mod from GitHub. Here we're installing to a workspace called `demo` but, because the integration was created in a Pipes organization, any workspace can use it.

<div>
  <img alt="install from main" src="/images/blog/2024-05-github-custom-mods/install-mod-from-branch.png" />
</div>

The picker only shows the one authorized repo. You can install from a branch, defaulting to main,  or from a tag that defines a semantic versioning constraint.

## View the dashboard

Once it's installed, we can view that dashboard in Pipes.

<div style={{width:"60%"}}>
  <img alt="dashboard as bar chart" src="/images/blog/2024-05-github-custom-mods/bucket-versioning-1.png" />
</div>

Now here's the really fun part: let's push a change.

## Update the mod

We'll change the bar chart to a pie chart.

<div style={{width:"60%"}}>
  <img alt="change bar to pie chart" src="/images/blog/2024-05-github-custom-mods/git-diff.png" />
</div>

Commit the change, and push it.

Pipes detects the change, updates the mod, and immediately refreshes the dashboard to display the updated chart.

<div style={{width:"60%"}}>
  <img alt="dashboard as bar char" src="/images/blog/2024-05-github-custom-mods/bucket-versioning-2.png" />
</div>

It really is that easy, and that fast, to deploy a change and live-update it!

## Developing for Pipes

Pipes' vast collection of pre-built benchmarks and dashboards will meet many of your needs right out of the box. But if you haven't yet discovered the power of custom mods in Pipes, now's a great time to get started. You'll find that building custom mods, using HCL and SQL, is straightforward and enjoyable. It's a powerful way to tailor data visualizations and benchmarks to your needs. So give it a try, and [let us know](https://turbot.com/community/join) how it goes!

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/a7Op2_LLm7E"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Pipes GitHub integration"
>
</iframe>
</div>





In our previous blog post, [Real-time AWS v3.0.0 CIS Benchmark assessments](https://turbot.com/guardrails/blog/2024/05/real-time-aws-cis-benchmark), we showed how Turbot Guardrails enables continuous monitoring of your AWS environment against the CIS v3.0.0 Benchmark. Guardrails discovers changes in real-time, instantly evaluates compliance, sends alerts when misconfigurations are detected, and takes quick actions to immediately apply fixes.

But what if you could automatically fix those misconfigurations and maintain a secure and compliant environment without manual intervention? With Turbot Guardrails' auto-remediation policies, you can.

## Effortless CIS compliance with auto-remediation
Traditional security incident management relies on manual processes to address misconfigurations identified in periodic assessments. This approach is time-consuming, error-prone, and leaves your environment vulnerable between scans.

Turbot Guardrails takes CIS assessments to the next level with auto-remediation policies. Guardrails detects configuration drift in real-time, evaluates the change for a misconfiguration, and can automatically take action to fix the issue and bring the resource back into compliance with the AWS CIS v3.0.0 Benchmark.

**Key benefits of Turbot Guardrails' auto-remediation policies:**
* **Continuous compliance:** As soon as a misconfiguration is detected, Guardrails automatically remediates the issue, ensuring your environment remains compliant with CIS recommendations at all times.
* **Reduced security operations overhead:** Auto-close incidents. Eliminate the need for manual intervention and free up your team to focus on higher-value tasks. Guardrails handles the heavy lifting of maintaining compliance.
* **Customizable policies:** Define auto-remediation policies that align with your organization's specific requirements and risk posture. Set different policies for different accounts, regions, resources, or per specific resource configurations. Set time-based exceptions to ignore recommendations per cloud account or per resource when not applicable.
* **Audit trail:** Guardrails keeps a detailed record of all auto-remediation actions taken, providing a complete audit trail for compliance reporting and troubleshooting.
* **Comprehensive coverage:** Guardrails provides over 10,000+ security and operational policies OOTB for AWS, Azure, GCP, Kubernetes and ServiceNow, so you can centralize your security posture management efforts across your entire cloud infrastructure.


## How to enable auto-remediation for AWS CIS v3.0.0 using Guardrails

### Prerequisite: Install the AWS CIS mods
If you haven't already, install the `@turbot/cis` and the `@turbot/aws-cisv3-0` mods, and enable the `AWS > CIS v3.0` policy to `Check: All CIS Benchmarks` as described in our previous [blog post]((https://turbot.com/guardrails/blog/2024/05/real-time-aws-cis-benchmark)). Installing the CIS mods will also ensure you have the appropriate `aws-*` dependent mods.

### Remediation for a control
You can focus on specific CIS Benchmark recommendations and set policies that map to them.  For example, `2.1.4 Ensure that S3 Buckets are configured with 'Block public access (bucket settings)'` has a number of buckets that are not meeting the related CIS recommendation.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS section AWS v3 2.1.4" src="/images/blog/2024-05-instant-aws-cis-v3-compliance/guardrails-cis-section-aws-cis-2-1-4.png" />
</div>

To remediate, use the following `AWS > S3 > Bucket > Public Access Block` policies to enforce the bucket settings that block public access:

First, set the `AWS > S3 > Bucket > Public Access Block > Settings` to block each of the four settings:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS remediation policy settings for section AWS v3 2.1.4" src="/images/blog/2024-05-instant-aws-cis-v3-compliance/guardrails-cis-remediation-aws-cis-2-1-4-settings.png" />
</div>

Then set the `AWS > S3 > Bucket > Public Access Block` to `Enforce: Per Public Access Block > Settings`:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS remediation policy enforcement for section AWS v3 2.1.4" src="/images/blog/2024-05-instant-aws-cis-v3-compliance/guardrails-cis-remediation-aws-cis-2-1-4-enforce.png" />
</div>

Once the policy is set to enforce, Guardrails will immediately remediate all applicable S3 buckets in scope:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS remediation activity for section AWS v3 2.1.4" src="/images/blog/2024-05-instant-aws-cis-v3-compliance/guardrails-cis-remediation-activity-aws-cis-2-1-4.png" />
</div>

All the buckets were remediated and now the CIS recommendation 2.1.4 is showing 'green' for the controls:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS now green for section AWS v3 2.1.4" src="/images/blog/2024-05-instant-aws-cis-v3-compliance/guardrails-cis-now-green-aws-cis-2-1-4.png" />
</div>

Now that enforcement policies are set, Guardrails will continuously monitor and take action whenever a new bucket is created or an existing bucket is misconfigured.

### Remediation for a CIS section
While you can set your own policies to remediate CIS recommendations, you can also leverage the collection of open-source policy samples available in the [Guardrails Samples repository](https://github.com/turbot/guardrails-samples). These policy samples are [Smart Folders](https://github.com/turbot/guardrails-samples/tree/main/cis_enforcements/aws) which are a collection of policies that can be applied to your cloud resources.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Smart Folders" src="/images/blog/2024-05-instant-aws-cis-v3-compliance/guardrails-cis-remediation-smart-folders.png" />
</div>

For example, in the Acme environment, there are a number of networking resources which don't adhere to the AWS CIS v3.0.0 Section 5 recommendations.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Section 5 Alerts" src="/images/blog/2024-05-instant-aws-cis-v3-compliance/guardrails-cis-remediation-section-5-alerts.png" />
</div>

Using `AWS CIS v3.0.0 Section 5 - Networking` Smart Folder as an example, we can apply a group of preset policies to cover the Section 5 recommendations. 

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Section 5 Smart Folder" src="/images/blog/2024-05-instant-aws-cis-v3-compliance/guardrails-cis-remediation-section-5-smart-folder.png" />
</div>

To apply the Smart Folder policies, you can attach it to any resource. In this case we attach it to the `Acme` folder which contains a few AWS accounts.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS Remediation Smart Folder Attachment" src="/images/blog/2024-05-instant-aws-cis-v3-compliance/guardrails-cis-remediation-smart-folder-attachment.png" />
</div>

Once the Smart Folder is attached, Guardrails will immediately remediate all applicable networking resources in scope:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS remediation activity for section 5 of AWS v3.0.0" src="/images/blog/2024-05-instant-aws-cis-v3-compliance/guardrails-cis-remediation-activity-aws-cis-section-5.png" />
</div>

All the networking resources were remediated and now Section 5 of the CIS Benchmark is showing 'green' for the controls:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Guardrails CIS now green for section 5 of AWS v3.0.0" src="/images/blog/2024-05-instant-aws-cis-v3-compliance/guardrails-cis-now-green-aws-cis-section-5.png" />
</div>


## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/5_ye51phJhk"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Get to green with real-time remediation for AWS CIS v3.0.0"
>
</iframe>
</div>

## Level up your compliance game with Guardrails

Take your AWS CIS v3.0.0 Benchmark compliance to the next level with Turbot Guardrails' auto-remediation policies. With continuous monitoring, instant remediation, and detailed audit trails, Guardrails makes it effortless to maintain a secure and compliant cloud environment.

Try it now with a [14-day free trial](https://turbot.com/start) and see how easy it is to keep your AWS environment aligned with CIS best practices.

Looking for Azure & GCP CIS guardrails? Guardrails also supports auto-remediation policies and covers the latest CIS benchmark assessments for [Azure](/guardrails/blog/2024/05/real-time-azure-cis-benchmark) and [GCP](/guardrails/blog/2024/05/real-time-gcp-cis-benchmark).



[Center for Internet Security](https://cisecurity.org) (CIS) benchmarks for AWS are widely used by cloud teams as guidance to configure their AWS accounts securely. The latest iteration, AWS CIS v3.0.0, includes over 60 recommendations to evaluate your posture against best practices.

Turbot Guardrails now supports the [AWS CIS v3.0.0](/guardrails/changelog/aws-cisv3-0-v5-0-0) Benchmark. Use it to enable continuous monitoring of your cloud environment's security posture, and to ensure compliance with industry standards as resources are created and modified.

## Go beyond CIS reporting with Guardrails

Traditional compliance assessments rely on periodic scans of your environment, which only give you a point-in-time view of your security posture. But the cloud is dynamic, resources change rapidly, misconfigurations and security gaps can happen anytime.

Turbot Guardrails evaluates your AWS resources against the CIS v3.0.0 Benchmark in real-time. As resources are created or modified, Guardrails instantly assesses them for compliance and provides a view of your security posture that's always current.

Key Features of Turbot Guardrails' CIS Benchmark controls:
1. **Instant Evaluation**. As new AWS resources are created or modified, Guardrails immediately assesses them against the relevant CIS controls, providing instant feedback on your compliance status.
2. **Real-time Alerts**. Stay informed about critical changes in your compliance posture with real-time alerts. Guardrails can send [notifications](/guardrails/docs/guides/notifications) to Slack, MS Teams, and email, so you and your team always know when benchmarks aren't green.
3. **Take action**. By default, Guardrails will alert on misconfigurations. You can use Guardrails' [quick actions](/guardrails/docs/guides/quick-actions) to fix mistakes, and you can set continuous enforcement.
4. **Controlled Attestations**. Some CIS controls are manual and require attestations. Track evidence of your reviews and set reminders to re-evaluate the control next audit period.
5. **Set Exceptions**. Not all organizations require every CIS Benchmark recommendation. Set time-based [exceptions](/guardrails/docs/concepts/policies/values-settings#exceptions.) to ignore recommendations per account or per resource when not applicable.
6. **Comprehensive Reporting**. See an always up-to-date view of your compliance status, from sections of the benchmark to per-resource compliance, in the Guardrails console. Generate detailed reports, including CSV exports for easy sharing and analysis.

## How to monitor AWS CIS v3.0.0 using Guardrails

### Install the AWS CIS mods
To get started, [install](/guardrails/docs/mods/guide/install) the [@turbot/cis](/guardrails/docs/mods/turbot/cis) and the [@turbot/aws-cisv3-0](/guardrails/docs/mods/aws/aws-cisv3-0) mods.

### Enable the AWS CIS Benchmark
Once installed, set the policy for `AWS > CIS v3.0` to `Check: All CIS Benchmarks`.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Enable AWS CIS v3.0.0 Policy" src="/images/blog/2024-05-real-time-aws-cis-benchmark/enable-aws-cis-v3-policy.png" />
</div>

Guardrails will immediately assess all applicable resources for compliance with AWS CIS Benchmark v3.0.0.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="AWS CIS v3.0.0 Benchmark" src="/images/blog/2024-05-real-time-aws-cis-benchmark/aws-cis-v3-benchmark.png" />
</div>

### Assess your CIS adherence
See control status for each benchmark section.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="AWS CIS v3.0.0 S3 recommendations" src="/images/blog/2024-05-real-time-aws-cis-benchmark/aws-cis-v3-s3-recommendations.png" />
</div>

In this example, `2.01.04 - Ensure that S3 Buckets are configured with 'Block public access (bucket settings)’` we can see all the AWS S3 buckets are reporting in `ALARM` and `OK` states.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="AWS CIS v3.0.0 2.1.4 recommendation" src="/images/blog/2024-05-real-time-aws-cis-benchmark/aws-cis-v3-2-1-4-recommendation.png" />
</div>

In some cases you may need to suppress or ignore the control on a resource. With any Guardrails policy you can set time-based exceptions for one or many resources.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="AWS CIS v3.0.0 Exception" src="/images/blog/2024-05-real-time-aws-cis-benchmark/aws-cis-v3-exception.png" />
</div>

In this case we set the policy to `Skip` the `acme-demo-5` bucket for 30 days. This bucket now shows it's in a skipped state and will be reassessed automatically after the expiration period.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="AWS CIS v3.0.0 Skipped Recommendation" src="/images/blog/2024-05-real-time-aws-cis-benchmark/aws-cis-v3-skipped-recommendation.png" />
</div>

### Instant AWS CIS alerts
When creating or updating cloud resources, Guardrails instantly provides feedback on the state of the recommendation. In this example we created `acme-demo-6` which was instantly discovered and evaluated for CIS compliance.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="AWS CIS v3.0.0 Alert" src="/images/blog/2024-05-real-time-aws-cis-benchmark/aws-cis-v3-alert.png" />
</div>

Beyond alerts in the Turbot Guardrails console, you and your team members can subscribe to [alerts](/guardrails/blog/2023/10/guardrails-notifications) via email, MS Teams or Slack alerts.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="AWS CIS v3.0.0 Alert in Slack" src="/images/blog/2024-05-real-time-aws-cis-benchmark/aws-cis-v3-alert-in-slack.png" />
</div>

### Take action on your CIS alerts

Guardrails [quick actions](/guardrails/blog/2022/08/quick-actions) provide direct links to the Guardrails console where you can immediately apply fixes. This enables workflows that keep human approvers in the loop. 

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="AWS CIS v3.0.0 Guardrails Quick Action" src="/images/blog/2024-05-real-time-aws-cis-benchmark/aws-cis-v3-guardrails-quick-action.png" />
</div>

In [Instant AWS CIS v3.0.0 Benchmark compliance](/guardrails/blog/2024/05/instant-aws-cis-v3-compliance) we show how you can go further by setting guardrails that continuously enforce CIS recommendations with no human intervention.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/c8j10t_0UZw"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Real-time AWS v3.0.0 CIS Benchmark assessments"
>
</iframe>
</div>

## Level up your security posture with Guardrails

Elevate your compliance game with Turbot Guardrails and experience real-time adherence to the AWS CIS v3.0.0 Benchmark. With real-time awareness of your security posture, and flexible reporting, Guardrails makes it quick and easy to address any deviation from best practices.

Turbot Guardrails also supports assessments for [Azure](/guardrails/blog/2024/05/real-time-azure-cis-benchmark) and [GCP](/guardrails/blog/2024/05/real-time-gcp-cis-benchmark). Try all the CIS Benchmarks, using the [14-day free trial](https://turbot.com/start), to gain instant feedback on your adherence to CIS recommendations.



[Center for Internet Security](https://cisecurity.org) (CIS) benchmarks for Azure are widely used by cloud teams as guidance to configure their Azure subscriptions securely. The latest iteration, Azure CIS v2.0.0, includes over 150 recommendations to evaluate your posture against best practices.

Turbot Guardrails now supports the [Azure CIS v2.0.0](/guardrails/changelog/azure-cisv2-0-v5-0-0) Benchmark. Use it to enable continuous monitoring of your cloud environment's security posture, and to ensure compliance with industry standards as resources are created and modified.

## Go beyond CIS reporting with Guardrails

Traditional compliance assessments rely on periodic scans of your environment, which only give you a point-in-time view of your security posture. But the cloud is dynamic, resources change rapidly, misconfigurations and security gaps can happen anytime.

Turbot Guardrails evaluates your Azure resources against the CIS v2.0.0 Benchmark in real-time. As resources are created or modified, Guardrails instantly assesses them for compliance and provides a view of your security posture that's always current.

Key Features of Turbot Guardrails' CIS Benchmark controls:
1. **Instant Evaluation**. As new AWS resources are created or modified, Guardrails immediately assesses them against the relevant CIS controls, providing instant feedback on your compliance status.
2. **Real-time Alerts**. Stay informed about critical changes in your compliance posture with real-time alerts. Guardrails can send [notifications](/guardrails/docs/guides/notifications) to Slack, MS Teams, and email, so you and your team always know when benchmarks aren't green.
3. **Take action**. By default, Guardrails will alert on misconfigurations. You can use Guardrails' [quick actions](/guardrails/docs/guides/quick-actions) to fix mistakes, and you can set continuous enforcement.
4. **Controlled Attestations**. Some CIS controls are manual and require attestations. Track evidence of your reviews and set reminders to re-evaluate the control next audit period.
5. **Set Exceptions**. Not all organizations require every CIS Benchmark recommendation. Set time-based [exceptions](/guardrails/docs/concepts/policies/values-settings#exceptions.) to ignore recommendations per account or per resource when not applicable.
6. **Comprehensive Reporting**. See an always up-to-date view of your compliance status, from sections of the benchmark to per-resource compliance, in the Guardrails console. Generate detailed reports, including CSV exports for easy sharing and analysis.

## How to monitor Azure CIS v2.0.0 using Guardrails

### Install the Azure CIS mods
To get started, [install](/guardrails/docs/mods/guide/install) the [@turbot/cis](/guardrails/docs/mods/turbot/cis) and the [@turbot/azure-cisv2-0](/guardrails/docs/mods/azure/azure-cisv2-0) mods.

### Enable the Azure CIS Benchmark
Once installed, set the policy for `Azure > CIS v2.0` to `Check: All CIS Benchmarks`.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Enable Azure CIS v2.0.0 Policy" src="/images/blog/2024-05-real-time-azure-cis-benchmark/enable-azure-cis-v2-policy.png" />
</div>

Guardrails will immediately assess all applicable resources for compliance to Azure CIS Benchmark v2.0.0.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Azure CIS v2.0.0 Benchmark" src="/images/blog/2024-05-real-time-azure-cis-benchmark/azure-cis-v2-benchmark.png" />
</div>

### Assess your CIS adherence
See control status for each benchmark section.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Azure CIS v2.0.0 Storage Accounts recommendations" src="/images/blog/2024-05-real-time-azure-cis-benchmark/azure-cis-v2-storage-recommendations.png" />
</div>

In this example, `3.01 - Ensure that 'Secure transfer required' is set to 'Enabled'` we can see all the Azure Storage Accounts are reporting in `ALARM` and `OK` states.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Azure CIS v2.0.0 3.1 recommendation" src="/images/blog/2024-05-real-time-azure-cis-benchmark/azure-cis-v2-3-1-recommendation.png" />
</div>

In some cases you may need to suppress or ignore the control on a resource, with any Guardrails policy you can set time based exceptions for one or many resources.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Azure CIS v2.0.0 Exception" src="/images/blog/2024-05-real-time-azure-cis-benchmark/azure-cis-v2-exception.png" />
</div>

In this case we set the policy to `Skip` the `acmedemoaz2` bucket for 30 days. This bucket now shows it's in a skipped state and will be reassessed automatically after the expiration period.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Azure CIS v2.0.0 Skipped Recommendation" src="/images/blog/2024-05-real-time-azure-cis-benchmark/azure-cis-v2-skipped-recommendation.png" />
</div>

### Instant Azure CIS alerts

When creating or updating cloud resources, Guardrails instantly provides feedback on the state of the recommendation. In this example we created `acmedemoaz3` which was instantly discovered and evaluated for CIS compliance.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Azure CIS v2.0.0 Alert" src="/images/blog/2024-05-real-time-azure-cis-benchmark/azure-cis-v2-alert.png" />
</div>

Beyond alerts in the Turbot Guardrails console, you and your team members can subscribe to [alerts](/guardrails/blog/2023/10/guardrails-notifications) via email, MS Teams or Slack alerts.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Azure CIS v2.0.0 Alert in Slack" src="/images/blog/2024-05-real-time-azure-cis-benchmark/azure-cis-v2-alert-in-slack.png" />
</div>

### Take action on your CIS alerts
Guardrails [quick actions](/guardrails/blog/2022/08/quick-actions) provide direct links to the Guardrails console where you can immediately apply fixes. This enables workflows that keep human approvers in the loop. 

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Azure CIS v2.0.0 Guardrails Enforce Policy" src="/images/blog/2024-05-real-time-azure-cis-benchmark/azure-cis-v2-guardrails-enforce-policy.png" />
</div>

In [Instant AWS CIS v3.0.0 Benchmark compliance](/guardrails/blog/2024/05/instant-aws-cis-v3-compliance), using AWS as the example, we show how you can go further by setting guardrails to continuously enforce CIS recommendations with no human intervention.

## See it in action

<div className="flex justify-center">
<iframe 
class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/rUBUazeCNWs"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Real-time Azure v2.0.0 CIS Benchmark assessments"
>
</iframe>
</div>

## Level up your security posture with Guardrails

Elevate your compliance game with Turbot Guardrails and experience real-time adherence to the Azure CIS v2.0.0 Benchmark. With instant visibility into your security posture and flexible reporting options, Guardrails makes it easier and faster to address any deviations from best practices.

Turbot Guardrails also supports the latest CIS assessments for [AWS](/guardrails/blog/2024/05/real-time-aws-cis-benchmark) and [GCP](/guardrails/blog/2024/05/real-time-gcp-cis-benchmark). Try all the CIS Benchmarks, using the [14-day free trial](https://turbot.com/start), to gain instant feedback on your adherence to CIS recommendations.


[Center for Internet Security](https://cisecurity.org) (CIS) benchmarks for GCP are widely used by cloud teams as guidance to configure their GCP projects securely. The latest iteration, GCP CIS v2.0.0, includes over 80 recommendations to evaluate your posture against best practices.

Turbot Guardrails now supports the [GCP CIS v2.0.0](/guardrails/changelog/gcp-cisv2-0-v5-0-0) Benchmark. Use it to enable continuous monitoring of your cloud environment's security posture, and to ensure compliance with industry standards as resources are created and modified.

## Go beyond CIS reporting with Guardrails

Traditional compliance assessments rely on periodic scans of your environment, which only give you a point-in-time view of your security posture. But the cloud is dynamic, resources change rapidly, misconfigurations and security gaps can happen anytime.

Turbot Guardrails evaluates your GCP resources against the CIS v2.0.0 Benchmark in real-time. As resources are created or modified, Guardrails instantly assesses them for compliance, giving you an always up-to-date view of your security posture.

Key Features of Turbot Guardrails' CIS Benchmark controls:
1. **Instant Evaluation**. As new AWS resources are created or modified, Guardrails immediately assesses them against the relevant CIS controls, providing instant feedback on your compliance status.
2. **Real-time Alerts**. Stay informed about critical changes in your compliance posture with real-time alerts. Guardrails can send [notifications](/guardrails/docs/guides/notifications) to Slack, MS Teams, and email, so you and your team always know when benchmarks aren't green.
3. **Take action**. By default, Guardrails will alert on misconfigurations. You can use Guardrails' [quick actions](/guardrails/docs/guides/quick-actions) to fix mistakes, and you can set continuous enforcement.
4. **Controlled Attestations**. Some CIS controls are manual and require attestations. Track evidence of your reviews and set reminders to re-evaluate the control next audit period.
5. **Set Exceptions**. Not all organizations require every CIS Benchmark recommendation. Set time-based [exceptions](/guardrails/docs/concepts/policies/values-settings#exceptions.) to ignore recommendations per account or per resource when not applicable.
6. **Comprehensive Reporting**. See an always up-to-date view of your compliance status, from sections of the benchmark to per-resource compliance, in the Guardrails console. Generate detailed reports, including CSV exports for easy sharing and analysis.


## How to monitor GCP CIS v2.0.0 using Guardrails

### Install the GCP CIS mods
To get started, [install](/guardrails/docs/mods/guide/install) the [@turbot/cis](/guardrails/docs/mods/turbot/cis) and the [@turbot/gcp-cisv2-0](/guardrails/docs/mods/gcp/gcp-cisv2-0) mods.

### Enable the GCP CIS Benchmark
Once installed, set the policy for `GCP > CIS v2.0` to `Check: All CIS Benchmarks`.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Enable GCP CIS v2.0.0 Policy" src="/images/blog/2024-05-real-time-gcp-cis-benchmark/enable-gcp-cis-v2-policy.png" />
</div>

Guardrails will immediately assess all applicable resources for compliance to GCP CIS Benchmark v2.0.0.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GCP CIS v2.0.0 Benchmark" src="/images/blog/2024-05-real-time-gcp-cis-benchmark/gcp-cis-v2-benchmark.png" />
</div>

### Assess your CIS adherence
See control status for each benchmark section.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GCP CIS v2.0.0 Storage Accounts recommendations" src="/images/blog/2024-05-real-time-gcp-cis-benchmark/gcp-cis-v2-storage-recommendations.png" />
</div>

In this example, `5.02 - Ensure That Cloud Storage Buckets Have Uniform Bucket-Level Access Enabled` we can see all the GCP Storage Accounts are reporting in `ALARM` and `OK` states.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GCP CIS v2.0.0 5.2 recommendation" src="/images/blog/2024-05-real-time-gcp-cis-benchmark/gcp-cis-v2-5-2-recommendation.png" />
</div>

In some cases you may need to suppress or ignore the control on a resource, with any Guardrails policy you can set time based exceptions for one or many resources.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GCP CIS v2.0.0 Exception" src="/images/blog/2024-05-real-time-gcp-cis-benchmark/gcp-cis-v2-exception.png" />
</div>

In this case we set the policy to `Skip` the `acme-demo-turbot-gcp-2` bucket for 30 days. This bucket now shows it's in a skipped state and will be reassessed automatically after the expiration period.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GCP CIS v2.0.0 Skipped Recommendation" src="/images/blog/2024-05-real-time-gcp-cis-benchmark/gcp-cis-v2-skipped-recommendation.png" />
</div>

### Instant GCP CIS alerts
When creating or updating cloud resources, Guardrails instantly provides feedback on the state of the recommendation in the console. In this example we created `acme-demo-gcp` which was instantly discovered and evaluated for CIS compliance within seconds.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GCP CIS v2.0.0 Alert" src="/images/blog/2024-05-real-time-gcp-cis-benchmark/gcp-cis-v2-alert.png" />
</div>

Beyond alerts in the Turbot Guardrails console, you and your team members can subscribe to [alerts](/guardrails/blog/2023/10/guardrails-notifications) via email, MS Teams or Slack alerts.


<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GCP CIS v2.0.0 Alert in Slack" src="/images/blog/2024-05-real-time-gcp-cis-benchmark/gcp-cis-v2-alert-in-slack.png" />
</div>

### Take action on your CIS alerts
Guardrails [quick actions](/guardrails/blog/2022/08/quick-actions) provide direct links to the Guardrails console where you can immediately apply fixes. This enables workflows that keep human approvers in the loop. 


<div style={{width:"100%", textAlign:"center"}}>
  <img alt="GCP CIS v2.0.0 Guardrails Enforce Policy" src="/images/blog/2024-05-real-time-gcp-cis-benchmark/gcp-cis-v2-guardrails-enforce-policy.png" />
</div>

In [Instant AWS CIS v3.0.0 Benchmark compliance](/guardrails/blog/2024/05/instant-aws-cis-v3-compliance), using AWS as the example, we show how you can go further by setting guardrails that continuously enforce CIS recommendations with no human intervention.

## See it in action
p
<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/T_gEABDz_0I"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Real-time GCP v2.0.0 CIS Benchmark assessments"
>
</iframe>
</div>

## Level up your security posture with Guardrails

Elevate your compliance game with Turbot Guardrails and experience real-time adherence to the GCP CIS v2.0.0 Benchmark. With real-time awareness of your security posture, and flexible reporting, Guardrails makes it quick and easy to address any deviation from best practices.

Turbot Guardrails also supports the latest CIS assessments for [AWS](/guardrails/blog/2024/05/real-time-aws-cis-benchmark) and [Azure](/guardrails/blog/2024/05/real-time-azure-cis-benchmark). Try all the CIS Benchmarks, using the [14-day free trial](https://turbot.com/start), to gain instant feedback on your adherence to CIS recommendations.



When managing cloud configurations, Cloud Security Posture Management (CSPM) has become a well-known approach for identifying and remediating security and compliance issues in cloud environments. As more organizations adopt Kubernetes for their containerized workloads, a similar need arises for managing the security posture of Kubernetes clusters. Enter Kubernetes Security Posture Management (KSPM), a critical component of a Cloud-Native Application Protection Platform (CNAPP) strategy. KSPM solutions continuously monitor Kubernetes clusters, enforce security best practices, and detect policy violations. By automating the discovery and remediation of misconfigurations and compliance issues, KSPM helps organizations maintain a strong security posture across their Kubernetes deployments.

[Turbot Guardrails](https://turbot.com/guardrails) now expands its CSPM and CNAPP solutions for AWS, Azure, GCP, and ServiceNow, with KSPM guardrails for Kubernetes.

## Kubernetes Security Posture Management with Turbot Guardrails

Turbot Guardrails evaluates your Kubernetes resources as run-time changes occur, instantly assesses them for compliance, and provides a view of your security posture that's always current

**Key Features of Turbot Guardrails' KSPM:**

* **Real-Time Configuration Discovery:** Guardrails seamlessly integrates with your Kubernetes clusters, automatically discovering all configurations in real-time. This provides a comprehensive inventory of your Kubernetes assets, enabling you to track, search, and visualize changes across your clusters and subservices such as ConfigMaps, Deployments, Namespaces, Nodes, Pods, ReplicaSets, etc.
* **Continuous Policy Evaluation:** With Guardrails' powerful policy engine, your Kubernetes configurations are continuously evaluated against predefined security, compliance, and operational best practice policies. Any deviations or policy violations are instantly detected. Now you have real-time visibility into your Kubernetes security and operations posture.
* **Instant Alerts and Notifications:** When a policy violation or security issue is identified, Guardrails generates instant alerts and notifications. These can be sent to the Guardrails console, email, Slack, MS Teams, or any API endpoint, ensuring that your team is immediately informed and can take swift action to remediate the issue.
* **Comprehensive CSPM + KSPM coverage:** Guardrails integrates with any Kubernetes cluster, regardless of where its hosted. In addition Guardrails provides extensive security and operational controls for AWS, Azure, GCP, and ServiceNow, so you can centralize your security posture management efforts across your entire cloud infrastructure + Kubernetes clusters.

## How to enable KSPM Guardrails

### Install the Kubernetes mod
To get started, [install](/guardrails/docs/mods/guide/install) the [@turbot/kubernetes](/guardrails/docs/mods/turbot/kubernetes) mod.

### Connect your first Kubernetes cluster

Guardrails can connect with your Kubernetes cluster regardless of where it's hosted: in a cloud service provider (e.g. AWS, Azure, GCP), on-premise, or local to your device. 

Once the mod is installed, your Account Import page will now have an option to connect your Kubernetes clusters to Turbot Guardrails. The connect screen will generate a script to set up or derive your own to setup the Guardrails Kubernetes agent in your Kubernetes cluster using [kubectl](https://kubernetes.io/docs/tasks/tools) and [Helm](https://helm.sh/docs/intro/install). The enrollment secret can be generated via the API as well, and automatically rotate based on your defined [enrollment secret expiration policies](https://turbot.com/guardrails/docs/mods/turbot/osquery/policy#turbot--workspace--osquery--enroll-secret-expiration).

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Connect a Kubernetes cluster to Guardrails" src="/images/blog/2024-05-kubernetes-security-posture-management/kubernetes-cluster-to-guardrails.png" />
</div>

Guardrails will immediately discover the Kubernetes cluster and its related subservices such as ConfigMaps, Deployments, Namespaces, Nodes, Pods, ReplicaSets, etc. This will instantly update the Guardrails Configuration Management Database (CMDB) with the current configurations in run-time.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Kubernetes cluster configurations in Guardrails CMDB" src="/images/blog/2024-05-kubernetes-security-posture-management/kubernetes-cluster-in-guardrails-cmdb.png" />
</div>

### Assess your security and ops posture
Guardrails will detect and alert on security misconfigurations in your Kubernetes resources. What's more, it can check for operational best practices like correctly-provisioned and properly-labeled resources.

#### Security posture
A critical aspect of Kubernetes security posture management is detecting security misconfigurations. One common misconfiguration is running containers with privileged mode enabled. Privileged mode is a security context setting in Kubernetes that grants a container extensive permissions on the host system, essentially bypassing most security restriction which is a significant security risk because it increases the attack surface, weakens container isolation, and allows containers to perform privileged operations on the host.

Once the cluster was connected to Guardrails, Guardrails instantly checked whether the `nginx-deployment` resource was configured with `securityContext: true`.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Kubernetes deployment with privileged access alert" src="/images/blog/2024-05-kubernetes-security-posture-management/kubernetes-deployment-privileged-access-alert.png" />
</div>

#### Operational best practices
Guardrails continuously monitors any Kubernetes configuration drift, including ReplicaSets. Guardrails' policies to assess [Approved](https://turbot.com/guardrails/docs/concepts/guardrails/approved) configurations can be customized for any use case. For example, let's say you have a policy requiring ReplicaSets to be configured with 2 - 20 replicas. Using the `Kubernetes > ReplicaSet > Approved > Custom` policy we can configure a template of custom thresholds and alert messages.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Kubernetes ReplicaSet Approved Policy" src="/images/blog/2024-05-kubernetes-security-posture-management/kubernetes-replicaset-approved-policy.png" />
</div>

When the number of replicas drops from 3 to 1, Guardrails will detect this drift immediately and capture the diff history in the CMDB.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Kubernetes ReplicaSet Configuration Drift" src="/images/blog/2024-05-kubernetes-security-posture-management/kubernetes-replicaset-config-drift.png" />
</div>

As the CMDB is updated, Guardrails identifies that the replicas are below the minimum size requirements:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Kubernetes ReplicaSet Configuration Alert" src="/images/blog/2024-05-kubernetes-security-posture-management/kubernetes-replicaset-config-drift-alert.png" />
</div>

#### Instant Guardrails alerts
In addition to alerts in the Turbot Guardrails console, you and your team members can subscribe to alerts via email, MS Teams, or Slack.

The alerts shown above also went to to Slack, so the team can know immediately and react.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Kubernetes Configuration Alerts in Slack" src="/images/blog/2024-05-kubernetes-security-posture-management/kubernetes-replicaset-config-alert-slack.png" />
</div>

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/HKlW7oPPW-I"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Kubernetes Security Posture Management with Turbot Guardrails"
>
</iframe>
</div>

## Elevate your Kubernetes security posture with Turbot Guardrails

Kubernetes Security Posture Management (KSPM) is an essential aspect of a robust CNAPP strategy to proactively identify and mitigate security risks in their Kubernetes clusters.

With Turbot Guardrails' KSPM capabilities, you can automate the discovery, evaluation, and remediation of security and compliance issues. Real-time configuration drift detection, continuous policy evaluation, [real-time syncing to ServiceNow](https://turbot.com/guardrails/blog/2024/05/servicenow-kubernetes-discovery) and instant alerts provide the visibility and control needed to maintain a strong security and operational best practices posture.

[Get started](https://turbot.com/start) with a 14-day free trial of Turbot Guardrails to experience [KSPM Guardrails](https://turbot.com/guardrails/docs/integrations/kubernetes) for your Kubernetes deployments.



Turbot Guardrails just made it a lot easier to discover Kubernetes resources in ServiceNow's CMDB. The integration requires no additional ServiceNow modules or Kubernetes services. Instead, Turbot Guardrails enables a direct, low-cost way to get real-time Kubernetes discovery into ServiceNow.

This automated discovery integration has two key benefits. First, it enhances the accuracy of your ServiceNow CMDB data by surfacing comprehensive and timely [Kubernetes resource insights](https://turbot.com/guardrails/blog/2024/05/kubernetes-security-posture-management). Second, it significantly reduces the cost and overhead compared to native ServiceNow cloud discovery capabilities. This makes Turbot Guardrails an ideal solution for those new to cloud discovery or looking to augment existing ServiceNow Kubernetes integrations. Users can tap into broader coverage and more precise Kubernetes CMDB data without added complexity.

## Why an automated Kubernetes discovery for ServiceNow?

Manual and scheduled discovery of Kubernetes resources leads to inaccurate ServiceNow data. Kubernetes resources change too quickly for legacy discovery methods to keep up. As a result, CMDBs end up with blindspots and incorrect configurations.

When you automate continuous Kubernetes discovery, and integrate resources into ServiceNow, you:

* Eliminate blindspots by covering more Kubernetes sub-services beyond the native integrations
* Prevent missing and stale data with instant updates when resources change
* Enable flexible control over which resource data syncs to which CMDB CI tables
* Surface Kubernetes labels and metadata to enrich CMDB context
* Archive historical records of provisioned resources that get deleted

## Augmenting native Kubernetes discovery in ServiceNow

Native ServiceNow connectors and Kubernetes integrations are available to enable discovery of core Kubernetes services such as ConfigMap, Deployment, Namespace, Node, Pod, ReplicaSet, etc. But these have limitations including:

* Extra licensing and services
* Limited resource coverage
* Reliance on scheduled batch jobs rather than real-time scans
* Need for professional services and customization

The Turbot Guardrails Kubernetes ServiceNow integration augments native capabilities by:

* Point-and-click integration setup
* Discovering more resource types out of the box
* Reducing licensing dependency and service overhead
* Automatically updating CMDBs instantly when resources change
* Providing flexible control to configure synced data as needed
* Centralized management for other discovery scopes; [AWS](https://turbot.com/guardrails/blog/2024/01/servicenow-aws-discovery), [Azure](https://turbot.com/guardrails/blog/2024/03/servicenow-azure-discovery), & [GCP](https://turbot.com/guardrails/blog/2024/03/servicenow-gcp-discovery)

By combining automated discovery from Turbot Guardrails with ServiceNow and Kubernetes, you can accurately sync Kubernetes configs into ServiceNow's CMDB in real-time without added cost and complexity.

## Configuring automated Kubernetes cloud discovery for ServiceNow CMDB

Each Kubernetes resource type can be configured to sync to the ServiceNow CMDB. Most often you would set the scope of the policy across many Kubernetes resources from all your Kubernetes clusters. In this example we will show how to enable syncing Kubernetes ReplicaSets.

Set the Turbot Guardrails policy to “Enforce: Sync” and apply to all or specific Kubernetes clusters:

<img src="/images/blog/2024-05-servicenow-kubernetes-discovery/kubernetes-sync-policy.png" style={{ boxShadow: 'none' }}></img>

For the integration-enabled Kubernetes cluster, the following Kubernetes resources will be in scope for the Kubernetes discovery:

<img src="/images/blog/2024-05-servicenow-kubernetes-discovery/kubernetes-replicasets.png" style={{ boxShadow: 'none' }}></img>

Instantly the Kubernetes resources will be added to the associated ServiceNow CMDB table:

<img src="/images/blog/2024-05-servicenow-kubernetes-discovery/servicenow-cmdb-table-kubernetes-replicasets.png" style={{ boxShadow: 'none' }}></img>

As Kubernetes resources are added, updated, or deleted, Guardrails handles the configuration drift and keeps the ServiceNow CMDB updated.

For example, when an Kubernetes resource changes, Guardrails captures the configuration drift and updates the ServiceNow CMDB:

<img src="/images/blog/2024-05-servicenow-kubernetes-discovery/kubernetes-replicaset-config-drift.png" style={{ boxShadow: 'none' }}></img>

Kubernetes resource deletion can be managed as a complete synchronization — where the record in ServiceNow is deleted as well — or archived to retain its record with an archive status.

## Map Kubernetes resource details to ServiceNow CMDB CI tables

Each cloud resource type can be mapped to new or existing ServiceNow CMDB CI tables. You can also extend existing tables. In this example, the Kubernetes ReplicaSet resource type maps to a new `cmdb_ci_kubernetes_replicaset`.

```yaml
table:
  name: kubernetes_replicaset
  label: Kubernetes > ReplicaSet > 
  extendsTable: cmdb_ci
```

Mappings of which columns to create are simple to define in the Table Definition policy:

```yaml
columns:
- name: annotations
  size: 1000
  type: string
  label: Annotations
- name: cluster_name
  type: string
  label: Cluster Name
- name: labels
  size: 10000
  type: string
  label: Labels
- name: namespace
  label: Namespace
- name: replicas
  type: integer
  label: Replicas
- name: name
  size: 500
  type: string
  label: Name
- name: deployment
  size: 500
  type: string
  label: Deployment
```
Any data from Kubernetes clusters or Turbot Guardrails can be added. To associate more fields, add more ServiceNow CMDB table columns:

```yaml
- name: cluster_uid
  size: 500
  type: string
  label: Cluster UID
```

Adding custom ServiceNow CMDB CI fields is common. In this case we want to add a CMDB CI column to capture the Business Owner associated to the Kubernetes resource:

```yaml
- name: business_owner
  size: 500
  type: string
  label: Business Owner
```

Elevating labels into their own columns is possible, in this case we have a Kubernetes label for application name.

```yaml
- name: app_name
  size: 500
  type: string
  label: Application Name
```

To populate the added ServiceNow CMDB CI columns dynamically, the Configuration Item policy can define where the data comes from. In this case, this information can be pulled from the Kubernetes resource details such as the `cluster.uid`, data from the Guardrails CMDB for the `business_owner` field, and elevating the app label to `app_name`:

```
annotations: {{ $.resource.data.annotations | dump | safe }}
cluster_name: {{ $.resource.metadata.kubernetes.clusterName | dump | safe }}
deployment: {{ $.resource.data.owner_references[0].name | dump | safe }}
labels: {{ $.resource.data.labels | dump | safe }}
name: {{ $.resource.data.name | dump | safe }}
namespace: {{ $.resource.data.namespace | dump | safe }}
replicas: {{ $.resource.data.replicas | dump | safe }}

cluster_uid: {{ $.resource.metadata.kubernetes.clusterUid | dump | safe }}
business_owner: {{ $.folder.title }}
app_name: {{ $.resource.turbot.tags.app }}
```

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/JyYJBSsUl8g"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Kubernetes Discovery for ServiceNow with Turbot Guardrails"
>
</iframe>
</div>

## Modernize your ServiceNow CMDB with automated Kubernetes discovery

[Get started](https://turbot.com/start) with a 14-day free trial of Turbot Guardrails to experience [automated discovery](https://turbot.com/guardrails/docs/integrations/servicenow) and sync for Kubernetes resources into your ServiceNow instance. Real-time, comprehensive visibility within minutes at just $0.05 per resource per month.



Many DevOps challenges follow similar patterns. So we're creating a set of off-the-shelf Flowpipe mods to address these common scenarios head-on, starting with [AWS Thrifty](https://hub.flowpipe.io/mods/turbot/aws_thrifty). For example, suppose you want to detect and release unattached elastic IP addresses. Here are three basic patterns for doing that.

- **Wizard (with approval)**: Run Flowpipe on the command line, detect unattached EIPs, act on the choices `Skip` or `Release`.

- **Scheduled (with approval)**: Run on a schedule, detect, act on the choices `Skip` or `Release`.

- **Scheduled (no approval)**: Run on a schedule, detect and automatically `Release`.

You'll write no code to do any of those things, instead you'll configure mod variables such as:

- The database (Steampipe or other) that feeds the pipeline with the output of [a SQL query](https://hub.powerpipe.io/mods/turbot/aws_thrifty/controls/control.unattached_eips?context=benchmark.network#sql).

- A [notifier](/docs/reference/config-files/notifier) that binds the mod to a communication channel — an [integration](/docs/reference/config-files/integration) — that might be [email](/docs/reference/config-files/integration/email), [Slack](/docs/reference/config-files/integration/slack), or [MS Teams](/docs/reference/config-files/integration/msteams). 

- One more `approvers`, which are notifiers that obtain [action/approval decisions](/blog/human-interaction).

- Whether to run on a [trigger](/blog/query-trigger).

- The default action, e.g. `notify` or `release`.

## Run as a wizard, with Slack-based approval

To run this pipeline using Slack to receive notifications and take actions, [install the mod](https://hub.flowpipe.io/mods/turbot/aws_thrifty), start Steampipe (`steampipe service start`), configure a [Slack integration and notifier](/docs/reference/config-files/integration/slack#setting-up-as-slack-integration), and start the [Flowpipe server](/docs/run/server).

Now run this command.

```bash
flowpipe pipeline run detect_and_correct_vpc_eips_unattached \
  --arg approvers='["slack"]' \
  --arg host=local
```

In this case the pipeline detects one unattached EIP, and waits for a decision to be made in Slack channel.

<div>
  <img width="90%" alt="slack awaiting decision" src="/images/blog/2024-05-devsecops-mod-library/unattached_eip_1.png"/>
</div>

If we choose `Release`, the pipeline calls a [utility pipeline](https://hub.flowpipe.io/mods/turbot/aws/pipelines/aws.pipeline.release_eip) in the [AWS library mod](https://hub.flowpipe.io/mods/turbot/aws) to release the EIP, and the Slack message changes to confirm the action. 

<div>
  <img width="90%" alt="slack confirmed decision" src="/images/blog/2024-05-devsecops-mod-library/unattached_eip_2.png"/>
</div>

If there are more than one unattached EIPs, these Slack interactions will occur serially. To batch them, you can switch the `max_concurrency` from the default, which is 1, to your desired batch size. 

<div>
  <img width="90%" alt="slack confirmed decision" src="/images/blog/2024-05-devsecops-mod-library/unattached_eip_3.png"/>
</div>

Here `max_concurrency` is 3; we've chosen to release the first two but take no action on the third.


## Detect on a schedule, respond with human input

To use this pattern, activate a trigger to run the same query for unattached EIPs, but on a schedule instead of interactively. Because this requires setting a few variables, we'll package them into a file named for this scenario: `scheduled-interactive.fpvars`.

```hcl
notifier="slack"
approvers=["slack"]
vpc_eips_unattached_trigger_enabled=true
vpc_eips_unattached_trigger_schedule="* * * * *"
```

And then run the server, pointing to the `.fpvars` file and the server's public endpoint.

```
flowpipe server --var-file scheduled-interactive.fpvars --base-url=https://feline-just-gorilla.ngrok-free.app/ 
```

Note that if the trigger's query finds an unattached EIP that you choose to `Skip`, it won't be detected again because Flowpipe's [query trigger](/blog/query-trigger) remembers query results it has already seen. But if you change your mind and want to release an EIP that you skipped, you can use the wizard mode to do that. 

## Detect on a schedule, respond automatically

To use this pattern, we'll use a slightly different package of variables: `scheduled-automatic.fpvars`.

```hcl
notifier="slack"
approvers=[]
vpc_eips_unattached_default_action="release" 
vpc_eips_unattached_trigger_enabled=true
vpc_eips_unattached_trigger_schedule="* * * * *"
```

When `approvers` is the empty list, Flowpipe automically use the default action. Normally that's `notify` but here we override to `release`. Now, unattached EIPs detected on each run of the pipeline will be automatically released. It's as if there's a robot clicking the `Release` button for each detected EIP.

## Configuration-driven workflow

A detect-and-corect mod like this one doesn't require you to write any SQL or HCL. It's a tool that you configure and then use to perform a range of tasks you'd like to automate, with or without approval. Give `flowpipe-mod-aws-thrifty` a try, and please [let us know](https://turbot.com/community/join) how it goes.

## See it in action

<div className="flex justify-center">

<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/PH4YUun2lDU" 
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Control AWS costs with 29 FinOps workflows"
>
</iframe>
</div>




Steampipe now accelerates queries that involve only one of many aggregated connections. Here's a query of **~250,000 SQS queues spread across 10 AWS accounts** before the new optimization:

```sql
select count(*) from aws_sqs_queue where account_id = '767398119043'
```
```
Time: 18.4s. Rows returned: 1. Rows fetched: 248,456. Hydrate calls: 248,456. Scans: 10. Connections: 10.
```

Although the `767398119043` account is the only one of interest, Steampipe didn't know that, so it queried all 10 connections. Those subqueries run in parallel, and return a huge amount of data in just 18 seconds! But the query would be even faster, and lighter on API load, if Steampipe only needed to fetch rows for the target account.

Here are stats for the same query using v0.23.0 and the latest AWS plugin, which includes support for `connection-level qualifiers`:

```
Time: 2.3s. Rows returned: 1. Rows fetched: 8,500. Hydrate calls: 8,500.
```

In this case it's **8 times faster and fetches 96.5% fewer rows!** The query skips nine irrelevant connections, targets the only one of interest, fetches vastly fewer rows and finishes in a few seconds.

## New .verbose timing

To see these results in more detail, we'll turn on another new feature, [verbose timing](https://steampipe.io/docs/reference/dot-commands/timing#timing). Here's the verbose view without connection qualifers, showing that Steampipe fetches hundreds of thousands of rows that the query didn't require.

```
Time: 17.4s. Rows returned: 8,500. Rows fetched: 248,456. Hydrate calls: 0. Scans: 10. Connections: 10.

Scans:
  1) aws_sqs_queue.aws_pipeling_001: Time: 10.3s. Fetched: 119,209. Hydrates: 0. Quals: account_id=767398119043.
  2) aws_sqs_queue.aws_pipeling_002: Time: 6.6s. Fetched: 35,669. Hydrates: 0. Quals: account_id=767398119043.
  3) aws_sqs_queue.aws_pipeling_003: Time: 6.3s. Fetched: 34,070. Hydrates: 0. Quals: account_id=767398119043.
  4) aws_sqs_queue.aws_pipeling_005: Time: 3.9s. Fetched: 8,500. Hydrates: 0. Quals: account_id=767398119043.
  5) aws_sqs_queue.aws_pipeling_010: Time: 3.9s. Fetched: 8,500. Hydrates: 0. Quals: account_id=767398119043.
  6) aws_sqs_queue.aws_pipeling_009: Time: 3.9s. Fetched: 8,500. Hydrates: 0. Quals: account_id=767398119043.
  7) aws_sqs_queue.aws_pipeling_008: Time: 3.9s. Fetched: 8,500. Hydrates: 0. Quals: account_id=767398119043.
  8) aws_sqs_queue.aws_pipeling_007: Time: 3.9s. Fetched: 8,500. Hydrates: 0. Quals: account_id=767398119043.
  9) aws_sqs_queue.aws_pipeling_004: Time: 3.9s. Fetched: 8,498. Hydrates: 0. Quals: account_id=767398119043.
  10) aws_sqs_queue.aws_pipeling_006: Time: 3.8s. Fetched: 8,500. Hydrates: 0. Quals: account_id=767398119043.
```

With connection qualifiers in play, we can see that Steampipe targets the relevant connection with surgical precision.

```
Time: 3.4s. Rows returned: 1. Rows fetched: 8,500. Hydrate calls: 8,500.

Scans:
  1) aws_sqs_queue.aws_pipeling_004: Time: 3.4s. Fetched: 8,500. Hydrates: 8,500. Quals: account_id=767398119043.
```

Steampipe was already fast, even when fetching tons more data than necessary. Now that it can fetch only what's required for a connection-qualified query, it's blindingly fast.

## Plugins and mods updated to use connection qualifiers

So far we've updated the [AWS](https://hub.steampipe.io/plugins/turbot/aws), [Azure](https://hub.steampipe.io/plugins/turbot/azure), [GCP](https://hub.steampipe.io/plugins/turbot/gcp), [OCI](https://hub.steampipe.io/plugins/turbot/oci), [AliCloud](https://hub.steampipe.io/plugins/turbot/alicloud), [Kubernetes](https://hub.steampipe.io/plugins/turbot/kubernetes), [AzureAD](https://hub.steampipe.io/plugins/turbot/azuread), and [Microsoft365](https://hub.psteampipe.io/plugins/turbot/Microsoft365) plugins with support for connection qualifiers, and we'll continue to roll out these updates across the suite.

And we've updated the following mods to take advantage of the updated plugins: [AWS Insights](https://hub.powerpipe.io/mods/turbot/aws_insights), [Azure Insights](https://hub.powerpipe.io/mods/turbot/azure_insights), [GCP Insights](https://hub.powerpipe.io/mods/turbot/gcp_insights), [OCI Insights](https://hub.powerpipe.io/mods/turbot/oci_insights), and [Kubernetes Insights](https://hub.powerpipe.io/mods/turbot/kubernetes_insights). Insight mods, particularly detail screens and relationship graphs, see huge performance gains from these changes since they start from a single resource in a single account - watch the demo video below to see it in action.

## Steampipe's journey of optimization

This latest enhancement builds on a series of optimizations that have made Steampipe ever faster and more capable. [v0.21.0](https://steampipe.io/changelog/steampipe-cli-v0-21-0) added [rate limiters](https://steampipe.io/blog/memory-management-rate-limiters-diagnostics) which can speed up a query that triggers excessive throttling. [v0.20.0](https://steampipe.io/blog/release-0-20-0) made startup much faster. [v0.16.0](https://steampipe.io/blog/release-0-16-0) introduced a new architecture that made the time to first row much faster for environments that aggregate many connections.

What's next? It's hard to say, but every time we think we've gotten to the end of the optimization journey we find new ways to make it faster. Meanwhile, enjoy the latest speedup and [let us know](https://turbot.com/community/join) how it goes.

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/kXylCw4Ef7Q"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Connection-level qualifiers for faster queries and lower API load"
>
</iframe>
</div>




Our announcements this week span the breadth of Turbot offerings from Guardrails' ServiceNow discovery for Azure and GCP, to Flowpipe's query trigger and interactive inputs, to our newest open-source product: [Powerpipe](https://powerpipe.io).

## Watch the Wrap Party

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/0YbSwm1ELC8"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Turbot Launch Week 3 Wrap Party"
>
</iframe>
</div>
<div className="text-2xl font-bold pt-10">Full Week Recap</div>

## Monday Mar 4: ServiceNow discovery for Azure and GCP, gp3 migration, custom roles and groups

Building on the ServiceNow integration announced [last time](https://turbot.com/launch-week/02), Turbot Guardrails just made it a lot easier to discover [Azure](https://turbot.com/guardrails/blog/2024/03/servicenow-azure-discovery) and [GCP](https://turbot.com/guardrails/blog/2024/03/servicenow-gcp-discovery) resources in ServiceNow.

We've also enhanced the EBS volume controls to enable [automatic migration to `gp3`](https://turbot.com/guardrails/blog/2024/03/automate-ebs-optimization) for optimized performance and 20% cost savings.

Guardrails' multi-cloud identity management features have improved too. We've extended our time-based AWS IAM user management to include bringing your own [custom IAM groups](https://turbot.com/guardrails/blog/2024/03/aws-federated-access-custom-groups) for Guardrails to manage.

## Tuesday Mar 5: Flowpipe query triggers and interactive inputs

Flowpipe already had a [query step](https://flowpipe.io/docs/flowpipe-hcl/step/query) to acquire data from Steampipe or another database. Now you can also use [query trigger](https://flowpipe.io/blog/query-trigger) to watch a database table, fire when query results indicate new, deleted, or changed rows, and pass those changes to a pipeline.

With Flowpipe's new [input step](https://flowpipe.io/blog/human-interaction) connects your data, systems and now teams in your DevOps workflow. You can pause a pipeline to gather input from people, then continue when they respond. Supported interactions, such as pressing an `Approve` button or choosing a value from a list, work the same way in email or Slack.

## Wednesday Mar 6: Introducing Powerpipe!

[Powerpipe](https://powerpipe.io/blog/introducing-powerpipe) visualizes data on dashboards, and validates it with benchmarks, via the [mature ecosystem of mods](https://hub.powerpipe.io/mods) that Steampipe birthed. It's the same [dashboards-as-code](https://steampipe.io/blog/dashboards-as-code) technology that Steampipe users have enjoyed, plus new filtering and grouping for benchmarks. 

While Powerpipe can use Steampipe for these purposes, though, it isn't tied to Steampipe. Powerpipe's [database](https://powerpipe.io/docs/run#selecting-a-database) argument is the key that unlocks all your, data wherever it happens to live: in Steampipe, or in your own Postgres, SQLite,  MySQL, or DuckDB database. Now you can visualize all that data as [tables](https://powerpipe.io/docs/powerpipe-hcl/table), [charts](https://powerpipe.io/docs/powerpipe-hcl/chart), and [relationship graphs](https://powerpipe.io/docs/powerpipe-hcl/graph). And you can validate it with custom [benchmarks](https://powerpipe.io/docs/powerpipe-hcl/benchmark) and [controls](https://powerpipe.io/docs/powerpipe-hcl/control). 

In [Steampipe unbundled](https://steampipe.io/blog/steampipe-unbundled) we explain why we decoupled Steampipe's dashboards-as-code and benchmark features — the "mod" layer — from the Steampipe core. Unix philosophy holds that a software tool should do one thing well, and combine readily with others that abide by the same principle. Here's how Turbot's family of open-source products now aligns with that venerable rule.

|Component|Does one thing well|
|--|--|
| Steampipe| select * from cloud;|
| Flowpipe| Workflow for DevOps. |
| Powerpipe| Dashboards for DevOps. |

## Thursday Mar 7: Filtering and grouping benchmarks in Pipes

Available to [Pipes](https://turbot.com/pipes) users today, and to Steampipe users who [upgrade to Powerpipe](https://powerpipe.io/blog/migrating-from-steampipe), the new [filtering and grouping widgets](https://turbot.com/pipes/blog/2024/03/pipes-filter-and-group) enable you to create — and share — custom views of benchmark-style dashboards. 

Constrain the view with filters, reorganize it with groupings, and share insightful snapshots with your team. Pipes empowers you to analyze your compliance posture from every angle.

## And that's a wrap!

The announcements this week are individually exciting, but the Turbot story is bigger than the sum of its parts. Our products work together to deliver cloud security and intelligence with incomparable breadth, depth, power, speed, and versatility. And there's more to come. Enjoy the current batch of new products and features, [let us know](/community/join) how it goes, and stay tuned for our next Launch Week in a few months!



A benchmark like [AWS Compliance](https://hub.powerpipe.io/mods/turbot/aws_compliance) can produce a lot of data. There are dozens of dashboards, with more than a thousand controls, and you might need to run those controls over many AWS accounts. You've always been able to export the data for analysis, but it would be nice to filter and group that data right in the dashboard, and now you can.

One way to simplify things: [change the search path](https://powerpipe.io/docs/run/dashboard/search-path#change-the-search-path) in order to scope the dashboard's queries to one or several connections. 

<div>
  <img  style={{"width":"60%"}} alt="cis v300 with benchmark filter" src="/images/blog/2024-03-pipes-filter-and-group/cis_300_filtered_05.png" />
</div>

But let's say you want to query all your connections and reason over the entire data set. The new `Filter and Group` widgets enable you to create — and share! — customized views of benchmark results.

## Use Filter to constrain the view

Here's the `CIS v300` benchmark from the [AWS Compliance](https://hub.powerpipe.io/mods/turbot/aws_compliance) mod, filtered to show only alarms for `Identity and Access Management`.

<div>
  <img  style={{"width":"100%"}} alt="cis v300 with benchmark filter" src="/images/blog/2024-03-pipes-filter-and-group/cis_300_filtered_02.png" />
</div>

You can also filter on a dimension like `account_id` or `region`.

<div>
  <img style={{"width":"60%"}} alt="cis v300 with account filter" src="/images/blog/2024-03-pipes-filter-and-group/cis_300_filtered_03.png" />
</div>

## Share your custom view

At any point, you can [snapshot](https://powerpipe.io/docs/run/snapshots/interactive-snapshots) the dashboard and share the filtered view you've created. Your filters will be in effect for recipients of your link.

Note, though, that the snapshot preserves all the benchmark data. So people can add or remove their own filters to see things differently.

## Use Group to reorganize the view

Whether or not you use a filter to contrain the view, you can regroup to organize how results are shown. The default hierarchy is `Benchmark / Control / Result`. But here's an alternate view that uses the `account_id` dimension as the primary group.

<div>
  <img style={{"width":"100%"}} alt="cis v300 with account grouping" src="/images/blog/2024-03-pipes-filter-and-group/cis_300_filtered_04.png" />
</div>

Now you can review results by account. Of course you can continue to adjust the filters in this per-account grouping. And here as well you can take a snapshot at any time to share a custom view with your team.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/MleOf6olaCg"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Turbot Pipes filtering and grouping"
>
</iframe>
</div>

## Filter, group, and share your way to compliance clarity

With the new Filter and Group widgets in Turbot Pipes, you can easily create customized views of your benchmark results to focus on what matters most. Constrain the view with filters, reorganize it with groupings, and share insightful snapshots with your team. Pipes empowers you to analyze your compliance posture from every angle. So dive in, explore, and [let us know](https://turbot.com/community/join) how it goes!



## Introducing Powerpipe

Powerpipe is an open-source tool from [Turbot](https://turbot.com) that supports DevOps with:

**Benchmarks** - [5,000+ open-source controls](https://hub.powerpipe.io) from CIS, NIST, PCI, HIPAA, FedRamp and more. [Run instantly on your machine](/docs#run-security-and-compliance-benchmarks) or as part of your deployment pipeline.

**Relationship Diagrams** - The only dashboarding tool designed from the ground up to [visualize DevOps data](/docs#visualize-cloud-infrastructure). Explore your cloud, understand relationships, drill down to the details.

**Dashboards & Reports** - High-level dashboards provide a quick overview. Use them to highlight misconfigurations and hotspots. [Filter](/docs/run/benchmark/benchmark-dashboard#filtering--grouping), pivot, and [snapshot](/docs/run/snapshots) results.

**Code, not clicks** - Our dashboards are [code](/docs/powerpipe-hcl): version-controlled, composable, shareable, easy to edit — designed for the way you work. [Join](https://turbot.com/community/join) our open-source community!

## About Powerpipe dashboards and benchmarks

Powerpipe supports an ecosystem of [dashboards and benchmarks](https://hub.powerpipe.io/mods). These open-source components use HCL to define and compose elements that include [tables](/docs/powerpipe-hcl/table), [charts](/docs/powerpipe-hcl/chart), [relationship graphs](/docs/powerpipe-hcl/graph), [benchmarks](/docs/powerpipe-hcl/benchmark), and [controls](/docs/powerpipe-hcl/control). And they use SQL to query for the data that animates them, typically by way of a parallel ecosystem of [plugins](https://hub.steampipe.io/plugins) that translate cloud APIs to database tables.

For example, the [AWS Insights](https://hub.powerpipe.io/mods/turbot/aws_insights) mod provides 135+ overview and detail dashboards for AWS services: EBS, ECS, EC2, IAM, and more.

## Use the Powerpipe server to view dashboards

Likewise the [AWS Compliance](https://hub.powerpipe.io/mods/turbot/aws_compliance) mod provides 1100+ controls for Audit Manager Control Tower, AWS Foundational Security Best Practices, CIS, and many more.

Powerpipe [does not require Steampipe](/docs/run#selecting-a-database), but [mods](/docs/build) like `AWS Insights` rely on it. So to run the `AWS Insights` dashboards:

- Install [Powerpipe](https://powerpipe.io/downloads)
- Install [Steampipe](https://steampipe.io/downloads)
- Install the [AWS plugin for Steampipe](https://hub.steampipe.io/plugins/turbot/aws)

Then, in a fresh directory:

- `powerpipe mod init` (Powerpipe always runs in the context of a [mod](/docs/build).)

- `powerpipe mod install github.com/turbot/steampipe-mod-aws-insights` (There are commands for [mod management](/docs/build).)

- `steampipe service start` (Give Powerpipe access to Steampipe plugins.)

- `powerpipe server` (Many commands run client-only but dashboards use a server.)
- browse to http://localhost:9033

Dashboards are grouped by AWS service, here's the overview for EC2 instances.

<div>
<img src="/images/blog/2024-03-introducing-powerpipe/aws_ec2_dashboard.png" />
</div>

And here's the detail view for an instance, featuring a relationship graph that shows what it mounts, runs as, and connects to.

<div>
<img src="/images/blog/2024-03-introducing-powerpipe/aws_ec2_instance_detail.png" />
</div>

Although these dashboards use [Steampipe plugins](https://hub.steampipe.io/plugins), you can also connect Powerpipe to Postgres, SQLite, MySQL, or DuckDB. [This mod](https://hub.powerpipe.io/mods/turbot/rnacentral/dashboards), for example, visualizes biomedical data in a hosted instance of Postgres.

## Use the Powerpipe server to run benchmarks

To run the AWS Compliance benchmarks, follow the same recipe but use https://hub.powerpipe.io/mods/turbot/aws_compliance. These dashboards are grouped by benchmark, here's the CIS v300 benchmark.

<div>
<img src="/images/blog/2024-03-introducing-powerpipe/cis_v300_dashboard.png" />
</div>

You can adjust how results are filtered and grouped. Here we filter to just section 1, and group by `account_id`.

<div>
<img src="/images/blog/2024-03-introducing-powerpipe/cis_v300_dashboard_group_and_filter.png" />
</div>

## Use the Powerpipe CLI to run benchmarks

You can alternatively run that same benchmark, from the same folder. List the benchmarks.

```
powerpipe benchmark list
```

Run a benchmark.

```
powerpipe benchmark run cis_v300
```

The [documentation](/docs/reference/cli/benchmark#powerpipe-benchmark-run) covers a range of options including `--export` to save results in `asff`, `csv`, and other file formats, and `--snapshot` to create a snapshot [in Turbot Pipes](https://steampipe.io/blog/release-0-17-0#share-snapshots-with-steampipe-cloud).

## Compose and create dashboards and benchmarks

The thousands of pre-built dashboards and benchmarks on the hub are all available for [reuse and remixing](https://steampipe.io/blog/remixing-dashboards). You can use [mod management commands](/docs/reference/cli/mod) to [create your own mod](/docs/build/create-mod), install a local copy of one or more published mods, rearrange the resources they provide, and add your dashboards or benchmarks. Or you can create your own [dashboards](/docs/build/writing-dashboards) and [benchmarks](/docs/build/writing-dashboards) from scratch. In either case, you use HCL to define these elements, and SQL to fill them with data.

Here's a simple dashboard to chart Turbot blog posts by month. It uses the [RSS plugin](https://hub.steampipe.io/plugins/turbot/rss) to aggregate posts and the [chart](/docs/powerpipe-hcl/chart) resource to visualize the data.

```hcl
dashboard "blog_posts" {

  title = "Blog posts"

  chart {
    type = "column"
    title = "posts by month"
    sql = <<EOQ
      select
        to_char(published, 'YYYY-MM') as published,
        count(*)
      from
        rss_item
      where
        feed_link = 'https://turbot.com/blog/feed.xml'
     group by published
     order by published
    EOQ
    }

}
```

To replicate this example: install Powerpipe and Steampipe as above, install the RSS plugin (`steampipe plugin install rss`), start the Steampipe service, save this HCL code in a file (e.g. `dashboard.pp`), run `powerpipe server`, and visit http://localhost:9093. Note that Powerpipe watches its source files and reacts to edits of HCL and/or SQL. For example, in an auto-saving editor, try changing the chart type from `column` to `bar`. Then change the aggregation from `YYYY-MM` to `YYYY`. You'll see the chart reflect both changes in realtime.

And here's a benchmark that defines a single control to ensure that each post has at least one category.

```hcl
benchmark  "rss" {
  title    = "RSS compliance"
  children = [
    control.has_categories
  ]
}

control "has_categories" {
  title = "Ensure each item has one or more categories."
  sql = <<EOT
    select
      title as resource,
      case
        when jsonb_array_length(categories) > 0 then 'ok'
        else 'alarm'
      end as status,
      case
        when jsonb_array_length(categories) > 0 then 'Has ' || jsonb_array_length(categories) || ' categories'
        else 'Must have at least one category'
      end as reason,
      title,
      categories::text
    from
      rss_item
    where
      feed_link = 'https://turbot.com/blog/feed.xml'
    EOT
}
```

A [benchmark](/docs/powerpipe-hcl/benchmark) lists one or more child [controls](/docs/powerpipe-hcl/control). Each control runs a query that reports the [required columns](/docs/powerpipe-hcl/control#required-control-columns) `status`, `reason`, and `resource`, along with optional dimensions — in this case, `title` and `categories`.

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/-h6RSpvR0FE"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Powerpipe: Dashboards for DevOps"
>
</iframe>
</div>

## Data visualizations and compliance controls as code

Powerpipe is a flexible open-source platform to visualize and monitor your cloud infrastructure, ensure security compliance, and even build your own custom dashboards and benchmarks. With pre-built modules connecting to AWS, Azure, GCP, and more, along with easy ways to create and share your own mods, Powerpipe is an essential GitOps-friendly tool for DevOps teams wrangling cloud complexity. To get started, [download](/downloads) the tool, follow the [tutorial](/docs), explore the [mods](https://hub.powerpipe.io), then [let us know](https://turbot.com/community/join) how it goes!



Steampipe users have enjoyed the built-in dashboards-as-code features since [Steampipe CLI v0.13](https://steampipe.io/blog/release-0-13-0). With Powerpipe we wanted to make those same features available for other databases. That's one reason we've [decoupled](https://steampipe.io/blog/steampipe-unbundled) dashboards from the Steampipe CLI and dashboard server: so you can use Powerpipe with your own database.

Powerpipe includes the [43 Steampipe mods](https://hub.powerpipe.io/?engines=steampipe) to visualize AWS, Azure, GCP, GitHub, Terraform and more. And nine ready-to-use Powerpipe mods show you how to visualize data in [Postgres](https://hub.powerpipe.io/?engines=postgres), [SQLite](https://hub.powerpipe.io/?engines=sqlite), [DuckDB](https://hub.powerpipe.io/?engines=duckdb), and [MySQL](https://hub.powerpipe.io/?engines=mysql)!

If you're using Steampipe today to run dashboards and/or benchmarks, that all still works, but over time we'll deprecate these uses of Steampipe in favor of Powerpipe.

Powerpipe extends your Steampipe superpowers: it's database-agnostic, with dashboards that filter and group. Of course it also does what you've always been doing.

## Run Steampipe as usual

Let's assume you've [installed Steampipe](https://steampipe.io/downloads) along with [AWS plugin](https://hub.steampipe.io/plugins/turbot/aws) and are running Steampipe as a service (`steampipe service start`).

## Add Powerpipe

Now you [install Powerpipe](https://powerpipe.io/downloads).

Here are some common operations in Steampipe along with their Powerpipe counterparts.

## Run a dashboard

| action | Steampipe | Powerpipe |
| --- | --- | --- |
| Install AWS Insights | `steampipe mod install github.com/turbot/steampipe-mod-aws-insights` | `powerpipe mod install github.com/turbot/steampipe-mod-aws-insights` |
| Start dashboard server | `steampipe dashboard` | `powerpipe server` |
| Browse to dashboards | http://localhost:9194 | http://localhost:9033 |

See [Powerpipe dashboards](/docs/run/dashboard) for details.

## Run a benchmark

| action | Steampipe | Powerpipe |
| --- | --- | --- |
| Install AWS Compliance | `steampipe mod install github.com/turbot/steampipe-mod-aws-compliance` | `powerpipe mod install github.com/turbot/steampipe-mod-aws-compliance` |
| Start dashboard server | `steampipe dashboard` | `powerpipe server` |
| Browse to dashboards | http://localhost:9194 | http://localhost:9033 |
| Run a benchmark in your terminal | `steampipe check benchmark.cis_v300` | `powerpipe benchmark run cis_v300` |

See [Running benchmarks](/docs/run/benchmark) for details.

## Run a query

| action | Steampipe | Powerpipe |
| --- | --- | --- |
| List queries | `steampipe query list` | `powerpipe query list` |
| Run named query | `steampipe query query_name` | `powerpipe query query_name` |
| Run literal query | `steampipe query "sql statement"` | `powerpipe query run "sql statement"` |

See [Powerpipe query](/docs/reference/cli/query) for details.

## File extensions

| extension          | Steampipe | Powerpipe |
| ------------------ | --------- | --------- |
| HCL code for a mod | `.sp`     | `.pp`     |
| Config info        | `.spc`    | `.ppc`    |
| Snapshots          | `.sps`    | `.pps`    |
| Variables          | `.spvars` | `.ppvars` |

See [dashboards](/docs/reference/cli/dashboard), [benchmarks](/docs/reference/cli/benchmark), and [config files](/docs/reference/config-files) for details.

## Environment variables

| Steampipe                | Powerpipe                |
| ------------------------ | ------------------------ |
| `STEAMPIPE_LOG_LEVEL`    | `POWERPIPE_LOG_LEVEL`    |
| `STEAMPIPE_MAX_PARALLEL` | `POWERPIPE_MAX_PARALLEL` |
| `PIPES_TOKEN`            | `PIPES_TOKEN`            |

Many variables carry over from Steampipe, but Powerpipe adds new ones like `POWERPIPE_DATABASE`. See [Powerpipe Environment Variables](/docs/reference/env-vars) for details.

## Passing input variables

| Steampipe | Powerpipe |
| --- | --- |
| `steampipe query --var=instance_state="running"` | `powerpipe query --var=instance_state="running"` |
| `steampipe check all --var-file='tags.spvars'` | `powerpipe benchmark run all --var-file='tags.ppvars'` |

See [Passing Input Variables](/docs/build/mod-variables#passing-input-variables) for more examples.

## Change databases

Powerpipe defaults to a local Steampipe instance (`postgres://steampipe@127.0.0.1:9193/steampipe`) but you can change to another database in several ways.

### Using `--database`

```hcl
powerpipe benchmark run cis_v120 --database postgres://myusername:passworrd@mydbserver.mydomain.com:9193/steampipe
```

or

```hcl
powerpipe server --database  postgres://myusername:passworrd@mydbserver.mydomain.com:9193/steampipe
```

See [Selecting a database](/docs/run/#selecting-a-database) for more examples.

### Using a workspace

```hcl
workspace "my_workspace" {
  database = "postgres://my_username:passsord@db1.jpeterman.com:5432/steampipe"
}
```

or

```hcl
workspace "my_workspace" {
  database = "sqlite:./my_sqlite.db"
}
```

See [Managing workspaces](/docs/run/workspaces) for more examples.

## Use the Powerpipe Hub

The Steampipe Hub will redirect you to the [Powerpipe Hub](https://hub.powerpipe.io) where, under a fresh coat of paint, you'll find all your favorite mods.

## Read the Powerpipe docs

The Steampipe docs are also spruced up at [powerpipe.io/docs](https://powerpipe.io/docs).

## Enjoy the upgrade!

Rest assured that all v0.x versions of Steampipe mods will work in both Steampipe and Powerpipe. But note that from v1.0.0 onward, mods will be in Powerpipe format only.

Once you're settled in, we encourage you to try a few things. If you run benchmarks, turn on filters and grouping to slice and dice your benchmark results. If you build dashboards, check out some of the [new mods](https://hub.powerpipe.io/mods/turbot/rnacentral) that work with non-Steampipe databases and schemas. And as always, [let us know](https://turbot.com/community/join) your thoughts.



Unix philosophy holds that a software tool should do one thing well, and combine readily with other tools that abide by the same principle. Steampipe began that way. At launch in early 2021, Steampipe was a CLI that launched Postgres along with a foreign-data-wrapper extension that managed a suite of plugins — initially, [13 of them](https://web.archive.org/web/20210227040027/https://hub.steampipe.io/plugins) — that translate APIs to SQL-queryable tables. 

Then the first mods appeared, initially for compliance and cost control. Mods layered on top of plugins; they used HCL to define benchmarks and controls; they relied on plugin-enabled SQL queries to flow data into controls. The CLI added commands to run benchmarks and save results. 

Then Steampipe added a dashboard server that connected SQL queries to a new suite of HCL widgets for all-purpose data visualization. The CLI added more commands to run dashboards and save results. 

[Flowpipe](https://flowpipe.io) would have made Steampipe even more complicated, so we chose to make it a new, standalone workflow tool that can use Steampipe (among other databases) as one of its methods for acquiring data from cloud services. And that's been great! But it became clear that Steampipe itself needed to be refactored in order to align with the Unix prime directive. 
 
 The first step was to decouple Steampipe from its batteries-included Postgres database, so you could use plugins with your own database. That effort resulted in three new plugin flavors: for [Postgres](https://steampipe.io/blog/2023-12-postgres-extensions), for [SQLite](https://steampipe.io/blog/2023-12-sqlite-extensions), and for [standalone export](https://steampipe.io/blog/2023-12-steampipe-export). And it infused Steampipe's signature talent — zero-ETL live query of APIs — into a suite of single-purpose tools that you can pick up and use in any context.

 Now, with [Powerpipe](https://powerpipe.io), we've decoupled the core Steampipe engine from the Steampipe CLI and dashboard server. And as with Flowpipe, the queries that drive Powerpipe dashboards can connect to Steampipe, Postgres, SQLite, or others. 
 
 Here's a summary of the refactoring.

|Component|Does one thing well|
|--|--|
 Steampipe| select * from cloud;|
 Powerpipe| Dashboards for DevOps. |
|Flowpipe| Workflow for DevOps. |

If you're using Steampipe today to run dashboards and/or benchmarks, that all still works, but over time we'll deprecate these uses of Steampipe in favor of Powerpipe. In [Migrating from Steampipe to Powerpipe](https://powerpipe.io/blog/migrating-from-steampipe) we show what that transition looks like. Initially we expect most people will use Powerpipe's dashboard and benchmark features with the ecosystem of plugins and mods that Steampipe birthed, but the HCL+SQL layer is now decoupled from that ecosystem. You can build your own dashboards-as-code mods to visualize your data, from any source. And you can create your own benchmarks and controls — again, over any source of data — to define and enforce your own rules for security, compliance, and data integrity.

Likewise, if you're using Flowpipe today to run workflows, the [new query trigger](https://flowpipe.io/blog/query-trigger) underscores the value of refactoring Steampipe. The Flowpipe query trigger can watch any database for changes. If the changes you're looking for are best captured by way of Steampipe-managed plugins, that database will be Steampipe. But you can build workflows that rely on other databases too, and even compose solutions that mix and match Steampipe and other databases — which, again, can optionally tap into the Steampipe plugin ecosystem. 

Each of these tools — Steampipe, Flowpipe, and Powerpipe — is now well-defined and composable. Use any BI tool with Steampipe, use any database with Flowpipe or Powerpipe, run any of these single binaries separately or together in CI/CD pipelines. This is the kind of arrangement we know DevOps engineers prefer. And now, with a clearer set of components and interfaces, we're set to continue expanding and innovating the Pipes family. As always, we're eager to hear from you, so please do [let us know](https://turbot.com/community/join) your thoughts on this strategy and direction.



Workflows can gather — and automatically act on — all kinds of data, but sometimes you need a human in the loop. Here's a query that finds an IAM access key that's active beyond what our 90-day policy allows. 

```sql
select
  access_key_id,
  user_name,
  account_id,
  _ctx ->> 'connection_name' as conn_name
from
  aws_iam_access_key
where
  create_date < now() - interval '90 days'
  and status = 'Active'
```

It's nice to be able to find this non-compliant resource, but you also want to bring it into compliance, and that often requires somebody's input. Now, with the [input step](/docs/flowpipe-hcl/step/input), you can create a pipeline that will pause, ask for human input, and then continue based on that input.

## Require approval to deactivate an expired access key

This [sample workflow](https://hub.flowpipe.io/mods/turbot/deactivate-expired-aws-iam-access-keys-with-approval) kicks off with a [trigger](/blog/query-trigger) that uses that same query. 

```hcl
trigger "query" "list_expired_iam_access_keys" {
  title       = "List Expired IAM Access Keys"
  description = "List IAM access keys older than 90 days."
  database    = var.database
  schedule    = "* * * * *" # Every minute
  primary_key = "access_key_id"

  sql = <<-EOQ
    select
      access_key_id,
      user_name,
      account_id,
      _ctx ->> 'connection_name' as conn_name
    from
      aws_iam_access_key
    where
      create_date < now() - interval '90 days'
      and status = 'Active';
  EOQ

  capture "insert" {
    pipeline = pipeline.deactivate_expired_iam_access_keys_with_approval
    args = {
      access_keys = self.inserted_rows
    }
  }
}
```

The [capture block](/docs/flowpipe-hcl/trigger/query#capture) relays one or more expired keys to a pipeline that uses [for_each](/docs/flowpipe-hcl/step#for_each) to loop over one or more keys, calling a pipeline for each key. Here the input step in the per-key pipeline.

```hcl
step "input" "prompt_deactivate_expired_iam_access_key" {
  notifier = notifier[param.notifier]
  subject  = "Request to deactivate expired IAM access key ${param.access_key.access_key_id} for user ${param.access_key.user_name} [${param.acce
ss_key.account_id}]"
  prompt   = "Do you want to deactivate IAM access key ${param.access_key.access_key_id} belonging to ${param.access_key.user_name} [${param.acce
ss_key.account_id}]?"
  type     = "button"

  option "Deactivate" {
    label = "Deactivate"
    value = "deactivate"
    style = "ok"
  }

  option "Active" {
    label = "Keep expired key active"
    value = "active"
    style = "alert"
  }
}
```

Flowpipe's input step is generic. It defines what the choices are, and how they're represented — in this case, as buttons — but the input step says nothing about how they're rendered. That's just a configuration detail.

## Alternative renderings

Here's how the choices are rendered in email.

<div>
<img alt="buttons in email" src="/images/blog/2024-03-human-interaction/buttons-in-email.png"/>
</div>

And here's how they look in Slack.

<div>
<img alt="buttons in slack" src="/images/blog/2024-03-human-interaction/buttons-in-slack.png"/>
</div>

Buttons aren't the only [input type](/docs/flowpipe-hcl/step/input#input-types), you can also use `text`, `select`, and `multiselect`. Another sample, which [finds S3 buckets](https://hub.flowpipe.io/mods/turbot/add-s3-bucket-cost-center-tags) missing a required tag, then prompts with a hardcoded list of choices, uses `select`.

We'll circle back to the configuration that produces these effects, but first let's walk through the rest of this workflow. 

## Acting on the input

This step, which runs only if the person chose the `Deactivate` option, calls a pipeline [in the AWS mod for Flowpipe](https://hub.flowpipe.io/mods/turbot/aws) to deactivate the key. 

```hcl
step "pipeline" "deactivate_iam_access_key" {
  if       = step.input.prompt_deactivate_expired_iam_access_key.value == "deactivate"
  pipeline = aws.pipeline.update_iam_access_key
  args = {
    cred          = param.access_key.conn_name
    user_name     = param.access_key.user_name
    access_key_id = param.access_key.access_key_id
    status        = "Inactive"
  }
}
```

The next step sends a confirmation message.

```hcl
step "message" "notify_iam_access_key_deactivated" {
  if         = step.input.prompt_deactivate_expired_iam_access_key.value == "deactivate"
  depends_on = [step.pipeline.deactivate_iam_access_key]

  notifier = notifier[param.notifier]
  text     = "Deactivated IAM access key ${param.access_key.access_key_id} for user ${param.access_key.user_name} [${param.access_key.account_id}]"
}
```

The trigger's `var.database` could refer to any supported database — including Steamipe, Postgres, and SQLite — but since we're using the [AWS plugin](https://hub.steampipe.io/plugins/turbot/aws) we'll set that variable to `postgres://steampipe@localhost:9193/steampipe` and start Steampipe as a service.

```
steampipe service start
```

Then we'll start the Flowpipe server, with a base URL matching a webhook we've set up in Slack.

```
flowpipe server --base-url=https://9f6b-157-131-95-205.ngrok-free.app --verbose
```

Here we can see that the query trigger found a stale key and kicked off an interaction with Slack. 

<div>
<img alt="deactivation input started" src="/images/blog/2024-03-human-interaction/deactivation-input-step-started.png"/>
</div>


The pipeline's input step will now wait for a response. When a person presses either button, control returns to the pipeline. If `Deactivate` was the choice, the two steps we've seen will run: perform the deactivation, and send a confirmation message. Otherwise the workflow sends a message confirming that no action was taken. 

## Configure an integration with a notifier for Slack

Along with the `database` variable, this sample mod defines a `notifier` variable which we'll call `expired_access_key`. It's referenced from a config file we've named `integrations.fpc` which includes an [integration](/docs/reference/config-files/integration) that defines how Flowpipe talks to an external system, in this case Slack. It also includes a [notifier](/docs/reference/config-files/notifier) that binds to the integration and provides details specific to it — in this case, the channel (`security`) where the interaction will occur, and to which the confirmation message will be sent. 

```hcl
integration "slack" "default" {
  token = env("SLACK_TOKEN")
}

notifier "expired_access_key" {

  notify {
    integration = integration.slack.default
    channel     = "security"
  }

}
```

None of this configuration is hardwired in the sample mod. To perform the same interaction in email instead of Slack, you'd just write a different set of definitions.

```hcl
integration "email" "default" {
  smtp_tls      = "required"
  smtps_port    = 587
  smtp_host     = "smtp.gmail.com"
  smtp_username = "judell@example.com"
  smtp_password = env("MY_EMAIL_PASSWORD")
  from = "judell@example.com"
}

notifier "expired_access_key" {

  notify {
    integration = integration.email.default
    to = ["security@example.com"]
  }

}
```

You can even include several `notify` blocks in a `notifier` in order to send a message to multiple channels!

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/zBxNdOxnrRU"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Human interaction with Flowpipe"
>
</iframe>
</div>

## Build your own human-in-the-loop workflows

The `integration` and `notifier` blocks work with `input` and `message` steps to define how Flowpipe communicates and interacts with various external systems, so your pipelines can focus on what should happen, not how. It all adds up to a powerful and flexible toolkit for workflows that interact with people. Try building some interactive workflows, and [let us know](https://turbot.com/community/join) how it goes!




It's handy to run Flowpipe pipelines directly in your console, but pipelines also need to respond to events. <a href="https://flowpipe.io/docs/flowpipe-hcl/trigger">Triggers</a> are the way to do that. An <a href="https://flowpipe.io/docs/flowpipe-hcl/trigger/http">http</a> trigger relays an inbound webhook call so that, for example, a Slack integration can run  a Flowpipe pipeline. A <a href="https://flowpipe.io/docs/flowpipe-hcl/trigger">schedule</a> trigger runs a pipeline periodically. And a <a href="https://flowpipe.io/docs/flowpipe-hcl/trigger/query">query</a> trigger watches a database, firing in response to data changes. 

## A query trigger that watches for new AWS IAM access keys

Here's a workflow that's driven by a query trigger that periodically asks AWS if any new IAM access keys have appeared. If there are any, the trigger calls a notification pipeline.

<div>
<img width="40%" alt="access key workflow graph" src="/images/blog/2024-03-query-trigger/query_trigger_iam.png"/>
</div>

Here's the HCL definition of the trigger.

```hcl
trigger "query" "new_aws_iam_access_key" {
  description       = "Fire when a new key appears, call a notifier"
  database          = "postgres://steampipe@localhost:9193/steampipe"
  schedule          = "* * * * *"
  primary_key       = "access_key_id"
  sql               = <<EOQ
    select access_key_id, account_id, user_name, create_date
    from aws_iam_access_key
    where create_date > now() - interval '2 weeks'
  EOQ

  capture "insert" {
    pipeline = pipeline.notify_new_access_key
    args = {
      rows = self.inserted_rows
    }
  }

}
```

In this case we're using Steampipe, along with the AWS plugin, to query the Amazon API — so the `database` argument is a Postgres connection string. But pipelines can watch other databases including SQLite, MySQL, and DuckDB. And you could even use another distribution of the AWS plugin: a [foreign-data-wrapper extension](https://steampipe.io/blog/2023-12-postgres-extensions) in your own Postgres database, or a [virtual-table extension](https://steampipe.io/blog/2023-12-sqlite-extensions) in SQLite.

We're using a cron expression to run every minute, but could alternatively use a <a href="https://flowpipe.io/docs/flowpipe-hcl/trigger/schedule#arguments">named interval</a> like `"5m"`.

The primary key is optional, if omitted Flowpipe will use a hash of the row to detect change. In this case we're using the `access_key_id`; it's also possible to create a <a href="https://flowpipe.io/docs/flowpipe-hcl/trigger/query#composite-key">composite key</a> by concatenating columns.

The query finds one or more new access keys. The <a href="https://flowpipe.io/docs/flowpipe-hcl/trigger/query#all-capture-types">capture</a> block controls what happens when the query returns new rows. In this case it relays captured rows to a pipeline that sends a notification. You can also have capture blocks that watch for updates and deletes, and you can run a different pipeline for each set of affected rows. 

## Notify on new access keys

Here's the pipeline called by the trigger. The [notifier](/docs/reference/config-files/notifier) doesn't say anything about which communication medium to use, it could as easily be an email or Slack [integration](/docs/reference/config-files/integration), that's purely a configuration detail. 

```hcl
pipeline "notify_new_access_key" {
  description = "Loop over new access keys, send each to notifier."

  param "rows" {
    type = list
  }

  step "message" "notify" {
    notifier = notifier["new_access_key"]
    for_each = param.rows
    text = <<EOQ
      New access key for ${each.value.user_name}, ${each.value.account_id}: 
        ${each.value.access_key_id},  ${each.value.create_date}
      EOQ
  }  
  
}
```

## Run the workflow

As with any scheduled Flowpipe activity, we'll start Flowpipe in server mode

```
flowpipe server
```

Within one minute we can see that Flowpipe detected new data and fired to run the notification pipeline. 

<div>
<img alt="access key console" src="/images/blog/2024-03-query-trigger/access_key_console.png"/>
</div>


Note that Flowpipe watches your code for changes. If you alter the code while the pipeline is running, your changes take effect immediately.

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/TJZ5NCMohYo"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Data-driven DevOps workflows with query triggers and steps"
>
</iframe>
</div>


## Tap into the Flowpipe ecosystem

Flowpipe's query trigger enables powerful workflows that connect to databases and take action when data changes. You can use any of Steampipe's [140 plugins](https://hub.steampipe.io) to query cloud platforms like AWS and Azure, SaaS tools like Jira, security scanners, network tools, and more. Or you can connect to your own data sources. Likewise you can use any of Flowpipe's [library mods](https://hub.flowpipe.io/?type=library) to take actions in Slack, Teams, Email, Jira, AWS, Azure, and elsewhere, or create your own library pipelines that compose with one another and with published mods.

To get started with Flowpipe: [download](/downloads) the tool, follow the [tutorial](/docs), peruse the [library mods](https://hub.flowpipe.io/?type=library), and check out the [samples](https://hub.flowpipe.io/?type=sample). Then create a workflow that alerts on security events, enforces policies, gathers and publishes metrics, or does something else your DevOps team requires, and [let us know](https://turbot.com/community/join) how it goes!





A few years ago, AWS [announced](https://aws.amazon.com/about-aws/whats-new/2020/12/introducing-new-amazon-ebs-general-purpose-volumes-gp3/) the availability of Amazon EBS `gp3` volumes. These are the latest generation of general-purpose SSD-based EBS volumes. They provide baseline performance of 3,000 IOPS and 125MB/s network throughput, regardless of volume size. GP3 volumes can scale up to 16,000 IOPS and adjust performance independently of storage size. Cost savings range significantly when converting to `gp3`, for example: 
* `gp2` volumes <= 1TB in size will see a flat 20% cost reduction
* `gp2` volumes > 1TB will see 10-20% cost savings
* `io1` volumes <= 16K IOPS will see on average 67% savings

Despite these benefits, many new customers we come across have yet to take advantage of the latest generation of Amazon EBS volumes.

## Why migrate to gp3 volumes
After the `gp3` volume type was released, we analyzed our own architecture deployed for customers. We realized a [93% cost savings](https://turbot.com/guardrails/blog/2020/12/aws-ebs-cost-savings) by transitioning small, occasionally high-IOPS volumes to `gp3`, as opposed to overprovisioning larger `gp2` volumes for moderate IOPS.

Turbot Guardrails customers were already benefiting from `AWS > EC2 > Volume > Approved > Volume Type` conrols to remove any unapproved volume types. This control ensured `gp3` was the default standard moving forward. But AWS only made `gp3` the default in late 2023, and that leaves a lot of legacy volumes to be optimized.

Now with the introduction of our latest [AWS EC2 Volume configuration controls](https://turbot.com/guardrails/changelog/aws-ec2-v5-38-0), Guardrails can automatically convert volumes immediately and ensure they are optimized for cost and performance.

## How to migrate to gp3 volumes
To identify volumes that are not already `gp3`, you can run the AWS EBS Volume report.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="View AWS EBS Volume report" src="/images/blog/2024-03-automate-ebs-optimization/aws-ebs-volume-report.png" />
</div>

With our latest EBS volume controls, you can go beyond the report to automatically migrate to `gp3`. The following EC2 volume policies work together to enable the automated migration control:

```
AWS > EC2 > Volume > Performance Configuration:
   AWS > EC2 > Volume > Performance Configuration > IOPS Capacity
   AWS > EC2 > Volume > Performance Configuration > Throughput
   AWS > EC2 > Volume > Performance Configuration > Type
```
The policy type hierarchy allows you to set your configurations for IOPs, throughput and volume type. For a simple use case, first set `AWS > EC2 > Volume > Performance Configuration > Type` to `gp3`, and then set `AWS > EC2 > Volume > Performance Configuration` to `Enforce: Configured per Configuration > * policies`.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="AWS EC2 Volume Performance Configuration" src="/images/blog/2024-03-automate-ebs-optimization/aws-ec2-volume-performance-configuration-guardrail.png" />
</div>

Once enabled, Guardrails quickly identified the volumes and immediately converted each of them from `gp2` to `gp3` with their optimized throughput and IOPs, all while there was **no downtime** with the resources.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="" src="/images/blog/2024-03-automate-ebs-optimization/aws-ec2-volume-performance-configuration-remediation.png" />
</div>

 ## Using dynamic conditional logic for smarter gp3 conversions
With Turbot Guardrails, you can use [calculated policies](https://turbot.com/guardrails/docs/concepts/policies/taxonomy#calculated-policies) for smarter conversions. These policies enable you to set conditions based on IOPs, throughput, or volume type. For example, you can automatically convert any `gp2` or `io1` volume with less than 16,000 IOPs to `gp3`.

```
{%- if $.volume.volumeType and $.volume.iops -%}
 {%- if $.volume.volumeType in ['gp2', 'io1'] and $.volume.iops <= 16000 -%}
    gp3
  {%- else -%}
    {{ $.volume.volumeType }}
  {%- endif -%}
{%- endif -%}
```

For those not familiar with calculated policies: any metadata from your cloud provider — Turbot Guardrails or any other source — can be used as context in your calculated policy template.

## Automate your FinOps project for major cost savings

Recently a Turbot Guardrails customer with over 30,000 Guardrails-managed EBS volumes saw a major cost savings opportunity: their older `gp2` and `io1` volumes were candidates for migration to `gp3`. That would cut their annual AWS EBS volume cloud spend by 29%, translating to over $2 million in annualized savings. It was achieved via automation with **no downtime**. Guardrails continuously monitored the progress across hundreds of accounts, and made the initial cost savings estimate as well as status reporting easy for the project team. 

## See it in action
<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/6fwIYqtn6UQ"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Automatically optimize EBS volumes for 20% cost savings and higher performance"
>
</iframe>
</div>

## Start saving on your AWS cloud spend
Start optimizing your AWS cloud expenses today. Turbot Guardrails not only helps in converting to `gp3` for significant cost savings, it also enhances your cloud security posture and provides continuous [FinOps controls](https://turbot.com/guardrails/blog/2023/03/automations-to-reduce-cloud-spend) automation. Try Turbot Guardrails with a [14-day free trial](https://turbot.com/start) and see how much you can save.


Turbot Guardrails just made it a lot easier to discover Google Cloud Platform (GCP) resources in ServiceNow's CMDB. The integration requires no additional ServiceNow modules or GCP services. Instead, Turbot Guardrails enables a direct, low-cost way to get real-time GCP cloud discovery into ServiceNow.

This automated discovery integration has two key benefits. First, it enhances the accuracy of your ServiceNow CMDB data by surfacing comprehensive and timely GCP resource insights. Second, it significantly reduces the cost and overhead compared to native ServiceNow cloud discovery capabilities. This makes Turbot Guardrails an ideal solution for those new to cloud discovery or looking to augment existing ServiceNow GCP integrations. Users can tap into broader coverage and more precise GCP CMDB data without added complexity.

## Why an automated GCP discovery for ServiceNow?

Manual and scheduled discovery of GCP resources leads to inaccurate ServiceNow data. GCP resources change too quickly for legacy discovery methods to keep up. As a result, CMDBs end up with blindspots and incorrect configurations.

When you automate continuous GCP discovery, and integrate resources into ServiceNow, you:

* Eliminate blindspots by covering more GCP services beyond the native integrations
* Prevent missing and stale data with instant updates when resources change
* Enable flexible control over which resource data syncs to which CMDB CI tables
* Surface GCP tags and metadata to enrich CMDB context
* Archive historical records of provisioned resources that get deleted


## Augmenting native GCP discovery in ServiceNow

Native ServiceNow connectors and GCP integrations are available to enable discovery of core GCP services such as Compute, Storage, and Networking. But these have limitations including:

* Extra licensing and services
* Limited resource coverage
* Reliance on scheduled batch jobs rather than real-time scans
* Need for professional services and customization

The Turbot Guardrails GCP ServiceNow integration augments native capabilities by:

* Point-and-click integration setup
* Discovering IaaS & PaaS GCP resource types out of the box
* Reducing licensing dependency and service overhead
* Automatically updating CMDBs instantly when resources change
* Providing flexible control to configure synced data as needed
* Centralized management for other discovery scopes; [AWS](https://turbot.com/guardrails/blog/2024/01/servicenow-aws-discovery), [Azure](https://turbot.com/guardrails/blog/2024/03/servicenow-azure-discovery), & [Kubernetes](https://turbot.com/guardrails/blog/2024/05/servicenow-kubernetes-discovery)

By combining automated discovery from Turbot Guardrails with ServiceNow and GCP, you can accurately sync GCP configs into ServiceNow's CMDB in real-time without added cost and complexity.

## Configuring automated GCP cloud discovery for ServiceNow CMDB

Each GCP resource type can be configured to sync to the ServiceNow CMDB. Most often you would set the scope of the policy across many GCP resources from all your GCP projects. In this example we will show how to enable syncing GCP Storage Buckets.

Set the Turbot Guardrails policy to “Enforce: Sync” and apply to all or specific GCP projects:

<img src="/images/blog/2024-03-gcp-cmdb-sync-guardrails-servicenow-integration/gcp-sync-policy.jpg" style={{ boxShadow: 'none' }}></img>

For the integration-enabled GCP project, the following GCP resources will be in scope for the GCP discovery:

<img src="/images/blog/2024-03-gcp-cmdb-sync-guardrails-servicenow-integration/gcp-storage-buckets.jpg" style={{ boxShadow: 'none' }}></img>

Instantly the GCP resources will be added to the associated ServiceNow CMDB table:

<img src="/images/blog/2024-03-gcp-cmdb-sync-guardrails-servicenow-integration/servicenow-cmdb-table-gcp-storage-buckets.jpg" style={{ boxShadow: 'none' }}></img>

As GCP resources are added, updated, or deleted, Guardrails handles the configuration drift and keeps the ServiceNow CMDB updated.

For example, when an GCP resource changes, Guardrails captures the configuration drift and updates the ServiceNow CMDB:

<img src="/images/blog/2024-03-gcp-cmdb-sync-guardrails-servicenow-integration/turbot-guardrails-gcp-configuration-drift-capture.jpg" style={{ boxShadow: 'none' }}></img>

GCP resource deletion can be managed as a complete synchronization — where the record in ServiceNow is deleted as well — or archived to retain its record with an archive status.

## Map GCP resource details to ServiceNow CMDB CI tables

Each cloud resource type can be mapped to new or existing ServiceNow CMDB CI tables. You can also extend existing tables. In this example, the GCP Storage Bucket resource type maps to a new `cmdb_ci_gcp_storage_bucket`.

```yaml
table:
  name: cmdb_ci_gcp_storage_bucket
  label: GCP > Storage > Bucket
  extendsTable: cmdb_ci_cloud_storage_account
```

Mappings of which columns to create are simple to define in the Table Definition policy:

```yaml
columns:
- name: bucket_name
  label: Bucket Name
- name: project
  type: string
  label: Project
- name: region
  label: Region
- name: tags
  size: 1000
  type: string
  label: Cloud Tags
```
Any data from GCP can be added. To associate more fields, add more ServiceNow CMDB table columns:

```yaml
- name: versioning_enabled
  type: boolean
  label: Versioning Enabled
```

Adding custom ServiceNow CMDB CI fields is common. In this case we want to add a CMDB CI column to capture the Application ID associated from ServiceNow to the GCP resource:

```yaml
- name: app_id
  size: 1000
  type: string
  label: Application ID
```

To populate the added ServiceNow CMDB CI columns dynamically, the Configuration Item policy can define where the data comes from. In this case, this information can be pulled from the GCP resource details such as the `versioning.enabled` and the `app_id` tag:

```
bucket_name: {{ $.resource.data.name }}
project: {{ $.resource.metadata.gcp.projectId }}
region: {{ $.resource.metadata.gcp.regionName }}
tags: {{ $.resource.turbot.tags }}

versioning_enabled: {{ $.resource.data.versioning.enabled }}
app_id: {{$.resource.turbot.tags.app_id }}
```

## See Turbot Guardrails GCP discovery for ServiceNow in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/SMkkLroN3DA"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="GCP Discovery for ServiceNow with Turbot Guardrails Demo"
>
</iframe>
</div>

## Modernize your ServiceNow CMDB with automated GCP discovery

[Get started](https://turbot.com/start) with a 14-day free trial of Turbot Guardrails to experience [automated discovery](https://turbot.com/guardrails/docs/integrations/servicenow) and sync for GCP resources into your ServiceNow instance. Real-time, comprehensive visibility within minutes at just $0.05 per resource per month.



Turbot Guardrails just made it a lot easier to discover Azure resources in ServiceNow's CMDB. The integration requires no additional ServiceNow modules or Azure services. Instead, Turbot Guardrails enables a direct, low-cost way to get real-time Azure cloud discovery into ServiceNow.

This automated discovery integration has two key benefits. First, it enhances the accuracy of your ServiceNow CMDB data by surfacing comprehensive and timely Azure resource insights. Second, it significantly reduces the cost and overhead compared to native ServiceNow cloud discovery. It's an ideal solution if you're new to cloud discovery, or if you're looking to augment existing ServiceNow Azure integrations. Users can tap into broader coverage and more precise Azure CMDB data without added complexity.

## Why an automated Azure discovery for ServiceNow?

Manual and scheduled discovery of Azure resources leads to inaccurate ServiceNow data. Azure resources change too quickly for legacy discovery methods to keep up. As a result, CMDBs end up with blindspots and incorrect configurations.

When you automate continuous Azure discovery, and integrate resources into ServiceNow, you:

* Eliminate blindspots by covering more Azure services beyond the native integrations
* Prevent missing and stale data with instant updates when resources change
* Enable flexible control over which resource data syncs to which CMDB CI tables
* Surface Azure tags and metadata to enrich CMDB context
* Archive historical records of provisioned resources that get deleted

## Augmenting native Azure discovery in ServiceNow

Native ServiceNow connectors and Azure integrations are available to enable discovery of core Azure services such as Compute, Storage, and Networking. But these have limitations including:

* Extra licensing and services
* Limited resource coverage
* Reliance on scheduled batch jobs rather than real-time scans
* Need for professional services and customization

The Turbot Guardrails Azure ServiceNow integration augments native capabilities by:

* Point-and-click integration setup
* Discovering IaaS & PaaS Azure resource types out of the box
* Reducing licensing dependency and service overhead
* Automatically updating CMDBs instantly when resources change
* Providing flexible control to configure synced data as needed
* Centralized management for other discovery scopes; [AWS](https://turbot.com/guardrails/blog/2024/01/servicenow-aws-discovery), [GCP](https://turbot.com/guardrails/blog/2024/03/servicenow-gcp-discovery), & [Kubernetes](https://turbot.com/guardrails/blog/2024/05/servicenow-kubernetes-discovery)

By combining automated discovery from Turbot Guardrails with ServiceNow and Azure, you can accurately sync Azure configs into ServiceNow's CMDB in real-time without added cost and complexity.

## Configuring automated Azure cloud discovery for ServiceNow CMDB

Each Azure resource type can be configured to sync to the ServiceNow CMDB. Most often you would set the scope of the policy across many Azure resources from all your Azure subscriptions. In this example we will show how to enable syncing Azure Storage Accounts.

Simply set the Turbot Guardrails policy to “Enforce: Sync” and apply to all or specific Azure subscriptions:

<img src="/images/blog/2024-03-azure-cmdb-sync-guardrails-servicenow-integration/sync-policy.jpg" style={{ boxShadow: 'none' }}></img>

For the integration-enabled Azure subscription, the following Azure resources will be in scope for the Azure discovery:

<img src="/images/blog/2024-03-azure-cmdb-sync-guardrails-servicenow-integration/azure-storage-accounts.jpg" style={{ boxShadow: 'none' }}></img>

Instantly the Azure resources will be added to the associated ServiceNow CMDB table:

<img src="/images/blog/2024-03-azure-cmdb-sync-guardrails-servicenow-integration/servicenow-cmdb-table-azure-storage-accounts.jpg" style={{ boxShadow: 'none' }}></img>

As Azure resources are added, updated, or deleted, Turbot Guardrails handles the configuration drift and keeps the ServiceNow CMDB updated.

For example, when an Azure resource changes, Guardrails captures the configuration drift and updates the ServiceNow CMDB:

<img src="/images/blog/2024-03-azure-cmdb-sync-guardrails-servicenow-integration/turbot-guardrails-azure-configuration-drift-capture.jpg" style={{ boxShadow: 'none' }}></img>

Azure resource deletion can be managed as a complete synchronization — where the record in ServiceNow is deleted as well — or archived to retain its record with an archive status.

## Map Azure resource details to ServiceNow CMDB CI tables

Each cloud resource type can be mapped to new or existing ServiceNow CMDB CI tables. You can also extend existing tables. In this example, the Azure Storage Account resource type maps to a new `cmdb_ci_azure_storage_account`.

```yaml
table:
  name: cmdb_ci_azure_storage_account
  label: Azure > Storage > Storage Account
  extendsTable: cmdb_ci_cloud_storage_account
```

Mappings of which columns to create are simple to define in the Table Definition policy:

```yaml
columns:
- name: storage_account_name
  label: Storage Account Name
- name: subscription_id
  label: Subscription ID
- name: resource_group
  size: 200
  type: string
  label: Resource Group
- name: region
  label: Region
- name: tags
  size: 1000
  type: string
  label: Cloud Tags
```
Any data from Azure can be added. To associate more fields, add more ServiceNow CMDB table columns:

```yaml
- name: minimum_tls_version
  label: Minimum TLS Version
```

Adding custom ServiceNow CMDB CI fields is common. In this case we want to add a CMDB CI column to capture the Application ID associated from ServiceNow to the Azure resource:

```yaml
- name: app_id
  size: 1000
  type: string
  label: Application ID
```

To populate the added ServiceNow CMDB CI columns dynamically, the Configuration Item policy can define where the data comes from. In this case, this information can be pulled from the Azure resource details such as the `minimumTlsVersion` and the `app_id` tag:

```
storage_account_name: {{ $.resource.data.name }}
subscription_id: {{ $.resource.metadata.azure.subscriptionId }}
resource_group: {{ $.resource.metadata.azure.resourceGroupName }}
region: {{ $.resource.data.location }}
tags: {{ $.resource.turbot.tags }}

minimum_tls_version: {{ $.resource.data.minimumTlsVersion }}
app_id: {{$.resource.turbot.tags.app_id }}
```

## See Turbot Guardrails Azure discovery for ServiceNow in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/LF7_wizDaAk"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Azure Discovery for ServiceNow with Turbot Guardrails Demo"
>
</iframe>
</div>

## Modernize your ServiceNow CMDB with automated Azure discovery

[Get started](https://turbot.com/start) with a 14-day free trial of Turbot Guardrails to experience [automated discovery](https://turbot.com/guardrails/docs/integrations/servicenow) and sync for Azure resources into your ServiceNow instance. Real-time, comprehensive visibility within minutes at just $0.05 per resource per month.



[Turbot Guardrails](https://turbot.com/guardrails) is known for its ability to secure and optimize cloud resources instantly. It provides real-time defense that detects resource drift and remediates misconfigurations. 

## Multi-directory federated access for multi-cloud

<div style={{width:"100%", textAlign:"center"}} >
  <img alt="Multi-directory federated access for multi-cloud" src="/images/blog/2024-03-aws-federated-access-custom-groups/turbot-guardrails-identity-management.png" />
</div>


Guardrails supports multi-directory aggregation by centralizing the management of one or more SAML, LDAP, and local directories. This joins identity profiles in a unified way, whether using your own internal directory or aggregating multiple directories for break-glass access, secondary authentication via an identity provider if issues arise, or trusted [3rd party access](https://turbot.com/guardrails/blog/2017/05/enterprise-collaboration-patterns). These enterprise-grade identity features scale with you.  

## Manage time-based role-based access controls (RBAC)
Each user or group from the associated directories can be assigned RBAC or group-based permissions.  In Guardrails, you have a single pane of glass to manage roles and groups across directories, cloud providers, and cloud accounts. In the example below, resources can be assigned permissions across Guardrails, AWS, Azure and GCP.  When setting at a Folder level, permissions automatically inherit to descendant cloud accounts, or apply explicitly to specific cloud accounts.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Manage time-based role-based access controls (RBAC)" src="/images/blog/2024-03-aws-federated-access-custom-groups/time-based-rbac-controls.png" />
</div>

Permissions can be assigned in the Turbot Guardrails [console](https://turbot.com/guardrails/docs/guides/iam/permission-assignment), via [GraphQL API](https://turbot.com/guardrails/docs/reference/graphql), or by way of the [Guardrails Terraform Provider](https://turbot.com/guardrails/docs/reference/terraform).  When setting permissions, you choose the resource scope, the identities, and the permissions to be granted. Set criteria to act immediately, or later with pre-approval. Any permission grant(s) can be set to expire.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Grant time-based role-based access controls (RBAC)" src="/images/blog/2024-03-aws-federated-access-custom-groups/grant-time-based-rbac-controls.png" />
</div>

### Consistent multi-cloud role definitions with extensibility

Guardrails comes packaged with over 1000+ predefined roles, defined consistently at the ReadOnly, Operator, and Admin levels. Summarized policies like `AWS/Admin` include all enabled AWS services; per-service roles are also available, such as `AWS/S3/Admin`.  Each level can be enabled in one — or across hundreds of — cloud accounts. Guardrails does the heavy lifting of PIM/PAM automation across environments.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Consistent multi-cloud role definitions" src="/images/blog/2024-03-aws-federated-access-custom-groups/consistent-multi-cloud-role-definitions.png" />
</div>

While it's simplest to use Guardrails' role definitions out-of-the-box, you can modify these roles to include or exclude whatever IAM actions are applicable:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Extensible multi-cloud role definitions" src="/images/blog/2024-03-aws-federated-access-custom-groups/extensible-multi-cloud-role-definitions.png" />
</div>

And you may consider wrapping preventative controls to restrict over-permissive elevated access. Guardrails can manage IAM Boundary policies, and/or Lockdown policies, to explicitly apply `Deny` `Not Action` IAM policies. In this example there are friendly restrictive policies to enable for S3 actions:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Preventative cloud identity controls" src="/images/blog/2024-03-aws-federated-access-custom-groups/preventative-cloud-identity-controls.png" />
</div>

### Bring your own custom roles or deploy through Guardrails Stacks

Guardrails also enables you to bring your own provisioned roles into your cloud accounts, and associate them to Guardrails for unified management of permissions.

Simply associate the `AWS > Turbot > Permissions > Custom Role Levels` policy with a list of roles for Guardrails to begin managing.

Alternatively you can deploy and manage roles using Guardrails' Stacks policies.  Stacks will deploy your Terraform configurations, and manage any configuration drift in real-time. When applying Stacks, you can assign across multiple cloud accounts and different accounts with variance:

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Bring your own custom roles or deploy through Guardrails Stacks" src="/images/blog/2024-03-aws-federated-access-custom-groups/custom-iam-roles-through-guardrails-stacks.png" />
</div>

### Authenticate to Guardrails, federate into AWS

Guardrails can serve as a central authentication hub that federates multiple AWS, Azure, and GCP cloud accounts. When roles are assigned in Guardrails, users can select which account role to assume access - whether a Guardrails-managed role or a custom assigned role. Quick access to these accounts is available in the Guardrails UI, with a user profile summary that lists accounts for easy point-and-click authentication. Developers can also programmatically obtain credentials via the [Turbot Guardrails CLI](https://turbot.com/guardrails/docs/reference/cli) or Guardrails API.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Authenticate to Guardrails, federate into AWS" src="/images/blog/2024-03-aws-federated-access-custom-groups/authenticate-to-guardrails-federate-into-aws.png" />
</div>

## Manage time-based group-based access controls

AWS prefers that IAM roles act as separate IAM resources with associated permissions and users. Roles have built-in features to help switch roles across accounts, authenticate in time-based sessions, and define the permissions that individuals can assume when they login. Guardrails takes full advantage of IAM roles, and adds management and time-based access controls consistent with multi-cloud accounts.  

Turbot Guardrails supports both user-based and role-based identity management across multiple cloud accounts, applying RBAC permissions to Users, Roles and now also custom IAM Groups.

For some limited scenarios, IAM users can provide for more flexibility than roles — for example, when you need to tie individuals and system accounts to long-standing authentication credentials, or apply granular permissions. Turbot Guardrails is the only cloud authentication platform on the market that can federate third party identity — including Active Directory, Ping, and Okta — with AWS IAM users. End users never directly manage their associated IAM user entities; Guardrails automation ensures that they cannot become the IAM user from outside the organization's trusted network.

Without an automated cloud governance platform this is very hard to manage at scale, since users are local to each account, policies can vary, and it's a huge challenge to track changes across 10s to 1000s of identities.  Guardrails makes it easy, giving you the same RBAC capabilities but with a more flexible permission model.

### AWS IAM Group management guardrails

AWS IAM Groups simplify permissions management by enabling admins to assign common permission sets to groups of IAM users. For example, an admin could create a 'security_ops' group and grant it permissions to audit one or more AWS accounts. You then only need to add IAM users to the 'security_ops' group to give them the ability to audit the specified accounts.

Alternatively AWS IAM Roles can act as separate IAM resources with associated permissions. Turbot Guardrails supports both  user-based and role-based identity management across multiple cloud accounts, applying RBAC permissions to Users, Roles and now also IAM Groups.

### Managing time-based IAM user and policies associations

To manage AWS IAM groups with Guardrails, you can either bring your own existing AWS IAM Groups for Guardrails to manage, or use Guardrails to deploy and manage AWS IAM Groups.

Initially, configure your `AWS > Turbot > Permissions` policy to `Enforce: User Mode`

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Enable Turbot Guardrails User Mode management" src="/images/blog/2024-03-aws-federated-access-custom-groups/enable-guardrails-user-mode.png" />
</div>

### Manage your own AWS IAM groups with Guardrails

After enabling User Mode, associate your existing AWS IAM groups, such as `security_ops` and `cloud_team_admins`.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Associate your AWS IAM Groups to Guardrails" src="/images/blog/2024-03-aws-federated-access-custom-groups/associate-aws-iam-groups-to-guardrails.png" />
</div>

Once associated, these groups can be assigned to users in the Turbot Guardrails [console](https://turbot.com/guardrails/docs/guides/iam/permission-assignment), via [GraphQL API](https://turbot.com/guardrails/docs/reference/graphql), or by way of the [Guardrails Terraform Provider](https://turbot.com/guardrails/docs/reference/terraform).

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Assign AWS IAM Groups with Guardrails" src="/images/blog/2024-03-aws-federated-access-custom-groups/assign-aws-iam-groups-to-users.png" />
</div>

Guardrails continuously monitors IAM user groups, protecting against accidental removal and maintaining approved permissions consistently.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Detect and Remediate Permission Drift" src="/images/blog/2024-03-aws-federated-access-custom-groups/detect-and-remediate-permission-drift.png" />
</div>

### Deploying and managing AWS IAM Groups with Guardrails

With [Guardrails Stacks](https://turbot.com/guardrails/docs/concepts/guardrails/configured), deploying and managing AWS IAM groups is as straightforward as managing other cloud resources. These Stacks utilize Terraform to define policies that ensure infrastructure immutability. If changes or deletions occur, Turbot automatically restores the original state, correcting any drift.

For example, a Stacks policy can be used to create an `audit_team` AWS IAM group and attach the AWS-managed `SecurityAudit` role to it.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Deploy AWS IAM Groups IaC using Guardrails Stacks" src="/images/blog/2024-03-aws-federated-access-custom-groups/deploy-aws-iam-groups-iac-guardrails-stacks.png" />
</div>

## Audit AWS IAM activity

Turbot Guardrails captures all user activity, including both cloud provider actions and Guardrails’ own operations, centrally. This enables comprehensive reporting on permission grants to users, groups, or roles. For instance, you can track how `Bob` grants `Adam` & `Barry` access to the `audit_team `AWS IAM group.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Audit AWS IAM activity with Guardrails Reporting" src="/images/blog/2024-03-aws-federated-access-custom-groups/audit-aws-iam-activity-with-guardrails-reporting.png" />
</div>

Additionally, Guardrails can visualize configuration drift resulting from cloud resource changes, such as adding users to the `audit_team` group.

<div style={{width:"100%", textAlign:"center"}}>
  <img alt="Audit AWS IAM configuration drift with Guardrails Reporting" src="/images/blog/2024-03-aws-federated-access-custom-groups/audit-aws-iam-configuration-drift-with-guardrails-reporting.png" />
</div>

## See it in action
<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/l2_4WeefTpc"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Multi-directory federated access to AWS now with custom groups"
>
</iframe>
</div>

## Elevate your cloud management with time-based identity controls

With Guardrails' identity management, role-based or group-based access controls will enhance the security and efficiency of your cloud governance operations.

Try Turbot Guardrails with a [14-day free trial](https://turbot.com/start) to see how you can elevate your cloud identity management across AWS, Azure and GCP.



The [Center for Internet Security](https://cisecurity.org) (CIS) just released an updated version (v3.0) of their CIS AWS Benchmark. The new version of the benchmark removes 2 recommendations, updates MITRE mappings, and improves 20 recommendations' descriptions & remediation steps.

## Our take on the latest major changes

- **Removed - 3.3 Ensure the S3 bucket used to store CloudTrail logs is not publicly accessible:** 
This recommendation is redundant to the existing [2.1.4 Ensure that S3 Buckets are configured with 'Block public access (bucket settings)'](https://hub.steampipe.io/mods/turbot/aws_compliance/controls/control.cis_v300_2_1_4?context=benchmark.cis_v300/benchmark.cis_v300_2/benchmark.cis_v300_2_1) control which already covers a recommendation to enable S3 block public access settings based on data sensitivity, least privilege, and use case.

- **Removed - 3.4 Ensure CloudTrail trails are integrated with CloudWatch Logs:** — 
The original intent for this recommendation was to ensure AWS account activity is being captured, monitored, and appropriately alarmed on. CloudWatch Logs is a native way to accomplish this, but you could also consider logging to AWS S3 which a SIEM tool like [Splunk](https://splunk.com) can collect logs from those S3 buckets. Also, this recommendation overlaps with the Monitoring section which was once very specific on using CloudWatch logs to filter and alert on issues. Overtime the Monitoring section has evolved to recognize other native solutions like [AWS Security Hub](https://aws.amazon.com/security-hub/), or the use of CSPM and CNAPP alternatives such as [Turbot Guardrails](https://turbot.com/guardrails) which provides real-time monitoring of AWS API events and instantly actions guardrails to correct misconfigurations.

Note: the removal of these two recommendations changes the numbering scheme from the AWS CIS v2.0 benchmark. e.g. what was once [3.5 Ensure AWS Config is enabled in all regions](https://hub.steampipe.io/mods/turbot/aws_compliance/controls/control.cis_v200_3_5?context=benchmark.cis_v200/benchmark.cis_v200_3) is now [3.3 Ensure AWS Config is enabled in all regions](https://hub.steampipe.io/mods/turbot/aws_compliance/controls/control.cis_v300_3_3?context=benchmark.cis_v300/benchmark.cis_v300_3).

## Try the new AWS CIS v3.0 controls!

The [Steampipe AWS Compliance mod](https://github.com/turbot/steampipe-mod-aws-compliance), is packed with hundreds of controls that check your AWS accounts for compliance with 25+ benchmarks including NIST, PCI, HIPAA, SOC2, FedRAMP and more, now includes new controls for AWS CIS v3.0. If you're new to Steampipe, [download](https://steampipe.io/downloads) Steampipe, [install and configure](https://hub.steampipe.io/plugins/turbot/aws#get-started) the AWS plugin, and run these commands.

```
steampipe plugin update aws
git clone https://github.com/turbot/steampipe-mod-aws-compliance.git
cd steampipe-mod-aws-compliance
steampipe check aws_compliance.benchmark.cis_v300
```

If you've already installed Steampipe and the AWS plugin, and cloned the AWS Compliance mod, then just do this.

```
cd steampipe-mod-aws-compliance
git pull
steampipe check aws_compliance.benchmark.cis_v300
```

Here's a sample report in the console.

<div style={{"marginTop":"20px","marginBottom":"20px"}}>
  <img  src="/images/blog/2024-02-cis-v30-aws-benchmark/cis-console.png" />
</div>

You can output results to formats including JSON, CSV, HTML, and ASFF, or use [custom output templates](https://steampipe.io/docs/develop/writing-control-output-templates) to create new output formats.

To view the AWS CIS v3.0 benchmark report in your browser, run this command in the same cloned repo.

```
steampipe dashboard
```

Then open `http://localhost:9194` in your browser and view the dashboard.

<div style={{"marginTop":"20px","marginBottom":"20px"}}>
<img src="/images/blog/2024-02-cis-v30-aws-benchmark/cis-dashboard.png" />
</div>

## Cloud intelligence for your AWS accounts

Steampipe also includes 200+ ready-to-use [AWS cloud intelligence dashboards](https://hub.steampipe.io/mods?q=aws) to gain insights, assess your cloud inventory, navigate resource relationships, evaluate cost optimization recommendations and ensure security posture best practices.

Getting started with Steampipe? Let us know how you go by sending a message in the #steampipe channel of our [Slack community](https://turbot.com/community/join).



[Turbot Pipes](https://turbot.com/pipes) is a cloud intelligence and security platform to gain insights from 80+ cloud services and visualize 700+ ready-to-use interactive dashboards to understand your cloud inventory, navigate resource relationships, evaluate cost optimization recommendations and ensure security posture best practices. With a Pipes always-free [Developer account](https://turbot.com/pipes/docs/plans/developer), you can instantly evaluate your cloud environments without the overhead of an expensive CSPM or CNAPP service.

[AWS CIS benchmarks](https://www.cisecurity.org/benchmark/amazon_web_services) are among the more widely used security benchmarks cloud teams use as guidance to configure their AWS accounts. In the [latest iteration](https://steampipe.io/blog/cis-v30-aws-benchmark), AWS CIS v3.0, includes over [60 recomendations](https://hub.steampipe.io/mods/turbot/aws_compliance/controls/benchmark.cis_v300). Turbot Pipes leverages the Steampipe [AWS Compliance mod](https://hub.steampipe.io/mods/turbot/aws_compliance) which includes the latest CIS benchmark along with 25 other security and compliance control frameworks (e.g. NIST, SOC2, PCI, HIPAA, etc) to evaluate AWS accounts.

It takes only a couple of minutes to evaluate your AWS account for the latest CIS benchmark. Below we will walkthrough how to create your always-free Pipes Developer account, connect your first AWS account, and run your first assessment.

## Run an AWS CIS v3.0 assessment

If you’re new to Pipes, sign up at [https://pipes.turbot.com](https://pipes.turbot.com). Then create your first [workspace](https://turbot.com/pipes/docs/workspaces).

Next, we'll [connect](https://turbot.com/pipes/docs/connections) an AWS account to Turbot Pipes, whether using a cross-account AWS IAM role or using an AWS IAM access key.

<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/docs/connections" target="_blank">
  <img alt="Add a Turbot Pipes connection" src="/images/blog/2024-02-aws-cis-v3-benchmark-dashboard/add-turbot-pipes-connection.png" />
  </a>
</div>

Next, [install](https://turbot.com/pipes/docs/mods) the AWS Compliance mod which contains the AWS CIS v3.0 dashboard. Instant one-click installation:

<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/docs/mods" target="_blank">
  <img alt="Install the Steampipe AWS Compliance mod" src="/images/blog/2024-02-aws-cis-v3-benchmark-dashboard/add-aws-compliance-mod.png" />
  </a>
</div>

Finally, go to the [Dashboard](https://turbot.com/pipes/docs/dashboards) page, select AWS CIS v3.0 from the list of available dashboards from the AWS Compliance section.

<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/docs/dashboards" target="_blank">
  <img alt="View the AWS CIS v3.0 dashboard in Turbot Pipes" src="/images/blog/2024-02-aws-cis-v3-benchmark-dashboard/aws-cis-v3-benchmark-dashboard.png" />
  </a>
</div>

## Share the AWS CIS v3.0.0 assessment with your team

In the free Developer account you can create point-in-time [snapshots](https://turbot.com/pipes/docs/dashboards#saving--sharing-snapshots) of your CIS dashboard and share it via a secure link with your team. 

Capture a one-time snapshot, or create a [schedule](https://turbot.com/pipes/docs/dashboards#scheduling-snapshots) to programmatically snapshot periodically.

<div style={{width:"80%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/docs/dashboards#scheduling-snapshots" target="_blank">
  <img alt="Create a recurring AWS CIS Assessment report in Turbot Pipes" src="/images/blog/2024-02-aws-cis-v3-benchmark-dashboard/aws-cis-v3-weekly-schedule.png" />
  </a>
</div>

Optionally send [summary notifications](https://turbot.com/pipes/blog/schedule-snapshots) with your team via Slack or Microsoft Teams.

<div style={{width:"80%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/blog/schedule-snapshots" target="_blank">
  <img alt="Send AWS CIS Assessment notifications to Slack or Microsoft Teams" src="/images/blog/2024-02-aws-cis-v3-benchmark-dashboard/aws-cis-v3-slack-report.png" />
  </a>
</div>

## Enable cloud intelligence for your organization
Use the always-free Turbot Pipes Developer tier to assess your own AWS accounts. When you are ready to bring your team and expand your usage, sign up for the [Pipes Team tier](https://turbot.com/pipes/pricing) in a 14-day free trial. 

We would love to hear your feedback as you go. Send us a message in the #pipes channel of our [Slack community](https://turbot.com/community/join).



For a prior [Hacktoberfest](https://steampipe.io/blog/hacktoberfest-2022) we needed to add the tag `hacktoberfest` to over hundred repos. The [tool](https://github.com/turbot/steampipe-samples/tree/main/all/github-add-or-remove-topic) we wrote to do that uses Turbot Pipes to build a list of the names of those repos, then uses the native Github API (by way of Go's [github.com/google/go-github](https://github.com/google/go-github)) to add a topic to each repo. 

Why use Turbot Pipes? Obviously we're biased, but it wasn't a gratuitous choice. As we'll see, Turbot Pipes isn't just a convenient way to query an API with SQL. Turbot Pipes uses Steampipe plugins which represent APIs in ways that dramatically simplify their use.

Our starting point was an existing tool, [hacktoberfest-repo-topic-apply](https://github.com/Hacktoberfest/hacktoberfest-repo-topic-apply/blob/main/main.go), which showed us how to create a native GitHub client in Go and use it to assign topics to repos. We could have adapted that tool's native-API-based method of listing repositories, then filtered the results to just those matching the pattern `turbot/steampipe-(plugin|mod)`. With Turbot Pipes that's a trivial task.


```sql
select 
  count(*) 
from 
  github_my_repository
448

select 
  count(*) 
from 
  github_my_repository 
where 
  full_name ~ 'turbot'
186

select 
  count(*) 
from 
  github_my_repository
where 
  full_name ~ 'turbot/steampipe-(plugin|mod)'
135

select 
  full_name
from 
  github_my_repository
where 
  full_name ~ 'turbot/steampipe-(plugin|mod)'
order by
  full_name
turbot/steampipe-mod-alicloud-compliance
turbot/steampipe-mod-alicloud-insights   
turbot/steampipe-mod-alicloud-thrifty    
...
```

It's harder to do this with the native APIs which handle personal and organizational repos differently. Turbot Pipes hides that underlying complexity to make things easy.


## Tables that consolidate views of resources from the perspective of an authenticated user

Note that we're using the table [github_my_repository](https://hub.steampipe.io/plugins/turbot/github/tables/github_my_repository). Per the documentation:

<blockquote class="docs">
You can own repositories individually, or you can share ownership of repositories with other people in an organization. The `github_my_repository` table will list repos you own, you collaborate on, or that belong to your organizations.

To query ANY repository, including public repos, use the `github_repository` table.
</blockquote>


This is a powerful pattern that's also implemented elsewhere. The [Microsoft 365](https://hub.steampipe.io/plugins/turbot/microsoft365) plugin provides both `microsoft_my_mail_message` and `microsoft_mail_message`; the [Google Workspace](https://hub.steampipe.io/plugins/turbot/googleworkspace) plugin provides `googleworkspace_my_gmail_message` and `googleworkspace_gmail`; other plugins work the same way. Where possible, Turbot Pipes consolidates views of resources from the perspective of an authenticated user. 


As an authenticated GitHub user, my API token grants me access to my own personal repos as well as those of a half-dozen GitHub organizations I belong to. A native API query across all of them, using `github.com/google/go-github`, would require me to use `Repositories.List` (configured by `RepositoriesListOptions`) as well as `Repositories.ListByOrg` (configured by `RepositoriesListByOrgOptions`, then combine the results. That's doable but not trivial. If you just want a list of names, you'd rather not have to learn those details. Using `github_my_repository` hides them so you can focus on the job you're doing with those API results.

## Using Turbot Pipes in a Go program

We could have written our tool in any language but, since we'd started with `hacktoberfest-repo-topic-apply`, we decided to continue with Go. At its core, the tool loops through an array of repo names and calls a function to assign a topic to each repo.


```go
for i, repoName := range repos {
	addTopic(ctx, client, ownerLogin, repoName, topic)
}
```

Our tool (see [steampipe-samples](https://github.com/turbot/steampipe-samples/tree/main/all/github-add-or-remove-topic)) includes two functions that return a list of names as a comma-separated string; the tool splits that string to produce the array. 

```go
names = queryPostgresForRepos()
// names = querySpcForRepos()

var repos = strings.Split(names, ",")
```

The two functions use the same SQL query, and they produce the same list of names. We show both to illustrate two complementary ways you can use Turbot Pipes in a program that's written in Go -- or in any other language such as [Python](/pipes/docs/connect/python), [JavaScript](/pipes/docs/connect/javascript), etc.

1. Use the language's database client to query Postgres from [Turbot Pipes](/pipes/docs/connect).

2. Use the language's HTTP client to [ship your query to Turbot Pipes Query API](/pipes/docs/develop/query-api) and receive a JSON response.

Here's the SQL query that's common to both functions.

```
var query = `
	with names as (
		select
			replace(name_with_owner, 'turbot/', '') as name -- replace with your ownerLogin
		from
			github_my_repository
		where
			name_with_owner ~ 'turbot/steampipe-(plugin|mod)' -- replace with your ownerLogin and a pattern
		order by
			name_with_owner
		)
		select
			array_to_string(array_agg(name), ',') as names 
		from
			names	
```

If we run the query in the Turbot Pipes console [interactive query editor](/pipes/docs/queries), the output looks like this.

```
steampipe-mod-alicloud-compliance,steampipe-mod-alicloud-insights,...
```

Let's look at how the two functions acquire that output.

## Using a database connection

For this approach, we used Go's `database/sql` package which (we learned) also requires that you import `github.com/lib/pq` for Postgres. In this example we're querying a local instance of Steampipe but could as easily query an instance running in a Turbot Pipes workspace.


```go
func queryPostgresForRepos() string {
    connStr := "postgresql://judell:f3**-****-**2c@acme-jon.usea1.db.pipes.turbot.com:9193/dea4px"
    db, err := sql.Open("postgres", psqlconn)
	  CheckError(err)
    rows, _ := db.Query(query)
    defer rows.Close()
	var names string
	for rows.Next() {
		err = rows.Scan(&names)
		CheckError(err)
	}
	return names
}

```

## Using a REST call

Here's the second method, using Turbot Pipes' [query API](/pipes/docs/develop/query-api).


```go
func querySpcForRepos() string {
	type Result struct {
		Items []struct {
			Names string `json:"names"`
		} `json:"items"`
	}
	uri := fmt.Sprintf("https://pipes.turbot.com/api/latest/org/acme/workspace/jon/query?sql=%s", url.PathEscape(query)) //-- use your handle and workspace
	req, err := http.NewRequest("GET", uri, nil)
	CheckError(err)
	req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", os.Getenv("PIPES_TOKEN")))
	resp, err := http.DefaultClient.Do(req)
	CheckError(err)
	defer resp.Body.Close()
	decoder := json.NewDecoder(resp.Body)
	var result Result
	err = decoder.Decode(&result)
	CheckError(err)
	return result.Items[0].Names
}
```

Go's static typing requires us to define a struct into which to decode the JSON response. In a dynamically-typed language like Python that wouldn't be necessary. 

The REST call uses a bearer token that's available in your Turbot Pipes profile -- e.g `https://pipes.turbot.com/user/judell/setting/token`.

If we use `curl` to make that REST call:

```bash
curl -H "Authorization: Bearer ${PIPES_TOKEN}"  \
  -d sql="with names as ( select replace(name_with_owner, 'turbot/', '') as name from github_my_repository where name_with_owner ~ 'turbot/steampipe-(plugin|mod)' order by name_with_owner ) select array_to_string(array_agg(name), ',') as names from names"  \
 https://pipes.turbot.com/api/latest/org/acme/workspace/jon/query
```

Here's the result:


```json
{"items":[{"names":"steampipe-mod-alicloud-compliance,steampipe-mod-alicloud-insights, ... steampipe-plugin-zendesk,steampipe-plugin-zoom"}]}
```

The function decodes that into the `Result` struct and returns the `Names` property of the zeroth element of `Items`.


## Turbot Pipes as a software component

Turbot Pipes enables you to treat infrastructure, SaaS apps, and custom data like a database. It gives you the power of SQL to unify, analyze, and take action on your environment. This makes it possible to be a software component that you can call from any programming language. 

Why do that? As we've seen, Steampipe's plugins can abstract native APIs in a way that makes them simpler to use than the constructs provided by SDKs like `github.com/google/go-github`. The Turbot Pipes leveaging Steampipe plugins doesn't just provide a SQL interface to an API, it organizes that interface to make you maximally productive. And it does that using patterns, like the `my` pattern, that you can learn once and then apply across many APIs.

If you decide that Turbot Pipes is the right tool for the job, which of the two methods we've shown should you use? It's your choice! If you're more comfortable using a database connection, do that. If you'd rather use the Turbot Pipes Query API, do that. Either way, you'll get the job done more easily than if you had to use native APIs directly. Give it a try, and [let us know](https://turbot.com/community/join) how it goes!



Discovering AWS resources in ServiceNow's CMDB just got easier with Turbot Guardrails. The integration requires no additional ServiceNow modules or AWS services. Instead, Turbot Guardrails enables a direct, low-cost way to get real-time AWS resource discovery into ServiceNow.

This automated discovery integration has two key benefits. First, it enhances the accuracy of your CMDB data by surfacing comprehensive and timely AWS resource insights. Second, it significantly reduces the cost and overhead compared to enabling native ServiceNow cloud discovery capabilities. This makes Turbot Guardrails an ideal solution for those new to cloud discovery or looking to augment existing ServiceNow AWS integrations. Users can tap into broader coverage and more precise AWS data without added complexity.

## Why an automated AWS discovery for ServiceNow?

Manual and scheduled discovery of AWS resources leads to inaccurate ServiceNow data. AWS resources change too quickly for legacy discovery methods to keep up. As a result, CMDBs end up with blindspots and incorrect configurations.

Automating continuous AWS discovery and integrating resources into ServiceNow enhances the accuracy and reliability of your CMDB:

* Eliminates blindspots by covering more AWS services beyond the native integrations
* Prevents missing and stale data with instant updates when resources change
* Allows flexible control over which resource data syncs to which CMDB CI tables
* Surfaces AWS tags and metadata to enrich CMDB context
* Archives historical records of provisioned resources that get deleted

## Augmenting native AWS discovery in ServiceNow
Native ServiceNow connectors & AWS integrations are available to enable discovery of core AWS services such as EC2, VPC, etc. However, these have limitations around:

* Requiring additional licensing and services
* Supporting limited resource coverage
* Relying on scheduled batch jobs rather than real-time scans
* Needing professional services and customization

The Turbot Guardrails integration augments native capabilities by:

* Point-and-click integration setup
* Discovering 100+ AWS resource types out of the box
* Reducing licensing dependency and service overhead
* Automatically updating CMDBs instantly when resources change
* Providing flexible control to configure synced data as needed
* * Centralized management for other discovery scopes; [Azure](https://turbot.com/guardrails/blog/2024/03/servicenow-azure-discovery), [GCP](https://turbot.com/guardrails/blog/2024/03/servicenow-gcp-discovery), & [Kubernetes](https://turbot.com/guardrails/blog/2024/05/servicenow-kubernetes-discovery)

By combining automated discovery from Turbot Guardrails with ServiceNow and AWS, you can accurately sync AWS configs into ServiceNow's CMDB in real-time without added cost & complexity.

## Configuring automated AWS resource discovery for ServiceNow CMDB

Each AWS resource type can be configured to sync to the ServiceNow CMDB. Most often you would set the scope of the policy across many AWS resources from all your AWS accounts. In this example we will show how to enable syncing AWS S3 Buckets.

Simply set the Turbot Guardrails policy to “Enforce: Sync” and apply to all or specific AWS accounts:

<img src="/images/blog/2023-12-cmdb-sync-guardrails-servicenow-integration/sync-policy.jpg" style={{ boxShadow: 'none' }}></img>

For the AWS account we enabled the integration for, the following AWS resources will be in scope for the AWS discovery:

<img src="/images/blog/2023-12-cmdb-sync-guardrails-servicenow-integration/aws-accounts.jpg" style={{ boxShadow: 'none' }}></img>

Instantly the AWS resources will be added to the associated ServiceNow CMDB table:

<img src="/images/blog/2023-12-cmdb-sync-guardrails-servicenow-integration/servicenow-cmdb-table-aws-s3-buckets.jpg" style={{ boxShadow: 'none' }}></img>

As AWS resources are added, updated, or deleted, Turbot Guardrails handles the configuration drift and keeps ServiceNow CMDB updated.

For example, when an AWS resource changes, Turbot Guardrails captures the configuration drift and updates ServiceNow CMDB:

<img src="/images/blog/2023-12-cmdb-sync-guardrails-servicenow-integration/turbot-guardrails-configuration-drift-capture.jpg" style={{ boxShadow: 'none' }}></img>

AWS resource deletion can be managed as a complete synchronization where the record in ServiceNow is deleted as well, or archived to retain its record with an archive status.

## Map AWS resource details to ServiceNow CMDB CI tables

Each cloud resource type can be mapped to new or existing ServiceNow CMDB CI tables. You can also extend off existing tables as well. In this example, the AWS S3 Bucket resource type maps to a new `cmdb_ci_aws_s3_bucket”.

```yaml
table:
  name: cmdb_aws_s3_bucket
  label: AWS > S3 > Bucket
  extendsTable: cmdb_ci_cloud_storage_account
```

Mappings of which columns to create are simple to define in the Table Definition policy:

```yaml
columns:
  - name: bucket_name
    label: Bucket Name
  - name: account_id
    label: Account ID
  - name: region
    label: Region
  - name: tags
    size: 1000
    type: string
    label: Cloud Tags
```
Any data from AWS can be added. To associate more fields simply add as additional ServiceNow CMDB table columns:

```yaml
- name: versioning_enabled
  label: Versioning Enabled
```

Adding custom ServiceNow CMDB CI fields is common. In this case we want to add a CMDB CI column to capture the Application ID associated from ServiceNow to the AWS resource:

```yaml
- name: app_id
  size: 1000
  type: string
  label: Application ID
```

To populate the added ServiceNow CMDB CI columns dynamically, the Configuration Item policy can define where the data comes from. In this case, this information can be pulled from the AWS resource details such as the `Versioning.Status` and the `app_id` tag:

```
bucket_name: {{ $.resource.data.Name }}
account_id: {{ $.resource.metadata.aws.accountId }}
region: {{ $.resource.metadata.aws.regionName }}
tags: {{ $.resource.turbot.tags }}

versioning_enabled: {{ $.resource.data.Versioning.Status }}
app_id: {{ $.resource.turbot.tags.app_id }}
```

## See Turbot Guardrails AWS discovery for ServiceNow in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/7-e1-tBO1ho"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Continuous Cloud CMDB Sync with ServiceNow & Turbot Guardrails Demo"
>
</iframe>
</div>

## Modernize your ServiceNow CMDB with automated AWS discovery

[Get started](https://turbot.com/start) with a 14-day free trial of Turbot Guardrails to experience [automated discovery](https://turbot.com/guardrails/docs/integrations/servicenow) and sync for AWS resources into your ServiceNow instance. Real-time, comprehensive visibility within minutes at just $0.05 per resource per month.



Our announcements this week span the breadth of Turbot offerings, from Turbot Guardrails' new ServiceNow integration, to the unbundling of Steampipe plugins for use with any Postgres or SQLite database (or as standalone export tools), to our new open-source cloud scripting engine Flowpipe, to Turbot Pipes' new enterprise tier.

## Watch the Wrap Party

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/yWjQ6jNUKh4"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Turbot Launch Week 2 Wrap Party"
>
</iframe>
</div>
<div className="text-2xl font-bold pt-10">Full Week Recap</div>

## Monday Dec 11: Turbot Guardrails and ServiceNow: Better together

Guardrails now integrates with ServiceNow in both directions.

**Guardrails → ServiceNow**. Guardrails users can sync AWS, Azure, and GCP resources to ServiceNow, in realtime, to improve visibility and accuracy. See [Continuous Cloud CMDB Sync with ServiceNow & Turbot Guardrails](https://turbot.com/guardrails/blog/2023/12/cmdb-sync-guardrails-servicenow-integration) for details and a demo.

**ServiceNow → Guardrails**. Sync ServiceNow data, such as Cost Centers and Apps, directly into Guardrails. Enrich your cloud resources, automations and security controls with context straight from your ServiceNow CMDB. See [ServiceNow + Guardrails: Context-aware cloud & security automation](https://turbot.com/guardrails/blog/2023/12/context-aware-guardrails-servicenow-integration) for details and a demo.

## Tuesday Dec 12: Zero-ETL: Steampipe plugins for any Postgres or SQLite database, and as standalone export tools

People have often asked: "Why can't we use Steampipe plugins in our own Postgres databases?" Now the answer is: "You can!" Every [Steampipe plugin](https://hub.steampipe.io/plugins) is now available as a standalone FDW that you can load as an extension in your own Postgres database. More than 100 new Postgres FDWs just appeared in the world, enabling any Postgres database to tap into the thousands of tables available in the growing Steampipe ecosystem. See [Zero-ETL for Postgres](https://steampipe.io/blog/2023-12-postgres-extensions) for details and a demo.

But wait, there's more! Plugins are now also available as SQLite extensions that map plugin-provided tables to virtual tables in the world's most widely deployed database. See [Zero-ETL for SQLite](https://steampipe.io/blog/2023-12-sqlite-extensions) for details and a demo.

And we didn't stop there. You can _also_ use plugins as standalone export tools. It's the zeroest of our zero-ETL offerings: no database required. See [Command-line Zero-ETL](https://steampipe.io/blog/2023-12-steampipe-export) for details and a demo.

## Wednesday Dec 13: Flowpipe: The cloud scripting engine for DevOps

What's a cloud scripting engine? In our view, it's a tool that enables you to:

- **Orchestrate your cloud**. Build simple steps into complex workflows. Run and test locally. Compose solutions across clouds using open source mods.

- **Connect people and tools**. Connect your cloud data to people and systems using email, chat & APIs. Workflow steps can even run containers, custom functions, and more.

- **Respond to events**. Run workflows manually or on a schedule. Trigger pipelines from webhooks or changes in data.

- **Use code, not clicks**. Build and deploy DevOps workflows like infrastructure. Code in HCL and deploy from version control.

That tool is [Flowpipe](https://flowpipe.io), a new open-source project from Turbot. Define workflows with HCL, with pipeline steps that use Flowpipe libraries, Steampipe queries, custom functions, or containers to get the job done.

It's supported by the Flowpipe Hub with [28 libraries](https://hub.flowpipe.io/?type=library) and [35 samples](https://hub.flowpipe.io/?type=sample) to get you up and running immediately.

See [Flowpipe: The cloud scripting engine](https://flowpipe.io/blog/introducing-flowpipe) for details and a demo.

## Thursday Dec 14: Turbot Pipes Enterprise

With support for multiple organizations, isolated tenants and SAML Authentication, Turbot Pipes Enterprise elevates the standard of cloud intelligence and security for enterprises. Building on the Team tier's features, it offers enhanced collaboration, enterprise-grade security, and improved scalability, making it ideal for larger organizations. And like the Team tier, the Enterprise tier offers a 14-day free trial. Check out [the details](https://turbot.com/pipes/blog/2023/12/turbot-pipes-enterprise) and give it a try!

## Friday Dec 15: Community highlights

2023 was a busy and productive year for our contributors. The flurry of activity during Hacktoberfest was just part of the story. Throughout the year, more than 100 contributors created or updated plugins and mods, improved the CLI and plugin SDK, wrote documentation, and gave talks or wrote blog posts about Steampipe. See [Community highlights](https://turbot.com/blog/2023/12/community-highlights-2023) for details, and thanks again to everyone who contributed time, thought, and effort to make Steampipe better. We look forward to hearing from you in 2024!

## And that's a wrap!

The announcements this week are individually exciting, but the Turbot story is bigger than the sum of its parts. Our products work together to deliver cloud security and intelligence with incomparable breadth, depth, power, speed, and versatility. And there's more to come. Enjoy the current batch of new products and features, [let us know](/community/join) how it goes, and stay tuned for our next Launch Week in a few months!



In October, the community again rose to the [Turbot Hacktoberfest](https://turbot.com/blog/2023/10/hacktoberfest-2023) challenge. Here's a summary of contributions.

- New plugins from
  François de Metz ([metabase](https://github.com/1024pix/steampipe-plugin-metabase/)),
  Gabriel Solz ([semgrep](https://github.com/gabrielsoltz/steampipe-plugin-semgrep)),
  Jeremy Planckeel ([opsgenie](https://github.com/jplanckeel/steampipe-plugin-opsgenie)), and.
  José Reyes ([postgres](https://github.com/jreyesr/steampipe-plugin-postgres))

- New tables from [Jeremy Planckeel](https://github.com/jplanckeel), [Lawrence Wakefield](https://github.com/lwakefield), and [Mike Keller](https://github.com/mkell43)

- A new mod from Gabriel Soltz ([aws-securityhub](https://github.com/gabrielsoltz/steampipe-mod-aws-securityhub))

- Plugin updates from [Adrien Beudin](https://github.com/beudbeud), [François de Metz](https://github.com/francois2metz/), [Jeremy Planckeel](https://github.com/jplanckeel), [Mridang Agarwalla](https://github.com/mridang), and [Patrick Decat](https://github.com/pdecat)

- A CLI fix from [Meet Rajesh Gor](https://github.com/Mr-Destructive)

- A mod update from [Sai Sirish Reddy](https://github.com/saisirishreddy)

- Documentation updates from [Patrick Decat](https://github.com/pdecat) and [Piotr Idzik](https://github.com/vil02)

- A conference talk about Steampipe by [Antonio Maradiaga](https://www.linkedin.com/in/ajmaradiaga/)

- Blog posts about Steampipe by [Fatih Keles](https://www.linkedin.com/in/fatihkeles/), [José Reyes](https://github.com/jreyesr), [Youngjin Baek](https://www.linkedin.com/in/youngjin-baek-8a2a74a4/), and [Zakaria el Bazi](https://www.linkedin.com/in/zakaria-elbazi/).

Looking back over the whole year, there were 108 commits from 57 community members.

New plugins, in addition to those created during Hacktoberfest, were [cohereai](https://hub.steampipe.io/plugins/mr-destructive/cohereai) ([Meet Rajesh Gor](https://github.com/Mr-Destructive)), [ip2location](https://github.com/ip2location/steampipe-plugin-ip2locationio) ([ip2location](https://github.com/ip2location)), [openstack](https://hub.steampipe.io/plugins/ernw/openstack) ([sven-ernw](https://github.com/sven-ernw)), and [tomba](https://github.com/tomba-io/steampipe-plugin-tomba) ([Mohamed Benr](https://github.com/benemohamed)).

Here's everyone who made one or more plugin contributions in 2023: [adrfrank](https://github.com/adrfrank), [ahlfors](https://github.com/ahlfors), [aminvielledebatAtBedrock](https://github.com/aminvielledebatAtBedrock), [AnykeyNL](https://github.com/AnykeyNL), [badideasforsale](https://github.com/badideasforsale), [beudbeud](https://github.com/beudbeud), [brentmitchell25](https://github.com/brentmitchell25), [eric-glb](https://github.com/eric-glb), [francois2metz](https://github.com/francois2metz), [gabrielsoltz](https://github.com/gabrielsoltz), [hileef](https://github.com/hileef), [jackdelab](https://github.com/jackdelab), [jacksgt](https://github.com/jacksgt), [janslow](https://github.com/janslow), [japborst](https://github.com/japborst), [jaredreisinger-drizly](https://github.com/jaredreisinger-drizly), [jplanckeel](https://github.com/jplanckeel), [jramosf](https://github.com/jramosf), [juandspy](https://github.com/juandspy), [keyolk](https://github.com/keyolk), [KTamas](https://github.com/KTamas), [LalitLab](https://github.com/LalitLab), [leonid-panich](https://github.com/leonid-panich), [lucasjellema](https://github.com/lucasjellema), [lwakefield](https://github.com/lwakefield), [marcus-crane](https://github.com/marcus-crane), [mdaguete](https://github.com/mdaguete), [mridang](https://github.com/mridang), [mrkwtz](https://github.com/mrkwtz), [muescha](https://github.com/muescha), [nickman](https://github.com/nickman), [Pankaj-SinghR](https://github.com/Pankaj-SinghR), [pdecat](https://github.com/pdecat), [rasta-rocket](https://github.com/rasta-rocket), [reva](https://github.com/reva), [rinzool](https://github.com/rinzool), [rollwagen](https://github.com/rollwagen), [scotti-fletcher](https://github.com/scotti-fletcher), [srgg](https://github.com/srgg), [tinder-tder](https://github.com/tinder-tder), [tylarb](https://github.com/tylarb), [villers](https://github.com/villers), [vmdude](https://github.com/vmdude), [Wade9320](https://github.com/Wade9320), [whume](https://github.com/whume).

The most prolific contributor in 2023 was Patrick Decat, with a total of 22 contributions to `steampipe-mod-aws-compliance`, `steampipe-plugin-kubernetes`, `steampipe-plugin-aws`, `steampipe-plugin-sdk`, `steampipe-plugin-azure`, `steampipe-plugin-bitbucket`, and `steampipe`.

The most prolific author was José Reyes who wrote an astonishing series of [13 posts](https://jreyesr.github.io/tags/steampipe/), each a code-rich deep dive into some aspect of Steampipe.

Thank to everyone who participated in the Turbot community this year! We look forward to hearing from you in 2024, and from the new folks who will [join](https://turbot.com/community/join) during the coming year.


We are excited to announce the launch of our new Enterprise Tier, a comprehensive set of features designed to meet the needs of enterprise customers using [Turbot Pipes](https://turbot.com/pipes). 

Turbot Pipes is a cloud intelligence and security platform tailored for DevOps. It offers Zero-ETL for streamlined access to cloud data, alongside dashboards-as-code capabilities for visualizing insights and security posture. Our SOC2 Type II certified platform enables collaboration among teams to share data-driven insights, allowing members to leverage connections, send report snapshots, and communicate query results to provide the necessary context for informed decision-making.

## Key Features of Turbot Pipes Enterprise
Turbot Pipes Enterprise elevates the standard of cloud intelligence and security for enterprises by building on the Team tier’s features. It offers enhanced collaboration, enterprise-grade security, and improved scalability, making it ideal for larger organizations:
- **Organization-wide cloud intelligence & security:** Enables tailored data management across business units and teams for sharing insights.
- **SAML Authentication:** Provides secure and seamless SSO user experience using your identity provider.
- **Multi-Organization RBAC:** Allows granular access permissions across organizations and workspaces to protect sensitive data.
- **Trusted Login Domains:** Significantly reduces unauthorized access by restricting logins to trusted domains.
- **Consolidated Usage and Billing:** Simplifies resource and financial tracking with tenant-level visibility plus per organization/workspace details.

## Transparent, Scalable Pricing
Turbot Pipes is free for Developers, allowing exploration and personal use at no cost. The Team tier elevates team productivity and coordination. The Enterprise tier provides organization-wide security starting at $249 per month, includes three users, 10,000 compute minutes, and 100GB of storage. Scale up as needed, adding more users and paying only for the additional compute and storage you require, with the flexibility to cancel anytime.
<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/pricing" target="_blank">
 <img alt="Turbot Pipes Pricing 2023" src="/images/blog/2023-12-turbot-pipes-enterprise/turbot_pipes_enterprise_pricing.png" />
  </a>
  </div>

For additional information, please visit our [Pricing](https://turbot.com/pipes/pricing) page.

## Get Started with a 14-day Free Trial
Getting started with Turbot Pipes is simple. Sign up for the Developer free tier at [pipes.turbot.com](https://pipes.turbot.com). When you're ready to explore further, unlock more features by upgrading to the Team or Enterprise tier starting with a 14-day free trial.

Have questions? Connect with us and your peers in the **#pipes** channel of our [Slack community](https://turbot.com/community/join).


## Introducing Flowpipe

Flowpipe is an open-source cloud scripting engine from [Turbot](https://turbot.com) that enables you to:

**Orchestrate your cloud**. Build simple steps into complex workflows. Run and test locally. Compose solutions across clouds using open source mods.

**Connect people and tools**. Connect your cloud data to people and systems using email, chat & APIs. Workflow steps can even run containers, custom functions, and more.

**Respond to events**. Run workflows manually or on a schedule. Trigger pipelines from webhooks or changes in data.

**Use code, not clicks**. Build and deploy DevOps workflows like infrastructure. Code in HCL and deploy from version control.

## Pipelines as code!

Here's a simple example: a two-step pipeline to get your IP address and location:

```hcl
pipeline "learn_flowpipe" {

  # Simple HTTP step to get data from a URL
  step "http" "get_ipv4" {
    url = "https://api.ipify.org?format=json"
  }

  # Run a nested pipeline from the reallyfreegeoip mod
  step "pipeline" "get_geo" {
    pipeline = reallyfreegeoip.pipeline.get_ip_geolocation
    args = {
      # Automatic dependency resolution determines step order
      ip_address = step.http.get_ipv4.response_body.ip
    }
  }

  output "ip_address" {
    value = step.http.get_ipv4.response_body.ip
  }

  output "latitude" {
    value = step.pipeline.get_geo.output.geolocation.latitude
  }

  output "longitude" {
    value = step.pipeline.get_geo.output.geolocation.longitude
  }
}
```

Pipelines are defined and [run](/docs/run/pipelines) on your local machine:

```sh
$ flowpipe pipeline run learn_flowpipe
```

![](/images/blog/2023-12-introducing-flowpipe/geolocation.png)

## Step into cloud scripting

DevOps professionals use a vast repertoire of tools, scripts and processes to get through the day. Flowpipe's step primitives embrace all of them:

| Type | Description |
| --- | --- |
| [container](/docs/flowpipe-hcl/step/container) | Run a Docker container. |
| [email](/docs/flowpipe-hcl/step/email) | Send an email. |
| [function](/docs/flowpipe-hcl/step/function) | Run an AWS Lambda-compatible function. |
| [http](/docs/flowpipe-hcl/step/http) | Make an HTTP request. |
| [pipeline](/docs/flowpipe-hcl/step/pipeline) | Run another Flowpipe pipeline. |
| [query](/docs/flowpipe-hcl/step/query) | Run a SQL query, works great with [Steampipe](https://steampipe.io)! |
| [sleep](/docs/flowpipe-hcl/step/sleep) | Wait for a defined time period. |
| [transform](/docs/flowpipe-hcl/step/transform) | Use HCL functions to transform data . |

For example, here is a [container step](/docs/flowpipe-hcl/step/container) to run the AWS CLI:

```hcl
pipeline "another_pipe" {

  step "container" "aws_s3_ls" {
    image      = "public.ecr.aws/aws-cli/aws-cli"
    cmd        = ["s3", "ls"]
    env = credentials.aws["my_profile"].env
  }

  output "buckets" {
    value = step.container.aws_s3_ls.stdout
  }
}
```

Don't want to run a container? Use a mod or a query step or a function step. There's more than one way to do it, choose the method that best enables you to orchestrate your cloud and coordinate your team.

## Keep control, in good times and bad

Pipelines may start simple, but we know that things get complicated at cloud scale. Flowpipe has you covered with:

- [timeouts, retries and error handling](/docs/build/write-pipelines/errors)
- [conditional logic](/docs/build/write-pipelines/conditionals)
- [iteration](/docs/build/write-pipelines/iteration) and [parallelism](/docs/build/write-pipelines/control-flow#control-flow)

For example, `for_each` will run a step multiple times in parallel for a collection:

```hcl
step "http" "add_a_user" {
  for_each = ["Jerry", "Elaine", "Newman"]
  url      = "https://myapi.local/api/v1/user"
  method   = "post"
  request_body = jsonencode({
    user_name = "${each.value}"
  })
}
```

And `retry` allows you to [retry the step](/docs/build/write-pipelines/errors#retry) when an error occurs:

```hcl
step "http" "my_request" {
  url    = "https://myapi.local/subscribe"
  method = "post"
  body   = jsonencode({
    name = param.subscriber
  })
  retry {
    max_attempts = 5
    strategy     = "exponential"
    min_interval = 100
    max_interval = 10000
  }
}
```

## Composable Mods and the Flowpipe Hub

Flowpipe mods are open-source composable pipelines so you can remix and reuse your code — or build on the great work of the community. The [Flowpipe Hub](https://hub.flowpipe.io) is a searchable directory of mods to discover and use. The source code for mods is [available on GitHub](https://github.com/topics/flowpipe-mod) if you'd like to learn or contribute.

[Flowpipe library mods](https://hub.flowpipe.io/?type=library) make it easy to work with common services including <a href="https://hub.flowpipe.io/mods/turbot/aws">AWS</a>, <a href="https://hub.flowpipe.io/mods/turbot/github">GitHub</a>, <a href="https://hub.flowpipe.io/mods/turbot/jira">Jira</a>, <a href="https://hub.flowpipe.io/mods/turbot/slack">Slack</a>, <a href="https://hub.flowpipe.io/mods/turbot/teams">Teams</a>, <a href="https://hub.flowpipe.io/mods/turbot/zendesk">Zendesk</a> ... and many more!

[Flowpipe sample mods](https://hub.flowpipe.io/?type=sample) are ready-to-run samples that demonstrate patterns and use of various library mods.

[Creating your own mod](/docs/build/#initializing-a-mod) is easy:

```sh
mkdir my_mod
cd my_mod
flowpipe mod init
```

[Install the AWS mod](/docs/build/mod-dependencies) as a dependency:

```sh
flowpipe mod install github.com/turbot/flowpipe-mod-aws
```

[Use the dependency](/docs/build/write-pipelines) in a pipeline step:

```sh
vi my_pipeline.fp
```

```hcl
pipeline "my_pipeline" {
  step "pipeline" "describe_ec2_instances" {
    pipeline = aws.pipeline.describe_ec2_instances
    args = {
      instance_type = "t2.micro"
      region        = "us-east-1"
    }
  }
}
```

and then run your pipeline!

## Schedules, Events and Triggers

DevOps is filled with routine work, which Flowpipe is happy to do on a schedule. Just setup a [schedule trigger](/docs/flowpipe-hcl/trigger/schedule) to run a pipeline at regular intervals:

```hcl
trigger "schedule" "daily_3pm" {
  schedule = "* 15 * * *"
  pipeline = pipeline.daily_task
}
```

Events are critical to the world of cloud scripting — we need to respond immediately to code pushes, infrastructure change, Slack messages, etc. So Flowpipe has a [http trigger](/docs/flowpipe-hcl/trigger/http) to handle incoming webhooks and run a pipeline:

```hcl
trigger "http" "my_webhook" {
  pipeline = pipeline.my_pipeline
  args     = {
    event = self.request_body
  }
}
```

## Cloud scripting is the sweet spot for DevOps automation

Flowpipe is built for DevOps teams. Express pipelines, steps, and triggers in HCL — the familiar DevOps language. Compose pipelines using mods, and mix in SQL, Python functions, or containers as needed. Develop, test and run it all on your local machine — no deployment surprises or long debug cycles. Then schedule the pipelines or respond to events in real-time.

To get started with Flowpipe: [download](/downloads) the tool, follow the [tutorial](/docs), peruse the [library mods](https://hub.flowpipe.io/?type=library), and check out the [samples](https://hub.flowpipe.io/?type=sample). Then [let us know](https://turbot.com/community/join) how it goes!


Your Postgres database just gained a new superpower: the ability to fill tables with data from cloud APIs. Actually there are [more than 100 of these superpowers](https://hub.steampipe.io/plugins), that's how many new Postgres foreign data wrapper extensions (FDWs) just appeared in the world. We'll use the [GitHub extension](https://hub.steampipe.io/plugins/turbot/github) to illustrate but they all work the same way. You write SQL like this.

```sql
select state, closed_at, title
from github.github_issue
where repository_full_name = 'turbot/steampipe-plugin-github'
order by state desc, closed_at desc;
```

And you get results like this.

```
+--------+---------------------------+-----------------------------------------------------------------------------------------------------------+
| state  | closed_at                 | title																								     |
+--------+---------------------------+-----------------------------------------------------------------------------------------------------------+
| OPEN   | <null>                    | A way to query for assignee on Pull Requests                                                              |
| OPEN   | <null>                    | Add `created_at` list key column and potentially others in `github_actions_repository_workflow_run` table |
| CLOSED | 2023-11-06T23:38:53-08:00 | Queries fail on GHE due to missing fields                                                                 |
| CLOSED | 2023-10-24T08:14:09-07:00 | 502 Bad Gateway with repos with large amount of  PRs
```

In this case we're asking the GitHub extension to tell us about [its own issues](https://github.com/turbot/steampipe-plugin-github/issues). The [github_issue table](https://hub.steampipe.io/plugins/turbot/github/tables/github_issue) is just one of 55 tables provided by the GitHub extension, and that extension is just one of [100+](https://hub.steampipe.io/plugins). Some represent APIs as a handful of tables, others as hundreds of tables; the [AWS extension](https://hub.steampipe.io/plugins/turbot/aws) leads the pack with over 450. There are more than 2000 tables in all, each clearly documented with copy/paste/run examples.

When you query one of these tables, the extension fills with live data from underlying APIs. It retains query results in its cache for five minutes, after which a repeat of the same query will again fetch from APIs. Why this ephemeral approach? It's our take on zero-ETL: pull data directly from sources, and use SQL as the uniform way to ask questions about their current state. APIs are fast enough to answer a wide range of such questions without needing to warehouse the data.

Ready to try? Let's get started.

## Install and configure the GitHub extension

Visit [Steampipe downloads](https://steampipe.io/downloads?install=postgres) to find the installer for your OS, and run it to put the extension's three files into your Postgres installation.

Now connect to your server as usual, using `psql` or another client, most typically as the postgres user. Then run these commands which are typical for any Postgres foreign data wrapper.

As with all Postgres extensions, you start like this:

```sql
CREATE EXTENSION steampipe_postgres_fdw_github;
```

To use a foreign data wrapper, you first create a server:

```sql
CREATE SERVER steampipe_github FOREIGN DATA WRAPPER steampipe_postgres_github OPTIONS (config 'token="ghp_..."');
```

Use OPTIONS to configure the extension to use your GitHub access token. (Alternatively, the standard environment variables used to configure a Steampipe plugin – `GITHUB_TOKEN` in this case – will work if you set them before starting your instance of Postgres.)

The tables provided by the extension will live in a schema, so define one:

```sql
CREATE SCHEMA github;
```

Now import the schema defined by the foreign server into the local schema you just created:

```
IMPORT FOREIGN SCHEMA github FROM SERVER steampipe_github INTO github;
```

Now run a query!

The foreign tables provided by the extension live in the `github schema`, so by default you'll refer to tables like `github.github_my_repository`. If you `set search_path = 'github'`, though, the schema becomes optional and you can write queries like this.

```sql
select
  count(*)
from
  github_my_repository

 count
-------
   447
```

If you have a lot of repos, the first run of that query will take a few seconds. The second run will return results instantly, though, because the extension includes a powerful and sophisticated [cache](https://steampipe.io/blog/release-0-10-0#caching-row-subsets).

And that's all there is to it! Every Steampipe foreign data wrapper works exactly like this one.

## What does Zero-ETL mean?

### Zero to query in seconds

It's quick and easy to install an extension, configure credentials, and immediately query the predefined foreign tables it provides.

Install the FDW for a [Steampipe plugin](https://hub.steampipe.io/plugins), configure credentials, and you're ready to go. Immediately query the provided tables, join across them, or join with you own tables and/or those provided by other Steampipe FDWs.

### Zero infrastructure

No data warehouse is required, and persistence is optional. Just write SQL queries to ask questions of cloud APIs and receive instant answers from live API calls.

Our zero-ETL approach stores nothing permanently by default; your foreign tables represent the current state of the systems whose APIs they pull from. But you do have the option to retain results in a Postgres materialized view. To do that, prepend `create materialized view TABLE_NAME` as to any query of a foreign table. You can then use refresh materialized view TABLE_NAME to fill the view with current data from underlying APIs.

### Zero roadblocks

These open-source extensions are available for [more than 100 services](https://hub.steampipe.io/plugins). Each provides prebuilt tables that reflect lessons learned by users and contributors as the suite has evolved and matured.

There are thousands of tables that embody consistent access patterns, and each table is well-documented with examples. Missing a table? Add one! Missing a plugin? Build one!

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/53HF8qQWwUE" 
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Zero-ETL for Postgres: Live-query cloud APIs with 100+ new FDWs"
>
</iframe>
</div>

## Tap into the Steampipe ecosystem

The Steampipe hub currently provides more that 100 extensions that all work like the ones you've seen here. There are extensions to query infrastructure ([AWS](https://hub.steampipe.io/plugins/turbot/aws), [Azure](https://hub.steampipe.io/plugins/turbot/azure), [GCP](https://hub.steampipe.io/plugins/turbot/gcp), [Kubernetes](https://hub.steampipe.io/plugins/turbot/kubernetes)), SaaS ([ClickUp](https://hub.steampipe.io/plugins/theapsgroup/clickup), [Jira](https://hub.steampipe.io/plugins/turbot/jira), [Google Workspace](https://hub.steampipe.io/plugins/turbot/googleworkspace), [Microsoft 365](https://hub.steampipe.io/plugins/turbot/microsoft365), [Salesforce](https://hub.steampipe.io/plugins/turbot/salesforce), [Slack](https://hub.steampipe.io/plugins/turbot/slack)), security tools ([1Password](https://hub.steampipe.io/plugins/turbot/onepassword), [CrowdStrike](https://hub.steampipe.io/plugins/turbot/crowdstrike), [PAN-OS](https://hub.steampipe.io/plugins/turbot/panos), [Trivy](https://hub.steampipe.io/plugins/turbot/trivy)), and IaC ([Ansible](https://hub.steampipe.io/plugins/turbot/ansible), [CloudFormation](https://hub.steampipe.io/plugins/turbot/awscfn), [Terraform](https://hub.steampipe.io/plugins/turbot/terraform)), directories ([Google Directory](https://hub.steampipe.io/plugins/turbot/googledirectory), [LDAP](https://hub.steampipe.io/plugins/turbot/ldap), [Okta](https://hub.steampipe.io/plugins/turbot/okta)), and network tools ([crt.sh](https://hub.steampipe.io/plugins/turbot/crtsh), [ipstack](https://hub.steampipe.io/plugins/turbot/ipstack), [Tailscale](https://hub.steampipe.io/plugins/turbot/tailscale)). The ecosystem keeps growing as we and our contributors create new Steampipe plugins that will now also be available as standalone Postgres extensions.

Are you ready to get started? Head over to [Steampipe downloads](https://steampipe.io/downloads?install=postgres), try a few of these extensions, and [let us know](https://turbot.com/community/) how it goes!



Your SQLite database just gained a new superpower: the ability to fill tables with data from cloud APIs. Actually there are [more than 100 of these superpowers](https://hub.steampipe.io/plugins), that's how many new SQLite extensions just appeared in the world. We'll use the [GitHub extension](https://hub.steampipe.io/plugins/turbot/github) to illustrate but they all work the same way. You write SQL like this.

```sql
select
  state, closed_at, title
from
  github.github_issue
where
  repository_full_name = 'turbot/steampipe-plugin-github'
order by state desc, closed_at desc;
```

And you get results like this.

```
+--------+---------------------------+-----------------------------------------------------------------------------------------------------------+
| state  | closed_at                 | title																								     |
+--------+---------------------------+-----------------------------------------------------------------------------------------------------------+
| OPEN   | <null>                    | A way to query for assignee on Pull Requests                                                              |
| OPEN   | <null>                    | Add `created_at` list key column and potentially others in `github_actions_repository_workflow_run` table |
| CLOSED | 2023-11-06T23:38:53-08:00 | Queries fail on GHE due to missing fields                                                                 |
| CLOSED | 2023-10-24T08:14:09-07:00 | 502 Bad Gateway with repos with large amount of  PRs
```

In this case we're asking the GitHub extension to tell us about [its own issues](https://github.com/turbot/steampipe-plugin-github/issues). The [github_issue table](https://hub.steampipe.io/plugins/turbot/github/tables/github_issue) is just one of 55 tables provided by the GitHub extension, and that extension is just one of [100+](https://hub.steampipe.io/plugins). Some represent APIs as a handful of tables, others as hundreds of tables; the [AWS extension](https://hub.steampipe.io/plugins/turbot/aws) leads the pack with over 450. There are more than 2000 tables in all, each clearly documented with copy/paste/run examples.

When you query one of these tables, the extension fills with live data from underlying APIs. It retains query results in its cache for five minutes, after which a repeat of the same query will again fetch from APIs. Why this ephemeral approach? It's our take on zero-ETL: pull data directly from sources, and use SQL as the uniform way to ask questions about the current state of those sources. APIs are fast enough to answer a wide range of such questions without needing to warehouse the data.

These queries can join with your own SQLite tables, and/or with queries of virtual tables provided by other extensions. In [Using Shodan to test AWS Public IPs](https://steampipe.io/blog/use-shodan-to-test-aws-public-ip) we showed how to query for AWS EC2 instances with public IP addresses deemed vulnerable by Shodan.

```sql
select
  instance_id,
  ports,
  vulns,
  security_groups
from
  aws_ec2_instance
left join
  shodan_host on public_ip_address = ip
where
  public_ip_address is not null;
```

Ready to try? Let's get started.

## Install and configure the GitHub extension

To install the extension, [download](https://steampipe.io/downloads?install=sqlite) the version for your OS and CPU architecture. Now it's the usual drill you'll be familiar with if you've ever loaded and used a SQLite extension. In the SQLite shell, with a version of SQLite that enables extension loading, do this.

```
.load /home/jon/anywhere-sqlite/steampipe_sqlite_extension_github.so
```

To configure the extension with your GitHub token, do this.

```
select github_configure('token='"ghp_…""');
```

Steampipe plugins often support multiple authentication methods that you configure in `.spc` files, or by way of environment variables, or by defaulting to the credentials you use with a CLI tool. With a standalone SQLite extension there's no `.spc` file. The standard env vars used to configure a Steampipe plugin will work if you set them before starting your instance of SQLite. So in this case if you've set `GITHUB_TOKEN` you can skip the `configure_github()` call.

## What does Zero-ETL mean?

### Zero to query in seconds

It's quick and easy to install an extension, configure credentials, and immediately query the predefined foreign tables it provides.

Install the SQLite extension for a [Steampipe plugin](https://hub.steampipe.io/plugins), configure credentials, and you're ready to go. Immediately query the provided tables, join across them, or join with you own tables and/or those provided by other Steampipe SQLite extensions.

### Zero infrastructure

No data warehouse is required, and persistence is optional. Just write SQL queries to ask questions of cloud APIs and receive instant answers from live API calls.

Our zero-ETL approach stores nothing permanently by default; your foreign tables represent the current state of the systems whose APIs they pull from. But you do have the option to retain results. To do that, prepend `create materialized view TABLE_NAME` as to any query of a foreign table.

### Zero roadblocks

These open-source extensions are available for [more than 100 services](https://hub.steampipe.io/plugins). Each provides prebuilt tables that reflect lessons learned by users and contributors as the suite has evolved and matured.

There are thousands of tables that embody consistent access patterns, and each table is well-documented with examples. Missing a table? Add one! Missing a plugin? Build one!

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/ZWZ11A9q6jw"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Steampipe SQLite extensions"
>
</iframe>
</div>

## Tap into the Steampipe ecosystem

The Steampipe hub currently provides more than 100 extensions that all work like the ones you've seen here. There are extensions to query infrastructure ([AWS](https://hub.steampipe.io/plugins/turbot/aws), [Azure](https://hub.steampipe.io/plugins/turbot/azure), [GCP](https://hub.steampipe.io/plugins/turbot/gcp), [Kubernetes](https://hub.steampipe.io/plugins/turbot/kubernetes)), SaaS ([ClickUp](https://hub.steampipe.io/plugins/theapsgroup/clickup), [Jira](https://hub.steampipe.io/plugins/turbot/jira), [Google Workspace](https://hub.steampipe.io/plugins/turbot/googleworkspace), [Microsoft 365](https://hub.steampipe.io/plugins/turbot/microsoft365), [Salesforce](https://hub.steampipe.io/plugins/turbot/salesforce), [Slack](https://hub.steampipe.io/plugins/turbot/slack)), security tools ([1Password](https://hub.steampipe.io/plugins/turbot/onepassword), [CrowdStrike](https://hub.steampipe.io/plugins/turbot/crowdstrike), [PAN-OS](https://hub.steampipe.io/plugins/turbot/panos), [Trivy](https://hub.steampipe.io/plugins/turbot/trivy)), and IaC ([Ansible](https://hub.steampipe.io/plugins/turbot/ansible), [CloudFormation](https://hub.steampipe.io/plugins/turbot/awscfn), [Terraform](https://hub.steampipe.io/plugins/turbot/terraform)), directories ([Google Directory](https://hub.steampipe.io/plugins/turbot/gcp), [LDAP](https://hub.steampipe.io/plugins/turbot/ldap), [Okta](https://hub.steampipe.io/plugins/turbot/okta)), and network tools ([crt.sh](https://hub.steampipe.io/plugins/turbot/crtsh), [ipstack](https://hub.steampipe.io/plugins/turbot/ipstack), [Tailscale](https://hub.steampipe.io/plugins/turbot/tailscale)). The ecosystem keeps growing as we and our contributors create new Steampipe plugins that will now also be available as standalone SQLite extensions.

Are you ready to get started? Head over to [Steampipe downloads](https://steampipe.io/downloads?install=sqlite), try a few of these extensions, and [let us know](https://steampipe.io/community/join) how it goes!



Here's the typical pattern for extracting data from a cloud API: find the wrapper library for your programming language, learn how it models the underlying API, then write a script to manipulate the model and retrieve the data you need. When you need data from another cloud API it's the same thing all over again: find a different library, learn how it models that API, and write a different kind of script specific to that API and library.

Here's a different approach. Let's export gists using the GitHub API.

```
./steampipe_export_github github_my_gist > gists.csv
```

Here's the resulting CSV file in a spreadsheet.

<p>
<img alt="steampipe export displayed in spreadsheet" src="/images/blog/2023-12-steampipe-export/spreadsheet.png" />
</p>

In this case we're asking `steampipe_export_github`, which is a variant of the [Steampipe plugin for GitHub](https://hub.steampipe.io/plugins/turbot/github), to export the [github_my_gist table](https://hub.steampipe.io/plugins/turbot/github/tables/github_my_gist) which is just one of 55 tables provided by the GitHub plugin. And that plugin is just one of [100+](https://hub.steampipe.io/plugins) that work the same way. Some represent APIs as a handful of tables, others as hundreds of tables; the [AWS plugin](https://hub.steampipe.io/plugins/turbot/aws) leads the pack with over 450. There are more than 2000 tables in all, each clearly documented.

When you export one of these tables, the tool fetches live data from underlying APIs. It's our take on zero-ETL: pull data directly from sources, no database required! APIs are fast enough to answer many questions without needing to warehouse the data.

## Install the GitHub exporter

Ready to try? To install any of these exporters, visit [Steampipe downloads](https://steampipe.io/downloads?install=export), copy the install command, and run it. Type the name of your plugin and you'll have a new tool called `steampipe_export_github` or `steampipe_export_aws` or `steampipe_export_slack`.

And that's all there is to it! Every Steampipe exporter works exactly like `steampipe_export_github`.

## What does Zero-ETL mean?

### Zero to query in seconds

It's quick and easy to install a export tool and immediately export the predefined foreign tables it provides.

Install the export tool for a [Steampipe plugin](https://hub.steampipe.io/plugins) and you're ready to go.

### Zero infrastructure

No data warehouse is required, and persistence is optional. Just extract live data from cloud APIs into your terminal. To save it, redirect output to a CSV file.

### Zero roadblocks

These open-source export tools are available for [more than 100 APIs and services](https://hub.steampipe.io/plugins). Each provides prebuilt tables that reflect lessons learned by users and contributors as the suite has evolved and matured.

There are thousands of tables that embody consistent access patterns, and each table is well-documented with examples. Missing a table? Add one! Missing a plugin? Build one!

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/bYtxaZdAvkI"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Steampipe Export"
>
</iframe>
</div>

## Tap into the Steampipe ecosystem

The Steampipe hub currently provides more than 100 export tools that work like the one you've seen here. Tools are available for cloud infrastructure ([AWS](https://hub.steampipe.io/plugins/turbot/aws), [Azure](https://hub.steampipe.io/plugins/turbot/azure), [GCP](https://hub.steampipe.io/plugins/turbot/gcp), [Kubernetes](https://hub.steampipe.io/plugins/turbot/kubernetes)), SaaS ([ClickUp](https://hub.steampipe.io/plugins/theapsgroup/clickup), [Jira](https://hub.steampipe.io/plugins/turbot/jira), [Google Workspace](https://hub.steampipe.io/plugins/turbot/googleworkspace), [Microsoft 365](https://hub.steampipe.io/plugins/turbot/microsoft365), [Salesforce](https://hub.steampipe.io/plugins/turbot/salesforce), [Slack](https://hub.steampipe.io/plugins/turbot/slack)), security tools ([1Password](https://hub.steampipe.io/plugins/turbot/onepassword), [CrowdStrike](https://hub.steampipe.io/plugins/turbot/crowdstrike), [PAN-OS](https://hub.steampipe.io/plugins/turbot/panos), [Trivy](https://hub.steampipe.io/plugins/turbot/trivy)), directories ([Google Directory](https://hub.steampipe.io/plugins/turbot/googledirectory), [LDAP](https://hub.steampipe.io/plugins/turbot/ldap), [Okta](https://hub.steampipe.io/plugins/turbot/okta)), and network tools ([crt.sh](https://hub.steampipe.io/plugins/turbot/crtsh), [ipstack](https://hub.steampipe.io/plugins/turbot/ipstack), [Tailscale](https://hub.steampipe.io/plugins/turbot/tailscale)). The ecosystem keeps growing as we and our contributors create new Steampipe plugins that will now also be available as standalone exporters.

Are you ready to get started? Head over to [Steampipe downloads](https://steampipe.io/downloads?install=export), try a few of these export tools, and [let us know](https://steampipe.io/community/join) how it goes!



The integration between ServiceNow CMDB and Turbot Guardrails bridges the gap between your business context with your cloud & security controls.

## Why sync ServiceNow CMDB data

Syncing ServiceNow CMDB data brings critical context to your cloud resources in Guardrails. Details like application ownership, data classification, and cost centers can then be used to:

- Automatically tag resources with application context
- Restrict access to groups, departments and business units
- Enable security controls based on data sensitivity
- Limit workload usage based on cost center budgets
- And more...

Turbot Guardrails has thousands of OOTB controls to ensure security and optimize operations. Adding this additional context improves cloud visibility, cost optimization, and governance to enable automated policy enforcement.

## How the integration works

Syncing data between systems is automated and continuous. Similar to how Guardrails captures cloud inventory and configs, now any ServiceNow CMDB table can sync in real-time as well.

For example, the ServiceNow Application table (cmdb_ci_appl):

<img src="/images/blog/2023-12-context-aware-guardrails-servicenow-integration/servicenow-cmdb-ci-appl-table.jpg" style={{ boxShadow: 'none' }}></img>

These applications are now visible and managed in Guardrails:

<img src="/images/blog/2023-12-context-aware-guardrails-servicenow-integration/turbot-guardrails-servicenow-cmdb-ci-appl-table.jpg" style={{ boxShadow: 'none' }}></img>

The integration keeps data in sync in real-time as changes occur:

<img src="/images/blog/2023-12-context-aware-guardrails-servicenow-integration/turbot-guardrails-servicenow-cmdb-ci-appl-record-update.jpg" style={{ boxShadow: 'none' }}></img>

## Enrich Guardrails Policies with Context

With ServiceNow data ingested, new possibilities open up for Guardrails automation, such as:

### Auto-Tag Resources with enterprise context

Identifying who is responsible for the application, which cost center, and other CMDB details is simple to auto-tag on the cloud resource. For example, this Turbot Guardrails calculated policy template associates ServiceNow data such as application's ID, name and data classification:

```
app_id: "{{ $.application.u_id }}"
app_name: "{{ $.application.u_name }}"
data_classification: "{{ $.application.u_data_classification }}"
```

Which then applies the keys and values to be tagged anytime a cloud resource is created or updated, or when the ServiceNow Application details are updated.

```
"app_id": "APP0001"
"app_name": "CMS App FLX"
"data_classification": "Restricted"
```

### Apply data protection rules with data sensitivity context

CMDB data can be used in policy logic to auto-configure resources appropriately. If an application is marked "Restricted" in ServiceNow, the associated AWS S3 Buckets can enforce versioning.

Using a calculated policy template, you can associate the context into the conditions of the policy enforcement:

```
{%- if $.resource.tags.data_classification == "Restricted" -%}
"Enforce: Enabled"
{%- else -%}
"Enforce: Disabled"
{%- endif -%}
```

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/_gMI_70Sq94"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="ServiceNow + Guardrails: Context-aware cloud & security automation Demo"
>
</iframe>
</div>

## Get started with context-aware guardrails

With the [integration](https://turbot.com/guardrails/docs/integrations/servicenow) between the ServiceNow and Turbot Guardrails, you can empower cloud resources with critical business context they've been lacking. This helps improve visibility, optimize costs, and enforce cloud governance best practices.

If you are new to Turbot Guardrails, reach out to us to [get started](https://turbot.com/start). For existing customers, simply follow the [ServiceNow integration guide](https://turbot.com/guardrails/docs/integrations/servicenow) to enable in your Turbot Guardrails workspace, and learn how you can also [sync cloud resources to ServiceNow](https://turbot.com/blog/2023/12/cmdb-sync-guardrails-servicenow-integration), further enriching your cloud & security controls.

Join our [Slack community](https://turbot.com/community/join) <strong>#guardrails</strong> channel to discuss your use cases and provide feedback as you leverage the integration.



Turbot Guardrails integration into ServiceNow injects continuous real-time cloud resource details, eliminating manual discovery gaps. Syncing occurs instantly across your multi-cloud environments as resources are created, updated or deleted. This real-time accuracy helps eliminate blindspots and empowers more responsive operations.

## Why continuous cloud sync is required

Manual and scheduled discovery leads to inaccurate data as the cloud is rapidly changing – resources come and go quickly which if it is not real-time will miss which resources have been used, what was their configuration at the time, and who made those changes.

Continuously syncing your cloud resources into ServiceNow enhances the CMDB into a living source of truth for hybrid and multi-cloud enterprises:

- Eliminate blindspots: comprehensive multi-cloud resource coverage.
- Improve accuracy: instant updates prevents missing and incomplete data.
- Flexible mapping: control what data is synced to which CMDB CI tables.
- Enrich context: surface tags as key resource data.

## How to sync cloud resources to ServiceNow CMDB

Each cloud resource type can be configured to sync to the ServiceNow CMDB. Most often you would set the scope of the policy across many cloud resources from all your cloud accounts. In this example we will show how to enable syncing AWS S3 Buckets.

Simply set the Turbot Guardrails policy to “Enforce: Sync” and apply to all or specific cloud accounts:

<img src="/images/blog/2023-12-cmdb-sync-guardrails-servicenow-integration/sync-policy.jpg" style={{ boxShadow: 'none' }}></img>

For the AWS account we enabled the sync for, the following resources will be in scope for the sync:

<img src="/images/blog/2023-12-cmdb-sync-guardrails-servicenow-integration/aws-accounts.jpg" style={{ boxShadow: 'none' }}></img>

Instantly the resources will be added to the associated ServiceNow CMDB table:

<img src="/images/blog/2023-12-cmdb-sync-guardrails-servicenow-integration/servicenow-cmdb-table-aws-s3-buckets.jpg" style={{ boxShadow: 'none' }}></img>

As cloud resources are added, updated, or deleted, Turbot Guardrails handles the configuration drift and keeps ServiceNow updated.

For example when a resource changes, Turbot captures the configuration drift and updates ServiceNow:

<img src="/images/blog/2023-12-cmdb-sync-guardrails-servicenow-integration/turbot-guardrails-configuration-drift-capture.jpg" style={{ boxShadow: 'none' }}></img>

Cloud resource deletion can be managed as a complete synchronization where the record in ServiceNow is deleted as well, or archived to retain its record with an archive status.

## Map cloud resource details to CMDB CI tables

Each cloud resource type can be mapped to new or existing CMDB CI tables. You can also extend off existing tables as well. In this example, the AWS S3 Bucket resource type maps to a new `cmdb_ci_aws_s3_bucket”.

```yaml
table:
  name: cmdb_aws_s3_bucket
  label: AWS > S3 > Bucket
  extendsTable: cmdb_ci_cloud_storage_account
```

Mappings of which columns to create are simple to define in the Table Definition policy:

```yaml
columns:
  - name: bucket_name
    label: Bucket Name
  - name: account_id
    label: Account ID
  - name: region
    label: Region
  - name: tags
    size: 1000
    type: string
    label: Cloud Tags
```

Any data from the cloud provider can be added. To associate more fields simply add as additional columns:

```yaml
- name: versioning_enabled
  label: Versioning Enabled
```

Adding custom fields is common. In this case we want to add a column to capture the Application ID associated from ServiceNow to the cloud resource:

```yaml
- name: app_id
  size: 1000
  type: string
  label: Application ID
```

To populate the added columns dynamically, the Configuration Item policy can define where the data comes from. In this case, this information can be pulled from the cloud resource details such as the `Versioning.Status` and the `app_id` tag:

```
bucket_name: {{ $.resource.data.Name }}
account_id: {{ $.resource.metadata.aws.accountId }}
region: {{ $.resource.metadata.aws.regionName }}
tags: {{ $.resource.turbot.tags }}

versioning_enabled: {{ $.resource.data.Versioning.Status }}
app_id: {{ $.resource.turbot.tags.app_id }}
```

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/7-e1-tBO1ho"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Continuous Cloud CMDB Sync with ServiceNow & Turbot Guardrails Demo"
>
</iframe>
</div>

## Modernize your ServiceNow CMDB

With the [integration](https://turbot.com/guardrails/docs/integrations/servicenow) between the ServiceNow and Turbot Guardrails, you can enable a continuous cloud sync to boost data accuracy and automate discoveries across your multi-cloud environments.

If you are new to Turbot Guardrails, reach out to us to [get started](https://turbot.com/start). For existing customers, simply follow the [ServiceNow integration guide](https://turbot.com/guardrails/docs/integrations/servicenow) to enable in your Turbot Guardrails workspace, and learn how you can also [sync ServiceNow data to Guardrails](https://turbot.com/blog/2023/12/context-aware-guardrails-servicenow-integration), further enriching your cloud & security controls.

Join our [Slack community](https://turbot.com/community/join) <strong>#guardrails</strong> channel to discuss your use cases and provide feedback as you leverage the integration.



We run a major open-source project, [Steampipe](https://steampipe.io), that's spawned [over 200 open-source repositories](https://github.com/turbot). A handful are for the core components, and the majority are for the [plugins](https://hub.steampipe.io/plugins) and [mods](https://hub.steampipe.io/mods) that layer on top of the core. Managing all these repos demands both technical rigor and human connection. That's why Turbot combines proven engineering practices with a focus on community. Here are seven key lessons we've learned along the way.

## Lesson 1: Respond instantly

**Challenge**: It's easy to miss issues and PRs that span hundreds of repos.

**Solution**: Monitor everything and alert the core team.

**Lesson Learned**: People really appreciate instant responses!

When a community member opens a new issue, or raises a PR, we use a GitHub action that runs a Steampipe query every hour and notifies a Slack channel when something new turns up. 

![](/images/blog/2023-10-repo-management/slack-alert.png)

When we scan all the repos we filter out members of our own organization so we can prioritize external contributors.

## Lesson 2: Early communication is essential

**Challenge**: Contributors' ideas may not align with project goals.

**Solution**: Communicate early and often.

**Lesson Learned**: Be crystal-clear about project goals and expectations to avoid painful misunderstandings. 

One memorable lesson came by way of a passionate user who [opened a PR](https://github.com/turbot/steampipe-plugin-aws/pull/847) that would simplify the management of AWS credentials, but create a dependency between Steampipe and the AWS CLI. The idea was appealing, but we were adamant about avoiding unnecessary dependencies. Unfortunately we didn't communicate that stance clearly at the outset. As a result, the enthusiastic contributor invested time and effort, believing it aligned with our goals. Only later, during a more thorough review, did we spot the problem.

Rejecting the contribution at that stage was painful for both the contributor and our team. We learned the hard way that transparency and clear communication about project goals and expectations are essential from the outset. It's crucial to steer enthusiastic contributors away from efforts that don't align with the project's vision.

## Lesson 3: Treat contributors like team members

**Challenge**: Incentivize contributors.

**Solution**: Prioritize external contributors and collaborate closely for quality contributions.

**Lesson Learned**: Being friendly, respectful, and direct fosters collaboration and accelerates progress.

Treating contributors like team members means more than just accepting their contributions. Solicit heir ideas, provide clear instructions, and help them understand what you want to be done. Don't hesitate to ask them to refine their work to meet the project's standards or goals — they will often gladly accept the challenge.

Here's one example: a user needed to specify a custom AWS endpoint, opened [an issue](https://github.com/turbot/steampipe-plugin-aws/issues/1050), and worked with the core team to refine a [pull request](https://github.com/turbot/steampipe-plugin-aws/pull/1053) that we approved and merged.

One valuable practice we've adopted is not waiting too long to merge a contributor's mostly-complete work. People who contribute to open source projects are often busy and are donating their time. Merging their contributions, even if they require minor adjustments, acknowledges their effort and allows the project to move forward swiftly. This approach gives contributors credit for their work and keeps the momentum going.

## Lesson 4: Age reports keep you honest

**Challenge**: Long-forgotten issues and PRs sap enthusiasm and hinder progress.

**Solution**: Use daily age reports to assess the "cost" of open issues and PRs and ensure systematic review.

**Lesson Learned**: It's painful but necessary to create age reports and regularly review them.

 Daily age reports are like a financial statement for open issues and PRs. We calculate the number of days each issue has been open, sum them up, and create a running tally of "days open" for the entire repository. 
 
 This approach helps us identify and prioritize old issues that have been open for an extended period. That happened, to our embarrassment, in [this case](https://github.com/turbot/steampipe-plugin-net/issues/48). The issue was opened last September, there was early engagement, but it fell off our radar and was only resolved a few weeks ago thanks to more systematic review of stale issues.

 The age reports also ensure we stay on top of new issues and PRs to prevent them from accumulating and becoming long-term challenges.

![Open PR and Issue Age Report](/images/blog/2023-10-repo-management/open-plugin-mod-pr-age.png)

## Lesson 5: Burndown is a must

**Challenge**: Long-standing issues and PRs can create bottlenecks.

**Solution**: Dedicate resources regularly to burn them down.

**Lesson Learned**: Visualize trends in order to bend them in the right direction.

We aim for continuous improvement, and to do that we need to measure progress. This isn't something easily done with GitHub alone, across a large set of repos, with filters for external contributors. We use trend charts built from daily snapshots of the age reports. 

![burndown trends](/images/blog/2023-10-repo-management/open-pr-issue-trends.png)

There's still room for more improvement but we've made good progress!

## Lesson 6: Consistency is the name of the game

**Challenge**: Maintaining consistency across repos is vital.

**Solution**: Implement benchmarks to enforce uniform settings, both public-facing and internal.

**Lesson Learned**: Automate your consistency checks!

Consistency is the bedrock of an efficient open-source project. To achieve it, we conduct checks across multiple dimensions:

- **Naming**: We ensure that repository names, control names, and table names follow a standardized format.

- **Formatting**: Our consistency checks extend to query examples, ensuring they adhere to established formatting standards.

- **Documentation**: We prioritize thorough, high-quality documentation, including column descriptions, query examples, and credential setup instructions.

- **Quality**: Our benchmarks include checks for tables that must handle pagination and error handling to maintain high-quality code.

- **Coverage**: We assess whether there are enough tables to make the repository useful and ensure that columns are complete.

- **Repository setup**: Consistency checks extend to public-facing aspects such as licenses, descriptions, tags, and social images.

![Consistency Checks](/images/blog/2023-10-repo-management/mod-checks.png)

## Lesson 7: Documentation is king

**Challenge**: Users need good examples that make our software easier to use.

**Solution**: Expect contributions to include high-quality documentation.

**Lesson Learned**: When you prioritize documentation you improve usability and reduce the support burden.

Documentation isn't just an afterthought; it's the backbone of user-friendly software. Prioritizing good descriptions for the columns of tables provided by plugins, for example, often inspires redesign of those tables to make them more usable. And with quality documentation, many support questions can be answered with a link.

## Be fast, friendly, and clever

When open source projects thrive it's because users bring new ideas — and contributors deliver implementations — that augment what the core team is able to do on its own. To realize those benefits at scale, you need a process that blends technical rigor and human connection. The process we've built embodies the lessons we've learned. We're confident that it helps us be better stewards of our community, but you don't need to take our word for it. You can meet our users and contributors in our [Slack community](/community/join) and learn more about how our community works.










Our announcements this week span the breadth of Turbot offerings, from the open-source Steampipe tool and its fast-growing collection of plugins and mods, to Turbot Pipes which lifts the Steampipe ecosystem into the cloud and adds team collaboration, pipelines, and persistent storage, to Turbot Guardrails which brings new notification and tagging features to our enterprise product.

## Watch the Wrap Party

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/OupJb4yHJ8I" 
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Launch Week Wrap Party"
>
</iframe>
</div>

<div className="text-2xl font-bold pt-10">Full Week Recap</div>

## Sunday Oct 1st: Hacktoberfest 2023

As a warmup exercise we kicked off [our second Hacktoberfest](https://turbot.com/blog/2023/10/hacktoberfest-2023). Last time we merged 38 contributions from 30 contributors representing 13 countries. We have since grown to 2,000 community members who are passionate engineers that love our open source projects as much as we do. Tune in next month for a recap of the 2023 event!

## Monday Oct 2nd: Pushing Steampipe beyond limits

In [v0.21.0: Dials for tuning Steampipe](https://steampipe.io/blog/memory-management-rate-limiters-diagnostics) we announced the latest version of the popular open-source tool that now also powers Turbot Pipes. We've always pushed Steamipe to run faster and more reliably, at scale, than you ever thought possible. Because API throttling and memory pressure can hinder your ability to scan cloud infrastructure, we've doubled down on the already-impressive mechanisms that enable Steampipe to handle huge workloads gracefully.

In [select \* from shell;](https://steampipe.io/blog/exec-plugin) we show how the [exec plugin](https://hub.steampipe.io/plugins/turbot/exec) broadens Steampipe’s core <em>select \* from cloud;</em> mission. Run it everywhere Steampipe can run, and use it to combine SQL with shell commands. The exec plugin was a key enabler for our new [Docker Compliance](https://hub.steampipe.io/mods/turbot/docker_compliance) mod; we know you’ll find other uses for this fundamental and versatile tool.

## Tuesday Oct 3rd: Go beyond reporting with Guardrails

Guardrails reports all the events and actions in your cloud environments. In [Notifications for Guardrails – Instant alerts, with one-click fixes](https://turbot.com/guardrails/blog/2023/10/guardrails-notifications) we announced custom notifications that filter alerts and direct them to the individuals and teams that need to know about specific misconfigurations and remediations. We provide templates for Slack, Teams, and email, and we show how you can easily tweak them with your own branding, links, and messaging.

In [Automate your tagging strategy with Guardrails](https://turbot.com/guardrails/blog/2023/10/guardrails-intelligent-tagging) we show how Guardrails' automated tagging, which has long enabled customers to classify and contextualize cloud resources – in order to visualize infrastructure, manage compliance, and contain costs – becomes even more powerful with new tag helpers for calculated policies.

## Wednesday Oct 4th: Query your infrastructure as code!

Steampipe's common model for code, data, and APIs enables DevSecOps pros to inventory cloud assets, control costs, check for compliance with standard security frameworks, investigate incidents, visualize cloud infrastructure, and answer ad-hoc questions.

In [select \* from code;](https://steampipe.io/blog/select-star-from-code) we show how Turbot's offerings work across all layers of your stack, enabling you to shift left easily, confidently, and comprehensively.

[Steampipe-powered GitOps](https://steampipe.io/blog/github-actions) announces a pair of new actions that enable you to run Steampipe queries and checks in GitHub workflows, flag IaC compliance issues with annotations that point directly to offending lines of YAML code, and push check results to Pipes workspaces where teams can evaluate progress toward compliance.

In [Interrogate Terraform](https://steampipe.io/blog/interrogate-terraform) we show how our unique ability to run SQL queries against both configurations and deployed infrastructure is now enhanced with the ability to interrogate plan and state files, and to run hundreds of new controls.

A companion article, [Interrogate Kubernetes](https://steampipe.io/blog/interrogate-kubernetes), tells the same story from a K8s perspective, featuring new tables that enable you to query helm charts.

## Thursday Oct 5th: Turbot Pipes: Cloud intelligence & security

In early 2022 we quietly launched a free preview of the product that would become Turbot Pipes. Many of you helped us refine it along the way, and now we celebrate the official launch in [Turbot Pipes: Free for Developers, great for Teams](https://turbot.com/pipes/blog/2023/10/announcing-turbot-pipes). Building on Steampipe's SQL-powered unification of cloud APIs and thriving ecosystem of plugins and mods, Pipes enables teams to work in shared spaces that enhance Steampipe-based queries, benchmarks, dashboards with scheduled pipelines, persistence, and collaboration.

You've long been able to persist snapshots of your Pipes workspaces. In [Datatank: Blow past API speed limits with scheduled data sync](https://turbot.com/pipes/blog/2023/10/datatank-launch) we announced a powerful – and complementary – new kind of persistence. Create a Datatank in your Pipes workspace to automatically synchronize live data with permanent tables that accelerate queries, benchmarks, and dashboards. Now you’ve got the best of both worlds: instant access to live data at scale when APIs can support it, and instant access to nearly-live data when they can’t.

In [Zapier for DevOps](https://turbot.com/pipes/blog/2023/10/zapier-launch) we round out the Pipes story for this Launch Week. Developers of Zapier "Zaps" use a point-and-click interface to build business workflows that interconnect thousands of services. Now they can inject Pipes into those workflows to query and join across data sources that Zapier doesn’t support, and use Pipes’ SQL superpower to make more effective use of Zapier’s portfolio of apps.

## Friday Oct 6th: Celebrating our community

We cherish our community of 2,000+ practitioners working together to conquer cloud complexity. In [How we manage 200+ open-source repos](https://turbot.com/blog/2023/10/repo-management/) we share the process we've built to help us respond more rapidly to your issues, bug reports, and contributions, and to track our progress as we improve.

In [Keep track of all Turbot product changes with the new Turbot Changelog](https://turbot.com/blog/2023/10/turbot-changelog/) we announce a new [changelog](https://turbot.com/changelog) that spans all our offerings so you can easily monitor everything that we ship. It's a lot! Now you can see everything at a glance, and also follow the action via [RSS](https://turbot.com/changelog/feed.xml) or the #changelog channel of our [Slack community](https://turbot.com/community/join).

## And that's a wrap!

The features we announced this week are individually exciting, but the Turbot story is bigger than the sum of its parts. Our products work together to deliver cloud security and intelligence with incomparable breadth, depth, power, speed, and versatility. And we're still just getting started. Enjoy the current batch of new features, [let us know](/community/join) how it goes, and stay tuned for our next Launch Week in a few months!



As Turbot's product suite expands with offerings such as [Steampipe](https://steampipe.io), [Turbot Guardrails](https://turbot.com/guardrails), and [Turbot Pipes](https://turbot.com/pipes), maintaining clear and direct communication becomes essential. We're thrilled to introduce the [Turbot Changelog](https://turbot.com/changelog), a one-stop, easily digestible resource for all updates and enhancements across our products.

The Turbot Changelog is your definitive source for updates across all Turbot products. In the past, tracking updates required navigating multiple pages or platforms. Now, with the centralized Changelog, all necessary information is consolidated and made accessible through various channels. You can filter updates by product, subscribe to RSS feeds for specific products, view recent updates within the Pipes and Guardrails interfaces, and receive Slack notifications in our [community’s](https://turbot.com/community/join) **#changelog** channel. The Changelog and associated RSS feeds are updated with each product release and enhancement, keeping you informed at all times.

## Unified Turbot Changelog
Our [Turbot Changelog](https://turbot.com/changelog) brings together updates for all Turbot products:

![](/images/blog/2023-10-turbot-changelog/changelog.png)

## Product level changelogs & RSS feeds
Plus, each product has its own unique URL and RSS subscription link:
- [Steampipe](https://steampipe.io/changelog) ([RSS](https://steampipe.io/changelog/feed.xml))
- [Turbot Pipes](https://turbot.com/pipes/changelog) ([RSS](https://turbot.com/pipes/changelog/feed.xml))
- [Turbot Guardrails](https://turbot.com/guardrails/changelog) ([RSS](https://turbot.com/guardrails/changelog/feed.xml))

![](/images/blog/2023-10-turbot-changelog/steampipe-changelog.png)

## Turbot Community changelog notifications
Stay connected, supported, and informed by joining our [Slack community](https://turbot.com/community/join) and following the #changelog channel. This channel, subscribed to all product RSS feeds, provides real-time updates and a space for Q&A in threads:

![](/images/blog/2023-10-turbot-changelog/community-changelog.png)

## Recent updates directly in-product
Instantly see the latest changes from the product; for example, [Turbot Pipes’ homepage](https://pipes.turbot.com) has the most recent Pipes & Steampipe Changelog items:

![](/images/blog/2023-10-turbot-changelog/product-changelog.png)

## How to leverage the Turbot Changelog

- **Universal Accessibility:** The Turbot Changelog is accessible to everyone. It's a transparent showcase of our commitment to continuous improvement. Anyone, from prospects to partners, can stay updated.

- **Share with your Team:** Share a feature, update, or enhancement with just a link. Keeping your team or network informed as there are announcements they are interested in.

- **Stay Updated in Real-time:** Subscribe to our RSS subscriptions to never miss an update. Be the first to know, whether it's a product-specific update or a general enhancement.

- **Join the Conversation:** More than a static list, our changelog is a community hub. Each update is automatically shared in our Slack community's [#changelog](https://turbot.com/community/join) channel, inviting collaborative discussions on the latest changes.


We're thrilled to announce the general availability of [Turbot Pipes](https://turbot.com/pipes)! Introducing a free Developer tier for individuals and a usage-based Team tier crafted for DevOps teams. Turbot Pipes is poised to revolutionize the way your team manages, secures, and collaborates on cloud services, acting as an invaluable ally for every DevOps team to empower them with data-driven insights essential for informed decision-making.

## Journey of Turbot Pipes
Developers love our open-source [Steampipe CLI](https://steampipe.io). Pipes enhances Steampipe as a centralized service with additional security, scalability, & collaboration features making it easier to adopt as a team. 

Since Pipes was quietly introduced through our docs, it has garnered adoption from over 1,500 users, ranging from hobbyists to Fortune 500 companies. It’s used for diverse applications, including [security assessments](https://turbot.com/pipes/blog/2022/02/pipes-for-security-posture-assessment) automation and as an integration point to power [GitOps pipelines](https://steampipe.io/blog/github-actions). Managed Service Providers and software firms have integrated Pipes into their systems, drawn by its Zero-ETL architecture to expand their breadth of service coverage. 

We've secured our [SOC2 Type II](https://turbot.com/pipes/blog/2023/05/soc2-2023) certification, introduced more [integration points](https://turbot.com/pipes/blog/2023/10/zapier-launch), expanded support for new [Steampipe plugins](https://hub.steampipe.io/plugins), and enhanced [data synchronization and reporting performance](https://turbot.com/pipes/blog/2023/10/zapier-launch).

## Cloud intelligence & security: crafted for DevOps
Turbot Pipes extends Steampipe’s capabilities, introducing features pivotal for team collaboration, workflow integration, enhanced security, and scalability:

- **Effortless Zero-ETL:** Serve as the central hub for all your cloud data, providing seamless integrations without the hassles of traditional ETL or database management overhead.
- **Robust Security & Operations:** Manage credentials securely with role-based access controls, conduct comprehensive security assessments of your cloud environments, and pinpoint opportunities for operational optimizations.
- **Enhanced Collaboration:** Facilitate sharing of data-driven insights, allowing team members to share connections, send report snapshots, and pipe query results amongst team members giving them the necessary context they need for informed decision-making.
- **Optimized Performance:** With persisted data storage, experience accelerated queries and reporting that not only empower your team to work faster but also streamline your workflows efficiently.

## Transparent pricing: tailored for DevOps
Turbot Pipes is free for Developers, allowing exploration and personal use at no cost. The Team tier, commencing at $49 per month, includes three users, 2,000 compute minutes, and 20GB of storage. Scale up as needed, adding more users and paying only for the additional compute and storage you require, with the flexibility to cancel anytime.

<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/pricing" target="_blank">
 <img alt="Turbot Pipes Pricing 2023" src="/images/blog/2023-10-announcing-turbot-pipes/pipes-pricing.png" />
  </a>
  </div>

For additional information, please visit our [Pricing](https://turbot.com/pipes/pricing) page. Current preview users will enjoy an extended transition period with increased usage limits; more information is in the Pricing [FAQ](https://turbot.com/pipes/pricing#i-am-on-the-free-preview-of-turbot-pipes-what-happens-now).

## Easy to start, easier to love
Getting started with Turbot Pipes is simple. Sign up for the Developer free tier at [pipes.turbot.com](https://pipes.turbot.com). When you're ready to explore further, upgrade to the Team tier starting with a 14-day free trial.

Join the conversation with peers in the **#pipes** channel of our [Slack community](https://turbot.com/community/join). Share your questions, discuss use cases, and let’s innovate together!



The [Turbot Pipes app for Zapier](https://zapier.com/apps/turbot-pipes), announced today, opens the world of DevOps data to Zap developers. If you're a DevOps pro, this new integration means you can enrich operational data with anything available from Zapier's vast catalog of connectors. Pipes' complementary connectors and data-wrangling superpowers kick Zap workflows into a whole new gear. Let's look at simple examples of two basic patterns: Pipes triggers a Zapier workflow, and Pipes runs as an intermediate step.

## Pipes as a Zap trigger: Notify when new AWS IAM access keys are created

In this scenario we want to be notified when a new AWS IAM access key is created. Here's the Pipes query we'll use.

```
select
  *
from
  aws_iam_access_key
where
  create_date > now() - interval '1 day' 
```

To use the query in a Zap, log in to Zapier, click `Create` → `New Zap`, name it `New access key`, and click `Trigger`. In the `Change Trigger` dialog box, search for `Turbot Pipes`, and select the app. You're then prompted to login to Pipes; do that.

There's only one choice for the trigger event, `New Row (Custom Query)`, choose that and click `Continue`.

Choose your workspace in the `Trigger` section, then enter your query, and click `Continue`.

![](/images/blog/2023-launch-zapier/zap-access-key-trigger.png)

In the `Test` section, click `Test Trigger`. Zapier will display up to three query results. If nothing is found, you probably don't have any fresh access keys so create one and then try again. When successful, Zapier will show you the results and invite you to select a record to continue building your Zap. Click `Continue with selected record`.

![](/images/blog/2023-launch-zapier/zap-access-key-continue-with-selected.png)

Now you're invited to choose the next app in the pipeline. We'll use Gmail to send the notification so search for it, select it, choose the `Send Email` action, and fill in `To` and `From` with your email address. You can customize the `Subject` with the `User Name` field of the sample record, and likewise customize the body to include `User Name` and `Create Date`. (Columns you asked for in the query are available in both contexts.) Then click `Test step`. You should soon receive an email like this:

![](/images/blog/2023-launch-zapier/zap-access-key-email.png)

If you want to actually use this Zap, click `Publish` to activate it.

Note that each action performed by a Zap costs you one task, and if you send a lot of emails this way you'll soon exhaust your plan's quota. SQL to the rescue! Your Pipes query can use Postgres functions (`concat`, `string_agg`) to combine multiple records into a single text value delimited by HTML tags. Then you can toggle the body type of the Zapier Gmail app from `plain` to `html` to send an HTML email.

## Pipes as a Zap action: Enrich a Jira issue

Now let's kick off a Zap with a Jira trigger. In Zapier's Jira app, set up a trigger on the `New issue` event. We'll pretend there's a convention for titling new issues: if the title contains an IP address (e.g. <em>Please investigate 10.84.1.0</em>), then Pipes will query `aws_vpc` for a match and, if there is one, will build a notification that enriches the bare IP address with data from `aws_vpc` and `aws_vpc_security_group`.

The setup for the trigger is similar to what we've already seen: start a new Zap, choose Jira, log in, choose the `New issue` event, choose your Jira project, and test to verify that Zapier can find a sample issue record.

Now we'll add a Pipes step and, this time, we'll write the query in the action instead of the trigger. The query starts by listing VPC ids and hosts, then looks for an IP addresses in the issue's `Summary` field. If there is one, the query uses the `vpc_id` to join with `aws_vpc_security_group` and report details from both. If there isn't one, the Zap ends here.

![](/images/blog/2023-launch-zapier/zap-jira-query.png)

Finally we create a Jira step that adds a comment to the issue that triggered the workflow.

![](/images/blog/2023-launch-zapier/zap-add-jira-comment.png)

When you test the Zap, you should find a new comment on the Jira issue.

![](/images/blog/2023-launch-zapier/zap-jira-issue-with-added-comment.png)

If your query returns multiple rows, you'll again burn one task for each. If you want to combine the rows there isn't an HTML option in this case, but you can use Postgres functions (`row_to_json`, `json_agg`) to convert the result set into a single value.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/6aJX4JdAjIE" 
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Zapier for DevOps: Use Pipes to unlock 5,000+ no-code automations"
>
</iframe>
</div>

## Get started with Turbot Pipes + Zapier

We've shown that, as a Zap developer, you can use Pipes to trigger a Zap with data that's otherwise unavailable, including from a vast collection of tables drawn from the major clouds ([AWS](https://hub.steampipe.io/plugins/turbot/aws/tables), [Azure](https://hub.steampipe.io/plugins/turbot/azure/tables), [GCP](https://hub.steampipe.io/plugins/turbot/aws/tables)) and others of interest to DevSecOps pros including [IPinfo](https://hub.steampipe.io/plugins/turbot/ipinfo/tables), [GitHub](https://hub.steampipe.io/plugins/turbot/github/tables), [Net](https://hub.steampipe.io/plugins/turbot/net/tables), [Shodan](https://hub.steampipe.io/plugins/turbot/shodan/tables), and many more. The Pipes queries you use in your Zap can query these Pipes-provided sources individually and also, to powerful effect, [join across](https://steampipe.io/blog/use-shodan-to-test-aws-public-ip) them.

We've also shown that you can use Pipes in the middle of a workflow to receive results from another app's trigger, enrich them with data from a Pipes query, and pass the enriched results along to a next step. 

But these are just the simplest possible examples of some basic patterns. The sky's the limit, so go build some workflows with Pipes + Zapier and [let us know](/community/join) how it goes!




Pipes enables users to [schedule and share](https://turbot.com/pipes/blog/2023/01/scheduled-queries-benchmarks-and-dashboards) persistent snapshots of their workspaces. Now it also provides a powerful – and complementary – form of persistence: Datatank. With this new feature you can automatically sync live data with persistent tables. Why? Although Pipes [parallelizes your queries](https://aws.amazon.com/blogs/opensource/querying-aws-at-scale-across-apis-regions-and-accounts/) and can pull data insanely fast, the laws of data physics still apply. If you `select * from aws_s3_bucket` for many accounts, you'll trigger a flurry of API calls. That's why our guidance has been [Not to Select *](https://steampipe.io/blog/selective-select). But no longer. With Datatank you can query broadly, in scheduled background tasks, without tapping your fingers waiting for results. The data lands in persistent tables that respond instantly, speed up benchmarks and dashboards, and stay as fresh as you need them to be. It's the best of both worlds: instant access to live data at scale when APIs can support it, and instant access to nearly-live data when they can't.

## Create a Datatank

From the `setting/connection` page in your workspace, click `New Connection` and choose the `Create Datatank` option. Then name it, describe it, and click the `Create Datatank` button. 


<img src= "/images/blog/2023-launch-datatank/create-datatank.png" />

## Synchronize a live table

You can create a Datatank in a workspace that uses the `db1.small` instance type.

<img src= "/images/blog/2023-launch-datatank/instance-type.png" />

When you create a new Datatank, you're prompted to add a table. It's as easy as picking a schema from the ones available in your workspace, and choosing a refresh frequency. In this example we choose the `all_aws` schema (which aggregates a set of [AWS connections](https://hub.steampipe.io/plugins/turbot/aws)), the `aws_s3_bucket` table, and `Daily` frequency. 

<img src= "/images/blog/2023-launch-datatank/sync-table.png" />

The query still takes as much time as it does, which in our demo example can be up to 30 seconds. Once it's done you'll see this report, which shows that the table built successfully from all three underlying connections. 

<img src= "/images/blog/2023-launch-datatank/successful-sync.png" />

Now, when you visit the query pane you'll see a new `my_datatank` schema and within it an `aws_s3_bucket` table that supersedes the corresponding live table. When you run `select * from aws_s3_bucket` here, the response will always be instant. Dashboards that [depend on the table](https://hub.steampipe.io/mods/turbot/aws_insights/dashboards?q=s3) will always be instant too.

<img style={{width:"65%"}} src= "/images/blog/2023-launch-datatank/fast-s3-query.png" />

The data will always be as fresh as you need it to be. Need more granular control beyond `Weekly/Daily/Hourly`? Write cron expressions to query on any schedule you like, and as often as you want. 

## Persist a view

Not every table can be synchronized this way. Some tables require qualifiers in `where` or `join .. on` clauses. For these cases, you can create a Datatank table from a query that specifies the qualifiers. To do that, we'll visit our `my_datatank`, click the `New Table` button, and choose the `Create Table From Query` option. 

<img src= "/images/blog/2023-launch-datatank/create-table-from-query.png" />

Here's a query to [aggregate max CPU utilization](https://hub.steampipe.io/plugins/turbot/aws/tables/aws_cloudwatch_metric_data_point#aggregate-maximum-cpu-utilization-of-all-ec2-instances-for-the-last-24-hrs) for EC2 instances. 

```sql
select
  id,
  label,
  timestamp,
  period,
  value,
  expression
from
  aws_cloudwatch_metric_data_point
where
  id = 'm1'
  and expression = 'select max(CPUUtilization) from schema("AWS/EC2", InstanceId)'
order by
  timestamp;
```

There are two required "quals": `id` and `expression`. Let's create the Datatank table `m1_max_cpu` and schedule it to run daily.


<img src= "/images/blog/2023-launch-datatank/table-from-query.png" />

You're prompted to use the `All at once` method which updates the table atomically from all aggregated connections. For more granular control you can choose the `Per connection` method to update connections independently and be more resilient to failure. Either way, the query takes as long as it initially does; thereafter you have a table that always responds instantly.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/aAKJaBbqDuw" 
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Datatank: Blow past API speed limits with scheduled data sync"
>
</iframe>
</div>


## Get started with Datatank

Steampipe's live-data approach broke new ground. Nobody, ourselves included, thought it would be possible to pull so much data so fast from cloud APIs. That remains a core strength of Turbot Pipes. If you can get live data instantly, you should; perfect freshness is ideal and you can often achieve it. Now, for those times when you can't, you can get nearly-live data instantly. It's effectively another cache that's under your control, easy to use, and able to speed up any slow query. We can't wait to [hear your stories](/community/join) once you put your foot down on the Datatank accelerator!




Infrastructure as code (IaC) has unleashed a Cambrian explosion of tools and techniques for modeling IT infrastructure and the policies governing it. From Terraform to Kubernetes to policy-as-code, we now define our desired infrastructure states declaratively. This evolution has enabled infrastructure automation, validation, and self-service. But it has also created a bewildering landscape of complex IaC formats and concepts. Terraform plans, Kubernetes manifests, and CloudFormation templates all model infrastructure differently. Under the hood though, there is a unified data model for IaC that represents cloud infrastructure as structured data. 

With the right lens you can extract a coherent graph of resources, relationships, and metadata — not only from IaC definitions, but also from live deployments. Steampipe is that lens: a single binary that plugs into your CI/CD pipelines, taps into the entire Turbot ecosystem of API connectors and security checks, and enables ad-hoc queries against both IaC resources and deployed infrastructure. Do it all without having to script multi-lingual conversations with APIs and CLIs. Like everything else about IaC, the language you use to reason over it should be declarative. Steampipe's core language, SQL, meets the need. 

## Query infrastructure as data

Let's look at some basic examples across the full range of the common data model, starting with this snippet of AWS-flavored Terraform.

```hcl
resource "aws_instance" "Turbot Bastion Host" {
  ami           = "ami-a1b2c3d4e5f6789ab" 
  instance_type = "t2.micro" 

  tags = {
    Name        = "Turbot Bastion Host"
    Environment = "Prod"
  }
}

resource "aws_instance" "Silverwater Bastion" {
  ami           = "ami-0f9e8d7c6b5a43210z" 
  instance_type = "t2.micro" 

  tags = {
    Name        = "Silverwater Bastion"
    Environment = "Dev"
  }
}
```

This query finds two rows in answer to the question "What are the AMIs for our bastions?"

```sql
select
  name,
  arguments ->> 'ami' as image_id
from
  terraform_resource
where
  arguments -> 'tags' ->> 'Name' ~ 'Bastion'
```

```
+---------------------+------------------------+
| name                | image_id               |
+---------------------+------------------------+
| Silverwater Bastion | ami-0f9e8d7c6b5a43210z |
| Turbot Bastion Host | ami-a1b2c3d4e5f6789ab  |
+---------------------+------------------------+
```

Once the instances are deployed, this query asks the same question of the live infrastructure. It returns the same rows, augmented with facts (`instance_id`, `instance_state`) known only at runtime.


```sql
select
  title as name, 
  image_id, 
  instance_id,
  instance_state
from
  aws_ec2_instance 
where 
  tags ->> 'Name' ~ 'Bastion'
```

```
+---------------------+------------------------+---------------------+----------------+
| name                | image_id               | instance_id         | instance_state |
+---------------------+------------------------+---------------------+----------------+
| Silverwater Bastion | ami-0f9e8d7c6b5a43210z | i-0957d10a9e54d560e | running        |
| Turbot Bastion Host | ami-a1b2c3d4e5f6789ab  | i-0b42074c10ba42ff0 | running        |
+---------------------+------------------------+---------------------+----------------+
```

The  two queries are not exactly the same, but they share a common model that abstracts both parsing static Terraform resources and making API calls to live infrastructure. 

Now let's look through a GCP-flavored Kubernetes lens. Here's part of the `values.yaml` file for a helm chart. 

```yml
images:
  repository: gcr.io/google-samples/microservices-demo

adService:
  create: true
  name: adservice
  resources:
    requests:
      cpu: 200m
      memory: 180Mi
    limits:
      cpu: 300m
      memory: 300Mi
```

This query uses the Kubernetes plugin to ask the helm chart: "What are the resources specified for the adservice app?"

```sql
select distinct 
  key_path, 
  value
from
  helm_value
where
  key_path::text ~ 'adService.resources'
```

```
+-------------------------------------+-------+
| key_path                            | value |
+-------------------------------------+-------+
| adService.resources.limits.cpu      | 300m  |
| adService.resources.limits.memory   | 300Mi |
| adService.resources.requests.cpu    | 200m  |
| adService.resources.requests.memory | 180Mi |
+-------------------------------------+-------+
```

This query also uses the Kubernetes plugin to ask the same question of the corresponding live Kubernetes deployment.

```sql
select
  distinct 
    namespace, 
    jsonb_pretty(template -> 'spec' -> 'containers' -> 0 -> 'resources') as resources
  from 
    kubernetes_deployment
  where 
    name ~ 'adservice'
  ```

```
+------------------------+---------------------------+
| namespace              | resources                 |
+------------------------+---------------------------+
| gke-microservices-demo | {                         |
|                        |     "limits": {           |
|                        |         "memory": "300Mi" |
|                        |     },                    |
|                        |     "requests": {         |
|                        |         "memory": "180Mi" |
|                        |     }                     |
|                        | }                         |
+------------------------+---------------------------+
```

This query asks: "Which node is running the app?"

```
with images as (
  select
    name,
    jsonb_array_elements_text(image.value->'names') a
  from
    kubernetes_node,
    jsonb_array_elements(images) as image
)
select
  name
from
  images
where
  image_name ~ 'microservices-demo' limit 1
```

```
+-----------------------------------------------------+
| name                                                |
+-----------------------------------------------------+
| gke-smyth-test-cluster-1-default-pool-42aa2e29-7264 |
+-----------------------------------------------------+
```

And now, switching from the Kubernetes plugin to the GCP plugin, this query asks: "What are the instance details?"

```sql
select
  machine_type_name,
  status,
  zone_name,
  network_interfaces -> 0 ->> 'networkIP' as networkIP
from
  gcp_compute_instance 
where 
  name = 'gke-smyth-test-cluster-1-default-pool-42aa2e29-7264'
```

```
+-------------------+---------+---------------+-------------+
| machine_type_name | status  | zone_name     | networkip   |
+-------------------+---------+---------------+-------------+
| e2-medium         | RUNNING | us-central1-c | 10.128.0.48 |
+-------------------+---------+---------------+-------------+
```

## Check infrastructure as data

Because the common model enables queries against both statically-defined resources and deployed resources, [compliance mods](https://hub.steampipe.io/mods?objectives=compliance) can check both types. For example, this Terraform code defines two AWS S3 buckets, one with logging and one without.

```hcl
provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "my_bucket_1" {
  bucket = "jon-test-project-foo-bucket-01"

  logging {
    target_bucket = aws_s3_bucket.my_bucket_2.bucket
    target_prefix = "log/"
  }
}

resource "aws_s3_bucket" "my_bucket_2" {
  bucket = "jon-test-project-foo-bucket-02"
}
```
The [AWS Compliance](https://hub.steampipe.io/mods/turbot/aws_compliance) mod includes a control ([s3_bucket_logging_enabled](https://hub.steampipe.io/mods/turbot/aws_compliance/controls/control.s3_bucket_logging_enabled)) that checks to ensure buckets will be logged. If we deploy these buckets and run the control, we'll see that bucket 2 is in the Alarm state. 

![](/images/blog/2023-10-select-star-from-code/aws-compliance-check-logging.png)

But why even let that happen? The corresponding [Terraform AWS Compliance](https://hub.steampipe.io/mods/turbot/terraform_aws_compliance) mod also includes an [s3_bucket_logging_enabled](https://hub.steampipe.io/mods/turbot/terraform_aws_compliance/controls/control.s3_bucket_logging_enabled) control. We can use that to detect the misconfiguration and fix it before the buckets ever get deployed.

![](/images/blog/2023-10-select-star-from-code/tf-aws-compliance-check-logging.png)

Mods that check statically-defined resources have long been available for both Terraform ([AWS](https://hub.steampipe.io/mods/turbot/terraform_aws_compliance), [Azure](https://hub.steampipe.io/mods/turbot/terraform_azure_compliance), [GCP](https://hub.steampipe.io/mods/turbot/terraform_gcp_compliance)) and [Kubernetes](https://hub.steampipe.io/mods/turbot/kubernetes_compliance). The queries that power all these controls exploit the power of the common model: it's just SQL, whether you're reasoning over static resources or live APIs.

Now we've broadened the scope of queries against static resources. In [Interrogate Terraform](https://steampipe.io/blog/interrogate-terraform) we showcase a new capability: you can now query plan files and state files along with `.tf` files. And in [Interrogate Kubernetes](https://steampipe.io/blog/interrogate-kubernetes) we discuss the analogous new ability to query helm charts and manifests along with live clusters. These new query targets make it even easier to reason over static resources and compare them to those you've deployed. It all follows logically from a unified model that abstracts the differences between static resources and live infrastructure. 

## GitOps for infrastructure as data

You've long been able to install Steampipe in [CI/CD pipelines](https://steampipe.io/docs/integrations/overview#cicd-pipelines) and use it to run checks and queries. That becomes even more powerful now that you can query and check Terraform plans and state files, as well as Kubernetes charts and manifests. But wait, there's more! We're also announcing [new GitHub actions](https://steampipe.io/blog/github-actions) that make it easier than ever to use Steampipe, along with plugins and mods, in GitHub workflows. Use the [Setup action](https://github.com/marketplace/actions/setup-steampipe) to install Steampipe itself, along with any of the plugins you'll need. Then use the [Check action](https://github.com/marketplace/actions/steampipe-check) to install mods that run your checks.

The Check action has a special trick up its sleeve. If a compliance mod detects a misconfiguration in a static resource that's included in pull request, the action will annotate the PR with the alarm reported by the control — at the line number in the resource file that triggered the alarm! Again, all this follows logically from a unified model that enables common declarative query over static resource and live APIs.

## select &#x002A; from cloud join select &#x002A; from code

Steampipe's original mantra was <em>select * from cloud</em>. We thought it was amazing to able to query within and across live APIs using basic SQL, and those of you who've experienced that tell us you agree. In an IaC world, though, resources you can see via APIs are defined in code, so we made it possible to query the code too. In three companion posts — [Interrogate Terraform](https://steampipe.io/blog/interrogate-terraform), [Interrogate Kubernetes](https://steampipe.io/blog/interrogate-kubernetes), and [Steampipe-powered GitHub actions](https://steampipe.io/blog/github-actions) — we explore the latest developments enabled by an IaC data model that unifies everything. 

Questions? Comments? Bring them to the [Turbot community](/community/join)! We'd love to know your thoughts.






When the [Terraform plugin](https://hub.steampipe.io/plugins/turbot/terraform) arrived in December 2021, it was an immediate game-changer. Now you could not only query live [AWS](https://hub.steampipe.io/plugins/turbot/aws), [Azure](https://hub.steampipe.io/plugins/turbot/azure), and [GCP](https://hub.steampipe.io/plugins/turbot/gcp) infrastructure, you could also write SQL to answer questions about the Terraform configurations that deploy that infrastructure. Then in January 2022 the other shoe dropped. New compliance mods enabled you to also check Terraform-configured [AWS](https://hub.steampipe.io/mods/turbot/terraform_aws_compliance), [Azure](https://hub.steampipe.io/mods/turbot/terraform_azure_compliance), and [GCP](https://hub.steampipe.io/mods/turbot/terraform_gcp_compliance) infrastructure. We've since added hundreds of controls to the suite of Terraform Compliance mods. And you can now query your plan and state files, as well as your `.tf` files.

## Query your `.tf` files

Let's first review the basics. You've always been able to run queries against static `.tf` files. Here's a query to answer the question: "How are public access blocks configured for S3 buckets?"

```sql
select
  attributes_std ->> 'block_public_acls' as block_public_acls, path 
from 
  terraform_resource
where 
  type = 'aws_s3_bucket_public_access_block'
```

```
+-------------------+-----------------------------------------------------------------------------------+
| block_public_acls | path                                                                              |
+-------------------+-----------------------------------------------------------------------------------+
| false             | /home/jon/terraform-examples/launchweek/bucket.tf                                 |
+-------------------+-----------------------------------------------------------------------------------+
```

And this query answers the question: "How many resources of each type are configured?"

```sql
select
  type,
  count(*)
from
  terraform_resource
group by 
  type 
order by
  count desc
```

```
+---------------------------------------------+-------+
| type                                        | count |
+---------------------------------------------+-------+
| aws_route53_record                          | 13    |
| google_bigquery_table                       | 11    |
| aws_iam_role_policy_attachment              | 7     |
| google_storage_bucket_object                | 7     |
| aws_security_group                          | 7     |
...
```

## Check `.tf` files for S3 bucket public access

The [Terraform AWS Compliance](https://hub.steampipe.io/mods/turbot/terraform_aws_compliance) mod uses this query capability to define controls. For example, as shown in the first query, the `block_public_acls` setting was false for one resource. Here's the offending code.

```
resource "aws_s3_bucket_public_access_block" "public_access_bucket_1" {
  bucket = aws_s3_bucket.my_bucket_1.id

  block_public_acls   = false # should be true
  block_public_policy = true
  ignore_public_acls = true
  restrict_public_buckets = true
}
```

There's a [control](https://hub.steampipe.io/mods/turbot/terraform_aws_compliance/controls/control.s3_bucket_public_access_blocked) to check for that misconfiguration.

```bash
steampipe check control.s3_bucket_public_access_blocked
```
![terraform public access alarm](/images/blog/2023-10-launch-terraform-scans/public-access-blocks-alarm.png)

After setting `block_public_acls` to `true` in the `.tf` file, we're all green.

![terraform public access ok](/images/blog/2023-10-launch-terraform-scans/public-access-blocks-ok.png)

## Check the plan

Static analysis of `.tf` files isn't the only tool you'd like to have in your Steampipe Terraform kit. Plan and state files are also rich sources of information. And now the Terraform plugin can interrogate those too. Why? Let's look at a few examples.

In this scenario, the settings are defined by variables. 

```hcl
variable "block_public_acls" {
  type        = bool
  default     = true
}

resource "aws_s3_bucket_public_access_block" "public_access_bucket_1" {
  bucket = aws_s3_bucket.my_bucket_1.id

  block_public_acls   = var.block_public_acls
  ...
}
```

Now somebody runs `terraform plan` in a way that injects a misconfiguration. 

```
terraform plan -var 'block_public_acls=false' -out=plan.tfplan
```

If you run the control again, it'll still be green because nothing has changed in the `.tf` file. Now let's make the plan queryable as well. 

```
terraform show -json plan.tfplan > plan.json
```

Now the control reports trouble. The `.tf` passes, but the plan reveals the problem.

![terraform public access plan alarm](/images/blog/2023-10-launch-terraform-scans/public-access-blocks-plan-alarm.png)

## Check the state file too!

But we're feeling reckless, and we deploy anyway. Now we're in worse trouble! 

![terraform public access plan and state alarm](/images/blog/2023-10-launch-terraform-scans/public-access-blocks-plan-and-state-alarm.png)

Let's  redo the plan without the `-var` to get the good default, and recheck.

![terraform public access plan ok state alarm](/images/blog/2023-10-launch-terraform-scans/public-access-blocks-plan-ok-state-alarm.png)

 Finally let's deploy the fix and check one more time.
 
 ![terraform public access tf and plan and state ok](/images/blog/2023-10-launch-terraform-scans/public-access-blocks-tf-and-plan-and-state-ok.png)

All green again!

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/JG8eOffr4jk"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Interrogate Terraform: Scan and query Terraform code, plans and state files."
>
</iframe>
</div>

## Tailor-made for GitOps shops

This unified ability to query and scan Terraform files (as well as deployed infrastructure) is available everywhere Steampipe can run. That includes [CI/CD pipelines](https://steampipe.io/docs/integrations/overview) where, for GitHub in particular, we have just announced [two new actions](https://steampipe.io/blog/github-actions) that make it easier than ever to run Steampipe queries and checks in GitHub workflows, and now also embed annotations in pull requests to flag compliance issues.

There are a million ways to use these new features and controls. Give them a try and [let us know](https://steampipe.io/community/join) how it goes!




Kubernetes environments are complex. It's a challenge to keep track of clusters, jobs, namespaces, deployments, nodes, pods, and roles, to  ensure everything is configured securely, and to visualize how the parts fit together. The [Kubernetes plugin](https://hub.steampipe.io/plugins/turbot/kubernetes) made it possible to write SQL queries to answer questions about your K8s environment. Building on that foundation, the [Kubernetes Compliance](https://hub.steampipe.io/mods/turbot/kubernetes_compliance) mod added benchmarks and controls for CIS, NSA, and CISA best practices. Then the [Kubernetes Insights](https://hub.steampipe.io/mods/turbot/kubernetes_insights/dashboards) mod delivered a wealth of instant answers to common questions, along with powerful [relationship graphs](https://steampipe.io/blog/k8s-insights) to help you visualize the environment.

At first all K8s queries — your own custom ones, and those that power benchmarks and dashboards — fetched data from live clusters. Now we've enhanced the plugin with the ability to query manifests and helm charts too. Let's look at some queries and controls that use the new capability with a simple helm chart [example](https://opensource.com/article/20/5/helm-charts).

## Query helm charts and Kubernetes deployments

This query returns two rows from the [kubernetes_deployment](https://hub.steampipe.io/plugins/turbot/kubernetes/tables/kubernetes_deployment) table. 

```sql
select
  source_type,
  name,
  creation_timestamp
from
  kubernetes_deployment
where
  name = 'cherry-chart'
order by
  creation_timestamp desc
```

```
+----------------------+--------------+---------------------------+
| source_type          | name         | creation_timestamp        |
+----------------------+--------------+---------------------------+
| helm_rendered:charts | cherry-chart | <null>                    |
| deployed             | cherry-chart | 2023-10-03T10:32:09-07:00 |
+----------------------+--------------+---------------------------+
```

Why two rows? We've configured the plugin's `source_types` argument as `["deployed", "helm"]`. So we're querying both the rendered helm chart — which doesn't know the `creation_timestamp` — and the deployment which does.

## Scan helm charts and Kubernetes deployments

The `Kubernetes Compliance` mod builds on this foundation. For example, it's a best practice not to use the default namespace for a `ServiceAccount`. There's a [control](https://hub.steampipe.io/mods/turbot/kubernetes_compliance/controls/control.service_account_default_namespace_used) to check for that, based on this [query](https://hub.steampipe.io/mods/turbot/kubernetes_compliance/queries/service_account_default_namespace_used).

```sql
select
  coalesce(uid, concat(path, ':', start_line)) as resource,
  case
    when namespace = 'default' then 'alarm'
    else 'ok'
  end as status,
  case
    when namespace = 'default' then name || ' uses default namespace.'
    else name || ' not using the default namespace.'
  end as reason,
  name as service_account_name,
  coalesce(context_name, '') as context_name,
  namespace,
  source_type,
  coalesce(path || ':' || start_line || '-' || end_line, '') as path
from
  kubernetes_service_account;
```

Running the control shows that the helm chart and deployment are both green: no default namespace.

![helm green deploy green](/images/blog/2023-10-launch-kube-scans/helm-green-deploy-green.png)

Now somebody improperly sets the namespace to `default` in `serviceaccount.yaml`.

```yaml
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: {{ include "buildachart.serviceAccountName" . }}
  labels:
{{ include "buildachart.labels" . | nindent 4 }}
{{- end -}}

```

Now the control shows a mismatch between the chart and the deployment.

![helm red deploy green](/images/blog/2023-10-launch-kube-scans/helm-red-deploy-green.png)

But we're going to deploy anyway. 

```
helm upgrade my-cherry-chart buildachart/
```

Now the chart and the deployment are both in the alarm state.

![helm red deploy red](/images/blog/2023-10-launch-kube-scans/helm-red-deploy-red.png)

Let's properly set a namespace in the helm chart.

```
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: cherry # must not be default!
  name: {{ include "buildachart.serviceAccountName" . }}
  labels:
{{ include "buildachart.labels" . | nindent 4 }}
{{- end -}}
```

Now we're back to a mixed result: the chart is green, but the deployment is still red.

![helm green deploy green](/images/blog/2023-10-launch-kube-scans/helm-green-deploy-red.png)

After redeploying, we're back to green all around.

![helm green deploy green](/images/blog/2023-10-launch-kube-scans/helm-green-deploy-green.png)

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/0ygbWKsuTwM"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Interrogate Kubernetes: Run checks and queries against your manifest files and helm charts."
>
</iframe>
</div>

## A wealth of queries and controls

We've looked at just one of the [790 controls](https://hub.steampipe.io/mods/turbot/kubernetes_compliance/controls) provided by the `Kubernetes Compliance` mod. Collectively they support best practices defined by [CIS Kubernetes v1.20](https://hub.steampipe.io/mods/turbot/kubernetes_compliance/controls/benchmark.cis_kube_v120), [CIS v1.7.0](https://hub.steampipe.io/mods/turbot/kubernetes_compliance/controls/benchmark.cis_v170), and [NSA and CISA Kubernetes Hardening Guidance v1.0](https://hub.steampipe.io/mods/turbot/kubernetes_compliance/controls/benchmark.nsa_cisa_v1). Those controls build on [781 queries](https://hub.steampipe.io/mods/turbot/kubernetes_compliance/queries). Use them out of the box, or [reuse and remix](https://steampipe.io/blog/remixing-dashboards) to suit your needs. And please do [let us know](https://steampipe.io/community/join) how it goes!




The GitHub [Setup](https://github.com/marketplace/actions/setup-steampipe) and [Check](https://github.com/marketplace/actions/steampipe-check) actions simplify installing Steampipe in a GitHub workflow, installing and configuring plugins, installing mods, and running benchmarks. You can also automate pushing snapshots to [Turbot Pipes](https://turbot.com/pipes) and annotating pull requests when controls raise alarms. Let's see how it all works.

## Setup Steampipe, as a GitHub Actions

It's easy enough to install Steampipe in a GitHub action, the command is [just a one-liner](https://steampipe.io/downloads). But when you need to install and configure a plugin, things can get messy if you have to manipulate your `.spc` file in a GitHub runner. The [Setup action](https://github.com/marketplace/actions/setup-steampipe) handles everything in a clean and simple way.

```yaml
- name: Setup Steampipe
  uses: turbot/steampipe-action-setup@v1
  with:
    plugin-connections: |
      connection "aws_dev" {
        plugin     = "aws"
        secret_key = "${{ secrets.AWS_ACCESS_KEY_ID_DEV }}"
        access_key = "${{ secrets.AWS_SECRET_ACCESS_KEY_DEV }}"
        regions    = ["*"]
      }

- name: Run queries
  run: |
    steampipe query "select account_id from aws_dev.aws_account"
```

That's it! Steampipe is installed, and the AWS plugin is installed and configured. If you need it, the Steampipe binary is available — for example, to run a query as shown here. But you won't need it to run benchmarks.

Thanks to [François de Metz](https://steampipe.io/blog/francois2metz) who built the first version of this action, then generously donated it to Turbot!

## Steampipe Check, as a GitHub Action

There are powerful mods available to check for [compliance](https://hub.steampipe.io/mods?objectives=compliance) and to display [insights dashboards](https://hub.steampipe.io/mods?objectives=compliance). The [Check](https://github.com/marketplace/actions/steampipe-check) action neatly manages all the moving parts required to install a mod, choose a branch, specify which checks to run, specify exports, create an artifact, create and upload a Pipes snapshot, and display the report's markdown export in GitHub. This example installs, configures, and runs [Terraform AWS Compliance](https://hub.steampipe.io/mods/turbot/terraform_aws_compliance).


```yaml
name: Check terraform_aws_compliance.benchmark.ec2
on:
  workflow_dispatch

jobs:
  aws_tf_compliance:
    runs-on: ubuntu-latest
    steps:
      - name: Repository Checkout
        uses: actions/checkout@v3
      - name: Steampipe Setup
        uses: turbot/steampipe-action-setup@v1
        with:
          plugin-connections: |
            connection "tf" {
              plugin = "terraform"
              paths  = [ "./path/to/dir/*.tf" ]
            }
      - name: Steampipe Check
        uses: turbot/steampipe-action-check@v1
        with:
          mod-url: https://github.com/turbot/steampipe-mod-terraform-aws-compliance
          checks: terraform_aws_compliance.benchmark.ec2
          snapshot-visibility: anyone_with_link
          pipes-token: ${{ secrets.PIPES_TOKEN }}
     
```

That's it! After the `Setup` action loads the [Terraform plugin](https://hub.steampipe.io/plugins/turbot/terraform) and points it at a set of Terraform files in the repo, the `Check` action loads the mod, runs the EC2 benchmark, and pushes a snapshot to Pipes.

Here's the GitHub view of the result. The artifact contains the action's default output (check results in CSV format) and also includes the check's markdown.

![](/images/blog/2023-launch-github-actions/basic-check.png)

And here's the dashboard it uploaded to Pipes.

![](/images/blog/2023-launch-github-actions/basic-check-pipes-dashboard.png)

## Annotations on pull requests

You can also trigger the `Check` action on pull requests. If your code changes raise alarms, the `Check` action will attach them to the pull request as annotations. Here's a Terraform resource that improperly sets `block_public_acls` to `false`.

```hcl
resource "aws_s3_bucket_public_access_block" "public_access_bucket_1" {
  bucket = aws_s3_bucket.my_bucket_1.id

  block_public_acls   = false # should be true
  block_public_policy = true
  ignore_public_acls = true
  restrict_public_buckets = true
}


resource "aws_s3_bucket" "my_bucket_1" {
  bucket = "test-project-foo-bucket-01"
}
```

And here's the workflow file we'll use.

```yaml
name: Annotate Pull Request Test
on:
  pull_request:
    types:
      - opened
    branches:
      - 'main'
    paths:
      - tf/*

jobs:
  compliance_with_annotations:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
      checks: write
    steps:
      - name: Repository Checkout
        uses: actions/checkout@v3
      - name: Steampipe Setup
        uses: turbot/steampipe-action-setup@v1
        with:
          plugin-connections: |
            connection "tf" {
              plugin = "terraform"
              paths  = [ "./path/to/dir/*.tf" ]
            }
      - name: Steampipe Check
        uses: turbot/steampipe-action-check@v1
        with:
          mod-url: https://github.com/turbot/steampipe-mod-terraform-aws-compliance
          github-token: ${{ secrets.GITHUB_TOKEN }}
          snapshot-visibility: anyone_with_link
          pipes-token: ${{ secrets.PIPES_TOKEN }}
```

The key difference: it triggers on pull requests, specifically those that affect Terraform files. It also adds required permissions and a GitHub token. And because it omits the `checks` argument, all the controls in the `Terraform AWS Compliance` will run. 

When the pull request lands, the Check action runs. Again it reports outputs, creates an artifact, and posts a snapshot. But now it also annotates the PR!

![](/images/blog/2023-launch-github-actions/check-with-annotations.png)

The misconfiguration we introduced in the PR shows up first, anchored to the place we made the  change: line 4 of `s3_bucket.tf`. But if there are more alarms — as in this case — you'd like to know about them, so the Check action posts those as annotations too.

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/EeOhJWX1bks"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Steampipe-powered GitOps"
>
</iframe>
</div>

## Get started with Steampipe Setup and Steampipe Check

Steampipe is happy to run in [any CI/CD pipeline](https://steampipe.io/docs/integrations/overview) in order to enable queries that use the growing [family of plugins](https://hub.steampipe.io/plugins), and to enable checks that use the parallel [family of mods](https://hub.steampipe.io/mods). If GitHub is your game, you can now take it to the next level with actions that make it easy to install, configure, and run Steampipe, then run compliance checks that integrate with both GitHub itself and with Turbot Pipes. If you haven't tried these new actions yet, give them a whirl and [let us know](/community/join) how it goes!



[Turbot Guardrails](http://turbot.com/guardrails), the leading platform for policy-driven compliance & real-time remediation for enterprise clouds, is constantly evolving based on feedback from our diverse customer base. Generating notifications from Turbot has been around for years with our [Firehose feature](https://turbot.com/guardrails/docs/mods/turbot/firehose-aws-sns), this works great for sending notifications programmatically from one system to another (e.g. a SEIM), but customers have also asked for a more direct way to involve individual users and teams in remediation of cloud governance issues using notifications.

Drumroll Please...

We are thrilled to unveil [Guardrails Notifications](https://turbot.com/guardrails/docs/guides/notifications). This powerful addition to the Guardrails platform ensures that you and your application teams stay informed in real-time about devops, compliance, and cloud security issues, and empowers those teams with quick-action capabilities to respond immediately. As patterns emerge from these human workflows, you can further authorize Guardrails to automatically take remediation actions on your behalf.

## Gorgeous default templates

To help you get started we have hand-crafted default templates (for both single and batch notifications) across Email, Slack and Teams, but feel free to jump into the weeds and use our template settings to customize and add your own branding, links or messaging

<img src="/images/blog/2023-10-guardrails-notifications/slack.png" style={{ boxShadow: 'none' }} />

## Integrated with Guardrails Quick Actions

Using Guardrails' [Quick Action feature](/guardrails/blog/2022/08/quick-actions), notifications can include direct links back to the Guardrails console to fix the alert on the spot. This enables workflows where humans can confirm automated actions before application, keeping them in the loop and in control of their cloud infrastructure.

<img src="/images/blog/2023-10-guardrails-notifications/quick-action.png" />

## Real-time notifications

Get individual alerts in real-time, and for those situations where alerts are flooding in, Guardrails automatically sends consolidated batch notifications to avoid notification fatigue:

<img src="/images/blog/2023-10-guardrails-notifications/email-batch.png" style={{ boxShadow: 'none' }} />

## Configurable to the core

Configure your notifications effortlessly using the Turbot Guardrails UI or seamlessly integrate with our Terraform provider for those who love Infrastructure as Code.

- **Configurable Channels:** Configure notifications for Slack, Microsoft Teams, and Email.
- **Configurable Triggers:** Receive alerts based on specific events like Cost overruns, Compliance gaps, or Security misconfigurations.

<img src="/images/blog/2023-10-guardrails-notifications/notification-rules.png" style={{ boxShadow: 'none' }} />

## Built for Cloud Operations

With this feature, we aim to give you even more visibility and control over your multi-cloud ecosystem along with the ability to loop-in teams that don't have access to the Guardrails console. Watch this quick demo to quickly get a feel for how you can use Notifications in your cloud governance workflows.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/LmyDTFqzDbk"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Turbot Guardrails Notifications Demo"
>
</iframe>
</div>

Questions about the new Notifications feature? Join our discussions in the #guardrails channel of [our Slack community](/community/join). We always value your feedback and look forward to hearing how your team leverages notifications!



Tagging strategies evolve over time, and are affected by mergers, acquisitions and mistakes during deployment. These issues accumulate large amounts of technical debt, making it difficult for organizations to rely on tag metadata for audit, support and billing purposes.

[Turbot Guardrails](http://turbot.com/guardrails) has an insanely powerful tagging automation solution that works across multiple cloud providers. Its event-driven architecture scales to monitor millions of cloud resources in real-time without the scanning limitations of other products. These features give you the same power and flexibility you would get from building your own tagging remediation code, without all of the deployment, monitoring and maintenance hassles that typically entails. Guardrails becomes an automated member of your cloud team that remediates and rectifies incorrect tag values in real-time **based on rules you establish**.

While context-aware tagging capabilities have been part of Turbot for years, we've been relentlessly improving (based on customer feedback) and making them easier to implement. Our recent updates introduce innovative solutions tailored for intricate tagging scenarios, empowered by our enhanced calculated policy engine.

## Mispellings & TypOs

The bane of most tagging strategies are common misspellings and typos; Guardrails' helper functions can automatically fix tag keys and values that are misspelled.  Guardrails' [calculated policy feature](https://turbot.com/guardrails/docs/concepts/policies/calculated-faq) now has a helper function named [transformMap](https://turbot.com/guardrails/docs/concepts/guardrails/tagging/tagging-helpers#rectify-and-cleanup-bad-keys-and-values). This function uses two inputs, the current list of tags/values (as a map object) and a set of rules to process on them.

The list of current tags for each resource is saved when Guardrails discovers it, and every time the resource is updated after:

```json
{
  "cost_center": "scranton-1138",
  "environment": "prd",
  "owner": "dwigt"
}
```

Rules define how Guardrails will handle current and future misspellings and typos:

```yaml
owner:
  values:
    dwight:
      incorrectValues:
        - dwigt
        - dwihgt
```

After processing the tag values against the rules above, Guardrails would automatically correct `"owner": "dwigt"` to `"owner": "dwight"`.

## Harmonization of changes in tag standards over time

The same function can be used to update or replace tags matching a previous standard to match a new standard. We will start again with our current tag values:

```json
{
  "cost_center": "scranton-1138",
  "environment": "prd",
  "owner": "dwight"
}
```

This time our rules will focus on changing our old key `cost_center` to our new standard value `costcenter` to make it more compatible across cloud resources that don't support underscore.  Also, we have standardized the `environment` tag to only accept the values `['production','development']`:

```yaml
costcenter:
  incorrectKeys:
    - cost_center
  replacementValue: undefined
environment:
  values:
    production:
      incorrectValues:
        - prd
        - /prod.*/gi
    development:
      incorrectValues:
        - /dev.*/gi
```

After processing the rules the resulting tag values would be applied as:

```json
{
  "costcenter": "scranton-1138",
  "environment": "production",
  "owner": "dwight"
}
```

## Automatically tagging who created a resource

When responding to an incident, time is of the essence. Knowing who created any given cloud resource enables operations teams to quickly reach out to the right individuals or teams, reducing Mean Time to Resolution (MTTR). Guardrail's now adds new `createdBy` and `createTimestamp` metadata to the Guardrails CMDB for every resource creation event. For each cloud provider we capture the IAM identity matching who created the resource from that provider's audit logging service:

**AWS**
- userIdentity.arn: The full ARN of the identity that created the resource.

**Azure**

- initiatedBy.userPrincipalName (Active Directory resources)
- caller (Azure resources)

**GCP**
- authenticationInfo.principalEmail

This metadata can then be used in your Guardrails tagging templates to apply the values directly onto the resource:

```nunjucks
{% if $.resource.metadata.createdBy %}
- "creator": "{{ $.resource.metadata.createdBy }}"
{% endif %}
{% if $.resource.metadata.createTimestamp %}
- "createTimestamp": "{{ $.resource.metadata.createTimestamp }}"
{% endif %}
```

## Context-aware tagging 

It is often desireable to enhance or validate existing resource tags with data from other sources. Using a [Turbot File]() we can import data from other systems and then use that data enhance or verify tags.

Input data from financial systems:

```yaml
# Approved Cloud Cost Centers:
new-york-0001:
  mgr: "David Wallace"
  dept: "Corporate" 
scranton-1138:
  mgr: "Michael Scott"
  dept: "Sales"
stamford-0124:
  mgr: "Josh Porter" 
  dept: "Sales"
```

This data can then be used to validate and add additional metadata to existing resources using a calculated policy template:

```nunjucks
{% approved_cc = $.resource.approved_cost_centers %}
{% cost_center = $.resource.tags['costcenter'] %}
{% if cost_center in approved_cc %}
- "validation": "Valid Cost Center"
- "department": "{{ approved_cc[cost_center]['dept'] }}"
- "owner": "{{ approved_cc[cost_center]['mgr'] }}"
{%- else -%}
- "validation": "Invalid Cost Center"
{%- endif -%}
```

After running our previous example through this logic our resulting tags would match the following:

```json
{
  "costcenter": "scranton-1138",
  "environment": "production",
  "owner": "Michael Scott",
  "department": "Sales",
  "validation": "Valid Cost Center"
}
```

## Intelligent tagging made easy

With Guardrails' intelligent tagging, the chaos of manual tagging is tamed. Guardrails ensures consistent, policy-driven tags across all cloud resources, automatically capturing and applying contextual insights. Operational overhead is minimized as your automated rules detect and rectify non-compliant tags in real-time. No more manual fixes, no more back and forth incident management; just accurate, insightful, real-time metadata.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/eSQ7V80StrQ" 
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Guardrails Intelligent Tagging"
>
</iframe>
</div>

**Questions about Guardrails' tagging capabilities?** Join our discussions in the #guardrails channel of [our Slack community](/community/join). We always value your feedback and look forward to hearing how your team leverages notifications!




Do you run Steampipe at scale, and/or in constrained environments? Now you can control the amount of memory Steampipe and its plugins can use. Do your queries sometimes trigger excessive throttling? Now you can define plugin-level limiters for API-call concurrency and request rates. Do you wonder which permissions are needed for a query? The new diagnostic mode will show you, and it'll help you fine-tune your limiters too. Let's see how it all works.

## Out of memory? Dial it down.

Steampipe can run anywhere: your laptop, an EC2 instance, a GitHub runner. In some environments, memory is a hard constraint. In Kubernetes, for example, the [OOM killer](https://medium.com/cloud-native-daily/title-demystifying-oom-killer-in-kubernetes-tracking-down-memory-issues-b5a4973fbd56) can suddenly and silently shut things down. Good news! Steampipe and its plugins now default to a 1024MB limit. You override any plugin's max memory in its [config file](https://steampipe.io/docs/managing/connections#connection-configuration-files).


```hcl
plugin "aws" {
  memory_max_mb = 2048 # megabytes
}
```

You can also use the new environment variables [STEAMPIPE_MEMORY_MAX_MB](/docs/reference/env-vars/steampipe_memory_max_mb) to set memory limits for the Steampipe process and [STEAMPIPE_PLUGIN_MEMORY_MAX_MB](/docs/reference/env-vars/steampipe_plugin_memory_max_mb) to do the same for plugins.

## Overwhelming your network? Dial it down.

Steampipe is [massively parallel](https://aws.amazon.com/blogs/opensource/querying-aws-at-scale-across-apis-regions-and-accounts/) and we've seen cases where it can take down a home network or reverse proxy. You can now tame that behavior with rate limiters. Here's a simple example that constrains the number of [List, Get, and Hydrate functions](/docs/develop/writing-plugins#hydrate-functions) that the AWS plugin can run in parallel.

```hcl
plugin "aws" {
  limiter "max_concurrency" {
    max_concurrency = 200
  }
}
```

See many more examples in the [documentation](/docs/reference/config-files/plugin).

## Getting throttled? Dial it down.

Steampipe plugins typically use exponential backoff/retry when API calls trigger throttling. Some amount of throttling is normal, but too much can cause problems. Now you can use [limiters](/docs/guides/limiter) to keep things calm. Pen testers can fly under the radar, and everyone can avoid being a noisy cloud neighbor. 

```hcl
plugin "aws" {
  limiter "aws_global" {
    bucket_size  = 1000
    fill_rate    = 1000
   }
}
```

Those settings will regulate the entire AWS plugin. `bucket_size` defines how many requests can be made per second, and `fill_rate` defines how many are added back per-second to refill the bucket. The two settings work together to implement a [token-bucket](https://en.wikipedia.org/wiki/Token_bucket) rate limiting algorithm.

What are the optimal values for these settings? It's hard to know in advance what's best for a given API, so you'll need to experiment. Happily, the new diagnostic mode will help you do that! 

## Diagnostic mode

Use the new [STEAMPIPE_DIAGNOSTIC_LEVEL](/docs/reference/env-vars/steampipe_diagnostic_level) environment variable to turn on Steampipe's version of [EXPLAIN ANALYZE](https://www.postgresql.org/docs/current/sql-explain.html). Let's dial down the above settings from 1000 to 10 and run Steampipe in diagnostic mode.


```
 $ STEAMPIPE_DIAGNOSTIC_LEVEL=ALL steampipe query

> select jsonb_pretty(_ctx) as _ctx, display_names, tags from aws_sns_topic
```

Here's a single row of output.

```
+------------------------------------------------------------+--------------+--------+
| _ctx                                                       | display_name | tags   |
+------------------------------------------------------------+--------------+--------+
| {                                                          |              | <null> |
|     "diagnostics": {                                       |              |        |
|         "calls": [                                         |              |        |
|             {                                              |              |        |
|                 "type": "list",                            |              |        |
|                 "scope_values": {                          |              |        |
|                     "table": "aws_sns_topic",              |              |        |
|                     "action": "ListTopics",                |              |        |
|                     "region": "us-east-1",                 |              |        |
|                     "service": "sns",                      |              |        |
|                     "connection": "sil",                   |              |        |
|                     "function_name": "listAwsSnsTopics"    |              |        |
|                 },                                         |              |        |
|                 "function_name": "listAwsSnsTopics",       |              |        |
|                 "rate_limiters": [                         |              |        |
|                     "aws_global"                           |              |        |
|                 ],                                         |              |        |
|                 "rate_limiter_delay_ms": 0                 |              |        |
|             },                                             |              |        |
|             {                                              |              |        |
|                 "type": "hydrate",                         |              |        |
|                 "scope_values": {                          |              |        |
|                     "table": "aws_sns_topic",              |              |        |
|                     "action": "GetTopicAttributes",        |              |        |
|                     "region": "us-east-1",                 |              |        |
|                     "service": "sns",                      |              |        |
|                     "connection": "sil",                   |              |        |
|                     "function_name": ""                    |              |        |
|                 },                                         |              |        |
|                 "function_name": "getTopicAttributes",     |              |        |
|                 "rate_limiters": [                         |              |        |
|                     "aws_global"                           |              |        |
|                 ],                                         |              |        |
|                 "rate_limiter_delay_ms": 5295              |              |        |
|             },                                             |              |        |
|             {                                              |              |        |
|                 "type": "hydrate",                         |              |        |
|                 "scope_values": {                          |              |        |
|                     "table": "aws_sns_topic",              |              |        |
|                     "action": "ListTagsForResource",       |              |        |
|                     "region": "us-east-1",                 |              |        |
|                     "service": "sns",                      |              |        |
|                     "connection": "sil",                   |              |        |
|                     "function_name": "listTagsForSnsTopic" |              |        |
|                 },                                         |              |        |
|                 "function_name": "listTagsForSnsTopic",    |              |        |
|                 "rate_limiters": [                         |              |        |
|                     "aws_global"                           |              |        |
|                 ],                                         |              |        |
|                 "rate_limiter_delay_ms": 5401              |              |        |
|             }                                              |              |        |
|         ]                                                  |              |        |
|     },                                                     |              |        |
|     "connection_name": "sil"                               |              |        |
| }                                                          |              |        |
```

The diagnostics information includes information about each Get, List, and Hydrate function that was called to fetch the row, including:

| Key                     | Description
|-------------------------|---------------------- 
| `type`                  | The type of function (`list`, `get`, or `hydrate`).
| `function_name`         | The name of the function.
| `scope_values`          | A map of scope names to values.  This includes the built-in scopes as well as any matrix qualifier scopes and function tags.
| `rate_limiters`         | A list of the rate limiters that are scoped to the function.
| `rate_limiter_delay_ms` | The amount of time (in milliseconds) that Steampipe waited before calling this function due to client-side (`limiter`) rate limiting.

The `_ctx` column shows that the `aws_global` limiter is in effect, and that it has delayed two of the sub-API calls that feed the `aws_sns_topic` table. 

Whether or not you've imposed limits, the `function_name` scope value is a key piece of information. Have you ever struggled to know which permissions are needed to query a particular table? The names of the functions that populate the table correspond to API calls; they'll help you figure that out.

What if you don't want to slow everything down? You can be much more precise. For example, the limits imposed by AWS on APIs can vary by region. To rate-limit just `us-east-1` traffic you can do this.

```hcl
plugin "aws" {
  limiter "us-east-1" {
    bucket_size  = 10
    fill_rate    = 10
    scope        = [ "region" ]
    where        = "'region = 'us-east-1'"
   }
}
```

A limiter defined with no scopes applies broadly. When you list one or more values in the `scope` argument you create a [unique limiter](/docs/guides/limiter#defining-limiters) for that set of scopes. With the AWS plugin, for example, you could create a limiter scoped to just `connection`, or `connection` and `region`, or any combination of available scopes — it's wildly flexible. 

And if the plugin implements [function tags](/docs/develop/writing-plugins#function-tags), as the AWS plugin does, you can even target a specific function call. 

```hcl
plugin "aws" {
  limiter "aws-sns-topic-us-east-1-" {
    bucket_size  = 10
    fill_rate    = 10
    scope        = [ "region" ]
    where        = "'region = 'us-east-1' and action = 'ListTagsForResource'"
   }
}
```

## The `plugin` block

All these examples use the new [plugin block](/docs/reference/config-files/plugin#plugins-and-connections) to set plugin-level options (in `.spc` files) that you can bind to connections. Steampipe will create a separate plugin process for each `plugin`/`connection` pair. 

In this example Steampipe creates two plugin processes. `aws_high` binds to `aws_prod_1` and `aws_prod_2`, sets a 2000MB maximum, and defines no limiters. `aws_low`  binds to `aws_dev_1` and `aws_dev_2`, sets a 500MB limit, and defines an `all_requests` limiter.

```hcl
plugin "aws_high" {
  memory_max_mb = 2000
  source        = "aws"
}
plugin "aws_low" {
  memory_max_mb = 500
  source        = "aws"
  limiter "all_requests" {
    bucket_size  = 100
    fill_rate    = 100
    max_concurrency = 50
  }
}
connection "aws_prod_1" {
  plugin  = plugin.aws_high
  profile = "prod1"
  regions = ["*"]
}
connection "aws_prod_2" {
  plugin  = plugin.aws_high
  profile = "prod2"
  regions = ["*"]
}
connection "aws_dev_1" {
  plugin  = plugin.aws_low
  profile = "dev1"
  regions = ["*"]
}
connection "aws_dev_2" {
  plugin  = plugin.aws_low
  profile = "dev2"
  regions = ["*"]
}
```

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/9J9X3xKUis0" 
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Steampipe v0.21.0 Release: Memory management, rate limiting, and diagnostics."
>
</iframe>
</div>

## Tune your Steampipe engine

New memory limits can regulate Steampipe's memory consumption overall, or per-plugin. Concurrency and rate limiters can control how plugins use APIs. And diagnostic mode helps you understand and fine-tune the diverse and flexible limiters you can create. Steampipe is a racecar that sometimes goes too fast. With these new mechanisms of control, and unprecedented visibility into the engine, you can now precisely calibrate its speed and optimize its performance. Read all about the new [plugin block](/docs/reference/config-files/plugin) and [limiters](/docs/guides/limiter), then give it all a try and [let us know](https://steampipe.io/community/join) how it goes!



Steampipe blurs the boundaries between code and data, to powerful effect. The [exec plugin](https://hub.steampipe.io/plugins/turbot/exec) continues that theme. Use it to write SQL queries that run commands on local or remote hosts, returning results as one or more rows. We'll look at a few basic examples, then show how we use the plugin to enable the new [Docker Compliance](https://hub.steampipe.io/mods/turbot/docker_compliance) mod.

## From Steampipe to your local shell and back again

If you install the plugin and do nothing else, this becomes possible.

```sql
select line from exec_command_line where command = 'uname -a'
+-------------------------------------------------------------------------------------------------------------------+
| line                                                                                                              |
+-------------------------------------------------------------------------------------------------------------------+
| Linux Indigo 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
+-------------------------------------------------------------------------------------------------------------------+
```

Multiple lines produce multiple rows.

```sql
select line from exec_command_line where command = 'ls -lr --time-style=+"%Y-%m"'
+-----------------------------------------------------------------------------------------+
| line                                                                                    |
+-----------------------------------------------------------------------------------------+
| drwxr-xr-x  6 jon jon     4096 2023-01 steampipe-mod-azure-insights                     |
| drwxr-xr-x  7 jon jon     4096 2023-07 steampipe-plugin-salesforce                      |
| drwxr-xr-x 15 jon jon     4096 2023-05 steampipe-plugin-sdk                             |
```

You can also use the `exec_command` table to bring an entire file into a column.

```sql
select stdout_output from exec_command where command = 'cat /etc/resolv.conf'
```

```
+-------------------------------------------------------------------------------------+
| output                                                                              |
+-------------------------------------------------------------------------------------+
| # This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8). |
| # Do not edit.                                                                      |
| ...                                                                                 |

| nameserver 127.0.0.53                                                               |
| options edns0 trust-ad                                                              |
| search .                                                                            |
+-------------------------------------------------------------------------------------+
```

## Connect to remote Linux or Windows hosts, traversing bastions and proxies

  The first example uses the default [exec_local](https://hub.steampipe.io/plugins/turbot/exec#configuration) connection. Now let's edit `exec.spc` to add a connection to a remote host.

``` 
  connection "exec_local" {
  plugin = "exec"
}

connection "exec_remote_linux" {
  plugin = "exec"
  protocol = "ssh"
  host = "my.host.net"
  user = "jon"
  private_key = "~/.ssh/my.host.net.id_rsa"
}
```

Now we can query the local and remote connections individually.

```
select line from exec_local.exec_command_line where command = 'uname -a'
+-------------------------------------------------------------------------------------------------------------------+
| line                                                                                                              |
+-------------------------------------------------------------------------------------------------------------------+
| Linux Indigo 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
+-------------------------------------------------------------------------------------------------------------------+
```

```
 select line from exec_remote_linux.exec_command_line where command = 'uname -a'
+-------------------------------------------------------------------------------------------------------+
| line                                                                                                  |
+-------------------------------------------------------------------------------------------------------+
| Linux hn 5.15.0-50-generic #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
+-------------------------------------------------------------------------------------------------------+
```

## Query multiple hosts with aggregators

You could use a SQL JOIN to combine those queries. But there's a much easier way! First create an aggregator.

```hcl
connection "exec_all" {
  plugin = "exec"
  type = "aggregator"
  connections = ["exec_local", "exec_remote_linux"]
}
```

Then query it.

```
select
  _ctx->>'connection_name' as connection, 
  line 
from
  exec_all.exec_command_line
where
  command = 'uname -a'
```

```
+-------------------+-------------------------------------------------------------------------------------------------------------------+
| connection        | line                                                                                                              |
+-------------------+-------------------------------------------------------------------------------------------------------------------+
| exec_remote_linux | Linux hn 5.15.0-50-generic #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux             |
| exec_local        | Linux Indigo 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
+-------------------+-------------------------------------------------------------------------------------------------------------------+
```

Here are the supported [connection modes](https://hub.steampipe.io/plugins/turbot/exec#configuration):

- Remote Linux via SSH

- Remote Windows via WinRM

- Remote Windows connection through WinRM ignoring certificate validation

- Remote Linux connection through SSH with in-file private key

- Remote Linux connection through SSH with password

- Remote Linux connection through SSH via bastion host

- Remote Linux connection through SSH via bastion host over proxy


## Analyze results with SQL

Of course SQL is also a great way to aggregate data. Let's ask and answer the question: "How many packages installed per day?"

```sql
select 
  _ctx->>'connection_name' as connection, 
  line as package_install_date, 
  count(*)
from
  exec_all.exec_command_line 
where 
  command = 'cat /var/log/dpkg.log | cut -d'' '' -f1'
group by 
  connection, line
order by
  connection, package_install_date
```

```
+-------------------+----------------------+-------+
| connection        | package_install_date | count |
+-------------------+----------------------+-------+
| exec_local        | 2023-09-05           | 11    |
| exec_local        | 2023-09-11           | 87    |
| exec_local        | 2023-09-13           | 67    |
| exec_remote_linux | 2023-09-05           | 107   |
| exec_remote_linux | 2023-09-08           | 35    |
| exec_remote_linux | 2023-09-13           | 52    |
+-------------------+----------------------+-------+
```

## Bringing it all together for Docker CIS scanning

Here's the CIS 1.6 benchmark running as a [dashboard](https://hub.steampipe.io/mods/turbot/docker_compliance/dashboards/benchmark.cis_v160). 

![CIS 1.6 dashboard](/images/blog/2023-10-exec-plugin/docker-compliance-dashboard.png)

The benchmark uses the [Docker](https://hub.steampipe.io/plugins/turbot/docker/) plugin for queries against the Docker layer (using tables like [docker_container](https://hub.steampipe.io/plugins/turbot/docker/tables/docker_container)), and also uses the `exec` plugin to query the OS layer. These queries support 33 controls that use the Docker plugin, and 56 that use the `exec` plugin.

Let's explore one of the `exec`-based controls. CIS Docker 1.6 advises that that the Docker sock file permissions should be set to 660 or more restrictively. Here's the definition of a [control](https://hub.steampipe.io/mods/turbot/docker_compliance/controls/control.cis_v160_3_16) that checks for that condition.

```
control "cis_v160_3_16" {
  title         = "3.16 Ensure that the Docker sock file permissions are set to 660 or more restrictively"
  description   = "You should verify that the Docker socket file has permissions of 660 or are configured more restrictively."
  query         = query.docker_sock_file_restrictive_permission
```

And here is the referenced query.

```sql
query "docker_sock_file_restrictive_permission" {
  sql = <<-EOQ
    ${local.os_hostname_sql}
     linux_output as (
      select
        stdout_output,
        _ctx ->> 'connection_name' as conn
      from
        exec_command,
        os_output
      where
        os_conn = _ctx ->> 'connection_name'
        and command = 'stat -c %a /var/run/docker.sock'
    ),
    darwin_output as (
      select
        stdout_output,
        _ctx ->> 'connection_name' as conn
      from
        exec_command,
        os_output
      where
        os_conn = _ctx ->> 'connection_name'
        and os_output.os = 'Darwin'
        and command = 'stat -f %Op /var/run/docker.sock'
    ),
    command_output as (
      select * from darwin_output
      union all
      select * from linux_output
    )
    select
      host as resource,
      case
        when o.stdout_output like '%No such file or directory%' then 'skip'
        when o.stdout_output like '%660%' then 'ok'
        else 'alarm'
      end as status,
      case
        when o.stdout_output like host || '%No such file or directory%' then ' recommendation is not applicable as the file is unavailable.'
        else host || ' docker.socket file permission set to ' || (btrim(o.stdout_output, E' \n\r\t')) || '.'
      end as reason
      ${replace(local.common_dimensions_qualifier_sql, "__QUALIFIER__", "h.")}
    from
      hostname as h,
      os_output as os,
      command_output as o
    where
      os.os_conn = h.host_conn
      and h.host_conn = o.conn;
  EOQ
}
```

Once you figure out  which shell commands to use, it's just the standard Steampipe pattern for a control that returns the [required columns](https://steampipe.io/docs/reference/mod-resources/control#required-control-columns) `resource`, `status`, and `reason`.

Like all other [mods](https://hub.steampipe.io/mods), `Docker Compliance` is available to [reuse and remix](https://steampipe.io/blog/remixing-dashboards). Are there other conditions you'd like to check? If there's a shell command that returns the necessary data, you can easily wrap it in a Steampipe control.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/fDCHJuWmH4k" 
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="select * from shell; Use SQL to execute commands on local or remote hosts!"
>
</iframe>
</div>

## What is an API anyway?

Steampipe nominally maps APIs to database tables. When you `select * from aws_sns_topic`, Steampipe winds up calling the AWS [ListTopics](https://docs.aws.amazon.com/sns/latest/api/API_ListTopics.html) API. Many Steampipe plugins work like that: a table corresponds to a specific API call. But some plugins work in a more general way. The [CSV plugin](https://hub.steampipe.io/plugins/turbot/csv) opens the door to a universe of dynamic data. The [net_http_request](https://hub.steampipe.io/plugins/turbot/net/tables/net_http_request) table of the [net plugin](https://hub.steampipe.io/plugins/turbot/net/) turns Steampipe into [an HTTP client](https://steampipe.io/blog/steampipe-is-an-http-client). The [Terraform](https://hub.steampipe.io/plugins/turbot/terraform) and [AWS CloudFormation](https://hub.steampipe.io/plugins/turbot/awscfn) plugins create a SQL facade over IaC config files. 

In that tradition, the [exec plugin](https://hub.steampipe.io/plugins/turbot/exec) creates a SQL facade over the universe of shell commands. It's a wide-open capability, we know you'll find creative uses for it, and we look forward to [hearing about them](/community/join).




Turbot is participating in [Hacktoberfest](https://hacktoberfest.com/) this year!

## Who is Turbot?

Turbot builds cloud intelligence & automated security platforms for DevOps teams:
- [Steampipe](https://steampipe.io) - select * from cloud;
- [Pipes](https://turbot.com/pipes) - Cloud intelligence and security
- [Guardrails](https://turbot.com/guardrails) - Go beyond reporting with Guardrails

We manage over 200 public open source repositories where we maintain open source samples and developer tools for our [Guardrails](https://github.com/search?q=topic%3Aturbot-guardrails+org%3Aturbot&type=Repositories) and [Pipes](https://github.com/search?q=topic%3Aturbot-pipes+org%3Aturbot&type=Repositories) products. As well as our open source project [Steampipe](https://github.com/search?q=topic%3Asteampipe+org%3Aturbot&type=repositories) and its related plugins and mod repositories.

## Why contribute? 

Be part of a growing community.  Last Hacktoberfest we merged 38 contributions from 30 contributors representing 13 countries. We have since grown to 2,000 community members who are passionate engineers that love our open source projects as much as we do:

<blockquote class="docs">
"I recently started using Steampipe and it has made my life about 30x better." - @bearjaws
</blockquote>

<blockquote class="docs">
"Steampipe is without a doubt one of the best open source tools for security compliance, operations and API orchestration." - @ashishsecdev
</blockquote>

<blockquote class="docs">
"The Steampipe is the best tool I have seen since Terraform." - @forty
</blockquote>

## What is Hacktoberfest?

DigitalOcean is in its 10th year organizing the overall [Hacktoberfest](https://hacktoberfest.com) event to celebrate the open source community. Hacktoberfest invites you to submit pull requests to open source projects. Those who submit multiple PRs will qualify for swag from DigitalOcean.

Turbot has marked all of our applicable public GitHub repos with the [hacktoberfest](https://github.com/search?q=org%3Aturbot+topic%3Ahacktoberfest) topic. If we accept one or more of your PRs, they'll be labeled `hacktoberfest-accepted` and will count towards the DigitalOcean swag campaign.

## Any special swag from Turbot?

Yes! In addition to the DigitalOcean event, Turbot is adding our own swag for contributions to our projects. We will offer unique Turbot stickers and t-shirts for no-code, low-code and high-code contributions. 

![](/images/blog/2024-swag-preview.jpg)

Here are examples of contributions to earn Turbot swag. Each swag item is limited to one per person.

**No-code** for a Turbot Hacktoberfest sticker:

* Publish a public facing blog post featuring Steampipe, Turbot Pipes, or Turbot Guardrails ([examples](https://steampipe.io/blog/news-and-reviews)).

* Record a publicly-accessible presentation, podcast, or talk about Steampipe, Turbot Pipes, or Turbot Guardrails.

* Submit an update to our Steampipe docs on [steampipe.io/docs](https://steampipe.io/docs) or [hub.steampipe.io](https://hub.steampipe.io).

* File a bug issue, with a detailed explanation of the testing you performed, to one of our public [Turbot repos](https://github.com/search?q=org%3Aturbot+topic%3Ahacktoberfest).

**Low-code** for a Turbot Hacktoberfest sticker:

* Raise a pull request for a bug fix or code improvement.

* Add a new column to an existing Steampipe table.

* Add a [new control](https://steampipe.io/docs/mods/writing-controls) to an existing Steampipe benchmark.

* Add a sample to our [Turbot Guardrails](https://github.com/turbot/guardrails-samples) or [Steampipe](https://github.com/turbot/steampipe-samples) repos.

**High-code** for a Turbot Hacktoberfest t-shirt:

* Add a [new Steampipe table](https://steampipe.io/docs/develop/writing-your-first-table) to an existing [plugin](https://hub.steampipe.io/plugins).

* Add a new [Steampipe dashboard](https://steampipe.io/docs/mods/writing-dashboards) to an existing [Insights mod](https://hub.steampipe.io/mods?q=insights).

* Add a new Steampipe benchmark to an existing [compliance mod](https://hub.steampipe.io/mods?objectives=compliance,security).

* Develop a [new Steampipe plugin](https://steampipe.io/docs/develop/writing-plugins) and publish it to [Steampipe Hub - Plugins](https://hub.steampipe.io/plugins).

* Develop a [new Steampipe mod](https://steampipe.io/docs/mods/overview) and publish it to [Steampipe Hub - Mods](https://hub.steampipe.io/mods).

* Participate in a company case study published on our [Turbot](https://turbot.com/blog?category=Case%20Study) or [Steampipe](https://steampipe.io/blog?category=Case%20Study) blog.

Inspire us! We'll also award prizes for innovative contributions that help move the community forward.

## When do you get your Turbot swag?

In the week following the event we will share links to redeem your swag. Most items will arrive in November, but depending on your location and t-shirt order time may take longer.

## Ready to contribute?

We can’t wait to see what incredible contributions you’ll make! If you have any questions, or need assistance, join us in our [Slack community](https://turbot.com/community/join) **#contributors** channel.


[Daemon](https://dae.mn) is a technology consultancy, based in the UK and South Africa, that helps customers assess their alignment with the best practices defined by Amazon's [AWS Well-Architected Framework](https://aws.amazon.com/architecture/well-architected). Their search for ways to streamline and accelerate the process turned up several commercial tools, including [Turbot](https://turbot.com), but their preference was for a free and open-source solution. Steampipe turned out to be just what they were looking for. "We saw a massive opportunity to use it, and also customize it," says Jean-Pierre Pienaar, senior consultant at Daemon.

The process begins with "pre-discovery": they gain access to a customer's AWS environments, identify the workloads to be reviewed, and explore them using a variety of vendor-provided and in-house tools. The goal here is to identify issues to be addressed by a Well-Architected assessment; this was a manual process that they realized Steampipe could help automate.

## Collaborative innovation

Because great minds think alike, they saw — before we announced our [AWS Well-Architected](https://hub.steampipe.io/mods/turbot/aws_well_architected) mod — that it would be possible to build a custom mod that maps controls from [AWS Compliance](https://hub.steampipe.io/mods/turbot/aws_compliance) to the questions asked in Well-Architected [Security](https://hub.steampipe.io/mods/turbot/aws_well_architected/controls/benchmark.well_architected_framework_security) pillar. Their first version of the custom mod mapped controls from `AWS Compliance` and added tags to align exporated outputs to the Well-Architected framework's questions and answers.

When they saw our announcement they contacted us to to see how we might collaborate. Inspired by their work, we released v0.8 of our Well-Architected mod. It incorporates their control mappings, and expands the tagging scheme to enable deeper integration with the Well-Architected tool's API which they're using to upload answers. We then wrote a [post](https://steampipe.io/blog/extend-well-architected-1) showing how anyone can extend the mod with new mappings. Following that model, Daemon have now successfully replaced their custom mod with [the official one](https://github.com/turbot/steampipe-mod-aws-well-architected). They, and others, are now in a position to add new mappings to existing controls, create new controls based on new queries, and address additional Well-Architected pillars.

## Facts versus emotions

Nathan Webster, principal consultant with Daemon, says the benefit of this approach isn't just automation of a manual checklist, though that is valuable. The Well-Architected benchmark delivers facts related to the questions asked, and that "helps take the emotion out of these conversations." In one case, a consulting client who needed to answer a question about whether backups are encrypted, and who believed that the answer was yes, learned otherwise. The benchmark revealed that while encryption was currently enabled, there were unencrypted legacy resources. That's need-to-know information, not a criticism of anyone's ability to configure AWS properly. And this data-driven approach is more likely to enagage technical partners who may otherwise tend to regard the assessment process as just a subjective checklist exercise. 

## Next steps

To close the loop, Daemon have created a script that converts the benchmark's [exported JSON](https://steampipe.io/docs/reference/cli/check#output-formats) to a mapping file and uploads the results directly into the AWS Well-Architected API. A copy of the results is also exported to HTML and stored in S3.  For controls in the alarm state, links to the failures in the report are noted in the Well-Architected review so they can be referenced in the review session with the client. The script is currently in customer trials; once proven, it will be shared with the community via Daemon's partnership with AWS.

Meanwhile, the Daemon team encourages others to help expand the Well-Architected mod. You'll need to apply your domain knowledge in order to map queries and controls to the framework, says Jean-Pierre Pienaar, but Steampipe makes it possible to do that "in a nice, complete way that's repeatable and scalable." If you dive in and give it a try, please [let us know](https://steampipe.io/community/join) how it goes!







Over the past decade, Turbot has focused relentlessly on providing our customers with industry- leading automation and data solutions for cloud operations, security and finance. Our passion for innovation in this space, and the consistent expansion of our product suite, have led us to an important crossroads. It’s time for our identity to reflect the evolution and future aspirations of our product suite.

Today we launch a comprehensive rebranding that embraces our history, while paving the way for an even brighter and more exciting future.

## Turbot Guardrails

Our flagship product, known until now as "Turbot", has been the cornerstone of our identity for 10 years. Many of you may recall first seeing Turbot at a trade show, proudly displaying our marketing tagline: "Go faster. Safely. With Turbot Guardrails". This wasn't just a catchphrase, but a commitment to provide tooling that enabled our customers to accelerate their cloud strategy, while remaining safe on their journey.

![](/images/blog/2023-07-introducing-turbot-guardrails-and-pipes/guardrails_booth.jpg)

Our signage at these shows would display “Guardrails” in such a prominent position that customers would often come up and say “Tell me about your Guardrails product”. 

Our inner control freaks have finally relented and we are happy to announce that our flagship product will now be known officially as "[Turbot Guardrails](/guardrails)", or "Guardrails" for short:

![](/images/blog/2023-07-introducing-turbot-guardrails-and-pipes/guardrails_wordmarks.svg)

This new identity embraces our marketing history and our promotion of the guardrails concept across the cloud operations and security domains.

## Steampipe Cloud becomes Turbot Pipes

A more recent product offering from Turbot, "Steampipe Cloud", will also transition its name to "[Turbot Pipes](/pipes)". This rebranding strengthens its connection to the Turbot family while acknowledging its relationship to Steampipe:

![](/images/blog/2023-07-introducing-turbot-guardrails-and-pipes/pipes_wordmarks.svg)

The open-source project "Steampipe" will retain its name, to demonstrate our commitment to the community that has built up around it, and will remain hosted on [steampipe.io](https://steampipe.io) to honor its unique role in our ecosystem. We will continue to invest heavily into building out the capabilities of Steampipe and fostering its community.

## Onward!

To codify these exciting changes, we have refreshed our websites to make it simpler to understand our product offerings and better distinguish between the different solutions we offer. This change isn't just about new names – it's about ensuring our brand clearly represents our dedication, innovation, and commitment to you, our users.

We are excited about the future of Turbot and want to express our deepest gratitude to you, our community, for your unwavering support and trust. You are more than just users; you are our partners in conquering cloud complexity!

Stay tuned for the exciting journey ahead!



In [Top 10 security items to improve in your AWS account](https://aws.amazon.com/blogs/security/top-10-security-items-to-improve-in-your-aws-account/), on the [AWS Security Blog](https://aws.amazon.com/blogs/security/), Nathan Case writes:

> If you're looking to improve your cloud security, a good place to start is to follow the top 10 most important cloud security tips that Stephen Schmidt, Chief Information Security Officer for AWS, laid out at AWS re:Invent 2019. Below are the tips, expanded to help you take action.

<div style={{"width":"70%","marginBottom":"1em"}}>
  <img alt="top-10-bullets" src="/images/blog/2023-06-aws-security-top-10/top-10-bullets.png" />
</div>

We agree! Let's look at how Steampipe can help you follow that advice. We'll develop a Steampipe benchmark that runs relevant controls — from mods like [AWS Compliance](https://hub.steampipe.io/mods/turbot/aws_compliance) and [AWS Perimeter](https://hub.steampipe.io/mods/turbot/aws_perimeter) — and produces a detailed report for all your accounts and regions. 

## Find relevant controls

The AWS Compliance mod is a rich source of controls that implement checks required by a broad set of frameworks and standards. On the Steampipe Hub's [controls page](https://hub.steampipe.io/mods/turbot/aws_compliance/controls) you can search for words that relate to a given security tip, for example `secrets` finds controls relevant to tip 3, *No hard-coding secrets*.

<div style={{"width":"50%","marginBottom":"1em"}}>
  <img alt="search-for-secrets" src="/images/blog/2023-06-aws-security-top-10/search-for-secrets.png" />
</div>

In this case, the first result is [CloudFormation stacks outputs should not have any secrets](https://hub.steampipe.io/mods/turbot/aws_compliance/controls/control.cloudformation_stack_output_no_secrets). 

<div style={{"width":"80%","marginBottom":"1em"}}>
  <img alt="cloud-formation-no-secrets" src="/images/blog/2023-06-aws-security-top-10/cloud-formation-no-secrets.png" />
</div>

It's clearly relevant, so we'll put its name — `aws_compliance.control.cloudformation_stack_output_no_secrets` — on our list.

Following the same approach, you can build up sets of control names relevant to each of the tips. Then it's time to roll them into a custom benchmark. 

## Create the benchmark skeleton

First, make and visit a directory.

```
$ mkdir aws-security-top-10
$ cd aws-security-top-10
```

Then, because this mod will use controls defined in the AWS Compliance mod, run the [steampipe mod install](https://steampipe.io/docs/reference/cli/mod) command to create a hidden local copy of that mod.

```
$ steampipe mod install github.com/turbot/steampipe-mod-aws-compliance

Installed 1 mod:

local
└── github.com/turbot/steampipe-mod-aws-compliance@v0.72.0
```

The command also creates an initial `mod.sp` file.

```
mod "local" {
  title = "aws-security-top-10"

  require {
    mod "github.com/turbot/steampipe-mod-aws-compliance" {
      version = "*"
    }
  }

}
```

You can change the title there to, for example, "AWS Top 10".

Now create another file, `top-10.sp`, in which to define the benchmark. Here's an initial skeleton.

```
benchmark "aws_top_10" {
  title       = "AWS Top 10"
  description = "Controls relevant to the top 10 security items."

  children = [
    benchmark.accurate_account_info,
    benchmark.use_mfa,
    benchmark.no_secrets,
    benchmark.limit_security_groups,
    benchmark.intentional_data_policies,
    benchmark.centralize_cloudtrail_logs,
    benchmark.validate_iam_roles,
    benchmark.take_action_on_findings,
    benchmark.rotate_keys
  ]

}

benchmark "accurate_account_info" {
  title = "1. Accurate account information"
  children = [
  ]
}

benchmark "use_mfa" {
  ...
}
```

A benchmark can contain benchmarks which in turn contain controls. The skeleton defines a top-level benchmark that enumerates a child benchmark for each of the security items. Each child benchmark has a list — initially empty -- of the controls we've gathered. 

## Put meat on the bones

Now it's just a matter of populating the lists with the names of the controls we've gathered. For example:

```
benchmark "use_mfa" {
  title = "2. Use multi-factor authentication (MFA)"
  children = [
    aws_compliance.control.iam_root_user_mfa_enabled,
    aws_compliance.control.iam_user_mfa_enabled,
    aws_compliance.control.iam_user_console_access_mfa_enabled,
  ]
}

```

Note that the AWS Compliance mod isn't the only source for controls. As we went through the list, we realized that [AWS Perimeter](https://hub.steampipe.io/mods/turbot/aws_perimeter) would have relevant controls too, such as [IAM role trust policy policies should prohibit public access](https://hub.steampipe.io/mods/turbot/aws_perimeter/controls/control.iam_role_trust_policy_prohibit_public_access). To use some of them, run [steampipe mod install](https://steampipe.io/docs/reference/cli/mod) again to gain access to them.

```
$ steampipe mod install github.com/turbot/steampipe-mod-aws-perimeter

Installed 1 mod:

local
└── github.com/turbot/steampipe-mod-aws-perimeter@v0.3.0

$ steampipe mod list

local
├── github.com/turbot/steampipe-mod-aws-compliance@v0.72.0
└── github.com/turbot/steampipe-mod-aws-perimeter@v0.3.0
```

Now we can refer to controls in both installed mods.

```
benchmark "validate_iam_roles" {
  title = "7. Validate IAM roles"
  children = [
    aws_compliance.control.cis_v150_1_20,
    aws_compliance.control.iam_access_analyzer_enabled_without_findings,
    aws_perimeter.control.iam_role_trust_policy_prohibit_public_access
  ]
}
```

## Run the benchmarks

To run the benchmark in your terminal:

```
$ steampipe check all
```

To run the benchmark in your browser, run `steampipe dashboard` and visit `https://localhost:9194`.

<div style={{"marginBottom":"1em"}}>
  <img alt="top-10-dashboard" src="/images/blog/2023-06-aws-security-top-10/aws-top-10-dashboard-benchmark.png" />
</div>

## Share the benchmark report

To share the report on Steampipe Cloud, first [login](https://steampipe.io/docs/reference/cli/login). Then:

```
steampipe dashboard --share benchmark.aws_top_10

Snapshot uploaded to https://cloud.steampipe.io/user/
udell/workspace/personal/snapshot/snap_cib1atb9rqbm1gfe1n10_04d5drrmy96xt3q63qhyhwweau
```

You can now view the benchmark at that URL, and you can [share](https://steampipe.io/docs/cloud/dashboards#saving--sharing-snapshots) the view with anyone you've invited into the workspace, or anyone to whom you give the URL.

## Where's #10?

We didn't include controls for `Tip #10: Being involved in the dev cycle` because none (as yet) are relevant. Instead, we'll humbly suggest a best practice: Use Steampipe! It's a great way to monitor your security posture and assure that it remains continuously effective.

## Next steps

We initially built this mod as tutorial, then the team decided to adopt it as an [offical mod](https://github.com/turbot/steampipe-mod-aws-security-top-10). We can imagine other Top 10 benchmarks, but we're sure you can too. If you build one, and would like to include it in the Steampipe hub, please [let us know](https://steampipe.io/community/join)!












In [part 1](https://steampipe.io/blog/extend-well-architected-1) we showed how you can extend the [AWS Well-Architected](https://hub.steampipe.io/mods/turbot/aws_well_architected) mod with a benchmark that helps you assess your infrastructure for enviromental impacts targeted by the [Sustainability](https://docs.aws.amazon.com/wellarchitected/latest/framework/sustainability.html) pillar. The AWS Well-Architected mod is an example of how to [reuse and remix](https://steampipe.io/blog/remixing-dashboards) existing benchmarks and dashboards: it cherrypicks controls from [AWS Compliance](https://hub.steampipe.io/mods/turbot/aws_compliance) which provides more than 780 controls. The Sustainability benchmark we added in part 1 follows the same strategy: find and reuse existing controls. But what if you want to check something that isn't covered by an existing control? It's easy to add your own custom controls, and in this post we'll show you how.

The [SUS 4](https://docs.aws.amazon.com/wellarchitected/latest/framework/sus-04.html) section of the Well-Architected framework relates to data. One of the recommended best practices is [SUS04-BP02 Use technologies that support data access and storage patterns](https://docs.aws.amazon.com/wellarchitected/latest/framework/sus_sus_data_a3.html). To align with that best practice, you may want to take advantage of [S3 Intelligent-Tiering storage](https://aws.amazon.com/s3/storage-classes/intelligent-tiering/). There isn't yet an existing control that checks whether the feature is enabled for your S3 buckets. So let's create one!


## Fork and clone the AWS Well-Architected mod

To get started, create a [fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo#forking-a-repository) of the mod in your GitHub account. Then clone it.

```
git clone https://github.com/judell/steampipe-mod-aws-well-architected
cd steampipe-mod-aws-well-architected
```

 And install the dependent mod, AWS Compliance.

```
$ steampipe mod install

Installed 1 mod:

aws_well_architected
└── github.com/turbot/steampipe-mod-aws-compliance@v0.67.0
```

If you do nothing else at this point, running `steampipe dashboard` will display both AWS Compliance and AWS Well-Architected just as happens when you use the AWS Well-Architected mod directly.

<div style={{"width":"60%"}}>

![waf 01](/images/blog/2023-06-extending-well-architected/waf-01.png)

</div>

## Define a new second-level benchmark for the Sustainability pillar

The existing pillars live in subdirectories of `well-architected-framework`. Create a new subdirectory there, called `sustainability`, and within it a new file, `sustainability.sp`:

```
locals {
  well_architected_framework_sustainability_common_tags = merge(local.well_architected_framework_common_tags, {
    pillar_id = "sustainability"
  })
}

benchmark "well_architected_framework_sustainability" {
  title       = "sustainability"
  description = "The Sustainability pillar addresses the long-term environmental, economic, and societal impact of your business activities."
  children    = [
    benchmark.well_architected_framework_sus04
  ]

  tags = local.well_architected_framework_sustainability_common_tags
}
```

We've yet to define `benchmark.well_architected_framework_sus04`. But first, refer to the new benchmark from the top-level `well_architected_framework.sp`:

```
benchmark "well_architected_framework" {
  title         = "AWS Well-Architected Framework"
  description   = "The AWS Well-Architected Framework describes key concepts, design principles, and architectural best practices for designing and running workloads in the cloud. By answering a few foundational questions, learn how well your architecture aligns with cloud best practices and gain guidance for making improvements."
  documentation = file("./well_architected_framework/docs/well_architected_framework_overview.md")

  children = [
    benchmark.well_architected_framework_operational_excellence,
    benchmark.well_architected_framework_reliability,
    benchmark.well_architected_framework_security,
    benchmark.well_architected_framework_sustainability
  ]

  tags = local.well_architected_framework_common_tags
}
```

## Define a third-level benchmark for a topic

Now create a new file, `sus04.sp`, which defines `benchmark.well_architected_framework_sus04` as cited above.

```
locals {
  well_architected_framework_sus04_common_tags = merge(local.well_architected_framework_sustainability_common_tags, {
    question_id = "sus_data"
  })
}

benchmark "well_architected_framework_sus04" {
  title       = "SUS04 How do you take advantage of data management policies and patterns to support your sustainability goals?"
  description = "Implement data management practices to reduce the provisioned storage required to support your workload, and the resources required to use it. Understand your data, and use storage technologies and configurations that more effectively support the business value of the data and how it’s used. Lifecycle data to more efficient, less performant storage when requirements decrease, and delete data that’s no longer required."
  children    = [
    control.s3_buckets_use_intelligent_tiering
  ]

  tags = merge(local.well_architected_framework_sus04_common_tags, {
    choice_id = "sus_sus_data_a3"
  })
}
```

Refer to the [Aligning tags](https://steampipe.io/blog/extend-well-architected-1#aligning-tags) to see how you can use the [aws_wellarchitected_answer](https://hub.steampipe.io/plugins/turbot/aws/tables/aws_wellarchitected_answer) table to find values for `question_id` and `choice_id`.

If we were cherrypicking existing controls, their names would be fully qualified in the `children` list. In part 1, for example, we reused a control from the [AWS Thrifty](https://hub.steampipe.io/mods/turbot/aws_thrifty) mod like so. 

```
  children = [
    aws_thrifty.control.ecs_cluster_low_utilization,
  ]
```

In this case, our custom control belongs to the mod we're creating so we don't need to fully qualify its name. Note that, as per this example from [the documentation](https://steampipe.io/docs/mods/mod-dependencies#mod-dependencies), you can mix both flavors in a benchmark's `children` list:

```
benchmark "my_mod_public_resources" {
  title       = "Public Resources"
  description = "Resources that are public."
  children = [
    aws_compliance.control.dms_replication_instance_not_publicly_accessible,
    aws_compliance.control.s3_bucket_restrict_public_write_access,
    control.my_mod_public_ec2,
  ]
}
```

## Define a new control

We've defined the `well_architected_framework_sus04` benchmark to expect a control called `s3_buckets_use_intelligent_tiering`. Like all controls, it will be powered by a query that returns [three required columns](https://steampipe.io/docs/reference/mod-resources/control#required-control-columns): `status`, `reason`, and `resource`. Here's a query that enumerates S3 buckets, and joins with the [aws_s3_bucket_intelligent_tiering_configuration] (https://hub.steampipe.io/plugins/turbot/aws/tables/aws_s3_bucket_intelligent_tiering_configuration) table.

```
query "s3_buckets_use_intelligent_tiering" {
  sql = <<EOQ
    select
      b.name as resource,
      case when i.status = 'Enabled' then 'ok' else 'alarm' end as status,
      case when i.status = 'Enabled' 
        then b.name || ' uses intelligent tiering.'
        else b.name || ' does not use intelligent tiering.'
      end as reason,
      b.account_id,
      b.region
    from
      aws_s3_bucket b
    left join
      aws_s3_bucket_intelligent_tiering_configuration i 
    on 
      b.name = i.bucket_name
  EOQ
}
```

The new control refers to that query.

```
control "s3_buckets_use_intelligent_tiering" {
  title         = "Use S3 intelligent tiering"
  description   = "S3 buckets should use intelligent tiering"
  query         = query.s3_buckets_use_intelligent_tiering
}
```

Per the [documentation](https://steampipe.io/docs/reference/mod-resources/control), a control can use either `query = ` as we're doing here to refer to a separately-defined query, or `sql =` to include the query within the control. We recommend the `query` approach, so that a query like `s3_buckets_use_intelligent_tiering` is also avaiable in other contexts.

And voilà! The new benchmark is up and running.

![benchmark with custom control](/images/blog/2023-07-extending-well-architected-2/intelligent-tiering-benchmark.png)

## Next steps

You can, of course, use your fork of the Well-Architected mod privately. But we hope you'll want to contribute new mappings that improve the Well-Architected mod for everyone. For example, here's a [pull request](https://github.com/turbot/steampipe-mod-aws-well-architected/pull/29) for the new benchmark and custom control we've defined here.

Do you have other ideas for extending the Well-Architected mod? Give them a try and [let us know](https://steampipe.io/community/join) how it goes!







The [Center for Internet Security](https://www.cisecurity.org/) (CIS) just released an updated version (v2.0) of their CIS AWS Benchmark. The new version of the benchmark includes 2 new recommendations, 1 removed, and updates to descriptions and remediation steps.

## Our take on the latest major changes

- **Added - 1.22 Ensure access to AWSCloudShellFullAccess is restricted:** AWS CloudShell allows running CLI commands with full access, including file upload/download and sudo permissions. This presents a potential data exfiltration channel for malicious cloud admins. Restricting access and denying file transfer permissions through a more restrictive IAM policy is recommended. Personally, CloudShell is a great way to run commands in your account quickly and [quickly enable tools like Steampipe](https://dev.to/aws-builders/instantly-query-aws-with-sql-in-cloudshell-hd0). However, with any highly privileged role, boundary and lockdown permissions should be considered for the least privilege restrictions.

- **Added - 5.6 Ensure that EC2 Metadata Service only allows IMDSv2:** — Instance Metadata Service Version 2 (IMDSv2), which is a session-oriented method which provides temporary, frequently rotated credentials. Version 2 adds new protections to mitigate vulnerabilities from open website application firewalls, open reverse proxies, SSRF, etc. This is a common recommendation used across many other [frameworks](https://hub.steampipe.io/mods/turbot/aws_compliance/controls/control.ec2_instance_uses_imdsv2) like NIST, FedRamp, and HIPAA.

- **Removed - 2.1.1 Ensure all S3 buckets employ encryption-at-rest:** — In Benchmark v1.5, the requirement for encryption-at-rest in S3 buckets (2.1.1) was removed since AWS now automatically encrypts all new objects using SSE-S3 as the default encryption setting since January 2023. Existing buckets with S3 Default Encryption remain unchanged, so you may want to still check for encryption-at-rest for legacy buckets. However, moving forward at a minimum all new buckets will be encrypted and you cannot remove encryption any longer. With the removal of the control, all other Storage recommendations have changed their IDs in section 2.0.

## Try the new AWS CIS v2.0 controls!

The [Steampipe AWS Compliance mod](https://github.com/turbot/steampipe-mod-aws-compliance), is packed with hundreds of controls that check your AWS accounts for compliance with 25 benchmarks including NIST, PCI, HIPAA, SOC2, FedRAMP and more, now includes new controls for AWS CIS v2.0. If you're new to Steampipe, [download](https://steampipe.io/downloads) Steampipe, [install and configure](https://hub.steampipe.io/plugins/turbot/aws#get-started) the AWS plugin, and run these commands.

```
steampipe plugin update aws
git clone https://github.com/turbot/steampipe-mod-aws-compliance.git
cd steampipe-mod-aws-compliance
steampipe check aws_compliance.benchmark.cis_v200
```

If you've already installed Steampipe and the AWS plugin, and cloned the AWS Compliance mod, then just do this.

```
cd steampipe-mod-aws-compliance
git pull
steampipe check aws_compliance.benchmark.cis_v200
```

Here's a sample report in the console.

<div style={{"marginTop":"20px","marginBottom":"20px"}}>
  <img  src="/images/blog/2023-07-cis-v20-aws-benchmark/cis-console.png" />
</div>

You can output results to formats including JSON, CSV, HTML, and ASFF, or use [custom output templates](https://steampipe.io/docs/develop/writing-control-output-templates) to create new output formats.

To view the AWS CIS v2.0 benchmark report in your browser, run this command in the same cloned repo.

```
steampipe dashboard
```

Then open `http://localhost:9194` in your browser and view the dashboard.

<div style={{"marginTop":"20px","marginBottom":"20px"}}>
<img src="/images/blog/2023-07-cis-v20-aws-benchmark/cis-dashboard.png" />
</div>

## We love open source!

Steampipe delivers a full suite of tools to build, execute and share cloud configuration, compliance, and security frameworks using HCL + SQL! We would love your help to expand the open source documentation and control coverage for CIS, PCI, HIPAA, NIST, and more. The best way to get started is to [join our Slack community](https://steampipe.io/community/join); we would love to talk to you!



The [GitHub plugin](https://github.com/turbot/steampipe-plugin-github) ranks second among all Steampipe plugins with respect to [stargazers](https://github.com/orgs/turbot/repositories?q=plugin&type=all&language=&sort=stargazers). Clearly it has served the community well. But there were concerns about its rapid consumption of your rate limit allowance. When running [GitHub Compliance](https://github.com/turbot/steampipe-mod-github-compliance) against our own organization, for example, we weren't able to complete the benchmark without exceeding the rate limit. That's now resolved by the v0.28 release we're announcing today. Thanks to GitHub's GraphQL-based v4 API, we've enabled the plugin to fetch a lot more data before hitting the limit. 

This is a major overhaul that entails both breaking changes to existing tables, as well as  some new tables. So if you have existing queries you'll want to review the [changes](#how-these-changes-affect-you) in order to align with v0.28. To help you orient to the new version, and for the benefit of anyone working on a v3-to-v4 transition, here's an overview of our initial research and migration strategy.

## Unblocking the GitHub Compliance and GitHub Sherlock mods

For an organization like ours, with many repos, it was challenging to run the [Compliance](https://hub.steampipe.io/mods/turbot/github_compliance) and [Sherlock](https://hub.steampipe.io/mods/turbot/github_sherlock) mods in a timely fashion — or even just to complete them. We're delighted to report that, with v0.28, we were finally able to complete the running of the `Compliance` mod, and use only about 1/5th of our rate limits in doing so.

<div style={{"width":"100%", "marginTop":"2em", "marginBottom":"2em"}}>
  <img alt="compliance mod run complete" src="/images/blog/2023-06-github-graphql/compliance-completed.png" />
</div>

The sentiment expressed by our CEO, Nathan Wallace, nicely sums up the progress made with these changes to the plugin.

> Awesome we can finally run that compliance mod!!!

## How these changes affect you

This is a major overhaul, with breaking changes across most tables, so you'll want to ensure that you have capacity to upgrade any queries and/or mods you depend on before upgrading the plugin.

Column changes are documented comprehensively in the following tracking issues:

- [Update github_tag table to use GraphQL API #238](https://github.com/turbot/steampipe-plugin-github/issues/238)
- [Update github_team & github_my_team tables to use GraphQL API #241](https://github.com/turbot/steampipe-plugin-github/issues/241)
- [Update github_team_member table to use GraphQL API #243](https://github.com/turbot/steampipe-plugin-github/issues/243)
- [Update github_branch table to use GraphQL API #244](https://github.com/turbot/steampipe-plugin-github/issues/244)
- [Update github_user table to use GraphQL API #248](https://github.com/turbot/steampipe-plugin-github/issues/248)
- [Update github_branch_protection table to use GraphQL API #251](https://github.com/turbot/steampipe-plugin-github/issues/251)
- [Update github_team_repository table to use GraphQL API #254](https://github.com/turbot/steampipe-plugin-github/issues/254)
- [Update github_repository table to use GraphQL API #256](https://github.com/turbot/steampipe-plugin-github/issues/256)
- [Update github_my_repository table to use GraphQL API #257](https://github.com/turbot/steampipe-plugin-github/issues/257)
- [Update github_organization table to use GraphQL API #258](https://github.com/turbot/steampipe-plugin-github/issues/258)
- [Update github_my_organization table to use GraphQL API #259](https://github.com/turbot/steampipe-plugin-github/issues/259)
- [Update github_issue & github_my_issue tables to use GraphQL API #260](https://github.com/turbot/steampipe-plugin-github/issues/260)
- [Update github_commit table to use GraphQL API #262](https://github.com/turbot/steampipe-plugin-github/issues/262)
- [Update github_community_profile table to use GraphQL API #266](https://github.com/turbot/steampipe-plugin-github/issues/266)
- [Update github_search_issue, github_search_pull_request, github_search_repository, github_search_user tables to use GraphQL API #268](https://github.com/turbot/steampipe-plugin-github/issues/268)
- [Update github_stargazer & github_my_star tables to use GraphQL API #270](https://github.com/turbot/steampipe-plugin-github/issues/270)
- [Update github_pull_request table to use GraphQL API #276](https://github.com/turbot/steampipe-plugin-github/issues/276)
- [Update github_organization_member table to include all user fields #279](https://github.com/turbot/steampipe-plugin-github/issues/279)

We've also included some new tables:
- [github_issue_comment](https://hub.steampipe.io/plugins/turbot/github/tables/github_issue_comment) ([#281](https://github.com/turbot/steampipe-plugin-github/pull/281))
- [github_pull_request_comment](https://hub.steampipe.io/plugins/turbot/github/tables/github_pull_request_comment) ([#281](https://github.com/turbot/steampipe-plugin-github/pull/281))
- [github_rate_limit_graphql](https://hub.steampipe.io/plugins/turbot/github/tables/github_rate_limit_graphql) ([#280](https://github.com/turbot/steampipe-plugin-github/pull/280))
- [github_repository_collaborator](https://hub.steampipe.io/plugins/turbot/github/tables/github_repository_collaborator) ([#280](https://github.com/turbot/steampipe-plugin-github/pull/280))
- [github_repository_deployment](https://hub.steampipe.io/plugins/turbot/github/tables/github_repository_deployment) ([#282](https://github.com/turbot/steampipe-plugin-github/pull/282))
- [github_repository_environment](https://hub.steampipe.io/plugins/turbot/github/tables/github_repository_environment) ([#282](https://github.com/turbot/steampipe-plugin-github/pull/282))

_Note: that when paging the repository resources, responses are a bit slower than they were with the previous REST-only tables. We hope GitHub will address this minor regression by doing work to improve performance in that case._

## Next steps

We're not done trying to make this plugin better! As the GraphQL API becomes more mature, we may revisit tables that still use REST primarily, or in a secondary way to hydrate extra data, looking for any performance improvements to be gained by swapping these tables to GraphQL. Meanwhile, once you determine there's a clear path to upgrading to v0.28, please give it a try and [let us know](https://steampipe.io/community/join) how it goes!












The v0.8 release of the [AWS Well-Architected](https://hub.steampipe.io/mods/turbot/aws_well_architected) mod includes support for three of the six pillars of the [AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html): [Operational Excellence](https://hub.steampipe.io/mods/turbot/aws_well_architected/controls/benchmark.well_architected_framework_operational_excellence), [Reliability](https://hub.steampipe.io/mods/turbot/aws_well_architected/controls/benchmark.well_architected_framework_reliability) and [Security](https://hub.steampipe.io/mods/turbot/aws_well_architected/controls/benchmark.well_architected_framework_security). In this post we'll show how you can extend the Well-Architected mod with a benchmark that helps you assess your infrastructure for enviromental impacts targeted by the [Sustainability](https://docs.aws.amazon.com/wellarchitected/latest/framework/sustainability.html) pillar. 

The AWS Well-Architected mod is an example of how to [reuse and remix](https://steampipe.io/blog/remixing-dashboards) existing benchmarks and dashboards: it cherrypicks controls from [AWS Compliance](https://hub.steampipe.io/mods/turbot/aws_compliance) which provides more than 780 controls. The Sustainability benchmark we'll add here will, similarly, reuse controls from [AWS Thrifty](https://hub.steampipe.io/mods/turbot/aws_thrifty) which provides 46 controls. Other sources include [AWS Perimeter](https://hub.steampipe.io/mods/turbot/aws_perimeter) (52 controls) and [AWS Tags](https://hub.steampipe.io/mods/turbot/aws_tags) (350 controls).

## Fork and clone the AWS Well-Architected mod

To get started, create a [fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo#forking-a-repository) of the mod in your GitHub account. Then clone it.

```
git clone https://github.com/judell/steampipe-mod-aws-well-architected
cd steampipe-mod-aws-well-architected
```



If you do nothing else at this point, running `steampipe dashboard` will display both AWS Compliance and AWS Well-Architected just as happens when you use the AWS Well-Architected mod directly.

<div style={{"width":"60%"}}>

![waf 01](/images/blog/2023-06-extending-well-architected/waf-01.png)

</div>

## Define a new second-level benchmark for the Sustainability pillar

The existing pillars live in subdirectories of `well-architected-framework`. Create a new subdirectory there, called `sustainability`.

<div style={{"width":"30%"}}>

![waf 02](/images/blog/2023-06-extending-well-architected/waf-02.png)

</div>

There we'll create a new file, `sustainability.sp`:

```
locals {
  well_architected_framework_sustainability_common_tags = merge(local.well_architected_framework_common_tags, {
    pillar_id = "sustainability"
  })
}

benchmark "well_architected_framework_sustainability" {
  title       = "sustainability"
  description = "The Sustainability pillar addresses the long-term environmental, economic, and societal impact of your business activities."
  children = [
    // tbd
  ]

  tags = local.well_architected_framework_sustainability_common_tags
}
```

And refer to the new benchmark from the top-level `well_architected_framework.sp`:

```
benchmark "well_architected_framework" {
  title       = "AWS Well-Architected Framework"
  description = "The AWS Well-Architected Framework describes key concepts, design principles, and architectural best practices for designing and running workloads in the cloud. By answering a few foundational questions, learn how well your architecture aligns with cloud best practices and gain guidance for making improvements."
  documentation = file("./well_architected_framework/docs/well_architected_framework_overview.md")

  children = [
    benchmark.well_architected_framework_operational_excellence,
    benchmark.well_architected_framework_reliability,
    benchmark.well_architected_framework_security,
    benchmark.well_architected_framework_sustainability
  ]

  tags = local.well_architected_framework_common_tags
}
```

## Define a third-level benchmark for a topic

The [Sustainability pillar](https://docs.aws.amazon.com/pdfs/wellarchitected/latest/sustainability-pillar/wellarchitected-sustainability-pillar.pdf) covers six topics. Here we'll focus on `SUS02: Alignment to demand.` Create a new file, `sus02.sp`:

```
locals {
  well_architected_framework_sus02_common_tags = merge(local.well_architected_framework_sustainability_common_tags, {
    question_id = "sus_user"
  })
}

benchmark "well_architected_framework_sus02" {
  title       = "SUS02 How do you align cloud resources to your demand?"
  description = "Scale infrastructure to continually match demand and verify that you use only the minimum resources required to support your users."
  children = [
  ]

  tags = local.well_architected_framework_sus02_common_tags
}
```

## Define a child benchmark for a best practice

We'll start with `SUS02-BP03 Stop the creation and maintenance of unused assets`. The children of this benchmark will be controls that identify underutilized resources. The relevant controls aren't in the AWS Compliance mod we're already using. But the tagline for AWS Thrifty is: "Be Thrifty on AWS! This mod checks for unused resources and opportunities to optimize your spend on AWS." Can we cherrypick controls that target unused assets? Let's search for likely candidates.

<div style={{"width":"50%"}}>

![waf 03](/images/blog/2023-06-extending-well-architected/waf-03.png)

</div>

We'll use [ECS Clusters with Low Utilization](https://hub.steampipe.io/mods/turbot/aws_thrifty/controls/control.ecs_cluster_low_utilization) for our example here. In order to cherrypick that control, we'll add Thrifty as another mod dependency. (If Thrifty were already a dependency, this step wouldn't be needed.)

```
steampipe mod install github.com/turbot/steampipe-mod-aws-thrifty

Installed 1 mod:

aws_well_architected
└── github.com/turbot/steampipe-mod-aws-thrifty@0.21.0
```

```
steampipe mod list

aws_well_architected
├── github.com/turbot/steampipe-mod-aws-compliance@v0.67.0
└── github.com/turbot/steampipe-mod-aws-thrifty@v0.21.0
```

On the [hub page](https://hub.steampipe.io/mods/turbot/aws_thrifty/controls/control.ecs_cluster_low_utilization) we discover the fully-qualified name of the control.

<div style={{"width":"70%"}}>

![waf 04](/images/blog/2023-06-extending-well-architected/waf-04.png)

</div>

We'll add that as the first child of `well_architected_framework_sus02`.

```
benchmark "well_architected_framework_sus02" {
  title       = "SUS02 How do you align cloud resources to your demand?"
  description = "Scale infrastructure to continually match demand and verify that you use only the minimum resources required to support your users."
  children = [
    benchmark.well_architected_framework_sus02_bp03
  ]

  tags = local.well_architected_framework_sus02_common_tags
}

benchmark "well_architected_framework_sus02_bp03" {
  title       = "BP03 Stop the creation and maintenance of unused assets"
  description = "Decommission unused assets in your workload to reduce the number of cloud resources required to support your demand and minimize waste."
  children = [
    aws_thrifty.control.ecs_cluster_low_utilization,
  ]

  tags = merge(local.well_architected_framework_sus02_common_tags, {
    choice_id = "sus_sus_user_a4"
    severity = "low"
  })
}
```

And voilà! The new benchmark is up and running.

![waf 06](/images/blog/2023-06-extending-well-architected/waf-06.png)

## Aligning tags

If your goal is to use the AWS Well-Architected mod to gather answers that you'll post back to the AWS Well-Architected Tool, you'll want the values of these tags to align with nomenclature used by that tool. How to discover those values? Steampipe to the rescue! You can query [aws_wellarchitected_answer](https://hub.steampipe.io/plugins/turbot/aws/tables/aws_wellarchitected_answer), one of eight tables that we've [added to the AWS plugin](https://github.com/turbot/steampipe-plugin-aws/blob/main/CHANGELOG.md#v01030-2023-05-18) for support the AWS Well-Architected API.

To find question ids and titles:

```
select
  pillar_id,
  question_id,
  ltrim(question_title) as question_title
from
  aws_wellarchitected_answer
where
  workload_id = 'b560b72391f8ca6332186d6b61dfbfc2'
  and pillar_id = 'sustainability'
```

```
+----------------+--------------+------------------------------------------------------------------------------------------------------------------+
| pillar_id      | question_id  | question_title                                                                                                   |
+----------------+--------------+------------------------------------------------------------------------------------------------------------------+
| sustainability | sus_dev      | How do your organizational processes support your sustainability goals?                                          |
| sustainability | sus_region   | How do you select Regions for your workload?                                                                     |
| sustainability | sus_software | How do you take advantage of software and architecture patterns to support your sustainability goals?            |
| sustainability | sus_hardware | How do you select and use cloud hardware and services in your architecture to support your sustainability goals? |
| sustainability | sus_data     | How do you take advantage of data management policies and patterns to support your sustainability goals?         |
| sustainability | sus_user     | How do you align cloud resources to your demand?                                                                 |
+----------------+--------------+------------------------------------------------------------------------------------------------------------------+
```

To find choice ids and titles:

```
with info as (
  select
    question_id,
    jsonb_array_elements(choices) as choice
  from
    aws_wellarchitected_answer
  where
   workload_id = 'b560b72391f8ca6332186d6b61dfbfc2'
   and pillar_id = 'sustainability'
)
select
  question_id,
  choice ->> 'ChoiceId' as choice_id,
  choice ->> 'Title' as choice_title
from
  info
where
  question_id = 'sus_user'
order by
  question_id, choice_id
```

```
+-------------+-----------------+-----------------------------------------------------------------------------------+
| question_id | choice_id       | choice_title                                                                      |
+-------------+-----------------+-----------------------------------------------------------------------------------+
| sus_user    | sus_sus_user_a2 | Scale workload infrastructure dynamically                                         |
| sus_user    | sus_sus_user_a3 | Align SLAs with sustainability goals                                              |
| sus_user    | sus_sus_user_a4 | Stop the creation and maintenance of unused assets                                |
| sus_user    | sus_sus_user_a5 | Optimize geographic placement of workloads based on their networking requirements |
| sus_user    | sus_sus_user_a6 | Optimize team member resources for activities performed                           |
| sus_user    | sus_sus_user_a7 | Implement buffering or throttling to flatten the demand curve                     |
| sus_user    | sus_sus_user_no | None of these                                                                     |
+-------------+-----------------+-----------------------------------------------------------------------------------+
```

The leaf benchmarks also include a `severity` tag. We take these from the [documentation](https://docs.aws.amazon.com/pdfs/wellarchitected/latest/sustainability-pillar/wellarchitected-sustainability-pillar.pdf#sustainability-pillar), which was also our source for the `sus02-bp03` in the control's name.

<div style={{"width":"70%"}}>

![waf 07](/images/blog/2023-06-extending-well-architected/waf-07.png)

</div>

Note that tags added this way will not appear in console or dashboard output, as do [common dimensons](https://steampipe.io/blog/mod-common-dimensions) like `account_id` and `region`. They will, however, appear in CSV and JSON [exports](https://steampipe.io/docs/reference/cli/check#output-formats).

## Next steps

You can, of course, use your fork of the Well-Architected mod privately. But we hope you'll want to contribute new mappings that improve the Well-Architected mod for everyone. For example, here's a [pull request](https://github.com/turbot/steampipe-mod-aws-well-architected/pull/27) for the changes we've shown here. To build on it, you can look for other controls that mention "low utilization" or "unused assets" and add them as additional children.

Note that you're not limited to existing controls. You can write your own controls, driven by your own queries, and we'll explore that topic in part 2 of this series. Meanwhile, give this approach a try and [let us know](https://steampipe.io/community/join) how it goes!












The [Center for Internet Security](https://www.cisecurity.org/) (CIS) just released an updated version (v1.2) of their CIS Oracle Cloud Infrastructure (OCI) Benchmark. The new version of the benchmark includes updated security guidance for services such as IAM, Networking, and Storage. We’ll review the new recommendations here.

## Our take on the new recommendations

### IAM
The common theme is aligning common controls from other Cloud Service Provider (CSP) benchmarks (AWS, Azure, etc):

- **1.5 Ensure IAM password policy expires passwords within 365 days** - Setting passwords to expire after 365 days through IAM password policies will at least automatically enforce password rotation and expiration. However most organizations we see will enforce <90 days for their duration.

- **1.6 Ensure IAM password policy prevents password reuse** - Common guidance across other CIS benchmarks, this control helps prevent potential security breaches and limits opportunities for malicious actors to compromise user accounts.

- **1.13 Ensure Dynamic Groups are used for OCI instances, OCI Cloud Databases and OCI Function to access OCI resources** - Dynamic Groups are one way to grant  appropriate IAM policies. While other benchmarks make similar recommendations too, in practice we see users set policies by roles or inline if they have appropriate controls and processes to manage at scale.

- **1.14 Ensure storage service-level admins cannot delete resources they manage** - To prevent unintended or unauthorized deletion of storage resources, a handy solution is to configure OCI IAM policies that grant OCI groups permissions to manage storage-related services without granting them the ability to delete those storage resources. With any preventative control, you take the penalty of the UX; you need a process to grant exceptions in a timely manner.

### Networking
All new controls are based on least privilege principles, and check that access is restricted to specific scopes:

- **2.6 Ensure Oracle Integration Cloud (OIC) access is restricted to allowed sources** - OIC instances (not to be confused with OCI!) are created within a secure private network managed by Oracle; they have public endpoints. Protect your OIC instances from unauthorized network access by configuring ingress filtering of network traffic and limiting access to your approved corporate IP addresses or Virtual Cloud Networks (VCN).

- **2.7 Ensure Oracle Analytics Cloud (OAC) access is restricted to allowed sources or deployed within a Virtual Cloud Network** - For a secure deployment, it is best to create new Oracle Analytics Cloud (OAC) instances within a VCN and configure Access Control Rules to restrict access to corporate IP addresses or existing VCNs for already deployed OAC instances.

- **2.8 Ensure Oracle Autonomous Shared Databases (ADB) access is restricted to allowed sources or deployed within a Virtual Cloud Network** - Most databases aren't meant to be accessible to the public. As a rule, assume your workloads are internal and make exceptions when public access is required.

### Storage

Storage services are elevated to their own section called "Storage," which offers guidance on utilizing Object Storage, Block Volumes, and File Storage Services:

- **4.1.2 Ensure Object Storage Buckets are encrypted with a Customer Managed Key**,
- **4.2.1 Ensure Block Volumes are encrypted with Customer Managed Keys (CMK)**,
- **4.2.2 Ensure Boot Volumes are encrypted with Customer Managed Key (CMK)**,
- **4.3.1 Ensure File Storage Systems are encrypted with Customer Managed Keys (CMK)**

All are similar recommendations to encrypt with CMKs vs Oracle Managed Keys. CMKs provide greater control than OCI-managed encryption keys; they enable you to  isolate encryption per resource. However, managing CMKs can be a lot of overhead if it is not required by your organizational Cryptography or Encryption policy. Using Oracle Managed Keys may suffice for your organization and would be easier to manage at scale.

- **4.1.3 Ensure Versioning is Enabled for Object Storage Buckets** - Enabling versioning for Object Storage buckets adds an extra layer of integrity to your objects. It enables you to handle accidental deletions by restoring prior revisions. While a good practice, the additional storage can cost more. So you should evaluate the tradeoff.

## Try the new OCI CIS v1.2 controls!

The [Steampipe OCI Compliance mod](https://hub.steampipe.io/mods/turbot/oci_compliance) is packed with over 80 controls and benchmarks that check your accounts for compliance with CIS v1.1 and CIS v1.2. If you're new to Steampipe, [download](https://steampipe.io/downloads) Steampipe, [install and configure](https://hub.steampipe.io/plugins/turbot/aws#get-started) the AWS plugin, and run these commands.

```
git clone https://github.com/turbot/steampipe-mod-oci-compliance
cd steampipe-mod-oci-compliance
steampipe check oci_compliance.benchmark.cis_v120
```

If you've already installed Steampipe and the OCI plugin, and cloned the OCI Compliance mod, then you can just pull the latest changes.

```
cd steampipe-mod-oci-compliance
git pull
steampipe check oci_compliance.benchmark.cis_v120
```

Here's a sample report in the console.

<div style={{"marginTop":"20px","marginBottom":"20px"}}>
  <img  src="/images/blog/2023-06-cis-oci-1-2/oci-cis-console.png" />
</div>

You can output results to formats including JSON, CSV, HTML, and ASFF, or use [custom output templates](https://steampipe.io/docs/develop/writing-control-output-templates) to create new output formats.

To view the OCI CIS v1.2 benchmark in your browser, run this command in the same cloned repo.

```
steampipe dashboard
```

Then open `http://localhost:9194` in your browser and view the dashboard.

<div style={{"marginTop":"20px","marginBottom":"20px"}}>
<img src="/images/blog/2023-06-cis-oci-1-2/oci-cis-dashboard.png" />
</div>

## We love open source!

Steampipe delivers a full suite of tools to build, execute and share cloud configuration, compliance, and security frameworks using HCL + SQL! We would love your help to expand the open source documentation and control coverage for CIS, PCI, HIPAA, NIST, and more. The best way to get started is to [join our Slack community](https://steampipe.io/community/join); we would love to talk to you!



[Turbot Pipes](http://pipes.turbot.com/) (Formerly known as Steampipe Cloud) is a cloud intelligence and security platform that enables users to leverage SQL-based analysis across all of their cloud services & data sources, gain insights, improve their security posture, and manage their workloads from a single pane. Optimized for scalability, central management, and team collaboration, Turbot Pipes works seamlessly with your Steampipe CLI queries, benchmarks, and dashboards.

Earning and maintaining our customers' trust is of the utmost importance to us at Turbot. Our operations, product, and engineering teams are dedicated to ensuring that our products are designed, architected, and developed with relentless focus on the security of our products and of our customers' data.  We invest heavily in security, and today we are excited to share that Turbot has once again obtained our annual SOC 2 Type II compliance for [Turbot Pipes](http://pipes.turbot.com/).

We have always delivered best-in-class security. This year, we have continued our security awareness training for all employees, maintained our [CIS Benchmark Certification](https://turbot.com/blog/2020/05/cis-securesuite-member), successfully passed multiple Well-Architected audits for our cloud partnerships and remained steadfast in our commitment to our [privacy policy](https://turbot.com/legal/privacy) covering GDPR, CCPA, and Privacy Shield requirements.

Achieving this certification once more affirms our commitment to meet and exceed our customers' expectations in data security, privacy, and compliance. Our annual SOC audits are conducted each April. We make the audit reports available each May, and our full SOC 2 Type II audit report is available to customers and prospects under NDA upon request. More information about our security practices can be found on our [Turbot Security](https://turbot.com/security) page.

We invite you to try Turbot Pipes and share your experiences with us in the #cloud channel of [our Slack community](/community/join)!


Steampipe Hacktoberfest 2022 Recap

Snapshot, share, and schedule queries in Turbot Pipes

Schedule snapshots of benchmarks and dashboards in Turbot Pipes

Contributor Spotlight: Graza Andersson

Contributor Spotlight: Ellis Valentiner

Contributor Spotlight: François de Metz