
### Pipes, Assemble!

Just like a certain heroic team coming together to defend the world, [Steampipe](https://steampipe.io), [Powerpipe](https://powerpipe.io), and [Flowpipe](https://flowpipe.io) are now united under the banner of **Pipes**. With Flowpipe officially becoming a [first-class citizen](/blog/2024/10/flowpipe-in-pipes) in the Pipes platform, your cloud tools have never been more powerful. Ready to join forces in your Pipes workspace, Flowpipe now works seamlessly alongside Steampipe and Powerpipe. Assemble your dream team of tools and get started with a single click - whether it’s your first Steampipe [query](https://turbot.com/pipes/docs/using/steampipe/query), Powerpipe [dashboard](https://turbot.com/pipes/docs/using/powerpipe/dashboards), or Flowpipe [pipeline](https://turbot.com/pipes/docs/using/flowpipe/pipelines).

### The Superpowers of Pipes

In the battle for cloud intelligence and security, Pipes delivers more than just tools - it gives you superpowers. Each component in Pipes has its own strengths, but they’re even more powerful together. While platforms like Splunk fight the good fight as monolithic titans, Pipes takes a different approach: empowering you to work with standalone tools or as a fully integrated, cloud-based system. Pipes is available in the cloud or right in your [terminal](https://turbot.com/pipes/docs/connect) - giving developers the flexibility to fight their cloud challenges on their terms. Assemble your team, build your solutions, and level up from individual developer efforts to full-blown organizational strength. 

### Seamless Collaboration Between Heroes

The key to any superhero team is seamless collaboration, and Pipes is no exception. When you use Steampipe, Powerpipe, or Flowpipe, it’s like watching your favorite heroes in action. Take, for example, a shared AWS [connection](https://turbot.com/pipes/docs/using/steampipe/connections) in Pipes. Once it’s set up, every tool in the Pipes suite can use it effortlessly. Steampipe can harness that connection for its queries, Powerpipe for its benchmarks and dashboards, and Flowpipe for enforcing security compliance. It’s all about using the right hero for the right job - no more setting up connections multiple times!

![](/images/blog/2024-10-pipes-assemble/pipes-assemble.png)

### Custom Mods: Your Secret Weapon

Every hero has a secret weapon, and in Pipes, custom mods are yours. For [Launch Week 5](https://turbot.com/launch-week/05), we introduced a [GitHub integration](https://turbot.com/pipes/blog/2024/05/github-custom-mods) that deploys [Powerpipe mods](https://turbot.com/pipes/docs/using/powerpipe/mods) from a branch or tag in a private repo. Now, with Flowpipe integrated, it works the same way for your custom [Flowpipe mods](https://turbot.com/pipes/docs/using/flowpipe/mods). Pipes automatically refreshes mods when changes are detected, ensuring your tools are always powered up and ready for action.

### One Unified Team, One Mission

Pipes isn’t just about powerful tools - it’s about teamwork. Every piece of Pipes speaks the same language: open source, as-code, and developer-friendly. Whether you’re writing SQL queries, building dashboards, or designing pipelines in HCL, it’s all part of the same mission. Together, Steampipe, Powerpipe, and Flowpipe give you everything you need to tackle cloud governance, security, and more. 

### What’s Next for the Team?

The Pipes team is just getting started. With Flowpipe now part of the action, the Pipes universe has room to grow, and we’re excited to see what’s next. More components? New powers? Who knows! But one thing’s for sure — Pipes is here to help you assemble the ultimate team for cloud security and intelligence. Ready to suit up? Take Pipes for a spin and [let us know](https://turbot.com/community/join) how it goes!



[Flowpipe](https://flowpipe.io) is the open-source solution for DevOps and security pros who use workflows-as-code to automate cloud operations. Today we're thrilled to announce that Flowpipe joins its open-source siblings [Steampipe](https://steampipe.io) and [Powerpipe](https://powerpipe.io) as a [component](/pipes/blog/2024/10/pipes-assemble) of the Pipes platform. Flowpipe features will gradually be rolling out across all Pipes users in the coming days.

To see it in action, we'll use an [AWS Thrifty](https://hub.flowpipe.io/mods/turbot/aws_thrifty/pipelines/aws_thrifty.pipeline.detect_and_correct_s3_buckets_without_lifecycle_policy) pipeline to remediate buckets that don't have a lifecycle policy. Let's see how that works in Pipes.

## An AWS-like collection of services

The first thing you'll notice, in a new Pipes workspace, is that all three core components — Steampipe, Powerpipe, and Flowpipe — are right there ready to use. In this case our job requires Flowpipe so that's the tool of choice. 

![new workspace with 3 components](/images/blog/2024-10-flowpipe-in-pipes/three-components.png)

## Easy mod installation

The Thrifty mod is available on the hub, but Pipes brings that interface into your workspace and enables point-and-click installation. 

![mod installation](/images/blog/2024-10-flowpipe-in-pipes/mod-install.png)

## Form-based configuration

You configure Flowpipe mods with variables which, when running the standalone tool, you edit by hand. Pipes automagically turns those variables into forms. In this case, we won't need to change any of the variables. The pipeline will ask for a decision about whether to remediate each non-compliant bucket, and we want that decision to happen in Slack, but that's the default notifier for this workspace so we are ready to go. The pipeline also needs a Steampipe AWS connection, in order to query for non-compliant buckets, but that comes for free as well. The organization that owns this workspace has set up the connection and permitted this workspace to use it. So we can just select the **Run Pipeline** button.

![pipeline ready to run](/images/blog/2024-10-flowpipe-in-pipes/run-pipeline.png)

## Slack-based approval

Here's a non-compliant bucket found by the pipeline. We've routed the approval decision to Slack, but could have used email instead, or MSTeams. We want this bucket to have a lifecycle policy so we choose `Apply lifecycle configuration`. What if the approver doesn't see the message right away? No problem, Pipes suspends the workflow so you won't be billed for the wait, and restarts it when approver replies.

![slack  approval](/images/blog/2024-10-flowpipe-in-pipes/slack-approval.png)

## Hands-free remediation

Flowpipe receives the reply, takes the action, and reports the outcome. Of course it can get old making decisions like this on a per-bucket basis. If you want all existing and new buckets to block public access, you can run the pipeline in a hands-free way, on a schedule. To do that you configure a couple of variables. For this Flowpipe needs to run in server mode, but Pipes takes care of that for you.


## Easy Slack integration

The default [notifier](https://flowpipe.io/docs/reference/config-files/notifier#notifiers) for this workspace is Slack. The notifier in turn refers to an [integration](https://flowpipe.io/docs/reference/config-files/integration#integration) that was already set up in the organization and available to this workspace. How did that happen? Let's create a new Slack integration to see how easy that is. Here are the types of integrations we can create.

![integrations screen](/images/blog/2024-10-flowpipe-in-pipes/integrations-available.png)

We've already seen the [AWS](/pipes/blog/2024/07/aws-org-sync), [Azure](/pipes/blog/2024/07/azure-tenant-sync), [GCP](/pipes/blog/2024/07/gcp-org-sync) integrations, which populate your Pipes organization or tenant with accounts, subscriptions, and projects from those platforms. And we've also seen the [GitHub](/pipes/blog/2024/05/github-custom-mods) integration, which uses Pipes' 3rd-party GitHub app to install mods into a workspace and automatically refresh them when changes are pushed to a branch or tag. The [Slack integration](https://turbot.com/pipes/docs/integrations/slack) works like the [GitHub integration](https://turbot.com/pipes/docs/integrations/github). You name it, install into your Slack instance as a 3rd-party Slack app, and pick the Slack channel where you want notifications sent. 

![slack app](/images/blog/2024-10-flowpipe-in-pipes/slack-app.png)

## Custom mods

The [Flowpipe Hub](https://hub.flowpipe.io) provides a growing collection of libraries and samples that capture a variety of common DevOps patterns, but everyone's situation is unique. It's often useful to write your own mods to suit your particular needs. How to install them? You may already have used the [GitHub](/pipes/blog/2024/05/github-custom-mods) to install Powerpipe mods into a workspace, and automatically refresh them when changes are pushed to a branch or tag. The same integration works exactly the same way for Flowpipe mods.

## Next-level workflows as code

As noted in [Pipes Assemble!](/pipes/blog/2024/10/pipes-assemble), Pipes is a platform on which Steampipe, Powerpipe, and now Flowpipe share common patterns and infrastructure. You can build powerful solutions with the standalone components, but Pipes brings them together smoothly, in a way that's easy to manage across your organization or tenant. If you're a Pipes user who hasn't yet tried Flowpipe, fire it up in a workspace and [let us know](https://turbot.com/community/join) how it goes.


## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/lwATElOgJ_4"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Run Flowpipe Workflows in Turbot Pipes"
>
</iframe>
</div>




You asked, we delivered. Now you can import your entire AWS organization, with just a few clicks, as a set of Pipes connections. This new integration dramatically simplifies connecting AWS resources to Pipes, and enables you to query and analyze your cloud infrastructure more thoroughly than ever before. 

## Get ready to import

Here's the AWS structure we'll import.

<div style={{width:"70%"}}>
  <img alt="aws org structure" src="/images/blog/2024-07-aws-org-sync/aws-org-structure.png" />
</div>


To create a new [AWS Integration](https://turbot.com/pipes/docs/integrations/aws), select `AWS` from the integration options.

<div>
  <img alt="select aws integration" src="/images/blog/2024-07-aws-org-sync/select-aws-integration.png" />
</div>

You can accept the default handle for the integration, `AWS`, or choose your own.

In `Discovery Settings`, you set up the authentication Pipes uses to discover your AWS OUs and accounts. You can use a cross-account role (recommended) or an access key pair; either way, test the connection to verify it works.

<div>
  <img alt="discovery settings" src="/images/blog/2024-07-aws-org-sync/aws-discovery-settings.png" />
</div>

In `Connection Settings`, you set up the trust relationship for discovered accounts. For this you'll need a cross-account role in each discovered account; Pipes can generate a CloudFormation StackSet to help automate the process. See [the docs](https://turbot.com/pipes/docs/integrations/aws) for details. Again, test to verify it works.

<div>
  <img alt="connection settings" src="/images/blog/2024-07-aws-org-sync/aws-connection-settings.png" />
</div>

## Set permissions

Now choose which workspaces — all, some, or none — are permitted to use the connections.

<div>
  <img alt="set permissions" src="/images/blog/2024-07-aws-org-sync/aws-set-permissions.png" />
</div>

Note the separation of concerns: Pipes administrators make connections available to workspaces, and workspace owners decide, on the workspace's `Advanced / Connections` page, which (if any) to activate for query.

## Import OUs and accounts

Now click `Create Integration` to import the discovered OUs and accounts. Pipes performs the import, and creates a tree that matches your AWS structure, so delegation of permissions works the same way in Pipes as it does in AWS.

<div>
  <img alt="imported tree" src="/images/blog/2024-07-aws-org-sync/aws-imported-tree.png" />
</div>

## Add connections to a workspace

Subject to the connections they are permitted to see, workspace owners can now use any OU that was imported into Pipes to add its corresponding set of connections to a workspace. In this example, we choose the `Pipeling Scale Testing` connection folder which contains 200 accounts. 

The workspace owner uses the `Add to schema` button to make the corresponding connections available for query.

<div>
  <img alt="add to schema" src="/images/blog/2024-07-aws-org-sync/aws-add-to-schema.png" />
</div>

When a Pipes connection folder contains two or more connections, Pipes automatically creates an aggregator for them. Here's `all_aws` which aggregates the 200 connections acquired from the `Pipeling Scale Testing` OU.

<div>
  <img alt="aws all" src="/images/blog/2024-07-aws-org-sync/aws-all.png" />
</div>

## Query the connections that were imported and added

Now, in the query pane, you can select and query any of the aggregated connections, or use the aggregator to query all of them. Here we query the aggregator.

<div>
  <img alt="query" src="/images/blog/2024-07-aws-org-sync/aws-query.png" />
</div>

And that's it! We've gone from a standing start to a queryable set of imported AWS connections in just minutes.

## Try it yourself!

This new AWS integration addresses a common challenge faced by our users with large AWS footprints. It would be time-consuming to do this by hand or with Terraform scripting. Now you can bring your whole AWS organization into Pipes quickly, easily, and automatically. And when OUs or accounts change, Pipes syncs the changes.

We're thrilled to offer this enhanced AWS support and can't wait to see how it accelerates your cloud operations and security workflows. As always, we welcome [your feedback](https://turbot.com/community/join) and we're to help as you explore this powerful new capability.


## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/923ssGdgIJ8"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Connect your AWS Organizations with Pipes"
>
</iframe>
</div>





You asked, we delivered. Now you can import your entire Azure tenant, with just a few clicks, as a set of Pipes connections. This new integration dramatically simplifies connecting Azure resources to Pipes, and enables you to query and analyze your cloud infrastructure more thoroughly than ever before. 

## Get ready to import

Here's the Azure structure we'll import.

<div>
  <img alt="azure tenant structure" src="/images/blog/2024-07-azure-tenant-sync/azure-tenant-structure.png" />
</div>


To create a new [Azure Integration](https://turbot.com/pipes/docs/integrations/azure), select `Azure` from the integration options.

<div>
  <img alt="select azure integration" src="/images/blog/2024-07-azure-tenant-sync/select-azure-integration.png" />
</div>

In `Setup Integration`, you can accept the default handle for the integration, `Azure`, or choose your own.

To authenticate, specify your environment (e.g. `Public Cloud`) and enter credentials for a role with at least `Global Reader` access. Then test to verify they work. 

<div>
  <img alt="setup integration" src="/images/blog/2024-07-azure-tenant-sync/setup-azure-integration.png" />
</div>

## Set permissions

Now choose which workspaces — all, some, or none — are permitted to use the connections.

<div>
  <img alt="set permissions" src="/images/blog/2024-07-aws-org-sync/aws-set-permissions.png" />
</div>

Note the separation of concerns: Pipes administrators make connections available to workspaces, and workspace owners decide, on the workspace's `Advanced / Connections` page, which (if any) to activate for query.

## Import management groups and subscriptions

Now click `Create Integration` to import the discovered management groups and subscriptions. Pipes performs the import, and creates a tree that matches your Azure structure, so delegation of permissions works the same way in Pipes as it does in Azure.

<div>
  <img alt="imported tree" src="/images/blog/2024-07-azure-tenant-sync/azure-imported-tree.png" />
</div>

## Add connections to a workspace

Subject to the connections they are permitted to see, workspace owners can now use any management group that was imported into Pipes to add its corresponding set of connections to a workspace. In this example, we choose the `Pipeling Scale Testing Small` connection folder which contains 60 subscriptions. 

The workspace owner uses the `Add to schema` button to make the corresponding connections available for query.

<div>
  <img alt="add to schema" src="/images/blog/2024-07-azure-tenant-sync/azure-add-to-schema.png" />
</div>

When a Pipes connection folder contains two or more connections, Pipes automatically creates an aggregator for them. Here's `all_azure` which aggregates the 60 subscriptions acquired from the `Pipeling Scale Testing` management group.

<div>
  <img alt="azure all" src="/images/blog/2024-07-azure-tenant-sync/all-azure.png" />
</div>

## Query the connections that were imported and added

Now, in the query pane, you can select and query any of the aggregated connections, or use the aggregator to query all of them. Here we query the aggregator.

<div>
  <img alt="query" src="/images/blog/2024-07-azure-tenant-sync/azure-query.png" />
</div>

And that's it! We've gone from a standing start to a queryable set of imported Azure connections in just minutes.

## Try it yourself!

This new Azure integration addresses a common challenge faced by our users with large Azure footprints. It would be time-consuming to do this by hand or with Terraform scripting. Now you can bring your whole Azure organization into Pipes quickly, easily, and automatically. And when management groups or subscriptions change, Pipes syncs the changes.

We're thrilled to offer this enhanced Azure support and can't wait to see how it accelerates your cloud operations and security workflows. As always, we welcome [your feedback](https://turbot.com/community/join) and we're to help as you explore this powerful new capability.



## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/K84Ow7wvK1s"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Connect your Azure Tenants with Pipes"
>
</iframe>
</div>




You asked, we delivered. Now you can import your entire GCP organization, with just a few clicks, as a set of Pipes connections. This new integration dramatically simplifies connecting GCP resources to Pipes, and enables you to query and analyze your cloud infrastructure more thoroughly than ever before. 

## Get ready to import

Here's the GCP structure we'll import.

<div>
  <img alt="gcp org structure" src="/images/blog/2024-07-gcp-org-sync/gcp-org-structure.png" />
</div>


To create a new [GCP Integration](https://turbot.com/pipes/docs/integrations/gcp), select `GCP` from the integration options.

<div>
  <img alt="select gcp integration" src="/images/blog/2024-07-gcp-org-sync/select-gcp-integration.png" />
</div>


In `Setup Integration`, you can accept the default handle for the integration, `GCP`, or choose your own.

To authenticate, drop the key file for a service account with `Viewer` permissions for the folder hierarchy you'll import. Then test to verify the credentials work. 

<div>
  <img alt="setup integration" src="/images/blog/2024-07-gcp-org-sync/setup-gcp-integration.png" />
</div>


## Set permissions

Now choose which workspaces — all, some, or none — are permitted to use the connections.

<div>
  <img alt="set permissions" src="/images/blog/2024-07-gcp-org-sync/gcp-set-permissions.png" />
</div>

Note the separation of concerns: Pipes administrators make connections available to workspaces, and workspace owners decide, on the workspace's `Advanced / Connections` page, which (if any) to activate for query.

## Import folders and projects

Now click `Create Integration` to import the discovered folders and projects. Pipes performs the import, and creates a tree that matches your GCP structure, so delegation of permissions works the same way in Pipes as it does in GCP.

<div>
  <img alt="imported tree" src="/images/blog/2024-07-gcp-org-sync/gcp-imported-tree.png" />
</div>

## Add connections to a workspace

Subject to the connections they are permitted to see, workspace owners can now use any folder that was imported into Pipes to add its corresponding set of connections to a workspace. In this example, we choose the `Pipeling Scale Testing Small` folder which contains 15 projects. 

The workspace owner uses the `Add to schema` button to make the corresponding connections available for query.

<div>
  <img alt="add to schema" src="/images/blog/2024-07-gcp-org-sync/gcp-add-to-schema.png" />
</div>

When a Pipes connection folder contains two or more connections, Pipes automatically creates an aggregator for them. Here's `all_gcp` which aggregates the 15 connections acquired from the `Pipeling Scale Testing Small` GCP folder.

<div>
  <img alt="gcp all" src="/images/blog/2024-07-gcp-org-sync/gcp-all.png" />
</div>

## Query the connections that were imported and added

Now, in the query pane, you can select and query any of the aggregated connections, or use the aggregator to query all of them. Here we query the aggregator.

<div>
  <img alt="query" src="/images/blog/2024-07-gcp-org-sync/gcp-query.png" />
</div>

And that's it! We've gone from a standing start to a queryable set of imported GCP connections in just minutes.

## Try it yourself!

This new GCP integration addresses a common challenge faced by our users with large GCP footprints. It would be time-consuming to do this by hand or with Terraform scripting. Now you can bring your whole GCP organization into Pipes quickly, easily, and automatically. And when folders or projects change, Pipes syncs the changes.

We're thrilled to offer this enhanced GCP support and can't wait to see how it accelerates your cloud operations and security workflows. As always, we welcome [your feedback](https://turbot.com/community/join) and we're to help as you explore this powerful new capability.


## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/mRk5iBe5MTA"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Connect your GCP Organizations with Pipes"
>
</iframe>
</div>





Pipes has long enabled you to install and use custom mods but you had to source them from public GitHub repositories, and it was a manual process to install and then update your mods. Now, with our new GitHub integration, you can install a custom mod from a public or private repo.

You can do this as an individual identity, as the owner of a Pipes [organization](https://turbot.com/pipes/docs/organizations) or — as a Pipes Enterprise user — in a Pipes [tenant](https://turbot.com/pipes/docs/tenants). In all these scenarios, you install your custom mod from a GitHub repo that you choose from a GitHub app. And when you push changes, Pipes automatically updates the mod.

To illustrate, we'll use a simple dashboard that counts S3 buckets with versioning enabled or disabled. As a  custom mod developer, you always want to test locally using Powerpipe to make sure the mod runs correctly. Then push to GitHub and create a GitHub integration in Pipes to gain access to it. 

## Create an integration

From the `Settings` page of your Pipes organization or tenant, click `New Integration`, choose `GitHub`, name the integration, and click `Create Integration`.

<div style={{width:"90%"}}>
  <img alt="new integration" src="/images/blog/2024-05-github-custom-mods/new-integration.png" />
</div>

## Choose your repo

You're redirected to GitHub, where you choose an identity and then pick the repos that Pipes can see. 

<div style={{width:"70%"}}>
  <img alt="choosing a repo for a custom mod" src="/images/blog/2024-05-github-custom-mods/authorize-the-repo.png" />
</div>


Typically you'll grant access to `All`, here we'll be more specific and just grant access to one  demo repo. 

## Install the mod

After installing the GitHub app with that configuration, you're redirected back to Pipes where you can now use the integration to install a mod from GitHub. Here we're installing to a workspace called `demo` but, because the integration was created in a Pipes organization, any workspace can use it.

<div>
  <img alt="install from main" src="/images/blog/2024-05-github-custom-mods/install-mod-from-branch.png" />
</div>

The picker only shows the one authorized repo. You can install from a branch, defaulting to main,  or from a tag that defines a semantic versioning constraint.

## View the dashboard

Once it's installed, we can view that dashboard in Pipes.

<div style={{width:"60%"}}>
  <img alt="dashboard as bar chart" src="/images/blog/2024-05-github-custom-mods/bucket-versioning-1.png" />
</div>

Now here's the really fun part: let's push a change.

## Update the mod

We'll change the bar chart to a pie chart.

<div style={{width:"60%"}}>
  <img alt="change bar to pie chart" src="/images/blog/2024-05-github-custom-mods/git-diff.png" />
</div>

Commit the change, and push it.

Pipes detects the change, updates the mod, and immediately refreshes the dashboard to display the updated chart.

<div style={{width:"60%"}}>
  <img alt="dashboard as bar char" src="/images/blog/2024-05-github-custom-mods/bucket-versioning-2.png" />
</div>

It really is that easy, and that fast, to deploy a change and live-update it!

## Developing for Pipes

Pipes' vast collection of pre-built benchmarks and dashboards will meet many of your needs right out of the box. But if you haven't yet discovered the power of custom mods in Pipes, now's a great time to get started. You'll find that building custom mods, using HCL and SQL, is straightforward and enjoyable. It's a powerful way to tailor data visualizations and benchmarks to your needs. So give it a try, and [let us know](https://turbot.com/community/join) how it goes!

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/a7Op2_LLm7E"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Pipes GitHub integration"
>
</iframe>
</div>





A benchmark like [AWS Compliance](https://hub.powerpipe.io/mods/turbot/aws_compliance) can produce a lot of data. There are dozens of dashboards, with more than a thousand controls, and you might need to run those controls over many AWS accounts. You've always been able to export the data for analysis, but it would be nice to filter and group that data right in the dashboard, and now you can.

One way to simplify things: [change the search path](https://powerpipe.io/docs/run/dashboard/search-path#change-the-search-path) in order to scope the dashboard's queries to one or several connections. 

<div>
  <img  style={{"width":"60%"}} alt="cis v300 with benchmark filter" src="/images/blog/2024-03-pipes-filter-and-group/cis_300_filtered_05.png" />
</div>

But let's say you want to query all your connections and reason over the entire data set. The new `Filter and Group` widgets enable you to create — and share! — customized views of benchmark results.

## Use Filter to constrain the view

Here's the `CIS v300` benchmark from the [AWS Compliance](https://hub.powerpipe.io/mods/turbot/aws_compliance) mod, filtered to show only alarms for `Identity and Access Management`.

<div>
  <img  style={{"width":"100%"}} alt="cis v300 with benchmark filter" src="/images/blog/2024-03-pipes-filter-and-group/cis_300_filtered_02.png" />
</div>

You can also filter on a dimension like `account_id` or `region`.

<div>
  <img style={{"width":"60%"}} alt="cis v300 with account filter" src="/images/blog/2024-03-pipes-filter-and-group/cis_300_filtered_03.png" />
</div>

## Share your custom view

At any point, you can [snapshot](https://powerpipe.io/docs/run/snapshots/interactive-snapshots) the dashboard and share the filtered view you've created. Your filters will be in effect for recipients of your link.

Note, though, that the snapshot preserves all the benchmark data. So people can add or remove their own filters to see things differently.

## Use Group to reorganize the view

Whether or not you use a filter to contrain the view, you can regroup to organize how results are shown. The default hierarchy is `Benchmark / Control / Result`. But here's an alternate view that uses the `account_id` dimension as the primary group.

<div>
  <img style={{"width":"100%"}} alt="cis v300 with account grouping" src="/images/blog/2024-03-pipes-filter-and-group/cis_300_filtered_04.png" />
</div>

Now you can review results by account. Of course you can continue to adjust the filters in this per-account grouping. And here as well you can take a snapshot at any time to share a custom view with your team.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/MleOf6olaCg"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Turbot Pipes filtering and grouping"
>
</iframe>
</div>

## Filter, group, and share your way to compliance clarity

With the new Filter and Group widgets in Turbot Pipes, you can easily create customized views of your benchmark results to focus on what matters most. Constrain the view with filters, reorganize it with groupings, and share insightful snapshots with your team. Pipes empowers you to analyze your compliance posture from every angle. So dive in, explore, and [let us know](https://turbot.com/community/join) how it goes!



[Turbot Pipes](https://turbot.com/pipes) is a cloud intelligence and security platform to gain insights from 80+ cloud services and visualize 700+ ready-to-use interactive dashboards to understand your cloud inventory, navigate resource relationships, evaluate cost optimization recommendations and ensure security posture best practices. With a Pipes always-free [Developer account](https://turbot.com/pipes/docs/plans/developer), you can instantly evaluate your cloud environments without the overhead of an expensive CSPM or CNAPP service.

[AWS CIS benchmarks](https://www.cisecurity.org/benchmark/amazon_web_services) are among the more widely used security benchmarks cloud teams use as guidance to configure their AWS accounts. In the [latest iteration](https://steampipe.io/blog/cis-v30-aws-benchmark), AWS CIS v3.0, includes over [60 recomendations](https://hub.steampipe.io/mods/turbot/aws_compliance/controls/benchmark.cis_v300). Turbot Pipes leverages the Steampipe [AWS Compliance mod](https://hub.steampipe.io/mods/turbot/aws_compliance) which includes the latest CIS benchmark along with 25 other security and compliance control frameworks (e.g. NIST, SOC2, PCI, HIPAA, etc) to evaluate AWS accounts.

It takes only a couple of minutes to evaluate your AWS account for the latest CIS benchmark. Below we will walkthrough how to create your always-free Pipes Developer account, connect your first AWS account, and run your first assessment.

## Run an AWS CIS v3.0 assessment

If you’re new to Pipes, sign up at [https://pipes.turbot.com](https://pipes.turbot.com). Then create your first [workspace](https://turbot.com/pipes/docs/workspaces).

Next, we'll [connect](https://turbot.com/pipes/docs/connections) an AWS account to Turbot Pipes, whether using a cross-account AWS IAM role or using an AWS IAM access key.

<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/docs/connections" target="_blank">
  <img alt="Add a Turbot Pipes connection" src="/images/blog/2024-02-aws-cis-v3-benchmark-dashboard/add-turbot-pipes-connection.png" />
  </a>
</div>

Next, [install](https://turbot.com/pipes/docs/mods) the AWS Compliance mod which contains the AWS CIS v3.0 dashboard. Instant one-click installation:

<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/docs/mods" target="_blank">
  <img alt="Install the Steampipe AWS Compliance mod" src="/images/blog/2024-02-aws-cis-v3-benchmark-dashboard/add-aws-compliance-mod.png" />
  </a>
</div>

Finally, go to the [Dashboard](https://turbot.com/pipes/docs/dashboards) page, select AWS CIS v3.0 from the list of available dashboards from the AWS Compliance section.

<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/docs/dashboards" target="_blank">
  <img alt="View the AWS CIS v3.0 dashboard in Turbot Pipes" src="/images/blog/2024-02-aws-cis-v3-benchmark-dashboard/aws-cis-v3-benchmark-dashboard.png" />
  </a>
</div>

## Share the AWS CIS v3.0.0 assessment with your team

In the free Developer account you can create point-in-time [snapshots](https://turbot.com/pipes/docs/dashboards#saving--sharing-snapshots) of your CIS dashboard and share it via a secure link with your team. 

Capture a one-time snapshot, or create a [schedule](https://turbot.com/pipes/docs/dashboards#scheduling-snapshots) to programmatically snapshot periodically.

<div style={{width:"80%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/docs/dashboards#scheduling-snapshots" target="_blank">
  <img alt="Create a recurring AWS CIS Assessment report in Turbot Pipes" src="/images/blog/2024-02-aws-cis-v3-benchmark-dashboard/aws-cis-v3-weekly-schedule.png" />
  </a>
</div>

Optionally send [summary notifications](https://turbot.com/pipes/blog/schedule-snapshots) with your team via Slack or Microsoft Teams.

<div style={{width:"80%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/blog/schedule-snapshots" target="_blank">
  <img alt="Send AWS CIS Assessment notifications to Slack or Microsoft Teams" src="/images/blog/2024-02-aws-cis-v3-benchmark-dashboard/aws-cis-v3-slack-report.png" />
  </a>
</div>

## Enable cloud intelligence for your organization
Use the always-free Turbot Pipes Developer tier to assess your own AWS accounts. When you are ready to bring your team and expand your usage, sign up for the [Pipes Team tier](https://turbot.com/pipes/pricing) in a 14-day free trial. 

We would love to hear your feedback as you go. Send us a message in the #pipes channel of our [Slack community](https://turbot.com/community/join).



For a prior [Hacktoberfest](https://steampipe.io/blog/hacktoberfest-2022) we needed to add the tag `hacktoberfest` to over hundred repos. The [tool](https://github.com/turbot/steampipe-samples/tree/main/all/github-add-or-remove-topic) we wrote to do that uses Turbot Pipes to build a list of the names of those repos, then uses the native Github API (by way of Go's [github.com/google/go-github](https://github.com/google/go-github)) to add a topic to each repo. 

Why use Turbot Pipes? Obviously we're biased, but it wasn't a gratuitous choice. As we'll see, Turbot Pipes isn't just a convenient way to query an API with SQL. Turbot Pipes uses Steampipe plugins which represent APIs in ways that dramatically simplify their use.

Our starting point was an existing tool, [hacktoberfest-repo-topic-apply](https://github.com/Hacktoberfest/hacktoberfest-repo-topic-apply/blob/main/main.go), which showed us how to create a native GitHub client in Go and use it to assign topics to repos. We could have adapted that tool's native-API-based method of listing repositories, then filtered the results to just those matching the pattern `turbot/steampipe-(plugin|mod)`. With Turbot Pipes that's a trivial task.


```sql
select 
  count(*) 
from 
  github_my_repository
448

select 
  count(*) 
from 
  github_my_repository 
where 
  full_name ~ 'turbot'
186

select 
  count(*) 
from 
  github_my_repository
where 
  full_name ~ 'turbot/steampipe-(plugin|mod)'
135

select 
  full_name
from 
  github_my_repository
where 
  full_name ~ 'turbot/steampipe-(plugin|mod)'
order by
  full_name
turbot/steampipe-mod-alicloud-compliance
turbot/steampipe-mod-alicloud-insights   
turbot/steampipe-mod-alicloud-thrifty    
...
```

It's harder to do this with the native APIs which handle personal and organizational repos differently. Turbot Pipes hides that underlying complexity to make things easy.


## Tables that consolidate views of resources from the perspective of an authenticated user

Note that we're using the table [github_my_repository](https://hub.steampipe.io/plugins/turbot/github/tables/github_my_repository). Per the documentation:

<blockquote class="docs">
You can own repositories individually, or you can share ownership of repositories with other people in an organization. The `github_my_repository` table will list repos you own, you collaborate on, or that belong to your organizations.

To query ANY repository, including public repos, use the `github_repository` table.
</blockquote>


This is a powerful pattern that's also implemented elsewhere. The [Microsoft 365](https://hub.steampipe.io/plugins/turbot/microsoft365) plugin provides both `microsoft_my_mail_message` and `microsoft_mail_message`; the [Google Workspace](https://hub.steampipe.io/plugins/turbot/googleworkspace) plugin provides `googleworkspace_my_gmail_message` and `googleworkspace_gmail`; other plugins work the same way. Where possible, Turbot Pipes consolidates views of resources from the perspective of an authenticated user. 


As an authenticated GitHub user, my API token grants me access to my own personal repos as well as those of a half-dozen GitHub organizations I belong to. A native API query across all of them, using `github.com/google/go-github`, would require me to use `Repositories.List` (configured by `RepositoriesListOptions`) as well as `Repositories.ListByOrg` (configured by `RepositoriesListByOrgOptions`, then combine the results. That's doable but not trivial. If you just want a list of names, you'd rather not have to learn those details. Using `github_my_repository` hides them so you can focus on the job you're doing with those API results.

## Using Turbot Pipes in a Go program

We could have written our tool in any language but, since we'd started with `hacktoberfest-repo-topic-apply`, we decided to continue with Go. At its core, the tool loops through an array of repo names and calls a function to assign a topic to each repo.


```go
for i, repoName := range repos {
	addTopic(ctx, client, ownerLogin, repoName, topic)
}
```

Our tool (see [steampipe-samples](https://github.com/turbot/steampipe-samples/tree/main/all/github-add-or-remove-topic)) includes two functions that return a list of names as a comma-separated string; the tool splits that string to produce the array. 

```go
names = queryPostgresForRepos()
// names = querySpcForRepos()

var repos = strings.Split(names, ",")
```

The two functions use the same SQL query, and they produce the same list of names. We show both to illustrate two complementary ways you can use Turbot Pipes in a program that's written in Go -- or in any other language such as [Python](/pipes/docs/connect/python), [JavaScript](/pipes/docs/connect/javascript), etc.

1. Use the language's database client to query Postgres from [Turbot Pipes](/pipes/docs/connect).

2. Use the language's HTTP client to [ship your query to Turbot Pipes Query API](/pipes/docs/develop/query-api) and receive a JSON response.

Here's the SQL query that's common to both functions.

```
var query = `
	with names as (
		select
			replace(name_with_owner, 'turbot/', '') as name -- replace with your ownerLogin
		from
			github_my_repository
		where
			name_with_owner ~ 'turbot/steampipe-(plugin|mod)' -- replace with your ownerLogin and a pattern
		order by
			name_with_owner
		)
		select
			array_to_string(array_agg(name), ',') as names 
		from
			names	
```

If we run the query in the Turbot Pipes console [interactive query editor](/pipes/docs/queries), the output looks like this.

```
steampipe-mod-alicloud-compliance,steampipe-mod-alicloud-insights,...
```

Let's look at how the two functions acquire that output.

## Using a database connection

For this approach, we used Go's `database/sql` package which (we learned) also requires that you import `github.com/lib/pq` for Postgres. In this example we're querying a local instance of Steampipe but could as easily query an instance running in a Turbot Pipes workspace.


```go
func queryPostgresForRepos() string {
    connStr := "postgresql://judell:f3**-****-**2c@acme-jon.usea1.db.pipes.turbot.com:9193/dea4px"
    db, err := sql.Open("postgres", psqlconn)
	  CheckError(err)
    rows, _ := db.Query(query)
    defer rows.Close()
	var names string
	for rows.Next() {
		err = rows.Scan(&names)
		CheckError(err)
	}
	return names
}

```

## Using a REST call

Here's the second method, using Turbot Pipes' [query API](/pipes/docs/develop/query-api).


```go
func querySpcForRepos() string {
	type Result struct {
		Items []struct {
			Names string `json:"names"`
		} `json:"items"`
	}
	uri := fmt.Sprintf("https://pipes.turbot.com/api/latest/org/acme/workspace/jon/query?sql=%s", url.PathEscape(query)) //-- use your handle and workspace
	req, err := http.NewRequest("GET", uri, nil)
	CheckError(err)
	req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", os.Getenv("PIPES_TOKEN")))
	resp, err := http.DefaultClient.Do(req)
	CheckError(err)
	defer resp.Body.Close()
	decoder := json.NewDecoder(resp.Body)
	var result Result
	err = decoder.Decode(&result)
	CheckError(err)
	return result.Items[0].Names
}
```

Go's static typing requires us to define a struct into which to decode the JSON response. In a dynamically-typed language like Python that wouldn't be necessary. 

The REST call uses a bearer token that's available in your Turbot Pipes profile -- e.g `https://pipes.turbot.com/user/judell/setting/token`.

If we use `curl` to make that REST call:

```bash
curl -H "Authorization: Bearer ${PIPES_TOKEN}"  \
  -d sql="with names as ( select replace(name_with_owner, 'turbot/', '') as name from github_my_repository where name_with_owner ~ 'turbot/steampipe-(plugin|mod)' order by name_with_owner ) select array_to_string(array_agg(name), ',') as names from names"  \
 https://pipes.turbot.com/api/latest/org/acme/workspace/jon/query
```

Here's the result:


```json
{"items":[{"names":"steampipe-mod-alicloud-compliance,steampipe-mod-alicloud-insights, ... steampipe-plugin-zendesk,steampipe-plugin-zoom"}]}
```

The function decodes that into the `Result` struct and returns the `Names` property of the zeroth element of `Items`.


## Turbot Pipes as a software component

Turbot Pipes enables you to treat infrastructure, SaaS apps, and custom data like a database. It gives you the power of SQL to unify, analyze, and take action on your environment. This makes it possible to be a software component that you can call from any programming language. 

Why do that? As we've seen, Steampipe's plugins can abstract native APIs in a way that makes them simpler to use than the constructs provided by SDKs like `github.com/google/go-github`. The Turbot Pipes leveaging Steampipe plugins doesn't just provide a SQL interface to an API, it organizes that interface to make you maximally productive. And it does that using patterns, like the `my` pattern, that you can learn once and then apply across many APIs.

If you decide that Turbot Pipes is the right tool for the job, which of the two methods we've shown should you use? It's your choice! If you're more comfortable using a database connection, do that. If you'd rather use the Turbot Pipes Query API, do that. Either way, you'll get the job done more easily than if you had to use native APIs directly. Give it a try, and [let us know](https://turbot.com/community/join) how it goes!


We are excited to announce the launch of our new Enterprise Tier, a comprehensive set of features designed to meet the needs of enterprise customers using [Turbot Pipes](https://turbot.com/pipes). 

Turbot Pipes is a cloud intelligence and security platform tailored for DevOps. It offers Zero-ETL for streamlined access to cloud data, alongside dashboards-as-code capabilities for visualizing insights and security posture. Our SOC2 Type II certified platform enables collaboration among teams to share data-driven insights, allowing members to leverage connections, send report snapshots, and communicate query results to provide the necessary context for informed decision-making.

## Key Features of Turbot Pipes Enterprise
Turbot Pipes Enterprise elevates the standard of cloud intelligence and security for enterprises by building on the Team tier’s features. It offers enhanced collaboration, enterprise-grade security, and improved scalability, making it ideal for larger organizations:
- **Organization-wide cloud intelligence & security:** Enables tailored data management across business units and teams for sharing insights.
- **SAML Authentication:** Provides secure and seamless SSO user experience using your identity provider.
- **Multi-Organization RBAC:** Allows granular access permissions across organizations and workspaces to protect sensitive data.
- **Trusted Login Domains:** Significantly reduces unauthorized access by restricting logins to trusted domains.
- **Consolidated Usage and Billing:** Simplifies resource and financial tracking with tenant-level visibility plus per organization/workspace details.

## Transparent, Scalable Pricing
Turbot Pipes is free for Developers, allowing exploration and personal use at no cost. The Team tier elevates team productivity and coordination. The Enterprise tier provides organization-wide security starting at $249 per month, includes three users, 10,000 compute minutes, and 100GB of storage. Scale up as needed, adding more users and paying only for the additional compute and storage you require, with the flexibility to cancel anytime.
<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/pricing" target="_blank">
 <img alt="Turbot Pipes Pricing 2023" src="/images/blog/2023-12-turbot-pipes-enterprise/turbot_pipes_enterprise_pricing.png" />
  </a>
  </div>

For additional information, please visit our [Pricing](https://turbot.com/pipes/pricing) page.

## Get Started with a 14-day Free Trial
Getting started with Turbot Pipes is simple. Sign up for the Developer free tier at [pipes.turbot.com](https://pipes.turbot.com). When you're ready to explore further, unlock more features by upgrading to the Team or Enterprise tier starting with a 14-day free trial.

Have questions? Connect with us and your peers in the **#pipes** channel of our [Slack community](https://turbot.com/community/join).


We're thrilled to announce the general availability of [Turbot Pipes](https://turbot.com/pipes)! Introducing a free Developer tier for individuals and a usage-based Team tier crafted for DevOps teams. Turbot Pipes is poised to revolutionize the way your team manages, secures, and collaborates on cloud services, acting as an invaluable ally for every DevOps team to empower them with data-driven insights essential for informed decision-making.

## Journey of Turbot Pipes
Developers love our open-source [Steampipe CLI](https://steampipe.io). Pipes enhances Steampipe as a centralized service with additional security, scalability, & collaboration features making it easier to adopt as a team. 

Since Pipes was quietly introduced through our docs, it has garnered adoption from over 1,500 users, ranging from hobbyists to Fortune 500 companies. It’s used for diverse applications, including [security assessments](https://turbot.com/pipes/blog/2022/02/pipes-for-security-posture-assessment) automation and as an integration point to power [GitOps pipelines](https://steampipe.io/blog/github-actions). Managed Service Providers and software firms have integrated Pipes into their systems, drawn by its Zero-ETL architecture to expand their breadth of service coverage. 

We've secured our [SOC2 Type II](https://turbot.com/pipes/blog/2023/05/soc2-2023) certification, introduced more [integration points](https://turbot.com/pipes/blog/2023/10/zapier-launch), expanded support for new [Steampipe plugins](https://hub.steampipe.io/plugins), and enhanced [data synchronization and reporting performance](https://turbot.com/pipes/blog/2023/10/zapier-launch).

## Cloud intelligence & security: crafted for DevOps
Turbot Pipes extends Steampipe’s capabilities, introducing features pivotal for team collaboration, workflow integration, enhanced security, and scalability:

- **Effortless Zero-ETL:** Serve as the central hub for all your cloud data, providing seamless integrations without the hassles of traditional ETL or database management overhead.
- **Robust Security & Operations:** Manage credentials securely with role-based access controls, conduct comprehensive security assessments of your cloud environments, and pinpoint opportunities for operational optimizations.
- **Enhanced Collaboration:** Facilitate sharing of data-driven insights, allowing team members to share connections, send report snapshots, and pipe query results amongst team members giving them the necessary context they need for informed decision-making.
- **Optimized Performance:** With persisted data storage, experience accelerated queries and reporting that not only empower your team to work faster but also streamline your workflows efficiently.

## Transparent pricing: tailored for DevOps
Turbot Pipes is free for Developers, allowing exploration and personal use at no cost. The Team tier, commencing at $49 per month, includes three users, 2,000 compute minutes, and 20GB of storage. Scale up as needed, adding more users and paying only for the additional compute and storage you require, with the flexibility to cancel anytime.

<div style={{width:"100%", textAlign:"center"}}>
  <a href="https://turbot.com/pipes/pricing" target="_blank">
 <img alt="Turbot Pipes Pricing 2023" src="/images/blog/2023-10-announcing-turbot-pipes/pipes-pricing.png" />
  </a>
  </div>

For additional information, please visit our [Pricing](https://turbot.com/pipes/pricing) page. Current preview users will enjoy an extended transition period with increased usage limits; more information is in the Pricing [FAQ](https://turbot.com/pipes/pricing#i-am-on-the-free-preview-of-turbot-pipes-what-happens-now).

## Easy to start, easier to love
Getting started with Turbot Pipes is simple. Sign up for the Developer free tier at [pipes.turbot.com](https://pipes.turbot.com). When you're ready to explore further, upgrade to the Team tier starting with a 14-day free trial.

Join the conversation with peers in the **#pipes** channel of our [Slack community](https://turbot.com/community/join). Share your questions, discuss use cases, and let’s innovate together!



The [Turbot Pipes app for Zapier](https://zapier.com/apps/turbot-pipes), announced today, opens the world of DevOps data to Zap developers. If you're a DevOps pro, this new integration means you can enrich operational data with anything available from Zapier's vast catalog of connectors. Pipes' complementary connectors and data-wrangling superpowers kick Zap workflows into a whole new gear. Let's look at simple examples of two basic patterns: Pipes triggers a Zapier workflow, and Pipes runs as an intermediate step.

## Pipes as a Zap trigger: Notify when new AWS IAM access keys are created

In this scenario we want to be notified when a new AWS IAM access key is created. Here's the Pipes query we'll use.

```
select
  *
from
  aws_iam_access_key
where
  create_date > now() - interval '1 day' 
```

To use the query in a Zap, log in to Zapier, click `Create` → `New Zap`, name it `New access key`, and click `Trigger`. In the `Change Trigger` dialog box, search for `Turbot Pipes`, and select the app. You're then prompted to login to Pipes; do that.

There's only one choice for the trigger event, `New Row (Custom Query)`, choose that and click `Continue`.

Choose your workspace in the `Trigger` section, then enter your query, and click `Continue`.

![](/images/blog/2023-launch-zapier/zap-access-key-trigger.png)

In the `Test` section, click `Test Trigger`. Zapier will display up to three query results. If nothing is found, you probably don't have any fresh access keys so create one and then try again. When successful, Zapier will show you the results and invite you to select a record to continue building your Zap. Click `Continue with selected record`.

![](/images/blog/2023-launch-zapier/zap-access-key-continue-with-selected.png)

Now you're invited to choose the next app in the pipeline. We'll use Gmail to send the notification so search for it, select it, choose the `Send Email` action, and fill in `To` and `From` with your email address. You can customize the `Subject` with the `User Name` field of the sample record, and likewise customize the body to include `User Name` and `Create Date`. (Columns you asked for in the query are available in both contexts.) Then click `Test step`. You should soon receive an email like this:

![](/images/blog/2023-launch-zapier/zap-access-key-email.png)

If you want to actually use this Zap, click `Publish` to activate it.

Note that each action performed by a Zap costs you one task, and if you send a lot of emails this way you'll soon exhaust your plan's quota. SQL to the rescue! Your Pipes query can use Postgres functions (`concat`, `string_agg`) to combine multiple records into a single text value delimited by HTML tags. Then you can toggle the body type of the Zapier Gmail app from `plain` to `html` to send an HTML email.

## Pipes as a Zap action: Enrich a Jira issue

Now let's kick off a Zap with a Jira trigger. In Zapier's Jira app, set up a trigger on the `New issue` event. We'll pretend there's a convention for titling new issues: if the title contains an IP address (e.g. <em>Please investigate 10.84.1.0</em>), then Pipes will query `aws_vpc` for a match and, if there is one, will build a notification that enriches the bare IP address with data from `aws_vpc` and `aws_vpc_security_group`.

The setup for the trigger is similar to what we've already seen: start a new Zap, choose Jira, log in, choose the `New issue` event, choose your Jira project, and test to verify that Zapier can find a sample issue record.

Now we'll add a Pipes step and, this time, we'll write the query in the action instead of the trigger. The query starts by listing VPC ids and hosts, then looks for an IP addresses in the issue's `Summary` field. If there is one, the query uses the `vpc_id` to join with `aws_vpc_security_group` and report details from both. If there isn't one, the Zap ends here.

![](/images/blog/2023-launch-zapier/zap-jira-query.png)

Finally we create a Jira step that adds a comment to the issue that triggered the workflow.

![](/images/blog/2023-launch-zapier/zap-add-jira-comment.png)

When you test the Zap, you should find a new comment on the Jira issue.

![](/images/blog/2023-launch-zapier/zap-jira-issue-with-added-comment.png)

If your query returns multiple rows, you'll again burn one task for each. If you want to combine the rows there isn't an HTML option in this case, but you can use Postgres functions (`row_to_json`, `json_agg`) to convert the result set into a single value.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/6aJX4JdAjIE" 
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Zapier for DevOps: Use Pipes to unlock 5,000+ no-code automations"
>
</iframe>
</div>

## Get started with Turbot Pipes + Zapier

We've shown that, as a Zap developer, you can use Pipes to trigger a Zap with data that's otherwise unavailable, including from a vast collection of tables drawn from the major clouds ([AWS](https://hub.steampipe.io/plugins/turbot/aws/tables), [Azure](https://hub.steampipe.io/plugins/turbot/azure/tables), [GCP](https://hub.steampipe.io/plugins/turbot/aws/tables)) and others of interest to DevSecOps pros including [IPinfo](https://hub.steampipe.io/plugins/turbot/ipinfo/tables), [GitHub](https://hub.steampipe.io/plugins/turbot/github/tables), [Net](https://hub.steampipe.io/plugins/turbot/net/tables), [Shodan](https://hub.steampipe.io/plugins/turbot/shodan/tables), and many more. The Pipes queries you use in your Zap can query these Pipes-provided sources individually and also, to powerful effect, [join across](https://steampipe.io/blog/use-shodan-to-test-aws-public-ip) them.

We've also shown that you can use Pipes in the middle of a workflow to receive results from another app's trigger, enrich them with data from a Pipes query, and pass the enriched results along to a next step. 

But these are just the simplest possible examples of some basic patterns. The sky's the limit, so go build some workflows with Pipes + Zapier and [let us know](/community/join) how it goes!




Pipes enables users to [schedule and share](https://turbot.com/pipes/blog/2023/01/scheduled-queries-benchmarks-and-dashboards) persistent snapshots of their workspaces. Now it also provides a powerful – and complementary – form of persistence: Datatank. With this new feature you can automatically sync live data with persistent tables. Why? Although Pipes [parallelizes your queries](https://aws.amazon.com/blogs/opensource/querying-aws-at-scale-across-apis-regions-and-accounts/) and can pull data insanely fast, the laws of data physics still apply. If you `select * from aws_s3_bucket` for many accounts, you'll trigger a flurry of API calls. That's why our guidance has been [Not to Select *](https://steampipe.io/blog/selective-select). But no longer. With Datatank you can query broadly, in scheduled background tasks, without tapping your fingers waiting for results. The data lands in persistent tables that respond instantly, speed up benchmarks and dashboards, and stay as fresh as you need them to be. It's the best of both worlds: instant access to live data at scale when APIs can support it, and instant access to nearly-live data when they can't.

## Create a Datatank

From the `setting/connection` page in your workspace, click `New Connection` and choose the `Create Datatank` option. Then name it, describe it, and click the `Create Datatank` button. 


<img src= "/images/blog/2023-launch-datatank/create-datatank.png" />

## Synchronize a live table

You can create a Datatank in a workspace that uses the `db1.small` instance type.

<img src= "/images/blog/2023-launch-datatank/instance-type.png" />

When you create a new Datatank, you're prompted to add a table. It's as easy as picking a schema from the ones available in your workspace, and choosing a refresh frequency. In this example we choose the `all_aws` schema (which aggregates a set of [AWS connections](https://hub.steampipe.io/plugins/turbot/aws)), the `aws_s3_bucket` table, and `Daily` frequency. 

<img src= "/images/blog/2023-launch-datatank/sync-table.png" />

The query still takes as much time as it does, which in our demo example can be up to 30 seconds. Once it's done you'll see this report, which shows that the table built successfully from all three underlying connections. 

<img src= "/images/blog/2023-launch-datatank/successful-sync.png" />

Now, when you visit the query pane you'll see a new `my_datatank` schema and within it an `aws_s3_bucket` table that supersedes the corresponding live table. When you run `select * from aws_s3_bucket` here, the response will always be instant. Dashboards that [depend on the table](https://hub.steampipe.io/mods/turbot/aws_insights/dashboards?q=s3) will always be instant too.

<img style={{width:"65%"}} src= "/images/blog/2023-launch-datatank/fast-s3-query.png" />

The data will always be as fresh as you need it to be. Need more granular control beyond `Weekly/Daily/Hourly`? Write cron expressions to query on any schedule you like, and as often as you want. 

## Persist a view

Not every table can be synchronized this way. Some tables require qualifiers in `where` or `join .. on` clauses. For these cases, you can create a Datatank table from a query that specifies the qualifiers. To do that, we'll visit our `my_datatank`, click the `New Table` button, and choose the `Create Table From Query` option. 

<img src= "/images/blog/2023-launch-datatank/create-table-from-query.png" />

Here's a query to [aggregate max CPU utilization](https://hub.steampipe.io/plugins/turbot/aws/tables/aws_cloudwatch_metric_data_point#aggregate-maximum-cpu-utilization-of-all-ec2-instances-for-the-last-24-hrs) for EC2 instances. 

```sql
select
  id,
  label,
  timestamp,
  period,
  value,
  expression
from
  aws_cloudwatch_metric_data_point
where
  id = 'm1'
  and expression = 'select max(CPUUtilization) from schema("AWS/EC2", InstanceId)'
order by
  timestamp;
```

There are two required "quals": `id` and `expression`. Let's create the Datatank table `m1_max_cpu` and schedule it to run daily.


<img src= "/images/blog/2023-launch-datatank/table-from-query.png" />

You're prompted to use the `All at once` method which updates the table atomically from all aggregated connections. For more granular control you can choose the `Per connection` method to update connections independently and be more resilient to failure. Either way, the query takes as long as it initially does; thereafter you have a table that always responds instantly.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/aAKJaBbqDuw" 
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Datatank: Blow past API speed limits with scheduled data sync"
>
</iframe>
</div>


## Get started with Datatank

Steampipe's live-data approach broke new ground. Nobody, ourselves included, thought it would be possible to pull so much data so fast from cloud APIs. That remains a core strength of Turbot Pipes. If you can get live data instantly, you should; perfect freshness is ideal and you can often achieve it. Now, for those times when you can't, you can get nearly-live data instantly. It's effectively another cache that's under your control, easy to use, and able to speed up any slow query. We can't wait to [hear your stories](/community/join) once you put your foot down on the Datatank accelerator!




[Turbot Pipes](http://pipes.turbot.com/) (Formerly known as Steampipe Cloud) is a cloud intelligence and security platform that enables users to leverage SQL-based analysis across all of their cloud services & data sources, gain insights, improve their security posture, and manage their workloads from a single pane. Optimized for scalability, central management, and team collaboration, Turbot Pipes works seamlessly with your Steampipe CLI queries, benchmarks, and dashboards.

Earning and maintaining our customers' trust is of the utmost importance to us at Turbot. Our operations, product, and engineering teams are dedicated to ensuring that our products are designed, architected, and developed with relentless focus on the security of our products and of our customers' data.  We invest heavily in security, and today we are excited to share that Turbot has once again obtained our annual SOC 2 Type II compliance for [Turbot Pipes](http://pipes.turbot.com/).

We have always delivered best-in-class security. This year, we have continued our security awareness training for all employees, maintained our [CIS Benchmark Certification](https://turbot.com/blog/2020/05/cis-securesuite-member), successfully passed multiple Well-Architected audits for our cloud partnerships and remained steadfast in our commitment to our [privacy policy](https://turbot.com/legal/privacy) covering GDPR, CCPA, and Privacy Shield requirements.

Achieving this certification once more affirms our commitment to meet and exceed our customers' expectations in data security, privacy, and compliance. Our annual SOC audits are conducted each April. We make the audit reports available each May, and our full SOC 2 Type II audit report is available to customers and prospects under NDA upon request. More information about our security practices can be found on our [Turbot Security](https://turbot.com/security) page.

We invite you to try Turbot Pipes and share your experiences with us in the #cloud channel of [our Slack community](/community/join)!



A Steampipe [aggregator](https://steampipe.io/docs/managing/connections#using-aggregators) bundles multiple connections for a plugin. For example, when you're using the [AWS](https://hub.steampipe.io/plugins/turbot/aws) plugin, Turbot Pipes automatically creates an aggregator called `aws_all`. When you write a query like `select * from aws_all.aws_s3_bucket`, Steampipe runs against all the connections bundled by the aggregator, and does so concurrently.

## A default aggregation

Here's a workspace with four connections: two for Dunder-Mifflin's Scranton office, and two for the New York office.

![](/images/blog/2023-05-pipes-aggregator/aws_all.png)

Turbot Pipes has automatically created the `aws_all` aggregator. Querying `aws_all` taps all four bundled connections.

## A custom aggregation

Now let's add an aggregator for just the Scranton office. We'll use the new `Create aggregator` feature.

![](/images/blog/2023-05-pipes-aggregator/create-aggregator-1.png)

We'll use wildcards to match just the Scranton connections.

![](/images/blog/2023-05-pipes-aggregator/create-aggregator-2.png)

Now we can query `aws_all` for all four connections, or `aws_scranton` for just the two Scranton connections.

![](/images/blog/2023-05-pipes-aggregator/create-aggregator-3.png)


We could similarly bundle the New York connections as `aws_newyork`. And, of course, we can still query the Scranton and New York connections individually.

## The power of aggregation

With Turbot Pipes aggregators you can group your connections together in varying contexts. Give it a try and share your experiences with us in the `#pipes` channel of [our Slack community](/community/join)!



In
[How To Save and share Turbot Pipes benchmarks and dashboards](/pipes/blog/2022/10/using-turbot-pipes-snapshots/)
we introduced [snapshots](/pipes/docs/dashboards#saving--sharing-snapshots):
saved renderings of dashboards that you can view instantly, share privately with
your team in Turbot Pipes, and optionally share to anyone with a link. The
[v0.17 release of Steampipe CLI](https://steampipe.io/blog/release-0-17-0) added
the ability to save snapshots from the CLI, view them locally, and share them to
cloud workspaces. Next we added the ability to
[schedule](/pipes/docs/dashboards#scheduling-snapshots) snapshots so you can
build a historical record of benchmark and dashboard runs, review changes over
time, and track progress toward compliance.

Now you can do the all these things with queries too! Here's an initial run of a
query of AWS EC2 instances.

<img alt="run a query" src="/images/blog/2023-03-scheduled-query/scheduled-query-initial-run.png" />

## Snapshot and share a query

To [take a snapshot](/pipes/docs/queries#saving-snapshots) of the query, click
the _Snap_ button at the top of the query screen after the results have fully
loaded. (The button will be disabled until then.)

You'll land here.

<img alt="run a query" src="/images/blog/2023-03-scheduled-query/scheduled-query-snap.png" />

Click the _Share_ button to share it, just as you share a benchmark run or
dashboard. If you're in a Turbot Pipes [organization](/pipes/docs/organizations)
you can share with members of your workspace, or use _Anyone with the link_ to
share globally. Otherwise you can still use _Anyone with the Link_ to share from
your personal workspace.

## Schedule a query snapshot

Alternatively you can [schedule](/pipes/docs/queries#scheduling-query-snapshots)
a query snapshot. Starting from the same query pane, with results fully loaded
as above, click _Schedule_. You'll land on the _Create scheduled snapshot_
screen.

<img alt="schedule the query" src="/images/blog/2023-03-scheduled-query/scheduled-query-create-pipeline.png" />

All Turbot Pipes accounts offer Weekly and Daily options. When using
[organizations](/pipes/docs/organizations#organizations) you can also choose
Hourly or Custom (using cron syntax). To notify your team with a summary of
results, add a webhook URL for Slack or Microsoft Teams. As with any scheduled
[pipeline](/pipes/docs/pipelines#viewing-and-editing-pipelines) you can
optionally adjust visibility, assign tags, and set up notification to Slack or
MS Teams. Once you make your choices and click _Schedule_ you'll land on the
pipeline page. Click _Run now_ to verify that your query runs and produces an
initial snapshot.

<img alt="schedule the query" src="/images/blog/2023-03-scheduled-query/scheduled-query-pipeline-created.png" />

That's it! Now this query will run every day, and you'll accumulate snapshots
that document your EC2 instances over time. Try scheduling some of your own
queries, and [let us know](/community/join) how it goes!



In [How To Save and share Turbot Pipes benchmarks and dashboards](/pipes/blog/2022/10/using-turbot-pipes-snapshots/) we introduced [snapshots](/pipes/docs/dashboards#saving--sharing-snapshots): saved renderings of dashboards that you can view instantly, share privately with your team in Turbot Pipes, and optionally share to anyone with a link. The [v0.17 release of Steampipe CLI](https://steampipe.io/blog/release-0-17-0) added the ability to save snapshots from the CLI, view them locally, and share them to pipes workspaces. You can use `cron` to schedule such snapshots but now Turbot Pipes provides an easier and built-in way: [scheduled snapshots](/pipes/docs/dashboards#scheduling-snapshots).

Scheduled snapshots provide you with a historical record of benchmark and dashboard runs. Use them to review changes over time, and to track progress toward compliance.

To create a scheduled snapshot, click `Schedule` in any dashboard view.

<div style={{"marginTop":"2em", "marginBottom":"2em"}}>
  <img alt="initiate a scheduled snapshot" src="/images/blog/2023-02-scheduled-snapshots/scheduled-snap-pci-initiate.png" />
</div>

You'll land on the `Create scheduled snapshot` screen.

## Configure your scheduled snapshot

<div style={{"width":"50%", "marginTop":"2em", "marginBottom":"2em"}}>
  <img alt="configure a scheduled snapshot" src="/images/blog/2023-02-scheduled-snapshots/scheduled-snap-pci-create.png" />
</div>

All Turbot Pipes accounts offer Weekly and Daily options. When using [organizations](/pipes/docs/organizations#organizations) you can also choose Hourly or Custom (using `cron` syntax).

To notify your team with a summary of results, add a Webhook URL for Slack or Microsoft Teams.

## Test your scheduled snapshot

Once you've scheduled the snapshot, you can use the `Run now` button to kick off an initial run.

<div style={{"marginTop":"2em", "marginBottom":"2em"}}>
  <img alt="run a scheduled snapshot now" src="/images/blog/2023-02-scheduled-snapshots/scheduled-snap-pci-created.png" />
</div>

If you set a webhook URL for notifications, your team will soon see a message like this in the channel bound to the webhook. The notification summarizes the dashboard's cards.

<div style={{"width":"50%", "marginTop":"2em", "marginBottom":"2em"}}>
  <img alt="view slack notification" src="/images/blog/2023-02-scheduled-snapshots/scheduled-snap-slack-notification.png" />
</div>

## Review your pipelines

Scheduled snapshots run in [pipelines](/pipes/docs/pipelines) that you can view using the `Pipeline` tab.

<div style={{"marginTop":"2em", "marginBottom":"2em"}}>
  <img alt="view pipelines" src="/images/blog/2023-02-scheduled-snapshots/scheduled-snap-pipelines.png" />
</div>

You can view all runs of a pipeline.

<div style={{"marginTop":"2em", "marginBottom":"2em"}}>
  <img alt="view pipeline runs" src="/images/blog/2023-02-scheduled-snapshots/scheduled-snap-all-runs.png" />
</div>

And you can view a detailed log for each run.

<div style={{"width":"50%", "marginTop":"2em", "marginBottom":"2em"}}>
  <img alt="view pipeline log" src="/images/blog/2023-02-scheduled-snapshots/scheduled-snap-log.png" />
</div>

## Visualize change over time

Your inventory of cloud resources is a moving target, and your compliance status is a work in progress. Benchmarks and dashboards show inventory and compliance at a moment in time. Now, with scheduled snapshots, you can see how things change over time.

Try scheduling some of your own benchmarks and dashboards, and [let us know](/community/join) how it goes!






Turbot Pipes provides a flexible and comprehensive scheduler that runs queries, benchmarks, and dashboards on a regular basis, and saves snapshots for review.

## Snapshot and Share a Benchmark, Dashboard, or Query
When you [take a snapshot](/pipes/blog/2022/10/using-turbot-pipes-snapshots) of a benchmark, dashboard, or query, the result is a saved rendering that displays instantly. Share a private link with your team in Turbot Pipes, or a magic link with anyone.

## Schedule a Snapshot
You can also [schedule snapshots](/pipes/blog/2023/02/schedule-snapshots). This is incredibly useful for building a historical record, reviewing changes over time, and tracking progress toward compliance. Turbot Pipes supports Weekly, Daily, Hourly, and Custom (with `cron`` syntax) scheduling options.

## Notify Your Team with Summarized Results
Turbot Pipes can keep your team in the loop by sending notifications with summarized results. Add a Webhook URL for Slack or Microsoft Teams, and your team will receive notifications that summarize a dashboard's cards or query results.

## Visualize Change Over Time
As your cloud resources change, so does your compliance status. Now, with scheduled snapshots, you can visualize how things change over time.

## Get started with Turbot Pipes
Turbot Pipes delivers an unmatched ability to query and visualize cloud resources. With scheduled snapshots, your team stays continually aware of the state of your cloud infrastructure.  [Connect with us](/start) to get started.






We've shown that you can [view dashboards and run benchmarks in Turbot Pipes](/pipes/blog/2022/09/turbot-pipes-mods/). If you want to analyze the raw data behind any dashboard panel, you can use its `Download` button to capture it in a CSV file. But what if you want to see the saved data exactly as Steampipe presented it? Now, thanks to [snapshots](/pipes/docs/dashboards#saving--sharing-snapshots), you can.

A snapshot preserves the output of a benchmark, or the rendering of a dashboard, at a moment in time. Use snapshots for your own records, or share them with others. Just as a picture is worth a thousand words, a fully-rendered dashboard is worth a thousand rows of CSV output!

To take a snapshot, click the `Snap` button at the top of the dashboard after it has fully loaded. (The button will be disabled until then.)

For example, here's a live benchmark ready to be snapped.

<div style={{"width":"80%", "marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-10-using-snapshots/ready-to-snap.png" />
</div>

After clicking `Snap` the button disappears, and the `View` button changes from `Live` to `Snapshot`.

<div style={{"width":"80%", "marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-10-using-snapshots/snapped.png" />
</div>

## View snapshots

Steampipe presents the snapshot as an interactive view in which you can unfold the benchmark results just as you do in a live view.

<div style={{"width":"80%", "marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-10-using-snapshots/view-snapshot-with-unfolding.png" />
</div>

To review all your snapshots, click the `Snapshots` button in the top right corner of your workspace.

<div style={{"width":"80%", "marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-10-using-snapshots/the-snapshots-button.jpg" />
</div>

Steampipe lists all snapshots for the workspace. Use the `Query` button to search your snapshots; the dropdown list provides examples to get you started.

<div style={{"width":"90%", "marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-10-using-snapshots/query-for-snapshots.png" />
</div>

## Share a snapshot

To share a snapshot, click the `Share` button.

<div style={{"width":"90%", "marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-10-using-snapshots/share-a-snapshot.png" />
</div>

If you are using Turbot Pipes [organizations](/pipes/docs/organizations) you can share with members of your workspace as in this example, or use `Anyone with the link` to share globally. If you're not in a Turbot Pipes organization you can still share globally from your personal workspace.

<div style={{"width":"50%", "marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-10-using-snapshots/share-dialog.png" />
</div>

Here's a [link to a snapshot](https://cloud.steampipe.io/org/steampipe-demo/workspace/sandbox/snapshot/snap_ccg0ah276o9o9sp67lg0_qmf5edanpf1m2mhj22lj7n7e) of a CIS v1.5.0 benchmark that we ran in sandbox environment. We've shared it globally, so you can view and explore the results.

[Try it!](https://cloud.steampipe.io/org/steampipe-demo/workspace/sandbox/snapshot/snap_ccg0ah276o9o9sp67lg0_qmf5edanpf1m2mhj22lj7n7e)

Then consider all what you *didn't* have to do:

- Install [Steampipe](https://steampipe.io/downloads).

- [Install and configure](https://hub.steampipe.io/plugins/turbot/aws#configuration) the AWS plugin.

- Install the [AWS Compliance mod](https://hub.steampipe.io/mods/turbot/aws_compliance).

- Run the CIS v1.5.0 benchmark.

All those steps are encapsulated in that link.

## Review snapshot history

Compliance with security and governance standards doesn't happen overnight. Typically you run Steampipe benchmarks, correct the most urgent problems, then iterate for days or weeks as you burn down the list. Snapshots can help you track progress on your journey to compliance. This
query finds all the CIS snapshots in a workspace.


<div style={{"marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-10-using-snapshots/viewing-history.png" />
</div>

## Get started

That's it! It's easy to capture, view, and share snapshots of your benchmarks and dashboards.

Want to try this for yourself? Turbot Pipes is available in preview, you can sign up at [pipes.turbot.com](http://pipes.turbot.com/). We're always eager for feedback; you can tell us about your experience in our
[Slack community's](/community/join) `#pipes` channel.


Steampipe [mods](https://hub.steampipe.io/mods) are packaged sets of resources
-- named queries, controls, benchmarks, or dashboards -- that work with the
database tables provided by a [plugin](https://hub.steampipe.io/plugins). Users
of Steampipe CLI install mods by cloning the mods' GitHub repos and invoking
`steampipe check` or `steampipe dashboard`.

In Turbot Pipes the process is [much simpler](/pipes/docs/mods).

## Get started

If you haven't already installed the
[Net](https://hub.steampipe.io/plugins/turbot/net) plugin, click
`Settings -> Connections -> Add Connection` and choose `Net`.

<div style={{"marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-09-cloud-mods/add-new-connection.jpg" />
</div>

Click `Create a new connection -> Create`.

<div style={{"marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-09-cloud-mods/create-new-connection.jpg" />
</div>

Then click `Add to workspace` and link the connection to your workspace.

<div style={{"marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-09-cloud-mods/add-connection-to-workspace.jpg" />
</div>

The `Settings -> Connections` tab in this Turbot Pipes workspace shows that
one plugin is installed.

<div style={{"marginTop":"2em", "marginBottom":"2em",  "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-09-cloud-mods/view-connections.jpg" />
</div>

The `Settings -> Mods` tab shows that no mods are installed.

<div style={{"marginTop":"2em", "marginBottom":"2em",  "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-09-cloud-mods/view-mods-initial.jpg" />
</div>

Click `Install Mods` to view a scrollable list of official mods that work with
the connections you've defined.
[Net Insights](https://hub.steampipe.io/mods/turbot/net_insights) is
recommended.

<div style={{"marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-09-cloud-mods/recommended-installable-mods.jpg" />
</div>

Click `Install Mods` to install the latest version of `Net Insights`.

## Run a Net Insights benchmark

Click `Dashboards` to view the benchmarks and dashboards provide by the mods
you've installed.

<div style={{"marginTop":"2em", "marginBottom":"2em",  "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef", "width":"80%"}}>
  <img src="/images/blog/2022-09-cloud-mods/dashboards-initial-view.jpg" />
</div>

They're initially grouped by mod name. Search for `Net Insights`, and click
`DNS Best Practices` to run that benchmark. Turbot Pipes reports progress for
each group of controls. Expand a group to see progress for the individual
controls in the group, and to view results.

<div style={{"marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-09-cloud-mods/running-dns-best-practices-benchmark.jpg" />
</div>

Note that the default domains checked by the benchmark are `microsoft.com` and
`github.com`. To check different domains, click
`Settings -> Mods -> net_insights` to open an editor that knows about the
variables you can set for this mod. Enter some other domains, for example
`whitehouse.gov` and `gov.uk`, to see how they fare on the benchmark!

<div style={{"marginTop":"2em", "marginBottom":"2em",  "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-09-cloud-mods/mod-variable-editor.jpg" />
</div>

## Run an AWS Insights dashboard

If you haven't already installed the
[AWS](https://hub.steampipe.io/plugins/turbot/aws) plugin, click
`Settings -> Connections -> Add Connection`, choose `AWS`, and authenticate
using one of the available methods: `Cross-Account Role` or `Access Key`.

As before, click `Settings -> Mods` to select from the list of mods that work
with the AWS plugin. Select `AWS Insights` and click `Install Mods`.

Some of the dashboards provided by `AWS Insights` focus on S3 resources. To find
them, search for `aws s3`.

<div style={{"marginTop":"2em", "marginBottom":"2em",  "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef", "width":"80%"}}>
  <img src="/images/blog/2022-09-cloud-mods/search-aws-s3.jpg" />
</div>

Click `AWS S3 Bucket Dashboard` to view that dashboard.

<div style={{"marginTop":"2em", "marginBottom":"2em", "borderStyle":"solid", "borderWidth":"1px", "borderColor":"#f3f1ef"}}>
  <img src="/images/blog/2022-09-cloud-mods/aws-s3-bucket-dashboard.jpg" />
</div>

That's it! It's easy to install mods that work with the connections in your
workspace, then use them to check your cloud resources for compliance with
security standards or visualize them on dashboards.

Want to try this for yourself? Turbot Pipes is available in preview, you can
sign up at [pipes.turbot.com](https://pipes.turbot.com). We're always eager
for feedback; you can tell us about your experience in our
[Slack community's](/community/join) `#pipes` channel.



[Steampipe](https://steampipe.io) is a groundbreaking open-source project that acts as a universal interface to APIs, allowing you to query cloud infrastructure, SaaS, logs, and more using SQL. Turbot Pipes integrates with Steampipe to provide an intuitive interface for powerful querying capabilities.

## Query Across APIs with SQL
With Steampipe, you can run SQL queries across different APIs. Whether it's AWS, Azure, GCP, or even SaaS products like GitHub and Salesforce, data-fetching becomes effortless. [Turbot Pipes](https://pipes.turbot.com/) enhances Steampipe with user-friendly cloud hosting to make Steampipe even more accessible.

## Perform Security Compliance Checks
Use Turbot Pipes to ensure that your cloud resources comply with security benchmarks such as CIS, NIST, and SOC2. Run these checks smoothly, and share reports with your team.

## Visualize Your Data
Turbot Pipes use [Steampipe mods](https://hub.steampipe.io/mods) to visualize cloud resources with prebuilt dashboards. You can also build custom dashboards using our [dashboards-as-code](https://steampipe.io/blog/dashboards-as-code) approach: write HCL to define dashboard widgets, and SQL to flow data into them.

## Choose from a Broad Set of Plugins
Turbot Pipes inherits Steampipe's open-source and community-driven [plugin ecosystem](https://hub.steampipe.io/plugins) which map APIs to tables that you can query with SQL. Plugin categories include Cloud, SaaS, Security, Identity, DevOps, Networking, IaC, Logs, Social Media, and more. Popular plugins include AWS, Azure, GCP, GitHub, and Slack.

## Enjoy Diverse Client Support
Query tables in your Turbot Pipes workspaces with [any tool or language that can connect to Postgres](/pipes/docs/connect). CLI tools include Steampipe's own [steampipe query](https://steampipe.io/docs/query/overview) as well as `psql` and `pgcli`. BI tools include Apache Superset, Metabase, and Tableau. Supported languages include Go, Python, JavaScript, and any other language with a Postgres driver.

## Monitor Costs and Resource Tagging
Use [open-source mods](https://hub.steampipe.io/mods) to keep track of cloud expenses, verify the consistency of tags applied to resources across different cloud providers, and thereby ensure organized, cost-effective operations.

## Gain Insights and Evaluate your Security Posture
The [open-source mods](https://hub.steampipe.io/mods) provide dashboards that visualize cloud resources using charts, tables, and [relationship graphs](https://steampipe.io/blog/k8s-insights). They also provide benchmarks that evaluate the security posture of your cloud environment using a [long list of frameworks and standards](https://hub.steampipe.io/mods/turbot/aws_compliance#references) including CIS, FedRAMP, GDPR, NIST, and SOC 2.

## Build Custom Plugins and Mods
Extend Steampipe's capabilities with [custom plugins](https://steampipe.io/docs/develop/writing-plugins) for your own APIs, and create [derivative versions](https://steampipe.io/blog/remixing-dashboards) of existing benchmarks and dashboards to meet your specific needs.

## Join a Thriving Community
Steampipe's [vibrant community](https://steampipe.io/community/join) thrives on feedback and involvement. You can ask questions, contribute plugins and mods, or even [join the Turbot team](/hiring).

## Get Started with Turbot Pipes and Steampipe
Turbot Pipes and Steampipe combine to deliver an unmatched ability to query and visualize cloud resources. [Connect with us](/start) to get started with Turbot Pipes and Steampipe.



Turbot Pipes uses [Steampipe](https://steampipe.io) to enable SQL queries of your cloud infrastructure, SaaS apps, logs, and more. As both a user and developer, you benefit from SQL's ability to [unify diverse APIs](https://www.infoworld.com/article/3648648/how-sql-can-unify-access-to-apis.html). Whether the data comes from cloud infra (AWS, Azure, GCP), SaaS apps (GitHub, Salesforce), or config data in Terraform files, it all resolves to SQL tables that you can query individually and join together.

## Perform Security Compliance Checks
Use Turbot Pipes to ensure that your cloud resources comply with security benchmarks such as CIS, NIST, and SOC2. Run these SQL-defined checks smoothly, and share reports with your team.

## Visualize Your Data
Use [mods](https://hub.steampipe.io/mods) to visualize cloud resources on a curated set of dashboards. You can also build custom dashboards using our [dashboards-as-code approach](https://steampipe.io/blog/dashboards-as-code): write HCL to define dashboard widgets, and SQL to flow data into them.

## Choose from a Broad Set of Plugins
Steampipe's community-driven plugin ecosystem maps APIs to tables that you can query with SQL. Plugin categories include Cloud, SaaS, Security, Identity, DevOps, Networking, IaC, Logs, Social Media, and more. Popular plugins include AWS, Azure, GCP, GitHub, and Slack.

## Enjoy Diverse Client Support
Query tables in your Steampipe workspaces with any SQL client that can connect to Postgres. CLI tools include Steampipe's own steampipe query as well as psql and pgcli. BI tools include Apache Superset, Metabase, and Tableau. Supported languages include Go, Python, JavaScript, and every other language with a Postgres driver.

## Monitor Costs and Resource Tagging
Use [mods](https://hub.steampipe.io) to keep track of cloud expenses, verify the consistency of tags applied to resources across different cloud providers, and thereby ensure organized, cost-effective operations.

## Gain Insights and Evaluate your Security Posture
The curated dashboards visualize cloud resources using charts, tables, and [relationship graphs](https://steampipe.io/blog/k8s-insights). They also provide benchmarks to evaluate the security posture of your cloud environment using a [long list](https://hub.steampipe.io/mods/turbot/aws_compliance#references) of frameworks and standards including CIS, FedRAMP, GDPR, NIST, and SOC 2.

## Build Custom Plugins and Mods
You can extend Steampipe's capabilities with custom plugins for your own APIs, as [Claranet France](https://steampipe.io/blog/claranet#a-steampipe-plugin-for-the-in-house-cmdb) did. And while the prebuilt mods are already deep and comprehensive, they're also easy to [reuse and remix](https://steampipe.io/blog/remixing-dashboards).

## Join a Thriving Community
Steampipe's [vibrant community](https://steampipe.io/community/join) thrives on feedback and involvement. You can ask questions, contribute by developing plugins, or even [join the Turbot team](/hiring).

## Get Started with Turbot Pipes
Turbot Pipes delivers an unmatched ability to query and visualize cloud resources. Harness the power of SQL for your APIs and cloud resources. [Connect with us](/start) to get started.




Cloud engineers and security experts appreciate Turbot Pipes' speed, flexibility, and seamless integration with the Steampipe CLI.

## Bundle your Connections
[Connection aggregators](https://steampipe.io/docs/managing/connections#querying-multiple-connections) bundle parallel connections for a plugin. If you're using the AWS plugin, Turbot Pipes creates an aggregator called `aws_all`. Now you can write queries like `select * from aws_all.aws_s3_bucket`, and Steampipe will concurrently run it against all the connections bundled by the aggregator. You'll be amazed at how fast it can be.

## Create a Custom Aggregator
Beyond the aggregators that Turbot Pipes creates automatically, you can define custom aggregators for subsets of connections. These can group your connections in a variety of useful ways.

## Reuse and Remix
Turbot Pipes delivers thousand of prebuilt queries, benchmarks, and dashboards. But don't stop there! You can [reuse and remix](https://steampipe.io/blog/remixing-dashboards) as well. Build custom mods locally, then [deploy](http://localhost:3000/pipes/docs/mods#installing-custom-mods) to Turbot Pipes.

## Connect your CI/CD Pipeline
When you run the Steampipe CLI in a [GitHub, GitLab, or Jenkins](https://steampipe.io/docs/integrations/overview#cicd-pipelines), you can use `--share` to send query results and benchmark/dashboard outputs to Turbot Pipes.

## Get Started with Turbot Pipes
Turbot Pipes delivers a flexible, fast, and unified experience across CLI and the Cloud. [Connect with us](/start) to get started.



Use Turbot Pipes to gain granular insights into cloud spending, helping to identify waste and improve your organization's financial security.

## Financial Metadata and Tagging
Tagging is a powerful feature that aids in cost allocation, tracking, and management. With Turbot Pipes, you can query resources based on tags to better understand the cost implications of different projects, departments, or other divisions within your organization, and you can check to make sure all resources within your organization have the correct tag keys and values.

## Find unused resources
Use Turbot Pipes to identify unused or underutilized resources that are silently adding to your cloud bill. These could include idle compute instances, unattached volumes, or unused IP addresses. Once identified, these resources can be decommissioned or repurposed, providing immediate cost savings.

## Financial Analysis and Planning
Use Turbot Pipes DB integration feature to compare spend over time to planned allocation and budgets, allowing you to better forecast growth and find savings opportunities.

## Service Cost Allocation
Up your game even further and use your Turbot Pipes cloud configuration data to measure the granular cost to deliver a service or feature, or integrate it directly with your billing or cost allocation system.

## Find older non-optimized resource types
Cloud providers are constantly releasing new, higher performance and lower-cost features. Use Turbot Pipes to identify these resources in your cloud infrastructure, providing an opportunity to replace or upgrade them with more cost-effective and efficient alternatives.

## Find misconfigured services
Misconfigured services not only pose a security risk but can also lead to unnecessary costs. With Turbot Pipes, you can query your cloud resources to identify misconfigurations, such as multiples of the same service configuration, resources in unused regions and features that are enabled but not used.

[Sign up](/pipes) for a free Turbot Pipes developer account today and embark on a journey toward unparalleled collaboration in cloud querying. Together, let's unleash the true potential of your cloud infrastructure!



Snapshot and share compliance reports, providing clear documentation of cloud infrastructure alignment with SOC 2, CIS, NIST 800-53, HIPAA, PCI, GxP, GDPR, FedRamp, and more.

## Broadest multi-cloud compliance and security reporting

Yes, we have the big 3: AWS, Azure and GCP, but Turbot pipes also supports compliance reporting for IBM Cloud, Oracle, Alibaba, GitHub, Kubernetes, SnowFlake, Tailscale, Terraform and more.

## Largest scope of supported benchmarks
Our AWS compliance mod supports 20+ industry control objectives, including AWS Foundational Security Best Practices, CIS, CISA, FedRAMP, FFIEC, GDPR, GxP, HIPAA, NIST, PCI, RBI, SOC 2.

## Open and customizable
Clone our open source compliance reporting repos and modify our built-in reporting for your organization's needs.

## Browsable benchmarks with drill down

{screenshot}

## Share the data with your team
Snapshot and share the browser-based view or export the data as CSV, Markdown or JSON.

[Sign up](/pipes) for a free Turbot Pipes developer account today and embark on a journey toward unparalleled collaboration in cloud querying. Together, let's unleash the true potential of your cloud infrastructure!



Turbot Pipes offers broader coverage and more depth than any other CSPM on the market – by a wide margin. With plugin support for 100+ cloud services, and thousands of pre-built open-source control objectives to run, Turbot Pipes protects your organization's multi-cloud infrastructure continuously and comprehensively.

## Use all the clouds
Turbot Pipes seamlessly integrates with over 100 cloud services, including AWS, Google Cloud, Azure, Oracle, IBM and many more. Whether you have a diverse multi-cloud environment or use a single cloud platform, our solution covers a wide range of services, ensuring that no corner of your infrastructure goes unassessed.

## Run pre-built benchmarks and controls
Use Turbot Pipes to assess your environment. There are thousands of pre-built open-source control ready to run. They embody current best practices shared by industry experts, plus our own domain expertise: Over the last decade, we have helped some of the world's largest companies tackle their compliance and security challenges. When you sign up for Turbot Pipes you tap into a broad and deep set of assessments tailored to industry best practices. Think of the time you'll save, versus creating your own security assessments from scratch!

## Be current, avoid surprises
We understand that cloud services and security best practices evolve at a rapid pace. That's why we constantly expand and update our library of controls. With Turbot Pipes, you'll have up-to-date security assessments for the latest cloud services and technologies, so you can stay ahead of emerging threats and industry changes.

## Customize with SQL
Turbot Pipes offers an extensive library of pre-built controls. But we know that every organization's security posture is unique, so you can also write custom controls that tailor assessment criteria for your needs. And it's [surprisingly easy](https://steampipe.io/docs/mods/writing-controls) to write a custom control! Enjoy the flexibility to define security benchmarks that reflect your industry, compliance standards, and internal policies.

## Give and receive
Turbot Pipes thrives on a vibrant community of contributors. We pool our collective expertise to foster a collaborative environment that benefits all of us. [Join](/community/join) our community, contribute your own controls, and access a wealth of assessments built by like-minded security professionals.

## Get started today with Turbot Pipes
When you assess the security posture of your multi-cloud infrastructure, don't settle for limited coverage. Turbot Pipes offers an unmatched breadth of coverage with plugins for 100+ cloud services and thousands of pre-built open-source controls, constantly updated to meet your evolving security needs.

[Sign up](/pipes) for a free Turbot Pipes developer account today and embark on a journey toward unparalleled collaboration in cloud querying. Together, let's unleash the true potential of your cloud infrastructure!



Prepare for third-party audits with confidence using Turbot Pipes, the ultimate solution for audit readiness. Gain unparalleled visibility into how your environment measures up against the most comprehensive set of compliance benchmarks available. Avoid surprises, ensure compliance, and streamline your audit process.

## Tap into comprehensive suites of compliance benchmarks
Turbot Pipes connects you with a vast library of benchmarks that check for compliance with a broad and deep set of industry standards and best practices. From regulatory requirements to security frameworks, Turbot Pipes covers a wide range of compliance domains. Rest assured that your environment will align with the most rigorous standards, so you can meet regulatory obligations and instill trust among your stakeholders.

## Avoid surprises
The last thing you want during an audit is an unexpected surprise. Turbot Pipes empowers you to proactively identify and address compliance gaps before they become audit issues. Comprehensive compliance benchmarks help you align your environment with industry standards, and avoid any potential surprises that could arise during the audit process. Prepare with confidence and demonstrate your commitment to compliance excellence.

## Streamline the audit process
Build confidence with auditors by answering their questions quickly and succinctly. Turbot Pipes empowers your team to respond to auditor inquiries rapidly, with only the information required.  Use Datatank to build a centralized pool of instantly-accessible audit-ready data, so you can provide concise, accurate responses. When you use Turbot Pipes to streamline the audit process, and show that you can navigate inquiries with ease, you build trust and instill confidence in your organization's compliance practices.

## Monitor compliance continuously
Audit readiness is an ongoing effort, not a one-time event. With Turbot Pipes you can establish continuous compliance monitoring, to ensure that your environment remains compliant even after the audit. By scheduling benchmarks to run weekly or daily, you can proactively monitor and address compliance issues, identify deviations, and implement corrective measures promptly.

## Remediate quickly
Empower your team to monitor and track corrective actions resulting from audit findings. Using Turbot Pipes' query-over-time capabilities, you can easily manage and track the implementation of recommended controls, remediation activities, and corrective measures identified during the audit process. Turbot Pipes provides a centralized platform to document and monitor the progress of each corrective action, ensuring accountability and transparency. By actively monitoring audit compliance corrective actions, you can demonstrate your commitment to addressing identified issues promptly, improve your overall compliance posture, and maintain a culture of continuous improvement.

## Build trust and credibility
An audits isn't just a checkbox exercise, it's also an opportunity to build trust and credibility with your stakeholders. Use Turbot Pipes to show everyone you're committed to meeting industry standards, elevate your audit performance, and strengthen your reputation as a compliant and trustworthy organization.

[Sign up](/pipes) for a free Turbot Pipes developer account today and embark on a journey toward unparalleled collaboration in cloud querying. Together, let's unleash the true potential of your cloud infrastructure!



Unlock the true potential of your cloud infrastructure with Turbot Pipes, a platform designed from the ground up for collaborative analysis of your cloud infrastructure. With Turbot Pipes, your teams can use SQL queries to delve into your organization's entire cloud infrastructure. Cloud operations and security professionals are empowered as never before.

## Make SQL a team sport

Members of a Turbot Pipes [organization](http://localhost:3000/pipes/docs/organizations) work together to craft queries that enable collective team awareness of the state of your multi-cloud infrastructure. Imagine the possibilities when your cloud operations and security teams collaborate in realtime, exchanging knowledge, refining queries, and uncovering hidden patterns.

## Be more efficient
When you capture organizational knowledge with shared SQL queries, teams work better together. Building on each others' insights, they accelerate resolution of critical issues and foster a culture of continuous improvement.

## Streamline cloud operations
Teams use shared queries to diagnose misconfigured infrastructure, find and eliminate unused resources, optimize infrastructure configurations, troubleshoot incidents, monitor system health, and make data-driven decisions to ensure optimal cloud performance. Turbot Pipes empowers your cloud operations team to become more proactive, responsive, and agile.

## Redefine cloud security
Collaboration matters everywhere, but especially in cloud security. Turbot Pipes provides security professionals with new tools to investigate, analyze, and respond to potential threats or vulnerabilities. Using shared queries, your team can swiftly identify security risks, detect anomalies, and perform comprehensive audits across your entire cloud infrastructure. Once the team has identified a threat pattern, they can use SQL again to rapidly find the offending configuration.

## Be flexible and safe
Turbot Pipes fosters collaboration, but not at the expense of the control you need! You can organize groups of accounts into access-controlled workspaces, and share configuration data without providing access to underlying cloud infrastructure. Empower your users with the ability to run and share queries securely, while retaining granular control over sensitive information.

## Get started today with Turbot Pipes
Experience the transformative power of collaborative querying with Turbot Pipes. Unlock new possibilities for your cloud operations and security teams by fostering collaboration, and harnessing the collective intelligence of your organization. Empower your teams to query your cloud infrastructure with SQL, make smarter decisions, and resolve issues faster.

[Sign up](/pipes) for a free Turbot Pipes developer account today and embark on a journey toward SQL-powered organizational knowledge. Together, let's unleash the true potential of your cloud infrastructure!



Use Turbot Pipes to efficiently monitor and analyze the configuration of your cloud resources over time. Gain unparalleled insights into resource allocation, service usage, tagging, and system health. Empower your organization to make data-driven decisions with confidence.

## Observe resource allocation
Turbot Pipes provides a clear view of resource allocation across your cloud infrastructure. With the ability to query the configuration of cloud resources over time, you can track changes in resource allocation, identify trends, and optimize your resource utilization. Gain a deep understanding of how resources are provisioned and allocated across your organization, empowering you to make informed decisions and ensure cost efficiency.

## Uncover usage patterns
Turbot Pipes enables you to go beyond surface-level observations and delve into the intricacies of service usage. By querying the configuration of your cloud resources, you can identify patterns, analyze usage trends, and gain insights into the consumption of various services. Use this information to optimize service provisioning, identify potential cost savings, and make informed decisions about resource allocation based on actual usage patterns.

## Check tagging practices and report inconsistencies
Disciplined tagging helps you organize and managing your cloud resources effectively. Use Turbot Pipes to gain visibility into your organization's tagging practices, identify inconsistencies or gaps, and enforce standardized tagging policies. With a clear view of tag usage and trends, you can enhance resource management, streamline operations, and improve compliance and governance practices.

## Streamline the reporting process
Turbot Pipes simplifies the operational reporting process, enabling you to generate and distribute reports quickly and effortlessly. Use its intuitive interface and powerful querying capabilities to define the parameters, filters, and timeframes for your reports. Turbot Pipes streamlines the reporting workflow, saving you time and effort while delivering comprehensive, accurate, and actionable insights.

## Make data-driven decisions
With Turbot Pipes, your organization can use operational reports to make data-driven decisions. Gain a holistic view of your cloud infrastructure, identify misconfigurations, optimize resource allocation, and align your operations with business objectives. Empower stakeholders across your organization with accurate and up-to-date operational reports that facilitate informed decision-making, enhance efficiency, and foster continuous improvement.

## Get started today with Turbot Pipes
Unlock the power of querying the configuration of your cloud resources over time. Gain valuable insights into resource allocation, service usage, tagging, and system health. Make informed decisions, optimize operations, and achieve operational excellence by harnessing the comprehensive operational reports provided by Turbot Pipes.

[Sign up](/pipes) for a free Turbot Pipes developer account today and embark on a journey toward unparalleled collaboration in cloud querying. Together, let's unleash the true potential of your cloud infrastructure!





Turbot Pipes is a groundbreaking platform that uses data to revolutionize the way your team manages and secures your cloud infrastructure. It offers a comprehensive solution for cloud intelligence that includes operational reporting, security posture assessment, and custom benchmarks and dashboards.

## Experience the power of data-driven cloud intelligence

At the core of Turbot Pipes is a commitment to data-driven cloud intelligence. By enabling you to use SQL queries to explore your organization's entire cloud infrastructure, Pipes empowers cloud operations and security professionals to collaborate, rapidly assess problems, and streamline operations.


## Prepare for audits with confidence

Use Turbot Pipes to prepare for third-party audits and avoid surprises. No more herding cats to gather the information needed for an audit, now you can schedule queries that automatically gather the information you need to present. Join across APIs to make correlations, for example between the IP addresses of EC2 instances and their locations as found by [ipinfo.io](https://hub.steampipe.io/plugins/turbot/ipinfo) or [ipstack](https://hub.steampipe.io/plugins/turbot/ipstack).

## Monitor compliance continuously
Set up continuous monitoring to ensure that your environment remains compliant even after the audit. By scheduling benchmarks to run weekly or daily, you can proactively monitor and address compliance issues, identify deviations, and implement fixes promptly. Turbot Pipes also empowers your team to monitor and track corrective actions resulting from audit findings, so you're always transparent and accountable. 

## Respond rapidly to threats
Turbot Pipes provides security professionals with new tools to investigate, analyze, and respond to potential threats or vulnerabilities. Empower your team to swiftly identify security risks, detect anomalies, and perform comprehensive audits across your entire cloud infrastructure.

## Use and extend thousands of benchmarks and dashboards

Pipes includes a broad and deep set of prebuilt benchmarks and dashboards that deliver 
 comprehensive, data-driven views of your cloud infrastructure. But don't stop there! With our [dashboards as code](https://steampipe.io/blog/dashboards-as-code) approach, you use HCL to define your own widgets and controls (or [remix](https://steampipe.io/blog/remixing-dashboards) existing ones), and SQL to flow data into them. It's all code, so you enjoy the benefits of version control and collaboration in environments like GitHub and GitLab. And you team can work together in Turbot Pipes to share queries and collectively explore your cloud resources.

## Get Started Today with Turbot Pipes

Experience the transformative power of data-driven cloud intelligence with Turbot Pipes. Use it to make informed decisions, optimize operations, and achieve operational excellence. [Sign up](/pipes) for a free Turbot Pipes developer account today. Together, let's use data to unleash the true potential of your cloud infrastructure!




Turbot Pipes provides a [huge set of benchmarks and dashboards](https://hub.steampipe.io/mods) out of the box. You can also build custom Turbot Pipes benchmarks and dashboards to gain unparalleled insights into resource allocation, service usage, role-based access control, and system health. Enable your organization to make data-driven decisions with confidence.

## Experience dashboards as code
With Turbot Pipes' [dashboards as code](https://steampipe.io/blog/dashboards-as-code) approach, you write HCL to define dashboard widgets, and  SQL to flow data into them. Enjoy the benefits of version control and collaboration as you deliver customized views of compliance and resource allocation across your cloud infrastructure.

## Remix Dashboards
You can easily [remix and reuse](https://steampipe.io/blog/remixing-dashboards) the batteries-included dashboards. And it's surprisingly easy to [add your own](https://steampipe.io/blog/extend-well-architected-1) custom benchmarks and controls.

## Use and create relationship graphs

Powerful relationship graphs make complex subsystems legible. Turbot Pipes provides many of these visualizations out of the box, e.g. [Kubernetes RBAC](https://steampipe.io/blog/kube-rbac-explorer). But don't stop there! You can also use HCL + SQL to make your own custom relationship graphs.

## Schedule snapshots

Schedule snapshots of your custom benchmarks and dashboards to track changes over time, review progress towards compliance, and share status with your team (or anyone with a link).

## Enjoy live editing

As you create a custom dashboad using HCL + SQL, you [see results immediately](https://steampipe.io/blog/dashboards-as-code#dashboards-as-code) in your favorite code editor.

## Make data-driven decisions
Use and customize Turbot Pipes to drive data-driven decisions. Gain a holistic view of your cloud infrastructure, identify misconfigurations, optimize resource allocation, and align your operations with business objectives.

## Get started today with Steampipe Dashboards
Make informed decisions, optimize operations, and achieve operational excellence by adding your own custom dashboards to the comprehensive built-in dashboards provided by Turbot Pipes.

[Sign up](/pipes) today for a free Turbot Pipes developer account today, and start building your own benchmarks and dashboards.





Steampipe, a new open source project from Turbot, enables cloud pros (e.g. software developers, operations engineers and security teams) to query their favorite cloud services with SQL. It has quickly become one of our favorite tools in-house and we hope it finds a way into your tool box as well.

The heart of Steampipe is an intuitive command line interface (CLI) that solves the challenges encountered when asking questions of cloud resources and services. Traditional tools and custom scripts that provide visibility into these services are cumbersome, inconsistent across providers and painful to maintain. Steampipe provides a consistent, explorable and interactive approach across IaaS, PaaS and SaaS services.

## The Challenge of Context Switching

Everyone that has worked with cloud service providers understands how hours can be wasted bouncing back and forth between tools to answer simple questions about your environment. The diagram below illustrates common challenges encountered when extracting data from cloud-based systems.

![traditional-workflow](/images/blog/2021-01-introducing-steampipe/traditional-workflow.png)



The traditional tools we use to gain insight and answer questions about our environments challenge us instead of empower us. Key challenges of the current capabilities:

1. Simple questions quickly devolve into software development projects.
2. Commercial solutions require you to grant access outside your organization or require deployment of complex infrastructure.
3. Tools that take snapshots of your environment, require synchronization and force you to work on stale data.

These barriers take the practitioners time and attention away from what is important: asking questions and finding answers.

## Steampipe's Approach

The goal of Steampipe is to simplify the workflow for discovery and querying of cloud based configuration information. Steampipe exposes your cloud configuration as a high-performance relational database. This allows you to explore the live configuration of running cloud assets without switching context. With Steampipe, SQL tables represent complex cloud resources such as IAM policies, security groups, databases, storage buckets, SSL certificates, etc..

![steampipe-workflow](/images/blog/2021-01-introducing-steampipe/steampipe-workflow.png)



## How Steampipe Works

We designed the architecture of Steampipe to be easy to understand, simple to install, and extensible. Our single step install process lets you get up to speed quickly with the Steampipe CLI on your local machine or deployed to a virtual machine.

![how-it-works-diagram](/images/blog/2021-01-introducing-steampipe/how-it-works-diagram.png)



When a user writes a Steampipe SQL query, Steampipe translates it into API calls that are executed in real-time across one or more cloud service APIs. The data returning is organized into tables using a PostgreSQL Foreign Data Wrapper (FDW); allowing the user to join, filter, and aggregate that data just like any other database table.

## Excited? We are too!

Steampipe v0.1.0 is [available for download today](https://steampipe.io/downloads). Install, query and get cloud work done. We can't wait to see what you query, and iterate based on your feedback. If you'd like to help expand the Steampipe universe, or even dive into the CLI code, the whole project is open source and we'd love to collaborate! Keep an eye on [https://hub.steampipe.io/](https://hub.steampipe.io) for the latest supported resource types from both Turbot Guardrails and the Steampipe community.

To learn more about Steampipe, please visit the project website at [https://steampipe.io/](https://steampipe.io). Our GitHub repo is here, and checkout [https://steampipe.io/docs](https://steampipe.io/docs) for a guided tour of Steampipe capabilities.

If you experience any issues, please report them on our [GitHub issue tracker](https://github.com/turbot/steampipe/issues) or post to the [Steampipe discussion forum](https://github.com/turbot/steampipe/discussions).

Free automated AWS CIS v3.0 assessments

Turbot Pipes as a software component

Aggregating connections in Turbot Pipes

Snapshot, share, and schedule queries in Turbot Pipes

Schedule snapshots of benchmarks and dashboards in Turbot Pipes

Save and share Turbot Pipes benchmarks and dashboards

View dashboards and run benchmarks in Turbot Pipes