You asked, we delivered. Now you can import your entire AWS organization, with just a few clicks, as a set of Pipes connections. This new integration dramatically simplifies connecting AWS resources to Pipes, and enables you to query and analyze your cloud infrastructure more thoroughly than ever before.
Get ready to import
Here's the AWS structure we'll import.
To create a new AWS Integration, select AWS from the integration options.
You can accept the default handle for the integration, AWS, or choose your own.
In Discovery Settings, you set up the authentication Pipes uses to discover your AWS OUs and accounts. You can use a cross-account role (recommended) or an access key pair; either way, test the connection to verify it works.
In Connection Settings, you set up the trust relationship for discovered accounts. For this you'll need a cross-account role in each discovered account; Pipes can generate a CloudFormation StackSet to help automate the process. See the docs for details. Again, test to verify it works.
Set permissions
Now choose which workspaces — all, some, or none — are permitted to use the connections.
Note the separation of concerns: Pipes administrators make connections available to workspaces, and workspace owners decide, on the workspace's Advanced / Connections page, which (if any) to activate for query.
Import OUs and accounts
Now click Create Integration to import the discovered OUs and accounts. Pipes performs the import, and creates a tree that matches your AWS structure, so delegation of permissions works the same way in Pipes as it does in AWS.
Add connections to a workspace
Subject to the connections they are permitted to see, workspace owners can now use any OU that was imported into Pipes to add its corresponding set of connections to a workspace. In this example, we choose the Pipeling Scale Testing connection folder which contains 200 accounts.
The workspace owner uses the Add to schema button to make the corresponding connections available for query.
When a Pipes connection folder contains two or more connections, Pipes automatically creates an aggregator for them. Here's all_aws which aggregates the 200 connections acquired from the Pipeling Scale Testing OU.
Query the connections that were imported and added
Now, in the query pane, you can select and query any of the aggregated connections, or use the aggregator to query all of them. Here we query the aggregator.
And that's it! We've gone from a standing start to a queryable set of imported AWS connections in just minutes.
Try it yourself!
This new AWS integration addresses a common challenge faced by our users with large AWS footprints. It would be time-consuming to do this by hand or with Terraform scripting. Now you can bring your whole AWS organization into Pipes quickly, easily, and automatically. And when OUs or accounts change, Pipes syncs the changes.
We're thrilled to offer this enhanced AWS support and can't wait to see how it accelerates your cloud operations and security workflows. As always, we welcome your feedback and we're to help as you explore this powerful new capability.