Run Flowpipe Workflows in Turbot Pipes

Flowpipe is the open-source solution for DevOps and security pros who use workflows-as-code to automate cloud operations. Today we're thrilled to announce that Flowpipe joins its open-source siblings Steampipe and Powerpipe as a component of the Pipes platform. Flowpipe features will gradually be rolling out across all Pipes users in the coming days.

To see it in action, we'll use an AWS Thrifty pipeline to remediate buckets that don't have a lifecycle policy. Let's see how that works in Pipes.

An AWS-like collection of services

The first thing you'll notice, in a new Pipes workspace, is that all three core components — Steampipe, Powerpipe, and Flowpipe — are right there ready to use. In this case our job requires Flowpipe so that's the tool of choice.

Easy mod installation

The Thrifty mod is available on the hub, but Pipes brings that interface into your workspace and enables point-and-click installation.

Form-based configuration

You configure Flowpipe mods with variables which, when running the standalone tool, you edit by hand. Pipes automagically turns those variables into forms. In this case, we won't need to change any of the variables. The pipeline will ask for a decision about whether to remediate each non-compliant bucket, and we want that decision to happen in Slack, but that's the default notifier for this workspace so we are ready to go. The pipeline also needs a Steampipe AWS connection, in order to query for non-compliant buckets, but that comes for free as well. The organization that owns this workspace has set up the connection and permitted this workspace to use it. So we can just select the Run Pipeline button.

Slack-based approval

Here's a non-compliant bucket found by the pipeline. We've routed the approval decision to Slack, but could have used email instead, or MSTeams. We want this bucket to have a lifecycle policy so we choose Apply lifecycle configuration. What if the approver doesn't see the message right away? No problem, Pipes suspends the workflow so you won't be billed for the wait, and restarts it when approver replies.

Hands-free remediation

Flowpipe receives the reply, takes the action, and reports the outcome. Of course it can get old making decisions like this on a per-bucket basis. If you want all existing and new buckets to block public access, you can run the pipeline in a hands-free way, on a schedule. To do that you configure a couple of variables. For this Flowpipe needs to run in server mode, but Pipes takes care of that for you.

Easy Slack integration

The default notifier for this workspace is Slack. The notifier in turn refers to an integration that was already set up in the organization and available to this workspace. How did that happen? Let's create a new Slack integration to see how easy that is. Here are the types of integrations we can create.

We've already seen the AWS, Azure, GCP integrations, which populate your Pipes organization or tenant with accounts, subscriptions, and projects from those platforms. And we've also seen the GitHub integration, which uses Pipes' 3rd-party GitHub app to install mods into a workspace and automatically refresh them when changes are pushed to a branch or tag. The Slack integration works like the GitHub integration. You name it, install into your Slack instance as a 3rd-party Slack app, and pick the Slack channel where you want notifications sent.

Custom mods

The Flowpipe Hub provides a growing collection of libraries and samples that capture a variety of common DevOps patterns, but everyone's situation is unique. It's often useful to write your own mods to suit your particular needs. How to install them? You may already have used the GitHub to install Powerpipe mods into a workspace, and automatically refresh them when changes are pushed to a branch or tag. The same integration works exactly the same way for Flowpipe mods.

Next-level workflows as code

As noted in Pipes Assemble!, Pipes is a platform on which Steampipe, Powerpipe, and now Flowpipe share common patterns and infrastructure. You can build powerful solutions with the standalone components, but Pipes brings them together smoothly, in a way that's easy to manage across your organization or tenant. If you're a Pipes user who hasn't yet tried Flowpipe, fire it up in a workspace and let us know how it goes.

See it in action