v1.0 for Steampipe, Powerpipe, Flowpipe, 116 plugins, and 44 mods

Today we're promoting all three of Turbot's open source components — Steampipe, Powerpipe, and Flowpipe — to v1.0.0. It's a major milestone on a journey that began with the launch of Steampipe v0.1.0 on January 21, 2021, along with an initial suite of 13 plugins featured on the then-new Steampipe Hub.

The first version of Steampipe's most popular plugin, AWS, provided 68 tables, already a formidable collection that has since grown to nearly 500. In the inaugural blog post we wrote:

Steampipe, a new open source project from Turbot, enables cloud pros (e.g. software developers, operations engineers and security teams) to query their favorite cloud services with SQL. It has quickly become one of our favorite tools in-house and we hope it finds a way into your toolbox as well.

Nearly four years later, that hope has become a reality. Here are a few top-line Steampipe stats:

The core query capability provided by Steampipe plugins was soon augmented with a suite of mods that focused, initially, on compliance checks for AWS, Azure, GCP, and a few others. These mods wrapped Steampipe queries in HCL-defined controls and benchmarks tailored to the Center for Internet Security recommendations. Today we offer compliance mods for Audit Manager Control Tower, AWS Foundational Security Best Practices, CIS, FedRAMP, FFIEC, GDPR, HIPAA, NIST, PCI, SOC 2, and many more, along with mods for asset inventory/insights, shift-left scanning, and cost management.

These continue to evolve rapidly. AWS Compliance, for example, now provides 1,229 controls and 30 dashboards, and adds support for security and compliance benchmarks on a regular basis.

Today all these mods join the v1.0.0 release party.

From benchmarks to dashboards

From the beginning we've standardized on two declarative languages: SQL to acquire data from clouds, HCL to report and visualize. That core pattern has carried through the suite of open source components, while evolving along the way. Initially, for example, the benchmarks we provided (and that you could compose or create yourself) used HCL to declare and combine controls. Then, with Dashboards as Code, we added new HCL syntax for tables, chart, infocards, inputs, relationship graphs, and more. What had been an engine for running benchmarks could now also power dashboards, all in the as-code, GitHub-friendly style of modern DevOps.

The arrival of relationship graphs showcased the value of this approach. Use SQL to query AWS for sets of resources related to, for example, a VPC. Then use HCL to visualize its full context.

Database-agnostic plugins

Originally, Steampipe plugins ran in a captive instance of Postgres equipped with a plugin manager and foreign data wrapper extension. People often asked: "Can we also run those plugins in our own databases?" We made it so, with "zero-ETL" distributions of plugins for your own instance of Postgres, or for SQLite. While we were at it, we also made them available as standalone exporters.

Now a bigger picture started to come into focus. The Steampipe ecosystem, as it grew and evolved, was becoming a suite of components that worked well together in their original contexts, but could be recombined to serve other purposes in other contexts.

Powerpipe emerges from Steampipe

Next, we realized that the benchmarks-and-dashboards subsystem included with Steampipe ought to be unbundled from the core query engine. These were logically two different components that worked well together, but needn't be joined at the hip. Once decoupled from Steampipe, for example, the benchmarks-and-dashboards subsystem could become database-agnostic just as plugins had. Thus was born Powerpipe, a standalone tool that runs all your benchmarks and dashboards but can also connect to any Postgres, or SQLite, or MySQL, or DuckDB.

Now you could build dashboards-as-code to visualize data in your own instance of Postgres, or even data living in Parquet files using DuckDB. Decoupled from Steampipe, Powerpipe could still serve its original purpose while also enabling wider ambitions.

Flowpipe delivers HCL-powered workflows for DevOps

All this visibility into resources and their configuration prompted users to ask: "How can we act on everything we can now see?" To build DevOps workflows that create, update, and delete you have to orchestrate scripting, CLI commands, Lambda functions, containers, and human approvals across multiple providers and identity frameworks. Bash and Python have served us well, but there had to be a better way.

Enter Flowpipe, Turbot's third major open source component, which debuted in December 2023. It's "like an HCL-powered alternative to Step Functions", said Aidan Steele. Noting Flowpipe's DAG-powered implicit parallelisation, he added: "I wish SFN had that!"

Like Steampipe and Powerpipe, Flowpipe is a self-contained binary that you can easily deploy locally, or in any CI/CD enviroment. Like its siblings, it's supported by a hub where you can find prebuilt workflows for common DevOps chores, supported by libraries that you can use to build your own workflows, and samples that capture common patterns.

And yes, these mods also join today's v1.0.0 release party.

Semantic versioning

We've held onto v0.x for a long time, allowing the tools to evolve and mature. Now that we see their shapes clearly we'll continue to move them forward, and we'll do that with regular releases that follow semantic versioning. And with a solid pattern in place, we're in a great position to grow the family of components.

Many components, one community

In Turbot's Slack community, nearly 7,000 users and contributors gather to discuss Steampipe, Powerpipe, and Flowpipe, along with the growing suites of mods they enable. You can use these core components individually to query cloud resources, report on them, and act on them. And they work even better together as when, for example, a Steampipe query feeds a Powerpipe dashboard or a Flowpipe pipeline.

Flipping the v1.0.0 switch is always an emotional moment. We're just getting started, but it was time to recognize the maturity of this ecosystem and celebrate with everyone who has traveled with us on this journey. Thanks, and stay tuned!