What is Steampipe?

Steampipe is open source software for interrogating your cloud. Run SQL queries, compliance controls and full governance benchmarks from the comfort of your CLI.

Steampipe’s codified operations framework gives you the power to test your cloud resources against security, compliance and cost benchmarks, and to build your own custom control frameworks.

Our multi-threaded Golang CLI makes your custom SQL controls blazing fast with unlimited integration options via our embedded PostgreSQL database.

{` select region, instance_state as state, instance_type as type from aws_ec2_instance; `} {` +-----------+---------+-----------+ | region | state | type | +-----------+---------+-----------+ | eu-west-1 | running | t3.medium | | eu-west-2 | running | m5a.large | | us-east-1 | running | t3.large | +-----------+---------+-----------+ `}

## tl;dr ** → ** **[Tables from CSV files](#tables-from-csv-files)** with support for other dynamic sources.
** → ** **[Improved control output for mods](#improved-control-output)** with summaries and new output formats.
** → ** **[7 new plugins and mods](#new-plugins-and-mods)**.
** → ** Even more goodies in the [full release notes](https://github.com/turbot/steampipe/blob/main/CHANGELOG.md#v090-2021-10-24).
## Tables from CSV files The APIs delivered by Steampipe plugins have so far worked with fixed schemas. With this release we now enable a different kind of plugin that maps tables on the fly. The first of these is the new [CSV plugin](https://hub.steampipe.io/plugins/turbot/csv). Point it at a directory containing CSV files, and a corresponding set of tables appears in Steampipe. Now spreadsheets can join the party, along with a world of other data sources that can export to CSV. Here's some data in a CSV file called `owners.csv`. ``` resource_type,owner ec2_instance,dwight@dundermifflin.com ec2_instance,jan@dundermifflin.com security_group,george@dundermifflin.com ``` If that file lives in a directory where you've told the CSV plugin to look for data, its contents are magically available as a table! {` select * from csv.owners o where o.resource_type = 'ec2_instance' `} {` +---------------+--------------------------+ | resource_type | owner | +---------------+--------------------------+ | ec2_instance | dwight@dundermifflin.com | | ec2_instance | jan@dundermifflin.com | +---------------+--------------------------+ `}
These dynamic tables will work hand-in-hand with the variables and query parameters introduced in [v0.8.0](https://steampipe.io/blog/release-0-8-0#variables--query-parameters-in-mods). Those features enable you to customize queries and mods with user-supplied data, such as required tags. Now you can also manage tag vocabularies in spreadsheets. For example, here's a query for the `owner` tag in the `aws_ec2_instance` table. {` select instance_id, tags ->> 'owner' as owner from aws_ec2_instance `} {` instance_id | owner --------------------+-------------------------- i-0e97f373db22dfa3f | dwight@dundermifflin.com i-0dc60dd191cb86539 | george@dundermifflin.com i-06ee5c096826de741 | jan@dundermifflin.com `}
To find instances whose `owner` tag isn't listed in `owners.csv`: {` select instance_id, tags ->> 'owner' as owner from aws_ec2_instance i where not exists ( select * from csv.owners o where o.resource_type = 'ec2_instance' and o.owner = i.tags ->> 'owner' ); `} {` +---------------------+--------------------------+ | instance_id | owner | +---------------------+--------------------------+ | i-022a51a815773780d | jan@dundermifflin.com | | i-03f3b66e057009f41 | dwight@dundermifflin.com | +---------------------+--------------------------+ `}
Beyond using spreadsheets in this way, you can now begin to inventory data assets that live in those places, and use Steampipe to define and check the integrity rules you want to enforce. We love spreadsheets because they're convenient, but we hate that they're uncontrolled. Now you can write Steampipe controls for your mission-critical spreadsheets! Note to developers: To write another plugin that creates tables from dynamically-discovered schemas, check out the new `SchemaMode` argument in the [plugin definition](https://steampipe.io/docs/develop/writing-plugins#plugin-definition). ## Improved control output When you run `steampipe check all` in a directory where you've installed one of the Steampipe [mods](https://hub.steampipe.io/mods), the output can be overwhelming. So we've added a nice summary to the end of the report. ![](/images/blog/2021-10-22-release-0-9-0/control-summary.jpg) We've also expanded the output formats. Along with `csv` and `json`, you can now output or export `html` and markdown (`md`). They look the same, here's an example of HTML output. ![](/images/blog/2021-10-22-release-0-9-0/html-control-output.jpg) ## New plugins and mods Since [our last release](/blog/release-0-8-0), we've added 6 new plugins: - [Code](https://hub.steampipe.io/plugins/turbot/code) - finds secrets in columns of other plugins' tables - [CSV](https://hub.steampipe.io/plugins/turbot/csv) - query tables defined in .CSV files - [Heroku](https://hub.steampipe.io/plugins/turbot/heroku) - query account, addon, domain, dyno, pipeline, etc - [IBM Cloud](https://hub.steampipe.io/plugins/turbot/ibm) - query account, access_group, iam_role, iam_user, etc - [Scaleway](https://hub.steampipe.io/plugins/turbot/scaleway) - query account_ssh_key, instance_server, rdb_database, etc - [Terraform Enterprise](https://hub.steampipe.io/plugins/turbot/tfe) - query organization, sentinel_policy, team, etc Major mod updates: * [Kubernetes Compliance](https://hub.steampipe.io/mods/turbot/kubernetes_compliance) with [NSA & CISA Hardening Guidance](https://hub.steampipe.io/mods/turbot/kubernetes_compliance/controls/benchmark.nsa_cisa_v1) * [Azure Compliance](https://hub.steampipe.io/mods/turbot/azure_compliance) now includes [NIST SP 800-53 Revision 5](https://hub.steampipe.io/mods/turbot/azure_compliance/controls/benchmark.nist_sp_800_53_rev_5) ## Let’s get building! Steampipe now delivers a full suite of tools to build, execute and share cloud configuration, compliance, and security frameworks using SQL, HCL and a little elbow grease! We would love your help to expand the open source documentation and control coverage for CIS, PCI, HIPAA, NIST… and the best way to get started is to [join our new Slack workspace](https://steampipe.io/community/join) and raise your hand; we would love to talk to you! For even more good stuff in v0.9.0, checkout the [full release notes on GitHub](https://github.com/turbot/steampipe/blob/main/CHANGELOG.md#v090-2021-10-21).

v0.9.0: Dynamic tables & improved control outputs