Launch Week 6 B-sides

As Launch Week 6 draws to a close, we wanted to take a moment to highlight some of the exciting updates and announcements that slipped under the radar this week across our Turbot products and open-source projects.

Guardrails: 70+ controls added

We continue to listen to customer feedback to add new cloud resource types and controls for security, compliance, and operational guardrails. Recently, we added 10 new resource types:

• AWS: RDS DB Cluster Parameter Group
• Azure: Container Registry, User Assigned Managed Identity, Metric Alerts, Private Link Service
• GCP: Vertex AI (Endpoint, Notebook Runtime Template), Dataplex (Lake, Task, Zone)

Accompanying them are over 70 controls from the common suite of Guardrails controls, including CMDB / Discovery, Tags, Stacks / Configured, Usage, and ServiceNow Cloud discovery sync.

Additional controls were added for auto-remediation of:

• AWS Manage Fargate FIPS Mode for GovCloud accounts
• AWS Support permission delegation to which roles
• Azure MySQL flexible services minimum TLS version
• Azure Storage Account Table logging policies
• GCP Service Account disable based on inactive status
• GCP Subnetwork Flow Log policy configuration
• and 65+ more

Guardrails: Improved Multi-Query in Calculated Policies

When using Guardrails calculated policies to retrieve data from multiple sources you previously had to write two queries. Common context fields such as the parent resource’s ID and URI, and the current resource’s ID, tags, and URI are now readily available. For example, previously for the parent resource's ID, then use that result in a second query to filter data. Now you can simply call the data field as $.resource.parent.turbot.id without the GraphQL query.

budgetData: resources(filter: "resourceTypeId:'tmod:@turbot/aws#/resource/types/budget' resourceId:{{ $.resource.parent.turbot.id }}")

Guardrails: Large scale improvements behind the scenes

In line with this Launch Week’s theme of making Guardrails easier and more efficient, we've introduced a number of mass improvements across Guardrails mods to boost performance, add flexibility, and simplify default policy behavior:

• Better performance: 40 Azure mods have been updated to support the latest Azure SDK versions, enhancing the discovery and management of 163 Azure resource types. The new SDK provides performance improvements and new data fields, which are included in the Guardrails CMDB.
• More flexibility: All supported AWS, Azure, GCP, and Kubernetes mods now include a new ServiceNow Import Set mode policy, allowing records to be shipped in either a single or multiple record format, offering more flexible integration options.
• Improved default behavior:
  • 36 GCP mods have been updated so the CMDB control for 142 GCP service resource types no longer depends on the API Enabled policy being set to Enforce: Enabled for the service.
  • 172 AWS mods have been updated for 431 AWS service resource types, adjusting the Approved > Usage policy for resource types to default to Approved instead of Approved if AWS > {service} > Enabled.

Steampipe: 20+ new tables and columns supported

Steampipe has added support for over 20 new tables and columns across several cloud providers and services. These updates allow for deeper insights and more comprehensive queries for your cloud resources.

New tables have been added for AWS, Azure, GCP, Linode, and Okta, including:

• AWS: New tables like aws_app_runner_service and aws_timestreamwrite_database.
• Azure: Support for azure_key_vault_certificate and azure_web_application_firewall_policy.
• GCP: Expanded support with tables for gcp_dataplex_lake, gcp_composer_environment, and gcp_vpc_access_connector.
• Okta: okta_authentication_policy
• Linode: linode_firewall and linode_node_balancer

Additionally, new columns have been added to existing tables, such as:

• AWS: Columns added to aws_health_event and aws_ec2_instance_type.
• Azure: Expanded data for azure_compute_virtual_machine and azure_postgresql_flexible_server.
• GCP: New columns for gcp_cloudfunctions_function and gcp_sql_database_instance.

Steampipe: Improved query JSON output format

Steampipe has updated its query JSON output format to include a columns property along with the rows property, enabling better handling of duplicate column names by appending a unique suffix. This update aligns all cases around a single output format, making queries more consistent across Steampipe, Powerpipe, and Pipes.

Example of the new JSON format:

{
  "columns": [
    { "name": "account_id", "data_type": "text" },
    { "name": "arn", "data_type": "text" }
  ],
  "rows": [
    { "account_id": "123456789012", "arn": "arn:aws:::123456789012" }
  ]
}

This standardized format now applies to:

• steampipe query --output json
• powerpipe query run --output json
• Snapshot data format
• Pipes query API

Powerpipe: New GitHub Actions

Powerpipe has introduced new GitHub Actions to automate cloud compliance and Infrastructure as Code (IaC) checks within your CI/CD pipelines.

• Powerpipe Action Setup: Simplifies the installation of Powerpipe and Steampipe, and sets up connections to your cloud accounts or IaC files. This action gets everything ready to run benchmarks and controls in subsequent workflow steps.
• Powerpipe Action Check: Executes benchmarks and controls from Powerpipe mods to scan for compliance issues across your cloud environments or IaC configurations. When used with IaC mods, this action can create GitHub annotations for any controls in an alarm state, helping you identify specific lines in your configuration files that are non-compliant.

Examples of what you can do:

• Run specific AWS compliance controls to check for unused EBS volumes or unencrypted S3 buckets.
• Execute cloud-wide security benchmarks (e.g., CIS or NIST) across multiple mods, including AWS, Azure, GCP, and Kubernetes.
• Automate Terraform compliance checks, ensuring your infrastructure configurations meet security and best practice standards before deployment.
• Check Kubernetes clusters for security best practices, such as ensuring encryption on secrets or restricting pod security policies.

By integrating these actions into your GitHub workflows, you can continuously monitor and enforce compliance for both cloud services, Kubernetes clusters, and IaC files, helping to catch issues early in the development lifecycle.

Note: the prior Steampipe Action Check v1.1.0 is now deprecated and will no longer be maintained.

Flowpipe: On-demand trigger execution

Flowpipe now supports on-demand trigger execution, enabling SQL queries to be scheduled, and capturing data changes for pipelines. This powerful feature enables you to run a query on a defined schedule and pass the results—such as inserted, updated, or deleted rows—to the relevant pipeline for further processing.

For example, a query trigger can check for expired AWS IAM access keys and pass the data to a pipeline for deletion:

trigger "query" "expired_access_keys" {
  database    = "postgres://steampipe@localhost:9193/steampipe"
  primary_key = "access_key_id"
  schedule    = "daily"

  sql = <<EOQ
      select
          access_key_id,
          user_name,
          create_date
      from
          aws_iam_access_key
      where
          create_date < now() - interval '90 days';
  EOQ

  capture "insert" {
    pipeline = pipeline.delete_expired_access_keys
    args = {
        access_keys = self.inserted_rows
    }
  }
}

Flowpipe keeps track of data in a SQLite database to detect new, updated, or deleted rows, and automatically triggers the pipeline based on these changes. This enables you to automate workflows such as handling access keys, DNS changes, or instance lifecycle management, all from a simple query.

Pipes: New connection - Prisma Cloud

Pipes now supports Prisma Cloud, allowing you to centrally query, report, and collaborate on over 26 resource types. These include:

• Alerts and Rules
• Compliance stats
• IAM permissions, roles, and users
• Inventory assets and workloads
• Vulnerability assets
• and more

This expands the growing list of security tools you can query and join data across, such as CrowdStrike, Splunk, Turbot Guardrails, Vanta, and more.

Pipes: GCP temporary credential support added

To avoid the use of long-term service account credentials, Pipes now supports temporary credentials through service account impersonation using the Service Account Token Creator role.

With this update, GCP integrations can use temporary credentials to securely access project data. To enable this in Pipes, choose Service Account Impersonation from the Access Mode dropdown, and grant the Service Account Token Creator role to the Pipes service. Additionally, label your project with the key turbot-pipes and follow the steps to automate role assignment using the provided Terraform plan or gcloud command. Click the Test Discovery button to verify the configuration.

Alternatively, you can still use a service key file by selecting Credentials from the Access Mode dropdown and uploading a JSON key pair for an existing GCP service account.

Pipes: 10+ new security benchmarks and controls supported

Powerpipe has added 4 new security benchmarks and 7 new controls to our AWS, Azure, and GCP compliance mods. These new benchmarks are available in the Powerpipe Hub and Turbot Pipes to assess your security posture and share status with your team. The latest benchmarks include:

• AWS Australian Cyber Security Center (ACSC) Essential Eight
• AWS CIS Compute Services v1.0.0
• AWS NIST 800-172
• GCP SOC2 2017

New controls span services like Azure Application Gateway, Azure Application Insights, Azure Log Analytics Workspace, and GCP AlloyDB Instance Logs, offering more detailed and comprehensive compliance coverage.

Hacktoberfest 2024: Mid-event update

Hacktoberfest 2024 is in full swing, and we’re excited to see incredible contributions from the community! This year, Turbot is participating in the global event, encouraging developers and enthusiasts to contribute to our open-source projects, including Steampipe, Powerpipe, and Flowpipe, along with our samples and docs repos for Turbot Pipes and Guardrails.

So far, we’ve seen a variety of contributions, ranging from bug fixes to feature enhancements across our repositories. A few highlights include:

• @dbermuehler - Adding Shield Tables and EPSS Score to the AWS plugin.
• @pdecat - New gcp_organization_project and gcp_compute_instance_group_manager tables in the GCP plugin.
• @oguzhan-yilmaz - creating a Steampipe + Powerpipe helm chart
• @brunorossi - leveraging AWS Steampipe FDWs in a custom AWS query tool via REST APIs.
• @l-teles - creating a new Steampipe plugin for Detectify
• and countless other cool features, improvements, doc fixes, blog posts, etc

It's inspiring to see the passion and creativity in these contributions. Whether it’s fixing a bug, adding new features, or creating insightful content, every effort helps make our open-source community stronger.

We’re looking forward to more great contributions in the final stretch of Hacktoberfest. If you haven’t joined yet, there’s still time to get involved and earn some exclusive Turbot swag!

Flip over to A-sides for the Wrap Up

Thank you for joining us for another exciting Launch Week! Check out the week's daily announcements summary in our Launch Week 6 Wrap Up post. Stay connected with us in our Slack community for our next Launch Week in a few months!