Pipes + GitHub: private repos, custom mods, instant deploys

Integrate Pipes with GitHub to develop custom Powerpipe mods.

Turbot Team
4 min. read - May 16, 2024
Integrate Pipes with GitHub to develop custom Powerpipe mods.

Pipes has long enabled you to install and use custom mods but you had to source them from public GitHub repositories, and it was a manual process to install and then update your mods. Now, with our new GitHub integration, you can install a custom mod from a public or private repo.

You can do this as an individual identity, as the owner of a Pipes organization or — as a Pipes Enterprise user — in a Pipes tenant. In all these scenarios, you install your custom mod from a GitHub repo that you choose from a GitHub app. And when you push changes, Pipes automatically updates the mod.

To illustrate, we'll use a simple dashboard that counts S3 buckets with versioning enabled or disabled. As a custom mod developer, you always want to test locally using Powerpipe to make sure the mod runs correctly. Then push to GitHub and create a GitHub integration in Pipes to gain access to it.

Create an integration

From the Settings page of your Pipes organization or tenant, click New Integration, choose GitHub, name the integration, and click Create Integration.

new integration

Choose your repo

You're redirected to GitHub, where you choose an identity and then pick the repos that Pipes can see.

choosing a repo for a custom mod

Typically you'll grant access to All, here we'll be more specific and just grant access to one demo repo.

Install the mod

After installing the GitHub app with that configuration, you're redirected back to Pipes where you can now use the integration to install a mod from GitHub. Here we're installing to a workspace called demo but, because the integration was created in a Pipes organization, any workspace can use it.

install from main

The picker only shows the one authorized repo. You can install from a branch, defaulting to main, or from a tag that defines a semantic versioning constraint.

View the dashboard

Once it's installed, we can view that dashboard in Pipes.

dashboard as bar chart

Now here's the really fun part: let's push a change.

Update the mod

We'll change the bar chart to a pie chart.

change bar to pie chart

Commit the change, and push it.

Pipes detects the change, updates the mod, and immediately refreshes the dashboard to display the updated chart.

dashboard as bar char

It really is that easy, and that fast, to deploy a change and live-update it!

Developing for Pipes

Pipes' vast collection of pre-built benchmarks and dashboards will meet many of your needs right out of the box. But if you haven't yet discovered the power of custom mods in Pipes, now's a great time to get started. You'll find that building custom mods, using HCL and SQL, is straightforward and enjoyable. It's a powerful way to tailor data visualizations and benchmarks to your needs. So give it a try, and let us know how it goes!

See it in action