Pipes + GitLab: private projects, custom mods, instant deploys

Pipes has long enabled you to install and use custom mods, but you had to source them from GitHub repositories. Now, with our new GitLab integration, you can install a custom mod from a private or public GitLab project as well.

You can do this as an individual identity, as the owner of a Pipes organization or (as a Pipes Enterprise user) in a Pipes tenant. In all these scenarios, you install your custom mod from a GitLab project that you choose from a the GitLab integration setup. And when you push changes, Pipes automatically updates the mod.

To illustrate, we'll use a simple dashboard that counts S3 buckets with versioning enabled or disabled. As a custom mod developer, you always want to test locally using Powerpipe to make sure the mod runs correctly. Then push to GitLab and create a GitLab integration in Pipes to gain access to it.

Create an integration

From the Settings page of your Pipes organization or tenant, click New Integration, choose GitLab, , and click Next.

GitLab Integration configuration management form shows up. Enter the following information:

• Handle for the integration. This handle should be meaningful and must be unique for all integrations in the tenant (including any org-level integrations).
• Host for your GitLab installation which defaults to gitlab.com.
• Token to be used by Pipes to discover projects. Please note that the token must be granted api scope which is required to create and manage webhooks that enables Pipes to keep the mods managed by the integration up-to-date. Pipes supports personal, group or project access tokens.
• Filter path denotes the path of the group/project where Pipes will setup the webhook and restrict project searches to. Note that this field is required when using a personal access token and must be the path to a single project. It is optional when using a group token and cannot be set when using a project token.

Click on Test after entering the appropriate information to verify whether your configuration is accurate before creating the integration. Click on Create once the test is successful.

Pipes will validate the configuration and set up the necessary webhook at the appropriate level. You will then be redirected back to the integration list page with the new GitLab integration set up and ready to use.

Install the mod

After installing the GitLab integration, you're ready to use it to install a mod from GitLab. Here we're installing to a workspace called mattydemo but, because the integration was created in a Pipes organization, any workspace can use it.

The picker only shows the one authorized project. You can install from a branch (defaulting is main) or from a tag that defines a semantic versioning constraint.

View the dashboard

Once it's installed, we can view the dashboard in Pipes.

Let's have some fun and push a change to our mod.

We'll change the mod to use a chart instead of a table:

diff --git a/dashboard.pp b/dashboard.pp
index de7b330..6f0acb5 100644
--- a/dashboard.pp
+++ b/dashboard.pp
@@ -1,6 +1,6 @@
 dashboard "s3_bucket_versioning" {
 
-  table {
+  chart {
     title = "s3_bucket_versioning"
     width = 4
     sql = <<EOQ

We commit this change and push it to GitLab.

Pipes detects the change, updates the mod, and immediately refreshes the dashboard to display the updated chart.

It really is that easy (and fast!) to deploy a change and update it live!

Developing for Pipes

The large collection of pre-built benchmarks and dashboards already available in Pipes will meet many of your needs right out of the box. However, if you haven't yet discovered the power of custom mods in Pipes, right now is a great time to get started. Building custom mods, using HCL and SQL, is straightforward and enjoyable way to customize data visualizations and benchmarks to your needs. So give it a try, and let us know how it goes!

See it in action