Pipes Compute Usage Drops 75% with Smarter Rate Limits
Turbot Pipes is now faster and cheaper to run thanks to smarter rate limiting that reduces throttling errors and saves customers up to 75% in compute usage.
Sometimes solving one customer's scale challenge unlocks a benefit for the entire community.
That's exactly what happened recently. While working with a large enterprise customer on demanding AWS workloads, we introduced custom Steampipe rate limiters into Turbot Pipes. The goal was to improve reliability at scale. The outcome? Dramatically reduced throttling errors, faster refresh jobs, and a surprising drop in compute usage across the board.
The challenge: Scaling IAM workloads
The customer was adding high-volume IAM tables to their Datatank, namely aws_iam_policy, aws_iam_policy_attachment, and aws_iam_role. At scale, these tables can hit AWS API throttling limits, leading to retries, wasted compute, and slower refreshes.
Jobs were finishing, but not as efficiently as they could.
The solution: Smarter rate limiters
In late June we rolled out custom Steampipe rate limiters for aws_iam_policy into pipes and then extended the approach to aws_iam_policy_attachment and aws_iam_role. Once we saw the success of these changes across some high volume workspaces we shipped a plugin update making these limiters the default for all workloads, including ad-hoc queries.
These changes are powered by Steampipe's rate limiter framework, recently enhanced with new memory management and diagnostic features.
By pacing requests intelligently, Pipes avoided AWS throttling errors. Counterintuitively, running slower at the right times made everything run faster overall.
Lower costs, better performance
For the enterprise customer that inspired the work, the results were clear:
- Compute usage dropped 60–75% within days of the release.
- Refreshes stabilized and completed with fewer retries.
- Overall performance improved, even as costs went down.
And because Pipes updates are platform-wide, all customers running workloads on these IAM tables saw the benefits automatically.
All Pipes customers benefit from:
- Lower costs: fewer wasted CPU minutes.
- Improved performance: jobs complete faster with fewer retries.
- Greater scalability: room to grow without running into throttling bottlenecks.
The big picture
What started as a scale optimization for one customer became a cost savings win for everyone.
It's a great example of how the Turbot ethos of working closely with customers on real-world scaling problems can unlock improvements that ripple across the platform.
Sometimes the fastest way forward is to slow down — just enough to avoid the throttle.
If you are new to Turbot Pipes, get started with an always free developer tier and a 14-day free trial for your team.