Common Troubleshooting Scenarios for Mods

Error: "Forbidden: Mod dist URL has expired"

Symptom

When trying to update a Type Installed control, an error message similar to "Forbidden: Mod dist URL has expired" appears. Re-running the control will not clear the error.

Cause

Turbot distributes mods using S3 pre-signed URLs. These URLs will expire after three days. As the controls belong to the mod, the entire mod must be refreshed to get a new URL.

Remediation

  1. Login to the Turbot Console as a user with Turbot/Owner permissions.
  2. Navigate to the Admin page by clicking the gear icon in the top right, then select the Mods tab.
  3. Find the mod that is in error in the list.
  4. Click the pencil icon. A dialog box with a number of versions will pop up. Click the version that is already installed.
  5. Click the Update Mod mod button.
  6. Turbot will begin to download the selected version of the mod and deploy it. This can take a moment depending on the size of the environment.
  7. Once the mod re-installation is complete, the Mod Health control will re-run. If the control does not go into the OK state, contact the Turbot support team at help@turbot.com.

Using Turbot Interval Policies

Turbot, by default, utilizes an event driven model to detect resource changes. In some rare circumstances this is either not possible or does not meet very specific business needs.

This introduces us to the concept of Turbot Intervals. Intervals can be used to force regular “ticking” of CMDB controls, which then trigger guardrails when misconfigurations are detected. While this method can solve many use cases, it is important to recognize the cost associated with doing so.

Why Intervals?

  • Ability to regularly update resource metadata which generate no events when changed, i.e. the number of available IP addresses in a subnet.
  • An organization requires resources to be scanned and logged on a regular basis.

Turbot Intervals come at a cost. Each “tick” represents a request from Turbot to describe a set of resources. As the number of resources in the environment goes up, so does the amount of requests sent to and received from the cloud service API endpoint. Similarly, setting a short interval triggers more requests.

Setting a Turbot Interval

In order to set a Turbot Interval policy, you must have Turbot/Admin permissions at the root level, CLI configured, and Terraform configured.

1. Either create a new Terraform configuration file or open an existing one in a text editor.

2. Paste the following code in the configuration file:

resource "turbot_policy_setting" "turbot_interval" {
  resource = "tmod:@turbot/aws-ec2#/control/types/volumeCmdb"
  type     = "tmod:@turbot/turbot#/policy/types/interval"
  value    = "days: 1"
  note     = "[Ticket CLOUD-152] Run the volume CMDB control on a scheduled interval"
}

Let's break this down:

  • resource - This is the resource where the policy will be set. Notice it is a Control for Volume CMDB. We can use any control here that needs to be triggered on a regular interval. For example, if the requirement was to refresh S3 Bucket CMDB entries, the control would then be tmod:@turbot/aws-s3#/control/types/s3AccountCmdb. Use the Mods Registry to find the correct control
  • type - The policy type is defining which policy is being created. This URI corresponds to the policy Turbot > Interval.
  • value - This defines the time in between control runs. It is recommended to start at the upper bound of the time requirements and slowly increment towards the desired time requirement. Additional keys that can be used are hours and minutes. Great care must be taken to not set too small of an interval!
  • note - While not required, it is highly recommended to set a note for future you (and anyone viewing said policy)!

3. Apply the configuration file.

4. That's it! CMDB controls will now be triggered for the desired resource type (EC2 volumes for this example) on the defined interval.

Important Note: This works well for new controls, but existing controls will require a control run to adhere to the new, defined interval. We suggest running all relevant resource controls which can be done all at once by utilizing a control run script.

WARNING!!

Cloud providers can and will throttle requests once the request rate is exceeded. Regular control runs at short intervals can also generate a considerable amount of “noise” in the environment.

Mod Interval policies are applied on controls themselves and apply for ALL resources defined by the control. For example, setting the policy Turbot > Interval policy on the tmod:@turbot/aws-ec2#/control/types/volumeCmdb control type will cause Turbot to poll ALL EC2 volumes in ALL accounts imported into Turbot. This CANNOT be set at a lower level in the hierarchy!

If setting a Turbot Mod Interval policy is required, it is recommended to start at the upper bound of the time frame. For example, start with 24 hours per tick to determine the impact on your environment and adjust downwards as necessary and as the environment allows it.

Turbot is not liable for misconfigurations of the Interval policy resulting in application downtime and throttling of service API. Great care must be taken when setting the interval to not overload the API endpoints.