Service Catalog Out of Sync

In this guide, you will:

• Use AWS Service Catalog to manage Turbot Guardrails Enterprise Database.
• Troubleshoot out-of-sync issues between Service Catalog and CloudFormation.

When working with the TED (Turbot Enterprise Database) stack, you may encounter an issue where the Service Catalog Provisioned Product, CloudFormation Stack, or the physical RDS instance becomes out of sync. This can prevent you from applying necessary changes or updates to the database, and may result in the TED provisioned product in the Service Catalog changing to a Tainted status.

Prerequisites

• Access to the Guardrails AWS account with Administrator Privileges.
• Familiarity with AWS Console, Service Catalog, and CloudFormation services.

Step 1: Check Provisioned Product Status

While executing the TED update, check if the provisioned product in the Service Catalog changing to a Tainted status.

Step 2: Check Stack Failure Details

In the selected provisioned product, select the Events tab, in UPDATE_PROVISIONED_PRODUCT section select View details.

This will bring up Error window with StatusReason.

Select View more in CloudFormation to navigate to CloudFormation stack. This will display Error Message as

Resource handler returned message: "Invalid storage size for engine name postgres and storage type gp3: 210 (Service: Rds, Status Code: 400, Request ID: d93b7008-0506-443d-a849-dc1c42b49656)"

Note

The most common cause of this issue is an increase in database storage due to auto-scaling or manual update of DB storage directly in AWS console. Attempting to modify TED while in this state will result in an error.

Step 3: Review Changes

• Open the AWS Console and navigate to the RDS service in the region where TED is deployed.

• Find the DB Instance associated with your TED stack.

• Navigate to the DB Instance Configuration and check for Storage

• Compare the actual Configuration settings of the RDS instance with the CloudFormation Parameter values in the TED stack.

Step 4: Find Root Cause

Select CloudFormation stack Events tab

Select Detect root cause

Step 5: Check CloudFormation Stack

CloudFormation stack does not allow updates when the Update button is grayed out. This indicates that the stack is in a state that prevents updates. Any changes attempted during this state will fail.

Step 6: Fix CloudFormation Stack

To proceed with aligning the Service Catalog TED stack with the actual configuration of the RDS instance, fix the CloudFormation stack from UPDATE_ROLLBACK_FAILED to UPDATE_ROLLBACK_COMPLETE.

From Stack actions dropdown select Continue update rollback.

A new window titled Continue update rollback will appear. In the Advanced Troubleshooting section, select check the box next to HivePrimary under Resources to skip - optional section to rollback for that specific resource. Select the Continue update rollback button.

This should successfully transition the stack to Update Rollback Complete, making it available for further updates with the Update now activated.

Step 7: Execute Service Catalog Stack Update

Navigate to Service Catalog and update the TED product. Ensure the parameter values match exactly with the current RDS DB instance storage setting. For instance, if RDS storage auto-scaled or manually updated from 200 GB to 225 GB, update the Service Catalog product's TED stack Allocated Storage in GB field to 225 GB to reflect the actual RDS value.

Initiate Service Catalog TED Update. CloudFormation will check for changes. If there are no discrepancies, the stack will transition to Update Complete.

Step 8: Review CloudFormation Stack

If you continue to encounter issues, please Open Support Ticket and attach the relevant information to assist you more efficiently.

• A screenshot of the CloudFormation Events tab for the TED stack.
• A screenshot of the Service Catalog with provisioned products.