[Quickbase US Status] Notice: Service Performance Degraded (Unplanned)

Incident
12/18/2024, 09:42am EST

[Quickbase US Status] Notice: Service Performance Degraded (Unplanned)

Status: closed
Start: 12/10/2024, 05:30pm EST
End: 12/10/2024, 08:20pm EST
Duration: 2 hours 50 minutes
Affected Components:
Quickbase Service - US Region Quickbase Pipelines - US Region
Update

12/10/2024, 05:30pm EST

12/10/2024, 05:30pm EST

Starting around 5:15 PM Eastern US Time, we are experiencing slow performance with Quickbase Pipelines.  We are actively investigating this issue and will provide another update as soon as we have one.

Update

12/10/2024, 06:25pm EST

12/10/2024, 06:25pm EST

Our vendor has acknowledged an issue on their end.  We are actively working with them to fix this issue as soon as possible.  We apologize for the inconvenience this issue may have caused.

Update

12/10/2024, 07:14pm EST

12/10/2024, 07:14pm EST

Our vendor is continuing to work to fix this issue as soon as possible.  We will provide another update as soon as we have one.

Update

12/10/2024, 08:14pm EST

12/10/2024, 08:14pm EST

We're seeing improvements in performance around 8 PM Eastern US Time.  We are still actively monitoring and working with our vendor on this issue.  We will provide another update as soon as we have one.

Resolved

12/10/2024, 08:20pm EST

12/10/2024, 08:20pm EST

Our vendor has confirmed a fix has been deployed on their end.  We will provide a root cause analysis once it is available.

This incident is closed. 

Root Cause

12/18/2024, 09:42am EST

12/18/2024, 09:42am EST

Our service provider for Quickbase Pipelines deployed a bug in their load balancing software that incorrectly allocated computing resources to a subset of their services.  Accordingly, some services were insufficiently provisioned and performance degradation was observed.  Their team was alerted of this issue and ultimately rolled back the deployment to fix this issue.  Additional monitoring has been implemented as well as deployment enhancements to prevent this issue from recurring.   

Our customers using Quickbase Pipelines during this incident may have experienced slower performance with their Pipelines.  We apologize for any inconvenience this issue may have caused. All Pipelines were executed through retry logic.

 

This incident is closed.