[Quickbase US Status] Notice: Service Functionality Degraded (Unplanned)

Incident

07/26/2024, 04:43pm EDT

[Quickbase US Status] Notice: Service Functionality Degraded (Unplanned)

Status: closed

Start: 06/20/2024, 02:34pm EDT

End: 06/20/2024, 04:15pm EDT

Duration: 1 hour 41 minutes

Affected Components:

Quickbase Service - US Region Quickbase Audit Logs - US Region Quickbase Automations - US Region Quickbase Billing - US Region Quickbase Pipelines - US Region Quickbase Platform Analytics - US Region Quickbase RESTful APIs - US Region Quickbase Sync - US Region Quickbase Webhooks - US Region

Update

06/20/2024, 02:34pm EDT

Starting around 2:34 PM Eastern US Time, some users interacting with the Quickbase platform may experience performance degradation or timeouts. We are investigating this issue and will provide another update as soon as we have one.

Update

06/20/2024, 03:00pm EDT

We're actively engaged with our vendor who has acknowledged an issue on their end. We are working on a resolution as soon as possible.

We will provide another update as soon as we have one.

Update

06/20/2024, 03:30pm EDT

We are continuing to work with our vendor on this issue. We will provide another update as soon as we have one.

Update

06/20/2024, 04:00pm EDT

We're seeing improvements in performance in the platform. Our vendor has indicated a fix will be deployed shortly.

We will provide another update as soon as we have one.

Update

06/20/2024, 04:06pm EDT

Our vendor has confirmed a fix has been deployed. We are continuing to monitor the platform.

Resolved

06/20/2024, 04:15pm EDT

Performance is back to normal. We will provide a root cause once we have it.

This incident is closed.

Root Cause

07/26/2024, 04:43pm EDT

Our DDOS protection provider experienced unexpected congestion in their network backbone. During mitigation of this incident, a bug in their Distributed Denial-of-Service (DDOS) rule incorrectly routed the traffic causing an infinite loop – prolonging the incident.

Their network team identified the issue and ultimately resolved by disabling the faulty DDOS rule and restarting the faulty services.

Additional monitoring has been implemented as well as design enhancements to prevent this issue from recurring.

We apologize for the inconvenience this issue caused.