[Quickbase US Status] Notice: Service Unavailable (Unplanned) - Down Time

Incident

03/28/2025, 01:15pm EDT

[Quickbase US Status] Notice: Service Unavailable (Unplanned) - Down Time

Status: closed

Start: 03/22/2025, 09:11am EDT

End: 03/22/2025, 09:56am EDT

Duration: 45 minutes

Affected Components:

Quickbase Service - US Region Quickbase Audit Logs - US Region Quickbase Automations - US Region Quickbase Billing - US Region Quickbase Pipelines - US Region Quickbase Platform Analytics - US Region Quickbase RESTful APIs - US Region Quickbase Sync - US Region Quickbase Webhooks - US Region

Update

03/22/2025, 09:11am EDT

As of 9:11 AM Eastern US Time, the Quickbase US platform is unavailable. We are actively working to restore service. We will post another update by 9:45 AM.

Update

03/22/2025, 09:40am EDT

As of 9:40 AM Eastern US Time, Quickbase US is available. However, performance will be slower than normal for up to 30 minutes. We will post another update by 10:00 AM.

Update

03/22/2025, 09:55am EDT

As of 9:55 AM Eastern US Time, the Quickbase US platform is almost at normal performance. We will post another update by 10:15 AM.

Resolved

03/22/2025, 09:56am EDT

As of 9:56 AM Eastern US Time, Quickbase US performance has returned to normal.

This incident is closed.

We will post a root cause once we have it.

Root Cause

03/28/2025, 01:15pm EDT

All times shown are Eastern US Time.

First, we realize we are an important part of your business and you expect Quickbase to be available and performing well when you need it. We apologize for the disruption this incident caused our customers.

On Saturday, March 22, starting at 8:15 AM, Quickbase was conducting planned maintenance on the software components that are part of the Quickbase load balancer (QLB) supporting the Quickbase US platform. The maintenance included updates to the software and a restart of each component. Quickbase routinely updates all the software components of the platform and these updates typically have little to no impact on customers actively using the platform. The QLB maintenance was expected to complete at 9:15 AM.

Towards the end of the maintenance, an unexpected problem developed. Between 9:11 AM and 9:14 AM, the majority of Quickbase US customer requests returned an error, HTTP 503, with text similar to "Your Quickbase app is temporarily unavailable" or "Quickbase is temporarily unavailable due to technical difficulties."

We detected this situation immediately, and at about 9:14 AM, we put the Quickbase US platform behind a maintenance page thereby intentionally blocking customer access. The message customers saw was:

Quickbase is Temporarily Unavailable

Quickbase is currently unavailable. We are working to have services back up as soon as possible. For more information, please see the Quickbase service page.

We investigated and fixed the problem, and re-opened access to Quickbase US at 9:40 AM. Between 9:40 AM and 9:56 AM, customers may have experienced slower than normal response times while the platform warmed up. This is normal and expected after a cold restart of the platform. By 9:56 AM, Quickbase US performance had returned to normal and the incident ended.

The root cause of this incident was a defect in one of the QLB software components. The defect was exposed due to a combination of the volume of traffic, and diversity of traffic in the Quickbase US platform. We did not discover the defect in our prior testing of QLB, nor did it materialize in the Quickbase EU platform.

In order to resolve this incident in the short term, while Quickbase US was behind the maintenance page, we rolled back the applicable QLB software component to the prior, known good, version. We are working on a fix for the bug, and considering different ways we can better recreate the traffic pattern that exposed the bug, in our testing environment.