[Quickbase US Status] Notice: Service Unavailable (Unplanned) - Down Time

Incident

03/07/2025, 08:30am EST

[Quickbase US Status] Notice: Service Unavailable (Unplanned) - Down Time

Status: closed

Start: 01/27/2025, 05:45am EST

End: 01/27/2025, 07:05am EST

Duration: 1 hour 20 minutes

Affected Components:

Quickbase Service - US Region Quickbase Audit Logs - US Region Quickbase Automations - US Region Quickbase Billing - US Region Quickbase Pipelines - US Region Quickbase Platform Analytics - US Region Quickbase RESTful APIs - US Region Quickbase Sync - US Region Quickbase Webhooks - US Region

Update

01/27/2025, 05:45am EST

Due to a problem with the new Quickbase load balancer (QLB) that is causing some Quickbase applications to not load correctly, we need to do emergency maintenance which will result in down time between 5:45 AM and 6:45 AM Eastern US Time.

We will post updates to this notice as the maintenance progresses.

Update

01/27/2025, 06:05am EST

As of 6:05 AM Eastern US Time, we are continuing to evaluate the problem that led us to shut down Quickbase US at 5:45 AM Eastern US Time.

We will provide another update at 6:20 AM Eastern US Time.

Update

01/27/2025, 06:20am EST

As of 6:20 AM Eastern US Time, we are starting the process of restarting Quickbase US.

We will provide another update at 6:35 AM Eastern US Time.

We now expect this emergency down time to last until 7:00 AM Eastern US Time.

Update

01/27/2025, 06:40am EST

As of 6:40 AM Eastern US Time, the restart of Quickbase US continues. We are hoping to open the platform to customers by 7:00 AM Eastern US Time.

We will provide another update at 7:00 AM Eastern US Time.

Resolved

01/27/2025, 07:05am EST

Quickbase US was available as of 6:47 AM Eastern US Time. We expect performance to be slower than normal for a few minutes following a restart of the platform. As of 7:05 AM Eastern US Time, Quickbase US performance returned to normal.

This incident is closed.

We will post a root cause once we have it.

Root Cause

03/07/2025, 08:30am EST

All times shown are Eastern US Time.

On January 27, between 12:22 AM and 5:45 AM, the potential existed for less than 1% of Quickbase US customers to receive the error "Your Quickbase app is temporary unavailable" with the HTTP error code of 503.

The root cause of this incident was a software defect in a hygienic process named “application restart process” that restarts each running Quickbase application at least once every 48-96 hours. Some apps failed to shut down properly and hence could not be restarted. The “application restart process” has been in place for several years and is controlled by the Quickbase Load Balancer. The Quickbase Load Balancer, or QLB, is responsible for routing all requests made to the platform to the appropriate computers for processing. We upgraded QLB on Saturday, January 25 during planned maintenance. The new version of QLB introduced the software defect in the “application restart process”. We did not discover the defect in our prior testing of QLB because we did not run it for more than 48 hours in any single test.

In order to resolve this incident in the short term, we made the decision to restart the Quickbase US platform with the aforementioned “application restart process” disabled, which occurred between 5:45 AM and 7:05 AM.

We implemented a fix for the “application restart process” defect on January 30 at 7:35 PM, and re-enabled the process at that time. No further issues with the process have been detected in the 5 weeks since.

We realize we are an important part of your business and you expect Quickbase to be available and performing well when you need it. We apologize for the disruption this incident caused our customers.