[Quickbase US Status] Notice: Service Unavailable (Unplanned) - Down Time

Incident
03/07/2025, 08:30am EST

[Quickbase US Status] Notice: Service Unavailable (Unplanned) - Down Time

Status: closed
Start: 01/30/2025, 07:13pm EST
End: 01/30/2025, 07:35pm EST
Duration: 22 minutes
Affected Components:
Quickbase Service - US Region Quickbase Audit Logs - US Region Quickbase Automations - US Region Quickbase Billing - US Region Quickbase Pipelines - US Region Quickbase Platform Analytics - US Region Quickbase RESTful APIs - US Region Quickbase Sync - US Region Quickbase Webhooks - US Region
Update

01/30/2025, 07:13pm EST

01/30/2025, 07:13pm EST

Between 7:13 PM and 7:35 PM Eastern US Time, the Quickbase US platform was unavailable due to an unexpected issue resulting from emergency maintenance that began at 7:10 PM Eastern US Time.

We began sending traffic to our maintenance page at 7:18 PM Eastern US Time.

Quickbase US was available as of 7:23 PM Eastern US Time.  It is normal for performance to be slower than normal for a few minutes following a restart of the platform.  As of 7:35 PM Eastern US Time, Quickbase US performance returned to normal.

We apologize for any inconvenience this unplanned down time may have caused.

Resolved

01/30/2025, 07:35pm EST

01/30/2025, 07:35pm EST

This incident is closed.

We will post a root cause once we have it.

Root Cause

03/07/2025, 08:30am EST

03/07/2025, 08:30am EST

First, we realize we are an important part of your business, and you expect Quickbase to be available and performing well when you need it.  We apologize for the disruptions that have occurred between January 27 and February 2.  We have not met the standard you expect of us, or that we expect of ourselves.

Please read on for an explanation of this specific incident.

All times shown are Eastern US Time

On Thursday, January 30, between 7:10 PM and 8:00 PM, the Quickbase team intended to conduct emergency maintenance to upgrade a component service of the Quickbase load balancer supporting the Quickbase US platform.  The maintenance was expected to cause limited degradation of functionality as described in this service page post.  Instead, early in the upgrade process, a problem was encountered which triggered this incident.

The root cause of this incident was a defect in the configuration of the service being upgraded.  The upgrade process is designed to prioritize the integrity and consistency of your application data.  When the upgrade process encountered the configuration defect at 7:13 PM, by design, it automatically shut down all instances of Quickbase’s proprietary in-memory database engine within one minute.  This resulted in all Quickbase applications becoming unavailable and the platform was effectively unusable.  The deployment process is then designed to automatically restart the instances of the in-memory database engine, which it properly did starting at 7:15 PM.  We refer to this as a cold restart of the platform.  The platform requires a warm-up period of about 20 minutes.  In the first 5-10 minutes of that period, we normally have the platform closed to customers because it is too slow to effectively use.  After triaging what was happening, at 7:18 PM, we closed access to Quickbase US and all customer requests would have returned a standard maintenance page indicating the platform was unavailable.  By 7:23 PM, ten minutes after the start of the incident, we opened access to Quickbase US, and by 7:35 PM, after the warm-up period fully completed, Quickbase US was performing normally and the incident ended.

After the incident ended on Thursday, we discovered the defect in the configuration.  In order to correct the defect, we have scheduled emergency down time maintenance for Sunday, February 2, between 6:10 AM and 8:00 AM.  See this service page post for more information.  It is essential that this maintenance occur quickly because all software services that make up the Quickbase platform must be able to be updated without causing unplanned down time, and all of the services are normally updated several times per year to enhance the functionality of the platform, fix defects, or to maintain a strong security posture.