[Quickbase US Status] Notice: Service Unavailable (Unplanned) - Down Time
[Quickbase US Status] Notice: Service Unavailable (Unplanned) - Down Time
01/30/2025, 07:13pm EST
01/30/2025, 07:13pm EST
Between 7:13 PM and 7:35 PM Eastern US Time, the Quickbase US platform was unavailable due to an unexpected issue resulting from emergency maintenance that began at 7:10 PM Eastern US Time.
We began sending traffic to our maintenance page at 7:18 PM Eastern US Time.
Quickbase US was available as of 7:23 PM Eastern US Time. It is normal for performance to be slower than normal for a few minutes following a restart of the platform. As of 7:35 PM Eastern US Time, Quickbase US performance returned to normal.
We apologize for any inconvenience this unplanned down time may have caused.
01/30/2025, 07:35pm EST
01/30/2025, 07:35pm EST
This incident is closed.
We will post a root cause once we have it.
03/07/2025, 08:30am EST
03/07/2025, 08:30am EST
First, we realize we are an important part of your business, and you expect Quickbase to be available and performing well when you need it. We apologize for the disruptions that have occurred between January 27 and February 2. We have not met the standard you expect of us, or that we expect of ourselves.
Please read on for an explanation of this specific incident.
All times shown are Eastern US Time
On Thursday, January 30, between 7:10 PM and 8:00 PM, the Quickbase team intended to conduct emergency maintenance to upgrade a component service of the Quickbase load balancer supporting the Quickbase US platform. The maintenance was expected to cause limited degradation of functionality as described in this service page post. Instead, early in the upgrade process, a problem was encountered which triggered this incident.
The root cause of this incident was a defect in the configuration of the service being upgraded. The upgrade process is designed to prioritize the integrity and consistency of your application data. When the upgrade process encountered the configuration defect at 7:13 PM, by design, it automatically shut down all instances of Quickbase’s proprietary in-memory database engine within one minute. This resulted in all Quickbase applications becoming unavailable and the platform was effectively unusable. The deployment process is then designed to automatically restart the instances of the in-memory database engine, which it properly did starting at 7:15 PM. We refer to this as a cold restart of the platform. The platform requires a warm-up period of about 20 minutes. In the first 5-10 minutes of that period, we normally have the platform closed to customers because it is too slow to effectively use. After triaging what was happening, at 7:18 PM, we closed access to Quickbase US and all customer requests would have returned a standard maintenance page indicating the platform was unavailable. By 7:23 PM, ten minutes after the start of the incident, we opened access to Quickbase US, and by 7:35 PM, after the warm-up period fully completed, Quickbase US was performing normally and the incident ended.
After the incident ended on Thursday, we discovered the defect in the configuration. In order to correct the defect, we have scheduled emergency down time maintenance for Sunday, February 2, between 6:10 AM and 8:00 AM. See this service page post for more information. It is essential that this maintenance occur quickly because all software services that make up the Quickbase platform must be able to be updated without causing unplanned down time, and all of the services are normally updated several times per year to enhance the functionality of the platform, fix defects, or to maintain a strong security posture.