[Quickbase US Status] Notice: Service Not Available (Unplanned) - Down Time

Incident
04/08/2023, 12:00pm EDT

[Quickbase US Status] Notice: Service Not Available (Unplanned) - Down Time

Status: closed
Start: 04/03/2023, 08:38am EDT
End: 04/03/2023, 09:04am EDT
Duration: 26 minutes
Affected Components:
Quickbase Service - US Region Quickbase Audit Logs - US Region Quickbase Automations - US Region Quickbase Billing - US Region Quickbase Pipelines - US Region Quickbase Platform Analytics - US Region Quickbase RESTful APIs - US Region Quickbase Sync - US Region Quickbase Webhooks - US Region
Update

04/03/2023, 08:38am EDT

04/03/2023, 08:38am EDT

As of 8:38 AM Eastern US Time, the Quickbase US platform is unavailable.  We are actively investigating and will provide an update within 30 minutes.

Update

04/03/2023, 09:00am EDT

04/03/2023, 09:00am EDT

As of 9:00 AM Eastern US Time, the Quickbase US platform is recovering.  We will post an update within 30 minutes.

Resolved

04/03/2023, 09:04am EDT

04/03/2023, 09:04am EDT

As of 9:04 AM Eastern US Time, request traffic levels to the Quickbase US platform have returned to normal but some requests may still be slow for less than 1% of applications.  See this post for more information on the performance issue impacting less than 1% of applications.

This incident is closed.

Root Cause

04/08/2023, 12:00pm EDT

04/08/2023, 12:00pm EDT

All times shown are Eastern US Time.

The Quickbase platform consists of several software services and databases which together provide the functionality on which our customers rely.  One of the software services routes requests within the platform, and that routing service crashed at 8:38 AM on Monday, April 3, 2023.  It is configured to automatically restart and did so at 8:44 AM.  Upon restart of the routing service, all Quickbase applications reload.  During that reload time, requests can be slow or result in error.  By 9:04 AM, the platform was performing normally with the exception of less than 1% of applications.  Those few applications underperformed until 10:10 AM as noted in this separate service page post.

Our analysis of the incident revealed that the routing service crashed due to a defect in the software code that evaluates and logs a specific error condition that can occur while the platform is operating.  We have not seen this defect in the 20+ year history of the routing service.  We made a configuration change to the routing service to prevent this specific issue from reoccurring.  Work has been underway for more than a year to replace the legacy routing service with a more resilient service which we expect to deploy later this year.

We realize we are an important part of your business and you expect Quickbase to be available and performing well when you need it. We apologize for the disruption this incident caused our customers.