[Quickbase US Status] Notice: Service Functionality Degraded (Unplanned)

Incident
09/27/2024, 04:28pm EDT

[Quickbase US Status] Notice: Service Functionality Degraded (Unplanned)

Status: closed
Start: 09/26/2024, 03:00pm EDT
End: 09/27/2024, 04:28pm EDT
Duration: 1 day 1 hour 28 minutes
Affected Components:
Quickbase Service - US Region Quickbase Service - EU Region Quickbase Pipelines - EU Region Quickbase Pipelines - US Region
Update

09/26/2024, 03:00pm EDT

09/26/2024, 03:00pm EDT

We are actively investigating an issue where some customers are receiving intermittent errors connecting to the Outlook channel.  After investigation, there are no changes to Pipelines services that correlate to the timing of these events.  Currently, we believe this might be related to a Microsoft service bulletin under ID MO896989.  Quickbase is continuing to investigate and deploying additional monitoring to check for this issue. We will provide an update by Friday, September 27, 2024 at 9:00 AM ET.

Update

09/27/2024, 08:59am EDT

09/27/2024, 08:59am EDT

We are continuing to investigate this issue and have not found any issues related to Pipelines. Currently this is impacting ~15% of outbound Outlook channel traffic.

Update

09/27/2024, 01:12pm EDT

09/27/2024, 01:12pm EDT

Quickbase has deployed additional logging elements to isolate the issue and convey additional information to Microsoft. We will provide an update by Friday, September 27, 2024 at 4:00 PM ET.

Update

09/27/2024, 02:48pm EDT

09/27/2024, 02:48pm EDT

We have begun receiving reports of similar behavior on other channels where, inconsistently, responses are being received about messaging formatting being in an incorrect type. We are continuing to investigate these issues both as it relates to Microsoft services and our own infrastructure.

Update

09/27/2024, 03:01pm EDT

09/27/2024, 03:01pm EDT

We are currently rolling out an update to the Pipelines infrastructure to mitigate this issue and will be monitoring for improvements.

Update

09/27/2024, 03:18pm EDT

09/27/2024, 03:18pm EDT

We are seeing errors subside with our most recent deployment. We will continue to monitor before resolving this issue.

Resolved

09/27/2024, 04:28pm EDT

09/27/2024, 04:28pm EDT

At this time, our logging continues to show reduced error volume and we no longer can reproduce this issue. Over the coming weeks, we will be performing a root cause analysis (RCA). A conclusion will be shared on this service posting once available.