We have fully caught up to the backlog of messages, and new files should be visible in the UI within a few seconds of uploading.
We really appreciate your patience and understanding during the long-running issue.
Posted Apr 13, 2023 - 13:16 PDT
Identified
We've identified and resolved the underlying issue. We're currently working through the backlog of messages.
Posted Apr 13, 2023 - 11:23 PDT
Update
Unfortunately the issue has gotten worse overnight. We are up to several hours worth of delay before ingestion. We have engaged GCP support and are trying to address the underlying infrastructure issue.
In the meantime, we are rolling out a new ingestion pathway for new files while we work on the backlog of unprocessed files. We will post updates here as we make progress.
We understand the frustration and apologize for the trouble this is causing. We are working as quickly as we can to resolve the issue.
Posted Apr 13, 2023 - 08:44 PDT
Update
We're continuing to look into this issue. Messages are failing to acknowledge in time despite successful processing. We've tuned our redelivery settings, which should lower the time to process files while we investigate the underlying infrastructure issue with our cloud provider.
Posted Apr 12, 2023 - 15:52 PDT
Investigating
File processing times are elevated due to an underlying issue with our infrastructure. We are working closely with GCP support to resolve the problem.
Unfortunately, this means that files can take up to an hour to render in the UI. We sincerely apologize for the inconvenience.