Resolved -
All messages in the backlog have been processed and we are back to sub-second file processing times. We apologize again for the inconvenience and appreciate your patience as we worked to resolve the issue.
Jan 25, 01:01 PST
Update -
We're still seeing a large volume of updates, so files are still taking an average of 30 minutes to process. We are currently deploying changes to make our file processing more efficient.
Jan 24, 23:25 PST
Update -
We have scaled up our file ingestion pipeline and are making progress through the backlog. We should be fully caught up in around 30 minutes.
Jan 24, 18:45 PST
Identified -
A large influx of files is causing a delay in processing new files. This should cause no disruption to existing runs or uploaded files, but you might experience delays before the files are visible in the UI.
We apologize for the inconvenience. We are working to resolve the issue as soon as possible.
Jan 24, 18:07 PST