Ingestion delay for run metrics
Incident Report for Weights and Biases
Resolved
This incident has been resolved.
Posted Sep 21, 2023 - 12:20 PDT
Update
We are continuing to monitor for any further issues.
Posted Sep 21, 2023 - 08:54 PDT
Monitoring
We have caught up with the backlog. All metrics should be visible from the UI and SDK.
Posted Sep 21, 2023 - 08:54 PDT
Identified
We are aware of an issue causing run metrics to not display in charts. We've identified the root cause as a burst of uploaded metrics and have scaled our ingestion workers up. We are rapidly catching up with the backlog and all metrics should be visible within a few minutes.
Posted Sep 21, 2023 - 08:29 PDT
This incident affected: Backend API.