The backfill job to restore links between artifacts and their originating runs has completed successfully.
New artifacts created between May 14th, 2 PM PST and May 16th, 12 PM PST should now be restored with links to their originating runs. Artifacts created before or after this period should remain unaffected by this incident.
Thank you for your patience as we addressed this issue. We will continue to monitor closely and are working on improving safeguards to prevent similar incidents in the future.
Posted May 19, 2025 - 20:59 PDT
Update
A job to backfill the links between artifacts and their originating runs is still ongoing.
This now affects only artifacts that were created on May 16, between 3AM - 12PM PST. Artifacts created before that period have been restored to their runs, while artifacts created after that period remain unaffected. We'll provide another update by 12AM PST today.
Posted May 19, 2025 - 18:06 PDT
Update
A job to backfill the links between artifacts and their originating runs is taking longer than previously expected to complete. This impacts only artifacts created between May 14th, 2 PM PST to May 16th, 12 PM PST, though some artifacts from that period have already been restored.
Any new artifacts logged since May 16th, 12 PM PST -- and onward -- remain unaffected and fully functional. We will share another update by 6PM PST today.
Posted May 19, 2025 - 12:59 PDT
Monitoring
We are seeing gradual restoration of affected artifacts created between May 14th, 2 PM PST to May 16th, 12 PM PST as the recovery continues.
Complete restoration is expected to take several more hours, and we'll continue to monitor progress on the recovery.
Posted May 16, 2025 - 19:38 PDT
Update
Newly created artifacts after May 16th 12PM and artifacts created before May 14th 2PM remain fully functional.
We are currently attempting to restore full functionality to affected artifacts committed between May 14th 2PM PST to May 16th 12 PM PST. We expect to have an update on the functionality of these affected artifacts later tonight.
Posted May 16, 2025 - 18:17 PDT
Update
For all new artifacts, full functionality should be restored. Artifacts created before this fix may not yet be linked to the runs that created them. We are actively working on restoring these links and will continue to provide further updates on progress.
Posted May 16, 2025 - 13:17 PDT
Update
Partial (core) functionality should be restored. A fix to restore full functionality is still ongoing. During this time, new artifacts will not be immediately linked to the runs that created them. These links will be restored on completion of the fix. We'll continue to provide updates on progress.
Posted May 16, 2025 - 08:59 PDT
Identified
Partial (core) functionality should be restored. A fix to restore full functionality is ongoing. We'll continue to provide updates on progress.
Posted May 15, 2025 - 16:12 PDT
Update
Partial (core) functionality should be restored, and investigation is going on to enable remaining functionality.
Posted May 15, 2025 - 09:55 PDT
Update
We are continuing to investigate this issue.
Posted May 14, 2025 - 18:44 PDT
Investigating
We are investigating an issue where recently uploaded artifacts are inaccessible via the UI or API.