Resolved -
The backfill job to restore links between artifacts and their originating runs has completed successfully.
New artifacts created between May 14th, 2 PM PST and May 16th, 12 PM PST should now be restored with links to their originating runs. Artifacts created before or after this period should remain unaffected by this incident.
Thank you for your patience as we addressed this issue. We will continue to monitor closely and are working on improving safeguards to prevent similar incidents in the future.
May 19, 20:59 PDT
Update -
A job to backfill the links between artifacts and their originating runs is still ongoing.
This now affects only artifacts that were created on May 16, between 3AM - 12PM PST. Artifacts created before that period have been restored to their runs, while artifacts created after that period remain unaffected. We'll provide another update by 12AM PST today.
May 19, 18:06 PDT
Update -
A job to backfill the links between artifacts and their originating runs is taking longer than previously expected to complete. This impacts only artifacts created between May 14th, 2 PM PST to May 16th, 12 PM PST, though some artifacts from that period have already been restored.
Any new artifacts logged since May 16th, 12 PM PST -- and onward -- remain unaffected and fully functional. We will share another update by 6PM PST today.
May 19, 12:59 PDT
Monitoring -
We are seeing gradual restoration of affected artifacts created between May 14th, 2 PM PST to May 16th, 12 PM PST as the recovery continues.
Complete restoration is expected to take several more hours, and we'll continue to monitor progress on the recovery.
May 16, 19:38 PDT
Update -
Newly created artifacts after May 16th 12PM and artifacts created before May 14th 2PM remain fully functional.
We are currently attempting to restore full functionality to affected artifacts committed between May 14th 2PM PST to May 16th 12 PM PST. We expect to have an update on the functionality of these affected artifacts later tonight.
May 16, 18:17 PDT
Update -
For all new artifacts, full functionality should be restored. Artifacts created before this fix may not yet be linked to the runs that created them. We are actively working on restoring these links and will continue to provide further updates on progress.
May 16, 13:17 PDT
Update -
Partial (core) functionality should be restored. A fix to restore full functionality is still ongoing. During this time, new artifacts will not be immediately linked to the runs that created them. These links will be restored on completion of the fix. We'll continue to provide updates on progress.
May 16, 08:59 PDT
Identified -
Partial (core) functionality should be restored. A fix to restore full functionality is ongoing. We'll continue to provide updates on progress.
May 15, 16:12 PDT
Update -
Partial (core) functionality should be restored, and investigation is going on to enable remaining functionality.
May 15, 09:55 PDT
Update -
We are continuing to investigate this issue.
May 14, 18:44 PDT
Investigating -
We are investigating an issue where recently uploaded artifacts are inaccessible via the UI or API.
May 14, 18:44 PDT
