One faulty GPU worker failed to process requests

Resolved

Operational

Started over 1 year agoMarch 28, 2025Lasted about 8 hoursMarch 28, 20255:01 PMMarch 29, 20251:10 AMUTC

Affected

API

Apps

Web app

Updates

Resolved
March 29, 2025 at 1:10 AMUTC
Resolved
March 29, 2025 at 1:10 AMUTC
- One GPU worker went into a bad state, it was restarted and returned to normal operation
- A mechanism to detect and auto-restart such states was developed and deployed.
Investigating
March 28, 2025 at 5:01 PMUTC
Investigating
March 28, 2025 at 5:01 PMUTC
Some of the requests to the nightly endpoint (experimental in app) are returning errors

FASHN - One faulty GPU worker failed to process requests – Incident details