Resolved
We have finished our monitoring phase, the outage is now fully resolved.
Monitoring
We have released a mitigation to restore Codex services while we await an outage update from AWS. Codex services are currently returning to a healthy state.
Identified
AWS has reported increased 5xx errors for Cloudfront customers, which is currently the root cause of the degraded Codex API performance.
Our team is treating this as an escalated all-hands outage, and working on restoring performance as soon as possible.
AWS service health can be found here: https://health.aws.amazon.com/health/status
Investigating
We're currently investigating increased error rates across Filter API endpoints.