Delayed handling of events

Incident Report for Spacelift

Resolved

Our monitors are reporting that all events have been processed.
The issue has been resolved.
Posted Oct 30, 2024 - 16:53 UTC

Identified

We have identified an issue with our object storage abstraction, we are rolling back to a previous release.
Posted Oct 30, 2024 - 16:28 UTC

Monitoring

We have introduced a quick fix for handling events.
Posted Oct 30, 2024 - 16:13 UTC

Investigating

We are seeing an increased error rate when handling webhooks for all customers.
This is causing a delay in runs being scheduled.
Posted Oct 30, 2024 - 15:40 UTC
This incident affected: Event processing and Public workers.