Depot - Builds unable to start – Incident details

All systems operational

Builds unable to start

Resolved
Major outage
Started 2 months agoLasted about 3 hours

Affected

Container builds

Degraded performance from 1:44 PM to 1:57 PM, Operational from 1:44 PM to 4:20 PM, Major outage from 1:57 PM to 2:59 PM, Degraded performance from 2:59 PM to 4:20 PM

us-east-1

Degraded performance from 1:44 PM to 1:57 PM, Major outage from 1:57 PM to 2:59 PM, Degraded performance from 2:59 PM to 4:20 PM

eu-central-1

Operational from 1:44 PM to 4:20 PM

GitHub Actions

Degraded performance from 1:44 PM to 1:57 PM, Major outage from 1:57 PM to 2:59 PM, Degraded performance from 2:59 PM to 4:16 PM, Operational from 4:16 PM to 4:20 PM

Depot-managed Actions Runners

Degraded performance from 1:44 PM to 1:57 PM, Major outage from 1:57 PM to 2:59 PM, Degraded performance from 2:59 PM to 4:16 PM, Operational from 4:16 PM to 4:20 PM

Updates
  • Resolved
    Update

    Build service is restored. We will continue monitoring.

  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Update

    We have deployed a cache fix for Docker image builds that will result in older projects using an empty cache for their first build.

  • Monitoring
    Update

    AWS has opened an incident on their side https://status.aws.amazon.com/#multipleservices-us-east-1_1720795596.

    We are observing GHA builds processing as expected now but are working on deploying the fix for Docker image builds.

  • Monitoring
    Monitoring

    We are monitoring as we process the backlog of queued jobs - the longest queue time we are now observing is less than 5 minutes.

  • Identified
    Identified

    We have deployed a mitigation for the EC2 API bug, we are starting to see some build service restored.

  • Investigating
    Update

    We have identified the outage as being related to AWS EC2 tag propagation on launch. We're working on a fix.

  • Investigating
    Investigating

    We are investigating some builds being unable to start