Depot - Increased latency and build failures – Incident details

All systems operational

Increased latency and build failures

Resolved
Partial outage 30 %
Started 22 days agoLasted about 1 hour

Affected

Container builds

Partial outage from 6:41 PM to 7:56 PM

us-east-1

Partial outage from 6:41 PM to 7:56 PM

GitHub Actions

Degraded performance from 6:41 PM to 7:26 PM, Partial outage from 7:26 PM to 7:36 PM, Operational from 7:36 PM to 7:56 PM

Depot-managed Actions Runners

Degraded performance from 6:41 PM to 7:26 PM, Partial outage from 7:26 PM to 7:36 PM, Operational from 7:36 PM to 7:56 PM

Updates
  • Resolved
    Resolved

    Container build service is restored.

  • Monitoring
    Monitoring

    GitHub Actions jobs are processing normally. We are continuing to monitor recovery of container builds in us-east-1.

  • Identified
    Identified

    We are seeing service begin to restore for both container builds and GitHub Actions jobs as capacity rebalances. We're continuing to monitor.

  • Investigating
    Investigating

    We are currently investigating increased latency in provisioning build infrastructure that is delaying GitHub Actions job starts and preventing some container build jobs from acquiring machines.