Depot - Increased latency in picking up GitHub Actions jobs – Incident details

All systems operational

Increased latency in picking up GitHub Actions jobs

Resolved
Partial outage
Started 4 months agoLasted about 5 hours

Affected

GitHub Actions

Degraded performance from 7:17 PM to 8:38 PM, Operational from 8:38 PM to 10:35 PM, Partial outage from 10:35 PM to 12:16 AM

Depot-managed Actions Runners

Degraded performance from 7:17 PM to 8:38 PM, Operational from 8:38 PM to 10:35 PM, Partial outage from 10:35 PM to 12:16 AM

Github.com - Actions

Updates
  • Resolved
    Resolved

    All queue times have returned to normal, marking as resolved.

  • Monitoring
    Monitoring

    We are seeing increased queue times as GitHub processes through the backlog of API requests.

  • Resolved
    Resolved

    New GitHub jobs are operational. GitHub does not appear to be re-queuing jobs that were unable to start during their downtime.

  • Monitoring
    Update

    All new GitHub Actions jobs are starting successfully. Jobs older than 40 minutes ago are still waiting on GitHub to process the backlog of jobs from the outage.

  • Monitoring
    Update

    We are observing new jobs starting, though GitHub has not yet processed all older jobs.

  • Monitoring
    Monitoring

    GitHub has opened an incident of its own that we are monitoring: https://www.githubstatus.com/incidents/f7jl0mdd2jr5

  • Identified
    Update

    We've identified the issue as GitHub webhooks not being delivered. We are monitoring their recovery. Jobs continue to start, but you may see a delay if GitHub continues to lag in delivering webhooks.

  • Identified
    Identified

    We believe this is a potential outage at GitHub as we are observing jobs actually processing and running, but we are not receiving webhooks from GitHub to tell us that.

  • Investigating
    Investigating

    We are currently investigating longer queue times to pick up GHA jobs.