Depot - GitHub Actions Outage – Incident details

GitHub Actions Outage

Resolved
Partial outage
Started about 2 months agoLasted about 3 hours

Affected

GitHub Actions

Github.com - Actions

Updates
  • Resolved
    Resolved

    All jobs are able to start on Depot runners. GitHub have announced that they have completed their failover, though job queue times may remain longer than usual.

  • Monitoring
    Update

    We are observing most Actions jobs able to start on Depot runners. We'll continue to monitor as GitHub recovers.

  • Monitoring
    Update

    GitHub continues to work on their region failover: https://www.githubstatus.com/incidents/m70hk23gx3nx.

    Some Actions jobs are able to start on Depot runners, however most API requests to the Actions API to retrieve work are failing with rate limit errors, it appears both the GitHub API and Actions control plane are overloaded by the effects of the outage.

  • Monitoring
    Monitoring

    GitHub is reporting that they are failing over to a secondary region. This appears to be caused by an outage of the Azure Central US region: https://azure.status.microsoft/en-us/status

  • Identified
    Identified

    We are still monitoring GitHub's outage. We have launched an Actions runner for each queued Actions job, and some jobs have been able to start, however many of the runners are online waiting for GitHub to send them work.

    Once GitHub resolves the outage upstream, we expect all queued jobs to be sent to the online runners.

  • Investigating
    Investigating

    GitHub is reporting an incident where they are unable to start Actions jobs: https://www.githubstatus.com/incidents/m70hk23gx3nx