Depot - Increased queue times for Actions jobs – Incident details

All systems operational

Increased queue times for Actions jobs

Resolved
Degraded performance
Started about 12 hours agoLasted about 8 hours

Affected

GitHub Actions

Operational from 10:00 PM to 1:22 AM, Degraded performance from 1:22 AM to 6:20 AM

Depot-managed Actions Runners

Operational from 10:00 PM to 1:22 AM, Degraded performance from 1:22 AM to 6:20 AM

Updates
  • Resolved
    Resolved

    Queue times have returned to normal for those runner types. It appears to have only impacted high-volume GitHub Actions orgs and appears to be the fallout of the GitHub Actions outage reported earlier today: https://www.githubstatus.com/incidents/crd2y6xy6knn

  • Monitoring
    Monitoring

    We have implemented a workaround caused by the GitHub outage that is causing extended queue times for 8 & 16 CPU runners. We are monitoring the results and checking on queue times across the system.

  • Update
    Update

    We have identified the root cause as the GitHub Actions outage they had earlier. We have reduced the blast radius of long queue times to 8 and 16 core CPU runners for Ubuntu 24.04. We are working to bring that runner type back within normal queue times.

  • Identified
    Identified

    We've identified the cause of the issue, and are working on a fix.

  • Investigating
    Investigating

    We are currently investigating an increase in queue times for some Actions jobs.