Depot - Additional latency in GitHub job queue times – Incident details

All systems operational

Additional latency in GitHub job queue times

Resolved
Degraded performance
Started 6 months agoLasted about 5 hours

Affected

GitHub Actions

Degraded performance from 3:23 PM to 8:00 PM

Depot-managed Actions Runners

Degraded performance from 3:23 PM to 8:00 PM

Github.com - Actions

Updates
  • Resolved
    Resolved

    GitHub seems to be generally delivering jobs in the expected amount of time, marking this as resolved.

  • Monitoring
    Monitoring

    We have deployed a workaround that periodically causes GitHub to re-evaluate its scheduling logic to force jobs to be assigned to runners. We will continue to monitor as GitHub fixes the issue upstream.

  • Identified
    Update

    We have pinned down the issue as a regression in the GitHub API that assigns work to an online runner. We are working on a workaround to trigger this loop from the outside.

  • Identified
    Update

    We have narrowed the issue down to a race condition in GitHub's API launching runners.

  • Identified
    Identified

    We have narrowed the issue down to a race condition in GitHub's API launching new runners.