Additional latency in GitHub job queue times

Resolved
Degraded performance
Started 16 days ago Lasted about 5 hours

Affected

Depot-managed GitHub Actions Runners
Github - Actions
Updates
  • Resolved
    Resolved

    GitHub seems to be generally delivering jobs in the expected amount of time, marking this as resolved.

  • Monitoring
    Monitoring

    We have deployed a workaround that periodically causes GitHub to re-evaluate its scheduling logic to force jobs to be assigned to runners. We will continue to monitor as GitHub fixes the issue upstream.

  • Identified
    Update

    We have pinned down the issue as a regression in the GitHub API that assigns work to an online runner. We are working on a workaround to trigger this loop from the outside.

  • Identified
    Update

    We have narrowed the issue down to a race condition in GitHub's API launching runners.

  • Identified
    Identified

    We have narrowed the issue down to a race condition in GitHub's API launching new runners.