Depot - Notice history

GitHub have reverted the bad deploy, jobs are running again.

All jobs are able to start on Depot runners. GitHub have announced that they have completed their failover, though job queue times may remain longer than usual.

We are observing most Actions jobs able to start on Depot runners. We'll continue to monitor as GitHub recovers.

GitHub continues to work on their region failover: https://www.githubstatus.com/incidents/m70hk23gx3nx.

Some Actions jobs are able to start on Depot runners, however most API requests to the Actions API to retrieve work are failing with rate limit errors, it appears both the GitHub API and Actions control plane are overloaded by the effects of the outage.

GitHub is reporting that they are failing over to a secondary region. This appears to be caused by an outage of the Azure Central US region: https://azure.status.microsoft/en-us/status

We are still monitoring GitHub's outage. We have launched an Actions runner for each queued Actions job, and some jobs have been able to start, however many of the runners are online waiting for GitHub to send them work.

Once GitHub resolves the outage upstream, we expect all queued jobs to be sent to the online runners.

GitHub jobs are processing normally.

We are still seeing some increased job queue times, but the backlog of jobs is processing.

GitHub have marked their incident as closed, but we have not yet seen full recovery of Actions webhooks. We will continue to monitor as they recover.

GitHub is now reporting a general outage, including for git operations: https://www.githubstatus.com/incidents/88003ngctf1z

Docker Hub issues appear to be resolved.

https://www.dockerstatus.com/pages/incident/533c6539221ae15e3f000031/6674ab18f483b5178cf6bdab

Docker has mitigated the issue. We still see some failures, but most builds are now succeeding. We will continue to monitor.

GitHub have marked the incident as resolved: https://www.githubstatus.com/incidents/lfrlwdg67fn8

Build service appears to be restored - we will continue to monitor GitHub's recovery. https://www.githubstatus.com/incidents/lfrlwdg67fn8

GitHub is reporting that they are in the process of applying a mitigation. We have not yet seen service restore. For more updates: https://www.githubstatus.com/incidents/lfrlwdg67fn8

GitHub have marked the incident as resolved.

We are observing all jobs able to successfully start. GitHub is still reporting that status updates are delayed for all Actions customers. https://www.githubstatus.com/incidents/nj47vccwm2zj

While GitHub are still investigating the incident, we are seeing some jobs able to complete, though their status is often not reflected correctly in the GitHub UI. https://www.githubstatus.com/incidents/nj47vccwm2zj

GitHub have identified the cause as network connectivity issues on their end, they are continuing to investigate: https://www.githubstatus.com/incidents/nj47vccwm2zj

GitHub have just opened an incident of their own for this outage: https://www.githubstatus.com/incidents/nj47vccwm2zj

This appears to be a GitHub Actions outage - both GitHub-hosted runners and Depot-hosted runners are currently experiencing issues starting jobs.

GitHub has opened an outage of its own that we are now monitoring https://www.githubstatus.com/incidents/2bglpcjvg7xb

We have identified the issue as being on GitHub's end. They are either not releasing jobs in a timely manner to online runners or not sending webhook events when jobs are in progress. We will continue to monitor queue times on our side.