Depot - Delays with Github Actions Jobs and Docker Builds – Incident details

We have brought container builds and GitHub Actions back to normal capacity. We appreciate your patience and will have a full post-incident on our blog in the coming days.

We will continue to monitor for anything that may come up.

We're seeing queue times beginning to return to normal for GitHub Actions.

Some container builds are failing to launch. We're investigating these, but in the meantime, you can try resetting your cache to clear out the bad state.

We have brought the system back to normal capacity, and backlogs are caught up for primary regions. Additional regions are being brought back up to normal capacity at the moment.

We will continue to monitor recovery.

We've added an additional workaround for an AWS CPU issue that is currently backing up the system under load. We're working to bring the system back to a normal state where the workaround can be removed so normal processing can restart.

We continue to work through bringing the system back to full capacity as we are processing a large backlog of work. You will see your jobs are still queued while we build out slack in the system to bring things back to full capacity.

We have rolled out an initial fix and are seeing initial recovery for container builds and github actions.

We have identified the root cause of the problem and are rolling out a fix across Depot now.