This incident has been resolved.
We are rolling out an additional fix to our API to help mitigate this issue.
We have detected an issue acquiring new build machines. We have deployed a fix and are monitoring recovery.
We have deployed a fix for machines failing to launch in eu-central. We are monitoring the fix.
We have identified the problem and are working on a fix. This is only impacting builds trying to acquire machines in the eu-central region at the moment.
We are investigating an issue with our API that is causing builds to not be able to acquire a machine.
We have deployed a mitigation and build service is restored.
We are currently investigating an incident preventing build machines from being acquired.
The upstream issue has been resolved.
A network issue at an upstream service provider is causing API requests to start builds to respond slowly. Build service is still functional. We are monitoring the situation.
Capacity issues appear to be resolved, we will continue to monitor.
Some builds in us-east-1 are failing to start due to capacity issues. We are working with AWS to acquire additional capacity.
No notices reported this month
Our database provider is working through an issue that is causing database connections to drop. We are monitoring the recovery: https://www.planetscalestatus.com/.
GitHub Actions has identified that the issue is capacity related: https://www.githubstatus.com/.
There is currently a GitHub Actions outage that is impacting OIDC token exchange. We are actively monitoring the situation. You can follow the outage here: https://www.githubstatus.com/.
API and build service are fully restored.
We have completed failover to a backup hosting provider, build service is restored. We will continue to monitor.
We're continuing to work on this incident and are in-progress with a failover to a backup hosting provider. We will update here shortly.
We've identified an outage with our API hosting provider, we're working with support to resolve.
We are investigating API performance issues.
Apr 2023 to Jun 2023