Builds
Mar 2023
This incident has been resolved, build service is restored.
We implemented a fix and are currently monitoring the result.
We are investigating an issue where some builds fail to acquire a builder.
Fast build launch times are restored.
Builds in us-east-1 are slower to start than usual due to capacity limitations. We are increasing capacity to meet the increase in demand.
Build service has been restored.
AWS is beginning to restore network service and we are seeing builds resume. We will continue to monitor recovery.
AWS is now reporting an increase in failure rates when launching in us-east-1, we are continuing to monitor.
We are investigating a network disruption affecting builders in us-east-1.
This incident has been resolved.
We have deployed a fix, builder connections are restored. We will monitor to ensure no issues remain.
We are investigating an issue connecting to builder machines.
Feb 2023
Database connectivity has been fully restored. We will continue to watch for any future issues. All build service is restored.
Builds are succeeding again, we are monitoring the rollout of the fix.
Our upstream database provider has identified the issue and is deploying a fix (https://www.planetscalestatus.com/incidents/3dvstlslt16b). We will update once the fix has applied to Depot's database.
Our database provider is currently experiencing an outage (https://www.planetscalestatus.com/incidents/3dvstlslt16b). This is affecting all builds.
We have deployed a fix, build service is restored.
We are investigating an incident preventing builds from starting.
Jan 2023
Connections have dropped back to normal latencies. We will be continuing to monitor for spikes.
We are monitoring increased latency in launching new builds that appears to be related to our earlier incident.
The issue is resolved, build service has been restored.
We pinpointed the problematic service and deployed a fix. Build connections are restored, and we are monitoring for connection failures.
We are investigating a failure to reconcile AWS API calls that is resulting in builders failing to launch.
We are currently investigating an issue causing some builders to fail to launch.
Database connectivity has been restored and all systems have recovered.
We have resolved the database connectivity issue and are monitoring the results.
We have identified an issue with our primary database.
We are investigating website errors on login.
Jan 2023 to Mar 2023
