Builds Experiencing Partial Outage

Builds

96.95 % uptime
Jul 202399.86% uptime
Aug 202399.72% uptime
Sep 202391.2% uptime
API
99.94 % uptime
Jul 202399.91% uptime
Aug 202399.92% uptime
Sep 2023100% uptime
Website
99.97 % uptime
Jul 202399.91% uptime
Aug 2023100% uptime
Sep 2023100% uptime
Github - Actions
Operational

Notice history

Sep 2023

Builders failing to launch
  • Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating

    We are currently investigating this incident.

Issues starting builds
  • Resolved

    We've deployed a fix to address this and are continuing to monitor it.

  • Investigating

    We are currently investigating an issue that is causing some builds to fail to start.

Aug 2023

Issue starting builds
  • Resolved

    We have deployed a fix, build service is restored.

  • Investigating

    We are currently investigating an issue preventing build machines from starting.

API Outage
  • Resolved

    A database fix is deployed, API service is restored

  • Identified

    We have identified a database issue as the outage cause and are working to remediate.

  • Investigating

    We are investigating an API outage.

Builders failing to launch in us-east
  • Resolved

    This incident has been resolved, build service is restored

  • Identified

    This capacity issue only exists in the us-east region, and we have capacity available in the eu-central region. You can change your region via the project settings.

  • Identified

    We have identified the issue and are working with our hosting provider to get additional capacity.

  • Investigating

    We are currently investigating an issue that is causing builders to fail to launch.

AWS us-east-1 outage

Jul 2023

Builds failing to start
  • Resolved

    Build service is restored.

  • Investigating

    We're currently investigating an issue causing builds to fail to start.

Database connectivity issue
  • Resolved

    Database service is restored, we are working with our database provider to understand the cause of the outage.

  • Investigating

    We are investigating reports of intermittent database connectivity issues.

Some builders unable to start
  • Resolved

    The fix has deployed, build service is restored.

  • Identified

    We have identified the cause and are deploying a fix.

  • Investigating

    We are investigating an incident preventing some build machines from launching.

Google Cloud Network Instability
  • Resolved

    Build service is operational. We are continuing to work with Google Cloud to ensure they have resolved the issue.

  • Identified

    Google Cloud started experiencing periodic load balancer failures about 5 hours ago. These failures are causing intermittent issues with build machines acquiring build information. We are monitoring the situation and are working with GCP support to resolve the issue.

Issue acquiring builders
  • Resolved

    Build service is restored, we will continue to monitor for any issues.

  • Monitoring

    The fix has deployed, we are manually restarting any build machines that were affected to apply the fix.

  • Identified

    We have identified the issue and are deploying a fix.

  • Investigating

    We are investigating reports of issues acquiring build machines.

Some builds failing to acquire builder
  • Resolved

    This incident has been resolved.

  • Monitoring

    The fix has deployed, we are monitoring but all builds are currently succeeding.

  • Identified

    We have identified the cause and have deployed a temporary fix.

  • Investigating

    We are investigating reports of issues with some builds failing to acquire builders

Database provider outage
  • Resolved

    Database service is restored, builds are functional again.

  • Monitoring

    Database service is beginning to restore, we are monitoring the progress.

  • Monitoring

    Our database provider is still offline, we are continuing to monitor.

  • Investigating

    We are currently investigating an outage with our primary database provider, all services are affected.

Jul 2023 to Sep 2023