All systems operational

Builds

99.82% uptime
Jul 2023 · 99.86%Aug · 99.72%Sep · 99.89%
Jul 202399.86% uptime
Aug 202399.72% uptime
Sep 202399.89% uptime
API
99.94% uptime
Jul 2023 · 99.91%Aug · 99.92%Sep · 100.0%
Jul 202399.91% uptime
Aug 202399.92% uptime
Sep 2023100.0% uptime
Website
99.97% uptime
Jul 2023 · 99.91%Aug · 100.0%Sep · 100.0%
Jul 202399.91% uptime
Aug 2023100.0% uptime
Sep 2023100.0% uptime
Github - Actions
Operational

Notice history

Sep 2023

failed to read dockerfile: failed to load cache key: no handler for git
  • Resolved
    Resolved

    The queue backlog is clear, build start time has returned to fast.

  • Monitoring
    Monitoring

    We are monitoring the rollout of the fix, we are currently waiting on a queue backlog

  • Resolved
    Resolved

    We have deployed a fix, the error has cleared

  • Investigating
    Investigating

    We are investigating an issue causing some builds to fail with the message "failed to read dockerfile: failed to load cache key: no handler for git".

Builders failing to launch
  • Resolved
    Resolved

    We've deployed a fix and will continue to monitor the results.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

Issues starting builds
  • Resolved
    Resolved

    We've deployed a fix to address this and are continuing to monitor it.

  • Investigating
    Investigating

    We are currently investigating an issue that is causing some builds to fail to start.

Aug 2023

Issue starting builds
  • Resolved
    Resolved

    We have deployed a fix, build service is restored.

  • Investigating
    Investigating

    We are currently investigating an issue preventing build machines from starting.

API Outage
  • Resolved
    Resolved

    A database fix is deployed, API service is restored

  • Identified
    Identified

    We have identified a database issue as the outage cause and are working to remediate.

  • Investigating
    Investigating

    We are investigating an API outage.

Builders failing to launch in us-east
  • Resolved
    Resolved

    This incident has been resolved, build service is restored

  • Identified
    Update

    This capacity issue only exists in the us-east region, and we have capacity available in the eu-central region. You can change your region via the project settings.

  • Identified
    Identified

    We have identified the issue and are working with our hosting provider to get additional capacity.

  • Investigating
    Investigating

    We are currently investigating an issue that is causing builders to fail to launch.

AWS us-east-1 outage

Jul 2023

Builds failing to start
  • Resolved
    Resolved

    Build service is restored.

  • Investigating
    Investigating

    We're currently investigating an issue causing builds to fail to start.

Database connectivity issue
  • Resolved
    Resolved

    Database service is restored, we are working with our database provider to understand the cause of the outage.

  • Investigating
    Investigating

    We are investigating reports of intermittent database connectivity issues.

Some builders unable to start
  • Resolved
    Resolved

    The fix has deployed, build service is restored.

  • Identified
    Identified

    We have identified the cause and are deploying a fix.

  • Investigating
    Investigating

    We are investigating an incident preventing some build machines from launching.

Google Cloud Network Instability
  • Resolved
    Resolved

    Build service is operational. We are continuing to work with Google Cloud to ensure they have resolved the issue.

  • Identified
    Identified

    Google Cloud started experiencing periodic load balancer failures about 5 hours ago. These failures are causing intermittent issues with build machines acquiring build information. We are monitoring the situation and are working with GCP support to resolve the issue.

Issue acquiring builders
  • Resolved
    Resolved

    Build service is restored, we will continue to monitor for any issues.

  • Monitoring
    Monitoring

    The fix has deployed, we are manually restarting any build machines that were affected to apply the fix.

  • Identified
    Identified

    We have identified the issue and are deploying a fix.

  • Investigating
    Investigating

    We are investigating reports of issues acquiring build machines.

Jul 2023 to Sep 2023