All systems operational

Builds

99.95% uptime
Feb 2024 · 99.95%Mar · 99.90%Apr · 100.0%
Feb 202499.95% uptime
Mar 202499.90% uptime
Apr 2024100.0% uptime
API
99.95% uptime
Feb 2024 · 99.98%Mar · 99.93%Apr · 99.93%
Feb 202499.98% uptime
Mar 202499.93% uptime
Apr 202499.93% uptime

Depot-managed GitHub Actions Runners

99.96% uptime
Feb 2024 · 100.0%Mar · 99.93%Apr · 99.95%
Feb 2024100.0% uptime
Mar 202499.93% uptime
Apr 202499.95% uptime
Website
99.96% uptime
Feb 2024 · 100.0%Mar · 99.95%Apr · 99.94%
Feb 2024100.0% uptime
Mar 202499.95% uptime
Apr 202499.94% uptime
Github - Actions
Operational

Notice history

Apr 2024

GitHub outage
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are monitoring the rollback GitHub has performed and are observing jobs being processed: https://www.githubstatus.com/

  • Investigating
    Investigating

    GitHub is currently experiencing a widespread outage that is impacting webhook delivery to our runners: https://www.githubstatus.com/

GitHub Actions Outage
  • Resolved
    Resolved

    GitHub has cleared their outage but we are continuing to monitor slower webhook delivery times from their side.

  • Identified
    Identified

    GitHub Actions is currently having an outage that is degrading performance across all of Actions. We are monitoring: https://www.githubstatus.com/incidents/xcjgp37zx5dt

Database outage
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Update

    We've gotten our database provider to clear the outage on their side. We're monitoring initial recovery.

  • Identified
    Update

    We've identified the cause as an outage at our database provider, we are working to restore service.

  • Identified
    Identified

    We've identified a performance degradation at our upstream database provider that we are working with them on.

  • Investigating
    Investigating

    We are currently investigating an issue causing increased latency when starting GitHub Actions and container builds.

Slow GitHub webhook delivery
  • Resolved
    Resolved

    We've fixed the underlying runner issue following the GitHub Actions outage. We are observing full recovery and will continue monitoring.

  • Identified
    Update

    We're investigating runners still failing to start following the GitHub Actions outage.

  • Identified
    Identified

    GitHub Actions is currently experiencing degraded performance: https://www.githubstatus.com/incidents/5ly0psff2s5d

Slow GitHub webhook delivery
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are observing some delay with GitHub sending webhooks that is increasing queue times for GitHub Actions jobs. We will monitor.

Mar 2024

Degraded performance when starting Docker and GitHub Actions builds
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing queue times return to normal, we're continuing to monitor

  • Identified
    Identified

    We have identified a database issue preventing builds from starting, we are working to resolve the issue.

  • Investigating
    Investigating

    We are investigating an issue causing longer queue times when starting Docker and GitHub Actions builds

Issue launching Docker builders
  • Resolved
    Resolved

    We've deployed a fix and are observing a full recovery. We will continue to monitor performance.

  • Investigating
    Update

    We're experiencing additional database load that is impacting Docker builds and GitHub Actions Runners.

  • Investigating
    Investigating

    We are currently investigating an issue preventing Docker builds from starting

Degraded performance
  • Resolved
    Resolved

    We have identified the problem and are monitoring recovery.

Issues starting builds in us-east-1
  • Resolved
    Resolved

    We are monitoring build recovery after a brief network disruption at our upstream database provider.

  • Investigating
    Investigating

    We are currently investigating some builds failing to start in us-east-1.

Feb 2024

Issues starting builds in us-east-1
  • Resolved
    Resolved

    We've rolled out a fix, and all builds are recovering. We will continue to monitor.

  • Monitoring
    Monitoring

    We are rolling out a fix and monitoring recovery.

  • Identified
    Identified

    We've identified an issue with API latency that is causing builds to take longer than usual to start.

  • Investigating
    Investigating

    We are currently investigating delays in machines launching for builds in us-east-1.

Issue starting builds in us-east-1
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Build service is restored, we are monitoring.

  • Investigating
    Investigating
    We are currently investigating this incident.
Issues pushing from the ephemeral registry
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We are currently investigating errors with running depot push to push an image from the ephemeral registry.

Issues receiving sign-in emails
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Update

    We are working to fail over our email delivery to an internal provider.

  • Identified
    Identified

    We've identified that our upstream email delivery provider is having an outage which is currently preventing sign-in emails from being delivered.

Issues acquiring builder in us-east-1
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing build service restore - this incident appears to have been caused by a database provider outage that has since recovered. We're continuing to monitor.

  • Investigating
    Investigating

    We are investigating an incident in us-east-1

Feb 2024 to Apr 2024