Depot - Notice history

Experiencing partial outage

100% - uptime

us-east-1 - Major outage

100% - uptime
Dec 2024 · 99.89%Jan 2025 · 99.91%Feb · 99.93%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

eu-central-1 - Major outage

100% - uptime
Dec 2024 · 99.99%Jan 2025 · 99.96%Feb · 99.93%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025
100% - uptime

Depot-managed Actions Runners - Major outage

100% - uptime
Dec 2024 · 99.97%Jan 2025 · 99.94%Feb · 99.93%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Github.com - Actions - Operational

API - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 99.96%Feb · 99.99%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Website - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 99.96%Feb · 99.99%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Notice history

Feb 2025

Website and API outage
  • Resolved
    Resolved

    Build service has fully restored.

  • Update
    Update

    Build and Action jobs are starting to process, we are working through the backlog of jobs from the disruption.

  • Identified
    Identified

    We have deployed a mitigation and are seeing service begin to be restored.

  • Investigating
    Investigating

    We are currently investigating an API and website outage.

Longer queue times for GitHub Actions
  • Resolved
    Resolved

    Queue times have returned to normal.

  • Investigating
    Investigating

    We are investigating alerts about longer queue times for Actions jobs.

GitHub Incident
  • Resolved
    Resolved

    GitHub have resolved their incident.

  • Investigating
    Investigating

    GitHub are currently experiencing an incident - affected GitHub jobs have increased queue time while GitHub works through the backlog of job notifications and should eventually be processed.

Longer queue times for some GitHub Actions jobs
  • Resolved
    Resolved

    GitHub have resolved their incident.

  • Monitoring
    Monitoring

    We are currently observing some increase in queue times for GitHub Actions jobs - GitHub are currently experiencing an incident that may be related. We are monitoring.

Application database outage
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result. Job are slowly being picked up.

  • Investigating
    Investigating

    We are currently investigating an incident affecting our primary application database.

Jan 2025

Application database outage
  • Resolved
    Resolved

    We are seeing full connectivity restored, builds are processing normally.

  • Monitoring
    Monitoring

    We are seeing database connectivity restore, builds are able to process now with degraded performance.

  • Investigating
    Investigating

    We are currently investigating an incident affecting our primary application database.

Some GitHub jobs waiting to start
  • Resolved
    Resolved

    GitHub have released all jobs to runners, builds are processing normally.

  • Identified
    Identified

    We are currently observing some GitHub Actions jobs waiting to be released by GitHub to their runners. This may indicate an upcoming GitHub outage, or may clear on its own. We will continue to monitor.

Some container builds failing to start
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    The fix has deployed, we are monitoring to ensure a full recovery.

  • Identified
    Identified

    We have identified a failing service preventing builder machines from acquiring disks and are deploying a fix.

  • Investigating
    Investigating

    We are currently investigating reports of some container builds failing to acquire build machines.

Increased queue time for some GitHub Actions jobs
  • Resolved
    Resolved

    All jobs are processing normally.

  • Identified
    Identified

    We are currently investigating longer queue times for some GitHub Actions jobs - these jobs appear to be waiting longer than expected for the GitHub Actions backend to release them to online runners. We will continue to monitor.

GitHub is investigating degraded Actions performance.
  • Resolved
    Resolved

    Queue times appear back to normal now.

  • Monitoring
    Monitoring

    GitHub has resolved the issue. We will continue to monitor before we also resolve.

  • Investigating
    Investigating

    We are seeing checkouts receiving 500 errors.

    Here is GitHub's current status for the issue: https://www.githubstatus.com/incidents/qd96yfgvmcf9

Dec 2024

Increased queue times for GitHub Actions jobs
  • Resolved
    Resolved

    The system continues to behave as expected and the incident is now resolved.

  • Monitoring
    Monitoring

    As of 21:47 UTC we disabled a feature flag that controlled the faulty provisioning system functionality. The system has now stabilized and jobs are processing normally.

    We will continue to monitor for any further issues.

  • Investigating
    Investigating
    We are currently investigating this incident.
Issues with GitHub jobs and container builds
  • Resolved
    Resolved

    This incident has been resolved, all jobs are processing normally.

  • Monitoring
    Monitoring

    Container builds in us-east-1 are also processing again, we are monitoring for a full recovery.

  • Identified
    Identified

    We are seeing some recovery for GitHub jobs, but separately are seeing increased start times for container builds in us-east-1.

  • Investigating
    Investigating

    We are currently investigating reports of GitHub jobs stuck in a queued state.

Degraded cache cluster performance for container image builds
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We have implemented a performance fix and monitoring recovery.

  • Investigating
    Investigating

    We are currently investigating an issue with our internal cache cluster for container layer cache that is degrading the performance of builds.

Increased queue times for GitHub Actions jobs
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    GitHub states they have reverted a change. We are starting to see job queue times get better. We will continue to monitor.

  • Identified
    Identified

    GitHub states it is having a disruption:

    https://www.githubstatus.com/incidents/4349zxvb8stp

  • Investigating
    Investigating
    We are currently investigating this incident.
Docker image builders failing to start
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    We have deployed a fix for container image builds in us-east-1 and are monitoring recovery. You can now switch back your projects to us-east in your project settings.

  • Monitoring
    Monitoring

    We are deploying a mitigation for AWS issues in us-east-1. GitHub Actions jobs continue to be able to process normally, as do container builds in the EU Central region.

  • Update
    Update

    We have isolated the outage to only impacting Docker image builders in us-east-1. As a workaround, you can change your project region in Depot to eu-central to bring builds back online.

  • Update
    Update

    We have identified an upstream API limit within AWS that has taken down the container image builders. We are seeing some builds that are able to connect and process but are still working on bringing full capacity back online.

  • Identified
    Identified

    This delays appear to be caused by an AWS EC2 API issue, we have raised a support case with AWS

  • Investigating
    Investigating
    We are currently investigating this incident.

Dec 2024 to Feb 2025

Next