Depot - Notice history

100% - uptime

us-east-1 - Operational

100% - uptime
May 2024 · 99.99%Jun · 99.99%Jul · 99.84%
May 2024
Jun 2024
Jul 2024

eu-central-1 - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024
100% - uptime

Depot-managed Actions Runners - Operational

100% - uptime
May 2024 · 99.93%Jun · 100.0%Jul · 99.63%
May 2024
Jun 2024
Jul 2024

Github.com - Actions - Operational

API - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

Website - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

Notice history

Jul 2024

GitHub Outage
  • Resolved
    Resolved

    GitHub jobs are processing normally.

  • Monitoring
    Update

    We are still seeing some increased job queue times, but the backlog of jobs is processing.

  • Monitoring
    Monitoring

    GitHub have marked their incident as closed, but we have not yet seen full recovery of Actions webhooks. We will continue to monitor as they recover.

  • Identified
    Update

    GitHub is now reporting a general outage, including for git operations: https://www.githubstatus.com/incidents/88003ngctf1z

  • Identified
    Identified

    We are currently observing increased delays receiving GitHub job webhooks from GitHub, which may cause increases in the queue times for some jobs, or may cause the Depot UI to incorrectly display jobs as queued when they have actually started running. We will continue to monitor.

Builds unable to start
  • Resolved
    Update

    Build service is restored. We will continue monitoring.

  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Update

    We have deployed a cache fix for Docker image builds that will result in older projects using an empty cache for their first build.

  • Monitoring
    Update

    AWS has opened an incident on their side https://status.aws.amazon.com/#multipleservices-us-east-1_1720795596.

    We are observing GHA builds processing as expected now but are working on deploying the fix for Docker image builds.

  • Monitoring
    Monitoring

    We are monitoring as we process the backlog of queued jobs - the longest queue time we are now observing is less than 5 minutes.

  • Identified
    Identified

    We have deployed a mitigation for the EC2 API bug, we are starting to see some build service restored.

  • Investigating
    Update

    We have identified the outage as being related to AWS EC2 tag propagation on launch. We're working on a fix.

  • Investigating
    Investigating

    We are investigating some builds being unable to start

GitHub Actions outage
  • Resolved
    Resolved

    GitHub is reporting that they have completed recovery. We are seeing all jobs able to start - if you have any jobs that are still queued, you may need to cancel and re-try them.

  • Monitoring
    Monitoring

    GitHub has started recovering some of the delayed webhooks, and we are seeing some Actions jobs able to start now. We'll continue to monitor as GitHub completes recovery.

  • Identified
    Identified

    We are investigating reports of GitHub jobs queued, but not yet sent from GitHub to Depot.

    GitHub reports issues delivering webhooks: https://www.githubstatus.com/incidents/5yx1d67vq9hg

Jun 2024

Docker Hub upstream outage
GitHub Actions upstream outage

May 2024

Increased GitHub job queue time
  • Resolved
    Resolved

    GitHub have marked the incident as resolved.

  • Monitoring
    Update

    We are observing all jobs able to successfully start. GitHub is still reporting that status updates are delayed for all Actions customers. https://www.githubstatus.com/incidents/nj47vccwm2zj

  • Monitoring
    Monitoring

    While GitHub are still investigating the incident, we are seeing some jobs able to complete, though their status is often not reflected correctly in the GitHub UI. https://www.githubstatus.com/incidents/nj47vccwm2zj

  • Identified
    Update

    GitHub have identified the cause as network connectivity issues on their end, they are continuing to investigate: https://www.githubstatus.com/incidents/nj47vccwm2zj

  • Identified
    Update

    GitHub have just opened an incident of their own for this outage: https://www.githubstatus.com/incidents/nj47vccwm2zj

  • Identified
    Update

    This appears to be a GitHub Actions outage - both GitHub-hosted runners and Depot-hosted runners are currently experiencing issues starting jobs.

  • Identified
    Identified

    We are observing GitHub taking an increased amount of time to release Actions jobs to runners.

GitHub Actions outage
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    GitHub has opened an outage of its own that we are now monitoring https://www.githubstatus.com/incidents/2bglpcjvg7xb

  • Identified
    Identified

    We have identified the issue as being on GitHub's end. They are either not releasing jobs in a timely manner to online runners or not sending webhook events when jobs are in progress. We will continue to monitor queue times on our side.

  • Investigating
    Investigating
    We are currently investigating this incident.

May 2024 to Jul 2024

Next