All systems operational

Builds

99.78% uptime
Jun 2023 · 99.78%Jul · 99.86%Aug · 99.72%
Jun 202399.78% uptime
Jul 202399.86% uptime
Aug 202399.72% uptime
API
99.93% uptime
Jun 2023 · 99.96%Jul · 99.91%Aug · 99.92%
Jun 202399.96% uptime
Jul 202399.91% uptime
Aug 202399.92% uptime

Depot-managed GitHub Actions Runners

100.0% uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%
Jun 2023100.0% uptime
Jul 2023100.0% uptime
Aug 2023100.0% uptime
Website
99.97% uptime
Jun 2023 · 100.0%Jul · 99.91%Aug · 100.0%
Jun 2023100.0% uptime
Jul 202399.91% uptime
Aug 2023100.0% uptime
Github - Actions
Operational

Notice history

Aug 2023

Issue starting builds
  • Resolved
    Resolved

    We have deployed a fix, build service is restored.

  • Investigating
    Investigating

    We are currently investigating an issue preventing build machines from starting.

API Outage
  • Resolved
    Resolved

    A database fix is deployed, API service is restored

  • Identified
    Identified

    We have identified a database issue as the outage cause and are working to remediate.

  • Investigating
    Investigating

    We are investigating an API outage.

Builders failing to launch in us-east
  • Resolved
    Resolved

    This incident has been resolved, build service is restored

  • Identified
    Update

    This capacity issue only exists in the us-east region, and we have capacity available in the eu-central region. You can change your region via the project settings.

  • Identified
    Identified

    We have identified the issue and are working with our hosting provider to get additional capacity.

  • Investigating
    Investigating

    We are currently investigating an issue that is causing builders to fail to launch.

AWS us-east-1 outage
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    All builds are now using primary infrastructure, we are monitoring to ensure that all service is restored.

  • Identified
    Update

    AWS is reporting that the us-east-1 outage has cleared, we are routing builds back to primary infrastructure.

  • Identified
    Update

    AWS is experiencing an outage in us-east-1 (https://health.aws.amazon.com/health/status). We are attempting a failover to backup infrastructure in another region.

  • Identified
    Update

    AWS is still experiencing issues in us-east-1, we are prepping to fail over to backup infra.

  • Identified
    Identified

    AWS us-east-1 is currently rate-limiting infrastructure updates, we are monitoring.

  • Investigating
    Investigating

    We are investigating lag in acquiring build machines, causing some build requests to time out.

Jul 2023

Builds failing to start
  • Resolved
    Resolved

    Build service is restored.

  • Investigating
    Investigating

    We're currently investigating an issue causing builds to fail to start.

Database connectivity issue
  • Resolved
    Resolved

    Database service is restored, we are working with our database provider to understand the cause of the outage.

  • Investigating
    Investigating

    We are investigating reports of intermittent database connectivity issues.

Some builders unable to start
  • Resolved
    Resolved

    The fix has deployed, build service is restored.

  • Identified
    Identified

    We have identified the cause and are deploying a fix.

  • Investigating
    Investigating

    We are investigating an incident preventing some build machines from launching.

Google Cloud Network Instability
  • Resolved
    Resolved

    Build service is operational. We are continuing to work with Google Cloud to ensure they have resolved the issue.

  • Identified
    Identified

    Google Cloud started experiencing periodic load balancer failures about 5 hours ago. These failures are causing intermittent issues with build machines acquiring build information. We are monitoring the situation and are working with GCP support to resolve the issue.

Issue acquiring builders
  • Resolved
    Resolved

    Build service is restored, we will continue to monitor for any issues.

  • Monitoring
    Monitoring

    The fix has deployed, we are manually restarting any build machines that were affected to apply the fix.

  • Identified
    Identified

    We have identified the issue and are deploying a fix.

  • Investigating
    Investigating

    We are investigating reports of issues acquiring build machines.

Jun 2023

Builds failing to acquire build machines
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Update

    We are rolling out an additional fix to our API to help mitigate this issue.

  • Monitoring
    Monitoring

    We have detected an issue acquiring new build machines. We have deployed a fix and are monitoring recovery.

Issues acquiring build machines
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We have deployed a fix for machines failing to launch in eu-central. We are monitoring the fix.

  • Identified
    Identified

    We have identified the problem and are working on a fix. This is only impacting builds trying to acquire machines in the eu-central region at the moment.

  • Investigating
    Investigating

    We are investigating an issue with our API that is causing builds to not be able to acquire a machine.

Issue acquiring build machines
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We have deployed a mitigation and build service is restored.

  • Investigating
    Investigating

    We are currently investigating an incident preventing build machines from being acquired.

Builds slow to start due to upstream network latency
  • Resolved
    Resolved

    The upstream issue has been resolved.

  • Identified
    Identified

    A network issue at an upstream service provider is causing API requests to start builds to respond slowly. Build service is still functional. We are monitoring the situation.

    https://status.posthog.com/incidents/q12bkzfnzlnr

Capacity issues acquiring builders
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Capacity issues appear to be resolved, we will continue to monitor.

  • Identified
    Identified

    Some builds in us-east-1 are failing to start due to capacity issues. We are working with AWS to acquire additional capacity.

Jun 2023 to Aug 2023