Appflow iOS builds intermittently failing

Incident Report for Ionic Services

Postmortem

On May 8th, the Appflow service experienced degraded performance from 3 AM EDT to 9:45 AM EDT. This was caused by an upgrade in our cloud hypervisor software specifically for Mac virtual machines running on Intel CPU Architecture.

Late evening May 7th, 2024, the Appflow team rolled out an upgrade to the Mac Hypervisor software on Intel Mac machines in the Appflow Build Cloud. This change introduced a new limitation to network requests within Mac virtual machines which subsequently caused timeouts on external requests within build executions. As a result, all builds running on the affected instances would fail or timeout.

On May 8th, 2024 at approximately 9:45 AM EDT, we rolled back all of the affected instances with the older version of the software to resolve the introduced problem.

Posted May 08, 2024 - 21:56 UTC

Resolved

This incident has been resolved.

Posted May 08, 2024 - 16:34 UTC

Monitoring

A fix has been implemented and we currently working on identifying the root cause. We expect that new iOS builds should no longer fail. Thank you for your patience.

Posted May 08, 2024 - 14:31 UTC

Investigating

Beginning around 5:00 am ET, degraded performance of the iOS build runners has been identified. iOS builds may be intermittently failing. We are working on investigating the issue and should have more details available shortly. Thank you for your patience.

Posted May 08, 2024 - 14:11 UTC

This incident affected: Ionic Appflow Build Service.