In the week of March 24, Ionic made some internal changes to GitHub configuration during the process of migrating to a new CI/CD setup. While deploying one service to test the changes, one of the required secrets was not correctly migrated. This was not immediately detected as the service's successful responses were heavily cached. Once the caches expired, the missing secret caused the service to respond with failure responses to permission checks. These failures were incorrectly interpreted by downstream services. This was the root cause of these services failing and resulting downtime with the Appflow dashboard. Once the required secret was migrated and caches were flushed, all service was restored.
Remediation CI/CD pipeline is updated to check for missing / incorrectly migrated secrets and fail fast under this scenario. Additional changes weee made to downstream services to not misinterpret failure responses.