High number of 5XX on the Machines API and dashboard

What happened

Jul 20 , 17:09 UTC Resolved - This incident has been resolved. Jul 20 , 13:34 UTC Update - We are still working on fixing degraded Managed Postgres clusters. Jul 20 , 09:14 UTC Update - Some Managed Postgres v1 clusters are degraded. We are working on fixing them. Managed Postgres v2 is unaffected. Jul 20 , 08:02 UTC Monitoring - A fix has been implemented and we are monitoring the results. Jul 20 , 07:52 UTC Update…

Summary assembled by rule from the sources below

Timeline

1. 17:09
   First appeared on Fly.io StatusFFly.io Status

   1. 07:10InvestigatingExisting machines are unaffected. We are investigating the issue.
   2. 07:48UpdateWe are continuing to investigate this issue.
   3. 07:51IdentifiedWe've identified an internal service providing authentication to our Machines API has failed, our team is currently looking at our options for restoring this service. Existing Machines/Apps will continue to run as normal. Thank you for your patience.
   4. 07:52UpdateWe've identified an internal service providing authentication to our Machines API has failed, our team is currently looking at our options for restoring this service. Existing Machines/Apps will continue to run as normal. Thank you for your patience.
   5. 08:02MonitoringA fix has been implemented and we are monitoring the results.
   6. 09:14UpdateSome Managed Postgres v1 clusters are degraded. We are working on fixing them. Managed Postgres v2 is unaffected.
   7. 13:34UpdateWe are still working on fixing degraded Managed Postgres clusters.
   8. 17:09ResolvedThis incident has been resolved.

Sources