401 errors due to JWT rejections

What happened

Aug 21 , 17:21 UTC Update - We are continuing to test and rollout the fixes for intermittent HTTP 401 errors. In most cases, waiting and refreshing is successful. We will monitor eu-central-2 over the weekend and are preparing for the fix to be pushed to all regions starting Monday. Thank you for your patience while we've worked on multiple fixes for this issue. Aug 20 , 16:37 UTC Update - As part of the incident re…

Summary assembled by rule from the sources below

Timeline

1. 17:21
   First appeared on Supabase StatusSSupabase Status

   1. 02:23IdentifiedWe have identified the root cause of newly refreshed JWTs being rejected by the API, resulting in HTTP 401 errors for affected sessions. We are working on a fix and will provide updates as things progress.
   2. 07:53UpdateOur teams are continuing to work on the fix and rollout is underway for selected customers. We will provide another update as we are applying the fix to all impacted users.
   3. 18:38UpdateFixes for this issue are being sequentially rolled out. The impact of the issue is limited to a subset of new projects that can experience it upon some JWT renewals. We will update the status when all the fix rollouts have completed.
   4. 16:37UpdateAs part of the incident remediation flow, we have separately identified "/lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found" error between 11:15 - 16:24 UTC today affecting PostgREST. This has been fixed and we can see the error rates have subsided.
   5. 17:21UpdateWe are continuing to test and rollout the fixes for intermittent HTTP 401 errors. In most cases, waiting and refreshing is successful. We will monitor eu-central-2 over the weekend and are preparing for the fix to be pushed to all regions starting Monday. Thank you for your patience while we've wor…

Sources