Three green guards and a dead database connection

Live chat stopped updating. So did typing indicators, unread badges, reactions, RSVPs, payment approvals and the rotation board — nine hooks in total, all of them quietly, for about three weeks.

Every monitor stayed green the whole time.

What each guard was actually watching

There were three. Each one was real, each one was working, and each one had been built in response to a specific past incident:

• The nightly backup ran pg_dump successfully every night. It proves Postgres answers queries.
• The keep-alive pinged PostgREST on a schedule so the Free-tier project wouldn't auto-pause from inactivity. It proves REST answers.
• The backup-freshness monitor checked that a recent backup release existed. It proves the backup job is still producing artifacts.

Between them they cover db, rest and storage. Realtime is a fourth surface, and nothing touched it.

That's the whole lesson, and it's narrower than "we needed more monitoring." We had monitoring. What we had was a set of probes each shaped like the last thing that went wrong. The union of incident-shaped probes is not coverage. It's a scar map.

The test that would have caught this is boring to run and uncomfortable to answer: list every external service the app depends on, then for each one ask which probe goes red if it dies. Do that honestly and the gaps are obvious in about five minutes.

The vendor's status field was lying

Supabase reported the project as PAUSING. That looked like the answer, and it was wrong.

select pg_postmaster_start_time();
-- 98 days of continuous uptime

Postgres had never stopped. The status field was stale from an unrelated billing block weeks earlier. Two fields from the same API, one true and one false, and no way to tell from the outside which was which.

What settled it was opening an actual Realtime channel and waiting for SUBSCRIBED. It returned TIMED_OUT. That's a behavioural probe — it tests the thing users depend on rather than the vendor's opinion of it, and it can't go stale in the way a status field can.

There's a nasty corollary here. A Free-tier project can wedge partway through an auto-pause: Realtime gets torn down first, and if REST traffic resumes before the pause completes, the pause never finishes. POST /restore then refuses, because restoring requires the project to actually be PAUSED. So a keep-alive that comes back late leaves you somewhere worse than one that never stopped at all — a state the management API has no verb for.

What replaced them

npm run health now enumerates every dependency and probes each one, with two rules that came directly out of this:

A skipped check says why it skipped. A probe that silently disappears when its credential is unset is the same false green one layer up. The output distinguishes PASS, FAIL and SKIP <name> <reason>, and counts skips separately so "everything passed" can't quietly mean "nothing ran."

Reachability and success are different assertions. The old HTTP classifier counted any 4xx as a pass, on the reasoning that it was testing reachability rather than authorization. That's defensible for an unauthenticated liveness ping and wrong for a request carrying a credential — a rotated key returns 401, and the probe would have stayed green while every user was locked out. It's a parameter now, chosen per call site.

The Realtime probe deliberately needs only the publishable key, so it doesn't require an admin credential to run in CI.

The part that wasn't a monitoring problem

One table, conversations, was subscribed to in the client and had never been added to the supabase_realtime publication. Realtime was delivering nothing for it, and always had been. The health check didn't catch that either, because it subscribes to messages — which is published.

Probing one table proves nothing about the others. So the durable fix isn't another probe, it's a drift guard: a test that scans the client source for table: "<x>" subscriptions and diffs them against every ALTER PUBLICATION ... ADD TABLE in the migrations. It fails on any subscription that isn't published.

That's the shape I keep coming back to. A behavioural test only covers what someone remembered to test. A guard derived from the source covers the next one too — including the one you'll add six months from now and forget to publish.

Splitting the signal

The Realtime probe first shipped as the last step of the nightly backup workflow, under if: always() so a probe failure couldn't discard an already-produced dump. The ordering was right and the placement was wrong: the backup went red every night for an unrelated open incident.

"The nightly backup is red" has to keep meaning exactly one thing. Ambiguity there is how a previous 40-night backup failure survived six weeks unnoticed. It's its own workflow now.

Frequently asked questions

+−How can a database be up and Realtime be down at the same time?

On Supabase they're separate services. A Free-tier project that starts auto-pausing tears down Realtime first; if REST traffic then resumes, the pause never completes. You end up with db, rest, auth and storage healthy, Realtime dead, and the project status field stuck on PAUSING indefinitely.

+−Why didn't the nightly backup catch it?

Because pg_dump only proves Postgres answers. The backup, the keep-alive ping and the backup-freshness monitor all probe db, rest or storage. Realtime is a fourth surface none of them touch, so nine features silently stopped updating with zero signal.

+−What does the fix actually check?

It opens a real Realtime channel and waits for SUBSCRIBED. Not a status field, not a TCP connection — the actual behaviour users depend on. It runs as its own workflow so a Realtime failure can't be confused with a backup failure.

+−Why not just trust the vendor's status API?

Because it was wrong. It reported PAUSING while pg_postmaster_start_time() showed the database had been running continuously for 98 days. Two fields from the same API, one true and one stale.