The Debezium failure that takes a system down is almost never the connector crashing. It is the disk filling up on a database you do not own, during a week when nothing changed.
How a replication slot pins the WAL
PostgreSQL logical replication works through a replication slot, which is a promise: the database will retain write-ahead log segments until the consumer confirms it has processed them. That promise is the entire point — it is what makes CDC durable across a connector restart.
It is also a loaded gun, because PostgreSQL keeps that promise unconditionally. A slot that stops advancing pins every WAL segment from that point forward. The database does not care that the consumer is gone. It will retain WAL until the volume is full and writes start failing.
You can watch it happen:
SELECT slot_name,
active,
pg_size_pretty(
pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)
) AS retained
FROM pg_replication_slots;If retained grows and never comes back down, you are on a clock.
The trap: a busy database with quiet tables
Here is the part that surprises people, because it inverts the intuition that low traffic means low risk.
Debezium advances the slot by committing an offset. It commits an offset when it produces a change event. It produces a change event when something happens on a table it captures.
Now imagine a large database. Dozens of schemas, constant write traffic, plenty of WAL. Your connector captures four tables in one schema, and those four tables are quiet — a reference table, a configuration table, something that changes during a release and then not again for a month.
The WAL keeps growing because the database is busy. The slot never advances because your tables are not. Nothing is broken, no error is logged, and the gap widens every hour.
This failure mode gets worse the more selective your capture list is, which is the opposite of what everyone assumes when they scope a CDC pipeline conservatively.
The heartbeat
The fix is built in and under-used. heartbeat.interval.ms makes Debezium emit a
periodic heartbeat event, which gives it something to commit an offset against
even when the captured tables are silent.
On its own that is not quite enough on PostgreSQL, because an offset commit with
no actual WAL activity in the publication may still not move
confirmed_flush_lsn where you need it. The reliable form pairs the heartbeat
with a query that writes to a table the connector does capture:
heartbeat.interval.ms=10000
heartbeat.action.query=INSERT INTO cdc.heartbeat (id, ts) \
VALUES (1, now()) \
ON CONFLICT (id) DO UPDATE SET ts = now()That single-row table gets an update every ten seconds, which produces WAL on a captured relation, which gives the slot something concrete to advance past. The table never grows. The cost is one small write per interval.
Two details worth getting right:
- The heartbeat table must be in the publication. If it is not, the write happens and the slot still does not move, and you will conclude the heartbeat does not work.
- Set the interval against your WAL headroom, not against a feeling. Ten seconds is comfortable. Thirty seconds is fine. Five minutes on a database producing gigabytes of WAL per hour is not a heartbeat, it is a slower leak.
What else stops the slot
The heartbeat solves the quiet-table case. These are the others, in rough order of how often I have seen them:
The connector is paused or failed and nobody noticed. A failed task in Kafka Connect does not stop the database from honouring the slot. Alert on connector task state, not just on consumer lag — lag looks healthy when nothing is being produced.
A long-running transaction. Logical decoding cannot emit a transaction until
it commits. One session sitting in an open transaction for hours holds the slot
behind it regardless of the heartbeat. Worth monitoring
pg_stat_activity for state = 'idle in transaction' alongside the slot lag.
The connector was deleted but the slot was not. Removing a connector from
Kafka Connect does not drop the slot. It becomes inactive, retains WAL forever,
and is invisible unless someone queries pg_replication_slots. On a database
owned by another team this is the one that produces an incident with your name
on it months later.
A snapshot that never finished. During an initial snapshot the connector is not streaming, so the slot sits at its creation LSN while the database keeps writing. On a large table this is a race between the snapshot and the disk.
The alerts that would have caught it
Three, and none of them are about Kafka.
-- 1. Retained WAL per slot. Page above a threshold set from actual disk headroom.
SELECT slot_name,
pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) AS retained_bytes
FROM pg_replication_slots;
-- 2. Inactive slots. Should be empty. Anything here is retaining WAL for nobody.
SELECT slot_name FROM pg_replication_slots WHERE NOT active;
-- 3. Slot age. A confirmed_flush_lsn that has not moved is the earliest signal.The first is the one that saves you. The second is the one that finds the forgotten slot before it becomes an incident. The third catches the quiet-table case before the disk does.
Operating on a database you do not own
Most of the difficulty is organizational rather than technical. The connector is yours; the database, the disk, the alerting and the on-call rotation belong to someone else. When the WAL fills, their pager fires for a problem your component caused.
A few things make that relationship survivable:
- Put the heartbeat table in your own schema. It is your artifact and it should be obvious whose it is.
- Name slots after the owning system, not after the connector version. The DBA who finds an orphan slot at 3am should be able to work out who to call from the name alone.
- Agree the WAL headroom explicitly and write down what happens when it is breached. "Drop the slot" is a legitimate answer — it costs you a resnapshot, which is far cheaper than a full disk on a shared database.
- Document that dropping the slot loses your position. Otherwise someone will drop it during an incident without knowing you will need a full snapshot afterwards, and the second incident will be worse than the first.
Why bother with CDC at all
Given all of the above, the honest question is whether the outbox pattern alone would be simpler. Often it is — if you own the writing application, an outbox table plus a poller avoids replication slots entirely.
CDC earns its complexity when you do not own the writer: a legacy application, a vendor product, a schema you cannot add a table to. That is exactly the situation where you also do not own the database, which is why the operational notes above matter more than the connector configuration does.
If you are running CDC into Kafka, the events it produces still land in a consumer that will see them more than once — idempotency keys are not enough when money moves covers what that consumer has to do about it.