Idempotency keys are not enough when money moves

Kafka gives you at-least-once delivery. Everyone knows this. The usual response is a deduplication table keyed on a message id, and for a lot of systems that is genuinely enough.

It stops being enough the moment the operation has consequences outside your database.

What the idempotency key actually protects

A deduplication table protects exactly one thing: the write you remembered to guard. Here is the version almost everyone writes first.

@KafkaListener(topics = "card-operations")
void onMessage(CardOperation operation) {
  if (processed.contains(operation.id())) {
    return;
  }
  applyToLedger(operation);
  notificationService.sendEmail(operation.customerId());
  processed.add(operation.id());
}

Three separate bugs live in those seven lines.

The gap between the check and the write. Two consumers in the same group will not see the same partition, but a rebalance during processing absolutely can hand the same offset to a new consumer while the old one is still running. Both pass the contains check. Both apply the operation.

The write and the marker are not atomic. If the process dies after applyToLedger and before processed.add, the redelivery applies it a second time. If it dies the other way round, the operation is silently lost — worse, because nothing will ever retry it.

The email is not covered at all. Even with perfect deduplication of the ledger write, the customer gets two emails, because the side effect happened before the marker was durable.

Fix one: the marker and the state change share a transaction

The deduplication record is not metadata about the processing. It is part of the processing, and it belongs in the same transaction as the state change.

@KafkaListener(topics = "card-operations")
@Transactional
void onMessage(CardOperation operation) {
  int inserted = processedRepository.insertIfAbsent(operation.id());
  if (inserted == 0) {
    return;                  // Already applied. Commit and move on.
  }
 
  ledger.apply(operation);
  outbox.append(NotificationRequested.from(operation));
}

insertIfAbsent is a single statement against a unique constraint — ON CONFLICT DO NOTHING returning the affected row count — not a read followed by a write. If the transaction rolls back, the marker rolls back with it, and the redelivery does the whole thing again cleanly. If it commits, both are durable together.

This is the same shape as the transactional outbox, for the same reason: the only way to make two facts agree is to write them in one transaction.

Fix two: side effects go through the outbox

The email is gone from the consumer. It is now a row in an outbox table, written in the same transaction as everything else, and relayed to Kafka by a separate process — Debezium reading the write-ahead log, or a poller, depending on what you are willing to operate.

That converts an unrecoverable side effect into a recoverable one. If the relay crashes mid-publish, it republishes; the notification service is itself an at-least-once consumer with its own deduplication, and the failure is contained inside a system you control rather than sitting in a mail provider's outbound queue.

The rule that falls out of this: a consumer transaction may write to its own database and nowhere else. No HTTP calls, no email, no third-party SDK, no publishing directly to another topic. Anything that reaches outside becomes a row that something else picks up.

Fix three: the key has to be a business key

This is the one that bites hardest, because the code looks correct.

// Looks fine. Is not.
var id = UUID.randomUUID();
producer.send(new CardOperation(id, cardId, amount));

If the producer generates a fresh id on every attempt, then a producer-side retry — a broker timeout where the write actually succeeded, say — publishes the same real-world operation twice under two different ids. The consumer deduplicates perfectly and still applies it twice, because as far as it can tell these are two different operations.

The key must be derived from the thing itself: the operation's own reference from the source system, or a deterministic hash of the fields that make it unique. The question to ask is not "is this message unique" but "what makes two messages the same real-world event", and that answer almost never lives in a UUID your code invented.

Fix four: make the operation naturally idempotent where you can

The strongest version of replay safety needs no deduplication at all, because applying the operation twice produces the same state as applying it once.

Where you can express the change as an assignment with a version guard, or as an insert protected by a unique constraint, the database enforces correctness and the deduplication table becomes an optimization rather than the thing standing between you and a double charge.

Some operations genuinely cannot be expressed that way. Those are the ones that need every fix above, and they are worth identifying explicitly rather than assuming the whole system is uniform.

Ordering is a separate problem

Idempotency stops duplicates. It does nothing about order.

Kafka guarantees order within a partition, which means order holds only if related messages share a partition key. If card operations are keyed by operation id rather than card id, two operations on the same card can land on different partitions and be processed out of order by different consumers. Neither is a duplicate. Both are applied exactly once. The final state is still wrong.

Key by the entity whose state is changing, not by the event. Where out-of-order delivery is still possible — across topics, or after a partition reassignment — carry a monotonic version on the entity and reject anything older than the current state.

The test that catches all of it

If there is one thing to take from this: make replay a first-class test, not a thought experiment.

@Test
void applyingTheSameOperationTwiceChangesNothing() {
  var operation = anOperation();
 
  consumer.onMessage(operation);
  var afterFirst = ledger.snapshotFor(operation.cardId());
 
  consumer.onMessage(operation);          // exact same message object
  var afterSecond = ledger.snapshotFor(operation.cardId());
 
  assertThat(afterSecond).isEqualTo(afterFirst);
  assertThat(outbox.entriesFor(operation.id())).hasSize(1);
}

Run it against a real database with Testcontainers, not a mock. The bugs described here live in the interaction between the transaction boundary and the unique constraint, and a mocked repository will happily let all of them through.

Once this passes, the harder question is what happens when the schema of that message changes and the consumer belongs to a different team — which is a different problem with a different set of answers, covered in Avro schema evolution when the consumer is a different company.