Academy

Save first, then publish: a simple rule for not losing work

One of the most common hidden bugs in event-driven systems is publishing the news before the fact is recorded. The right order — save first, then publish — removes that bug at the root.

The Felesh team 5 min

In event-driven systems, a subtle bug often lies in wait: publishing the news before the fact is recorded. The bug is silent. It rarely appears in testing, revealing itself only in production, under load. A strict ordering rule resolves it entirely: save first, then publish.

Same code, two databases, one raceTwo timelines from 0 to 60 ms, with the consumer reading at 20 ms in both. On the idle test database the write commits at 8 ms, before the read, so the reader finds A-7031. On the busy production database the write commits at 45 ms, after the read, so the reader finds nothing. Same code and same event; only the commit time moved.consumer reads at 20 mstest · idle dbcommitted 8 msreader finds A-7031prod · busy dbcommitted 45 msreader finds nothing0 ms60 mssame code, same event: only the commit time moved
The same code passes on an idle database and fails on a busy one: only the commit time moved, so a test never sees the race.

The shape of the problem

When a component finishes work, it must perform two actions: write the result to durable memory, and publish a completion event. If you invert this sequence — emitting the event before the write commits — a race condition opens. A downstream component hears the event, reacts immediately, and attempts to read the result. Because the save has not yet completed, the consumer reads stale or empty data. The failure cascades from there.

The event arrives before the commitA swim-lane timeline with producer, store and consumer. At 0 ms the producer publishes order.saved A-7031 while the write to the store is still running. At 3 ms the consumer hears it and reads A-7031 from the store, which is mid-write, and gets not found. The write commits at 12 ms, 9 ms after the read.publish first, save secondread 9 ms before the commitproducerstoreconsumerpublish: order.saved A-7031writing A-7031…hears it at 3 ms, reads A-7031A-7031: not foundcommitted at 12 ms0 ms3 ms12 ms
Published before the save, the event reaches the consumer at 3 ms; its read lands inside the 12 ms write and finds nothing.

The right order

The fix is strict linear execution: write the result to durable memory, ensure the commit has completed, and only then publish the event. Any component receiving the event is now guaranteed to find the corresponding data in memory. This enforces a core principle of event-driven architecture: the event is merely a trigger, and the truth is always in memory. If the truth is not yet written, the trigger must not fire.

Save, commit, then publishThree steps for the publisher: write A-7031 to the store, wait for the commit, publish order.saved A-7031. The store shows the row A-7031 paid 640 with a tick once committed. Only after that does the event reach the consumer, which then reads the row and finds it: paid, 640.the publisher, in this order1write A-7031 to store2wait for the commit3publish event A-7031storeA-7031 paid 640any reader can see itconsumerhears event A-7031reads: paid, 640no event leaves beforethe commit has landed
Write, wait for the commit, then publish: by the time the consumer hears of A-7031, the row is already there to read.

Why this rule matters

This ordering aligns with another architectural constraint: the event should carry no data, only an identifier. Including state in the event creates the temptation to emit it before persisting the data — which introduces the exact race condition. When the event carries only a reference, forcing consumers to query the data store for the truth, the “save first, then publish” pattern transitions from a best practice to a logical necessity.

State in the event versus an idTwo event envelopes side by side. The left one carries state: order.saved with id A-7031, status paid and total 640; it is easy to send before the save and the consumer trusts that copy. The right one carries only order.saved and id A-7031, with no status and no total, so the consumer must read the store, which means the save has to come first. The event is only a trigger; the truth is in the store.event carries stateevent carries an idorder.savedid: A-7031status: paidtotal: 640order.savedid: A-7031no status, no totaleasy to send before the saveconsumer trusts this copyconsumer must read the storeso the save has to come firstthe event is only a trigger; the truth is in the store
An event that carries state invites sending it before the save; an event that carries only an id makes every consumer read the stored truth.

At-least-once, not at-most-once

This approach carries a consequence. In durable systems, event delivery is almost always “at least once,” meaning an event may be delivered twice. Consumers must therefore be idempotent — processing the identical event twice must not produce an incorrect result. Combining strict publish-after-save ordering with idempotent consumers yields a system that neither drops messages nor corrupts state upon retry.

A retried event, two consumersA table with two deliveries of the same event, order.saved A-7031, and two consumers. The consumer that just acts charges 640 on delivery 1 and again on delivery 2, for 1280 in total. The idempotent consumer charges 640 on delivery 1 and notes A-7031 as seen; on delivery 2 it sees A-7031 before and skips it, for 640 in total, charged once.just actsno record of idschecks firstidempotentdelivery 1order.saveddelivery 2a retried copycharged in totalcharge 640charge 640note: A-7031 seen640 againalready seenskip it1280charged twice640charged oncea retry is safe only when the id is checked
A retried delivery charges 640 twice for a consumer that just acts, and is skipped by one that checks the id first.

Putting it together

The rule appears minor, but it forms the boundary between a system that remains resilient under load and one that fails irreproducibly. Enforce the order of operations — record the fact first, then announce the change — and an entire class of race conditions is eliminated. In designing durable systems, deterministic operational orderings matter most.