Idempotent upsert by version

Queues deliver at least once. That means duplicates, and after a retry, events out of order. A consumer that just writes whatever arrives last will sometimes go backwards in time. A version number on each event fixes that, if the consumer uses it.

Write apply_events(state, events):

• state is a dict: id -> {"version": int, "data": ...}.
• Each event is a dict with "id", "version" and "data". "data": None means the record was deleted.
• Apply an event only if its id isn’t in the state yet, or its version is greater than the stored version. Otherwise ignore it. Equal versions are duplicates.
• A delete is stored like any other event, as {"version": v, "data": None}. Keeping it (a tombstone) stops an older update from bringing the record back.
• Return a new dict. Don’t change state.

Applying the same batch twice must give the same result as applying it once.

>>> apply_events({}, [
...     {"id": "a", "version": 2, "data": "new"},
...     {"id": "a", "version": 1, "data": "old"},
... ])
{'a': {'version': 2, 'data': 'new'}}

Approach

Compare versions before writing. Newer wins, everything else is ignored.

def apply_events(state, events):
    out = dict(state)
    for e in events:
        cur = out.get(e["id"])
        if cur is None or e["version"] > cur["version"]:
            out[e["id"]] = {"version": e["version"], "data": e["data"]}
    return out

Why this is idempotent: after an event is applied, the stored version equals its version, so the same event arriving again fails the strict > check. Why order stops mattering: for each id, the final entry is the one with the highest version, and a maximum doesn’t care about order.

The tombstone is what makes deletes safe. If you removed the key on delete, a late version 1 update would find no entry and recreate the record.

Complexity

O(len(state) + len(events)). The copy is linear. Each event is one lookup and maybe one write.

Why this matters

This is a last-writer-wins register keyed by version, and it is how change-data-capture consumers, Kafka sinks and search-index updaters survive retries. MERGE ... WHEN MATCHED AND src.version > dst.version is the SQL form. Tombstones show up everywhere deletes meet replication: Cassandra keeps them, Kafka compacted topics keep them for a while, and vector stores need them so a deleted document doesn’t come back on the next re-index.