Stable content hash for a record

Two copies of the same event should get the same ID, even if one has its keys in a different order or a different ingested_at. A content hash gives you that ID. It only works if the bytes you hash are the same every time.

Write record_hash(record, ignore=()):

• record is a dict of JSON-friendly values: strings, numbers, bools, None, lists and nested dicts.
• Drop any top-level key named in ignore.
• Turn what’s left into canonical JSON: keys sorted at every level, no spaces ("," and ":" as separators), and non-ASCII characters written as-is rather than as \u escapes.
• Encode that string as UTF-8 and return the SHA-256 hex digest (64 lowercase hex characters).
• Don’t change the input dict.

>>> record_hash({"b": 1, "a": 2}) == record_hash({"a": 2, "b": 1})
True
>>> record_hash({"a": 1})
'015abd7f5cc57a2dd94b7590f04ad8084273905ee33ec5cebeae62276a97f862'

Approach

Pin down the bytes, then hash them.

import hashlib
import json


def record_hash(record, ignore=()):
    kept = {k: v for k, v in record.items() if k not in ignore}
    text = json.dumps(kept, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

Each option removes one source of drift. sort_keys removes key order. separators removes whitespace. ensure_ascii=False plus an explicit UTF-8 encode fixes how non-ASCII text becomes bytes. str(record) fails on all three: it keeps insertion order, and its format is Python’s, not a spec.

Complexity

O(size of the record) to serialise and hash, plus O(k log k) per dict level to sort its k keys.

Why this matters

Content hashes are how pipelines become idempotent. Use the hash as the primary key and a replayed event lands on the same row instead of making a new one. The same trick powers caches (hash the prompt, skip the model call on a hit), dedupe in object stores, and Git itself. The traps are all in “same bytes”: floats like 1.0 vs 1, key order, Unicode forms, and fields like ingested_at that change on every retry. That last one is why ignore exists.