Recover a log with a torn final record

A process appends a record to a log file and the power goes out halfway through the write. On restart, the last record is cut short, or its bytes are garbage. Recovery has to keep every complete record and find exactly where the good data ends.

Each record in the file is laid out like this:

| length: 4 bytes | crc: 4 bytes | payload: length bytes |

• length is the payload size, an unsigned 32-bit big-endian integer.
• crc is zlib.crc32(payload), unsigned 32-bit big-endian.
• payload is UTF-8 text.

Write read_log(data):

• data is a bytes object holding zero or more records back to back.
• Read records from the start. Stop at the first record that is incomplete (not enough bytes for the header or the payload) or whose checksum doesn’t match. Ignore everything from there on.
• Return {"records": [...], "valid_bytes": n}: the decoded payloads in order, and the byte offset where the good data ends. Truncating the file to n bytes makes it clean again.

>>> import struct, zlib
>>> def rec(s):
...     b = s.encode()
...     return struct.pack(">II", len(b), zlib.crc32(b)) + b
>>> read_log(rec("a") + rec("bc") + rec("def")[:5])
{'records': ['a', 'bc'], 'valid_bytes': 19}

Approach

Walk the bytes with an offset. Advance it only past records that are complete and verified.

import struct
import zlib


def read_log(data):
    records = []
    pos = 0
    n = len(data)
    while pos + 8 <= n:
        length, crc = struct.unpack_from(">II", data, pos)
        start = pos + 8
        end = start + length
        if end > n:
            break
        payload = data[start:end]
        if zlib.crc32(payload) != crc:
            break
        records.append(payload.decode("utf-8"))
        pos = end
    return {"records": records, "valid_bytes": pos}

Three checks, in order. Enough bytes for a header. Enough bytes for the payload the header promises. A matching checksum. The length check alone is not enough: a torn write can leave a header whose length field is garbage, or a full-length record whose last bytes were never flushed. The CRC catches both.

Stopping at the first bad record, instead of skipping it, matters. Once one length is wrong, every offset after it is wrong too, so later “records” can’t be trusted.

Complexity

O(n) in the size of the log. Each byte is sliced and checksummed once.

Why this matters

This is how write-ahead logs recover. Postgres, SQLite, Kafka segments and LevelDB all frame records with a length and a checksum, scan forward on startup, and truncate at the first bad frame. That truncation point is valid_bytes. The log stays append-only, a crash costs at most the record that was mid-write, and that record was never acknowledged to the client anyway.