Knowledge Guide
Home › Hands-On Builds › Backend Primitives

Build a Write-Ahead Log (WAL)

Build a Write-Ahead Log (WAL)

A write-ahead log is the single most important durability primitive in backend engineering: it's how databases, message queues, and replicated state machines survive a crash without losing acknowledged writes. The rule is in the name — before you mutate your in-memory state (or the main data file), you first append the change to a log and flush it to disk. If the process dies a millisecond later, recovery replays the log and nothing acknowledged is lost. Build one from an empty file and the phrase "we persist to a WAL" stops being hand-waving.

The spec

Milestones

The one idea that matters: fsync

A plain write() only copies bytes into the OS page cache — a crash loses them. fsync/fdatasync forces them to the physical device; that syscall returning is the exact moment the write becomes durable, and it's what you're really waiting on when you "acknowledge" a write. It's also the dominant cost, which is why real systems group-commit (batch many records into one fsync).

Reference — WAL core (Go)

type WAL struct{ f *os.File }

func Open(path string) (*WAL, error) {
    f, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR|os.O_APPEND, 0644)
    return &WAL{f}, err
}

// Append writes one framed record and fsyncs — returns only when durable.
func (w *WAL) Append(payload []byte) error {
    var hdr [8]byte
    binary.BigEndian.PutUint32(hdr[0:4], uint32(len(payload)))
    binary.BigEndian.PutUint32(hdr[4:8], crc32.ChecksumIEEE(payload))
    if _, err := w.f.Write(append(hdr[:], payload...)); err != nil {
        return err
    }
    return w.f.Sync() // <- the durability point
}

// Recover replays intact records; a torn or corrupt tail is discarded.
func Recover(path string) ([][]byte, error) {
    data, err := os.ReadFile(path)
    if err != nil { return nil, err }
    var recs [][]byte
    for off := 0; off+8 <= len(data); {
        n := binary.BigEndian.Uint32(data[off : off+4])
        crc := binary.BigEndian.Uint32(data[off+4 : off+8])
        end := off + 8 + int(n)
        if end > len(data) { break }                    // torn tail: partial write
        payload := data[off+8 : end]
        if crc32.ChecksumIEEE(payload) != crc { break }  // corruption
        recs = append(recs, payload)
        off = end
    }
    return recs, nil
}

Reference — WAL core (Java)

public final class WAL implements Closeable {
    private final FileChannel ch;
    public WAL(Path path) throws IOException {
        ch = FileChannel.open(path, CREATE, WRITE, READ, APPEND);
    }
    public void append(byte[] payload) throws IOException {
        CRC32 crc = new CRC32(); crc.update(payload);
        ByteBuffer buf = ByteBuffer.allocate(8 + payload.length);
        buf.putInt(payload.length).putInt((int) crc.getValue()).put(payload).flip();
        while (buf.hasRemaining()) ch.write(buf);
        ch.force(true);   // fsync — durability point
    }
    public static List<byte[]> recover(Path path) throws IOException {
        ByteBuffer b = ByteBuffer.wrap(Files.readAllBytes(path));
        List<byte[]> recs = new ArrayList<>();
        while (b.remaining() >= 8) {
            int len = b.getInt(), crc = b.getInt();
            if (b.remaining() < len) break;               // torn tail
            byte[] p = new byte[len]; b.get(p);
            CRC32 c = new CRC32(); c.update(p);
            if ((int) c.getValue() != crc) break;         // corruption
            recs.add(p);
        }
        return recs;
    }
}

Tests to write (this is where it gets real)

Extensions — how real systems go further

What you've mastered when this is done

🤖 Don't fully get this? Learn it with Claude

Stuck on Build a Write-Ahead Log (WAL)? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.

🎨 Explain it visually

Build the mental picture, not memorization.

I just read a lesson on **Build a Write-Ahead Log (WAL)** (Hands-On Builds) and want to truly understand it. Explain Build a Write-Ahead Log (WAL) from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🤔 Walk me through it (interactive)

Socratic — adapts to where you're stuck.

Teach me **Build a Write-Ahead Log (WAL)** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧪 Quiz me & fix my gaps

Active recall exposes what you missed.

Quiz me on **Build a Write-Ahead Log (WAL)** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧠 Make it stick

Intuition + hook + flashcards for long-term memory.

Help me remember **Build a Write-Ahead Log (WAL)** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

📝 My notes

← Build a Rate LimiterBuild an LRU + TTL Cache →