Build a Durable Job Queue

Build a Durable Job Queue (at-least-once + idempotency)

Every backend eventually needs "do this work later, reliably." Build the primitive behind SQS/Celery/Sidekiq: a durable queue with a visibility timeout, giving at-least-once delivery — and then make the consumer idempotent so at-least-once behaves like exactly-once in effect. The interview trap this defuses: "exactly-once delivery" is essentially impossible over an unreliable network; the real, shippable answer is at-least-once delivery + idempotent processing. Building it makes that answer yours.

The spec

• enqueue(job): durably append the job (reuse your WAL) and make it available. Returns after it's persisted.
• dequeue(): hand a job to a worker and make it invisible for a visibility timeout instead of deleting it.
• ack(id): the worker finished → delete the job for good.
• redelivery: if the timeout elapses with no ack (worker crashed), the job becomes visible again → at-least-once.
• idempotent consumer: because a job can be delivered more than once, processing must be safe to repeat.

The core idea: don't delete on dequeue — lease it

The whole reliability trick is that dequeue does not remove the job; it leases it (hides it for visibilityTimeout). Only ack deletes it. So a worker that crashes mid-job never loses it — the lease expires and another worker picks it up. The cost is duplicates (a slow worker's job can be redelivered while it's still running), which is exactly why the consumer must be idempotent.

Reference — broker (Go)

type Job struct{ ID string; Body string; visibleAt time.Time; acked bool }

type Queue struct {
    mu   sync.Mutex
    jobs []*Job                 // durable order (also appended to a WAL on enqueue)
    vis  time.Duration
}
func (q *Queue) Enqueue(body string) string {
    q.mu.Lock(); defer q.mu.Unlock()
    id := newID()
    q.jobs = append(q.jobs, &Job{ID: id, Body: body})   // + wal.Append("ENQ "+id+" "+body); fsync
    return id
}
func (q *Queue) Dequeue(now time.Time) (*Job, bool) {
    q.mu.Lock(); defer q.mu.Unlock()
    for _, j := range q.jobs {
        if !j.acked && now.After(j.visibleAt) {          // available (never leased, or lease expired)
            j.visibleAt = now.Add(q.vis)                  // LEASE: hide, don't delete → at-least-once
            return j, true
        }
    }
    return nil, false
}
func (q *Queue) Ack(id string) {
    q.mu.Lock(); defer q.mu.Unlock()
    for _, j := range q.jobs { if j.ID == id { j.acked = true } }  // + wal.Append("ACK "+id)
}

Reference — idempotent consumer (Java)

// The consumer makes duplicates harmless by recording processed IDs (durably, e.g. a UNIQUE key).
public final class IdempotentWorker {
    private final Set<String> processed = ConcurrentHashMap.newKeySet();  // in prod: a DB unique constraint
    private final Queue queue;

    public void runOnce() {
        queue.dequeue().ifPresent(job -> {
            if (!processed.add(job.id())) {      // already done → skip the side effect, still ack
                queue.ack(job.id()); return;     // this is what turns at-least-once into exactly-once *effect*
            }
            handle(job);                         // the real side effect — runs exactly once per id
            queue.ack(job.id());
        });
    }
}

Key point tested in interviews: idempotency lives in the consumer (a dedup key / unique constraint / conditional write), not the delivery layer. The broker guarantees at least once; the consumer guarantees at most one effect. Together: exactly-once effect. I validated this dedup behaviour (m1 applied once, duplicates skipped) before writing the code above.

Tests to write

• Happy path: enqueue → dequeue → ack → dequeue returns nothing.
• At-least-once on crash: dequeue but don't ack; advance time past the visibility timeout → the same job is dequeued again (not lost).
• No double-delivery while leased: within the timeout, a second dequeue does not return the leased job.
• Idempotency: feed the worker the same job id twice → the side effect runs once, both are acked.
• Durability: enqueue, kill the broker, restart from the WAL → the job is still there.

Extensions

• Dead-letter queue: after N redeliveries, move a poison job aside instead of looping forever.
• Delayed/scheduled jobs: visibleAt = now + delay — same mechanism gives you scheduling for free.
• Priority / fairness: multiple queues or a heap by priority.
• Ordering: per-key FIFO (partition by key, one in-flight per key) — how Kafka/SQS-FIFO preserve order.

What you've mastered

• Why lease-don't-delete gives at-least-once, and the duplicate it inevitably creates.
• That idempotency belongs in the consumer (dedup key / unique constraint) — the correct answer to "how do you get exactly-once?"
• Visibility timeouts, DLQs, and delayed delivery — the vocabulary of every managed queue.

I just read a lesson on **Build a Durable Job Queue** (Hands-On Builds) and want to truly understand it. Explain Build a Durable Job Queue from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Build a Durable Job Queue** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Build a Durable Job Queue** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Build a Durable Job Queue** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.