Testing the Concurrency in Your Design — Carving Seams & Forcing the Race

Testing the Concurrency in Your Design — Carving Seams & Forcing the Race

Elsewhere in this guide you learned to fix concurrency bugs: the seat-booking two-phase hold, SELECT ... FOR UPDATE on the last parking spot, TTL holds that expire, idempotency keys that dedupe a double-submit. Every one of those fixes is a claim — "under contention, exactly one writer wins." This page is about the claim you almost never see made in an interview or a PR: prove it. How do you write a test that fails when the lost-update bug is present and passes only when the fix is in?

The uncomfortable truth up front: you cannot prove a concurrency fix with a plain multi-threaded test. Spin up two threads, both book the seat, assert — and it will pass. It will pass on the buggy code too, most runs, because the OS scheduler almost never chooses the one interleaving that exposes the race. A green multi-threaded test is not evidence of correctness; it is evidence that the scheduler was kind today. Staff-level testing of concurrency is a design discipline, and it has exactly two moves.

1. The two moves: reach the race, then force it

Every testable concurrency claim decomposes into two independent problems, and conflating them is why most attempts are flaky.

• Reachability (a design problem) — carve seams. The contended moment lives inside a transaction, behind a clock, behind a repository. If your code calls Instant.now() directly and opens its own JDBC connection inline, there is no place for a test to stand. You must design seams — points where a test can substitute a controllable stand-in (a fake clock, a repository that pauses mid-transaction, an executor you drive). Michael Feathers' definition is exact: a seam is "a place where you can alter behavior in your program without editing in that place." No seam, no test.
• Determinism (a scheduling problem) — force the interleaving. Even with a seam, Thread.sleep to "line things up" is a guess about timing, and guesses are flaky. Instead you use a synchronization barrier (CountDownLatch, CyclicBarrier, Phaser) to hold every thread at the exact critical instant and release them together, so the contended interleaving fires on every run — not one run in ten thousand.

Get both and the lost-update bug becomes deterministic: red on the buggy code, green on the fix, every single time. That determinism — not the mere existence of threads — is what makes the test worth trusting.

2. The canonical test: two threads grab the last seat, exactly one wins

This is the test you write on the whiteboard. The latch releases both threads at once so their book() calls start as close together as possible — a single-count latch is a release gate, not a true rendezvous, so for a narrow critical section prefer the CyclicBarrier/Phaser version below, which forces every thread to arrive before any proceeds. The oracle is the count of successes. Note what we assert on: not "no exception," but exactly one success and one clean rejection. On buggy code (a naive read-then-write with no locking) both threads read "1 seat free," both write, and successes == 2 — the test goes red. That red is the whole point.

@Test
void twoThreadsGrabLastSeat_exactlyOneWins() throws Exception {
    // one seat left for show 42, seat 7
    seatRepo.insertAvailable(show(42), seat(7));

    var start = new CountDownLatch(1);          // release gate
    var done  = new CountDownLatch(2);          // join gate
    var successes = new AtomicInteger();
    var conflicts = new AtomicInteger();
    var pool = Executors.newFixedThreadPool(2);

    Runnable attempt = () -> {
        try {
            start.await();                       // park until released together
            bookingService.book(show(42), seat(7), someUser());
            successes.incrementAndGet();
        } catch (SeatAlreadyBookedException e) {
            conflicts.incrementAndGet();         // the expected loser
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        } finally {
            done.countDown();
        }
    };
    pool.submit(attempt);
    pool.submit(attempt);

    start.countDown();                           // fire the race
    assertTrue(done.await(5, TimeUnit.SECONDS)); // no thread hung on a lock
    pool.shutdownNow();

    assertEquals(1, successes.get(), "exactly one booking may succeed");
    assertEquals(1, conflicts.get(), "the other must be cleanly rejected");
    assertEquals(1, seatRepo.countBookings(show(42), seat(7)));  // DB-level oracle
}

Two subtleties that separate a real test from a demo. First, the affected-row-count is the truest oracle: the fix is usually a conditional write — UPDATE seats SET booked_by = ? WHERE id = ? AND booked_by IS NULL — and it returns rows-updated. The winner gets 1, the loser gets 0, and the service turns a 0 into SeatAlreadyBookedException. Asserting the count both at the service boundary and in the DB catches a fix that "returns success" but wrote nothing. Second, two threads is often enough to make the bug deterministic with a latch; bumping to N threads (below) buys confidence against subtler windows but is not what makes it reliable — the latch is.

The parking-lot last-spot race is the identical shape: N cars, one free spot, latch them at the allocation call, assert successes == 1 and that the lot's occupied count never exceeds capacity. Same skeleton, same oracle. Once you see that the "exactly one winner" invariant is reusable, you have a template for every scarce-resource race in the guide.

3. Scaling the pressure: a barrier for N threads

A CountDownLatch is single-use — perfect for one release. When you want N threads to rendezvous repeatedly (e.g. round after round of contention), a CyclicBarrier re-arms itself, and Phaser generalizes to a dynamic party size across phases. The mechanism is the same: everyone blocks at the barrier, and the barrier trips only when the last thread arrives, releasing all simultaneously.

int n = 16;
var barrier = new CyclicBarrier(n);             // all n meet, then go together
var successes = new AtomicInteger();
var pool = Executors.newFixedThreadPool(n);
var latch = new CountDownLatch(n);

for (int i = 0; i < n; i++) {
    pool.submit(() -> {
        try {
            barrier.await();                     // rendezvous: released as one burst
            allocator.claimLastSpot(lot(1));
            successes.incrementAndGet();
        } catch (NoSpotAvailableException expected) {
            // the n-1 losers land here
        } catch (Exception e) {
            throw new CompletionException(e);
        } finally {
            latch.countDown();
        }
    });
}
assertTrue(latch.await(5, TimeUnit.SECONDS));
pool.shutdownNow();
assertEquals(1, successes.get());               // one spot, one winner, regardless of n

Rule of thumb: latch when you release once, barrier when you release repeatedly, phaser when the party size changes per phase. All three replace Thread.sleep, which is a timing guess — it either wastes wall-clock time or, worse, releases threads in staggered order and quietly stops testing the race at all.

4. Seam #1: inject a Clock so TTL and hold-expiry are testable without sleeping

The seat-hold expires after 2 minutes; the idempotency key is valid for a window; the parking ticket's grace period ends. Every one of these is a time decision, and the instant you write Instant.now() inside the logic, you have made time untestable — the only way to test expiry is to actually wait, which is both slow and flaky. The seam is java.time.Clock: inject it, and a test can advance time by hours in nanoseconds.

// BEFORE — time is hard-wired; expiry can only be tested by waiting 2 real minutes
class SeatHoldService {
    Hold place(SeatId seat, UserId user) {
        Instant expiresAt = Instant.now().plus(Duration.ofMinutes(2));  // untestable
        return holds.save(new Hold(seat, user, expiresAt));
    }
    boolean isExpired(Hold h) {
        return Instant.now().isAfter(h.expiresAt());                    // untestable
    }
}

// AFTER — Clock is a seam. Production injects Clock.systemUTC(); tests inject a fake.
class SeatHoldService {
    private final Clock clock;
    private final HoldRepository holds;
    SeatHoldService(Clock clock, HoldRepository holds) {
        this.clock = clock; this.holds = holds;
    }
    Hold place(SeatId seat, UserId user) {
        Instant expiresAt = clock.instant().plus(Duration.ofMinutes(2));
        return holds.save(new Hold(seat, user, expiresAt));
    }
    boolean isExpired(Hold h) {
        return clock.instant().isAfter(h.expiresAt());
    }
}

Now expiry is deterministic and instant. Clock.fixed pins "now"; to advance it, use a mutable test clock (a few lines of your own, or Spock's MutableClock; note Spring deliberately has none):

@Test
void holdExpiresAfterTwoMinutes() {
    var tick = new MutableClock(Instant.parse("2026-07-11T10:00:00Z"), ZoneOffset.UTC);
    var service = new SeatHoldService(tick, holds);

    Hold h = service.place(seat(7), someUser());
    assertFalse(service.isExpired(h));           // t = 0:00

    tick.advance(Duration.ofMinutes(1));
    assertFalse(service.isExpired(h));           // t = 1:00, still held

    tick.advance(Duration.ofMinutes(1).plusSeconds(1));
    assertTrue(service.isExpired(h));            // t = 2:01, expired — no real waiting
}

The same seam makes a subtler concurrency claim testable: "a hold expiring at the exact moment another user tries to grab the seat." Advance the clock to the expiry boundary, then latch two threads — the reaper and the new booker — and assert the seat ends up owned by exactly one. That test is impossible with Instant.now() because you cannot pin the boundary.

5. Seam #2: a repository that pauses mid-transaction

Sometimes the race window is narrow — the read and the write are microseconds apart — and even a latch struggles to land two threads inside it reliably. The fix is to widen the window on purpose in the test. Because the repository is behind an interface (a seam), you can inject a fake that blocks the first caller after it reads but before it writes, holding the door open while the second caller races through. This makes the lost-update deterministic even for a hair-thin window, and it does so without any sleep.

/** Blocks the first reader after it reads count, until released — forcing the overlap. */
class PausingSeatRepository implements SeatRepository {
    private final SeatRepository real;
    private final CountDownLatch firstHasRead = new CountDownLatch(1);
    private final CountDownLatch mayProceed   = new CountDownLatch(1);
    private final AtomicBoolean firstCall     = new AtomicBoolean(true);

    PausingSeatRepository(SeatRepository real) { this.real = real; }

    @Override public int freeSeats(ShowId show) {
        int free = real.freeSeats(show);
        if (firstCall.getAndSet(false)) {
            firstHasRead.countDown();            // signal: I have read a stale count
            await(mayProceed);                   // ...and I will hold here
        }
        return free;
    }
    @Override public int tryBook(ShowId show, SeatId seat, UserId u) {
        return real.tryBook(show, seat, u);      // conditional UPDATE ... WHERE booked_by IS NULL
    }
    void releaseFirstReader() { mayProceed.countDown(); }
    void awaitFirstRead()     { await(firstHasRead); }
}

The test choreographs it: start thread A, wait for awaitFirstRead() (A now holds a stale "1 free"), start thread B and let it complete the booking, then releaseFirstReader() so A attempts its write against a seat B already took. With a correct conditional write, A's tryBook returns 0 rows and the service rejects it — successes == 1. With the buggy unconditional write, A overwrites B and you get a lost update. This is the most surgical way to prove a specific interleaving is handled; the cost is that the fake encodes intimate knowledge of the read-then-write sequence, so it is coupled to that code path.

6. Testing idempotency: same key twice, concurrently, one side effect

An idempotency key promises that a retried or double-clicked request produces exactly one insert / charge / email — even when both copies arrive at once (the hard case the guide's fix targets). The test fires the same key from two latched threads and asserts a single side effect. The DB-level oracle is decisive: one row, or a unique-constraint violation caught and translated into the cached original response.

@Test
void sameIdempotencyKeyConcurrently_oneSideEffect() throws Exception {
    var key = IdempotencyKey.of("order-4f2a");
    var start = new CountDownLatch(1);
    var done  = new CountDownLatch(2);
    var pool  = Executors.newFixedThreadPool(2);
    var responses = new ConcurrentLinkedQueue<OrderResult>();

    var failures = new ConcurrentLinkedQueue<Throwable>();
    Runnable submit = () -> {
        try {
            start.await();
            responses.add(orderService.placeOrder(key, cart()));  // both use SAME key
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        } catch (RuntimeException e) {
            failures.add(e);   // the loser leaking a 500 / duplicate-key is a FAILURE, not a silent drop
        } finally { done.countDown(); }
    };
    pool.submit(submit); pool.submit(submit);

    start.countDown();
    assertTrue(done.await(5, TimeUnit.SECONDS));
    pool.shutdownNow();

    // Exactly one order persisted...
    assertEquals(1, orderRepo.countByIdempotencyKey(key));
    // ...one charge issued...
    assertEquals(1, paymentGateway.chargeCount());
    // ...neither caller leaked an exception (the loser must get the cached result, not a 500)...
    assertTrue(failures.isEmpty(), () -> "a caller threw instead of returning: " + failures);
    // ...BOTH callers actually returned a result (guards against a masked drop)...
    assertEquals(2, responses.size(), "both callers must return a response");
    // ...and BOTH see the SAME order id (the second got the cached result, not an error).
    var ids = responses.stream().map(OrderResult::orderId).distinct().toList();
    assertEquals(1, ids.size(), "both callers must observe one identical order");
}

The assertion that catches the most bugs is the last one: it is not enough that only one row exists — the losing caller must receive the same result, not a 500 or a duplicate-key stack trace leaking out. That is the difference between "idempotent" and "merely deduplicated." The insert typically relies on a unique constraint on the key; the fake gateway's chargeCount() proves the external side effect fired once, which an in-memory row count alone would miss.

7. When the fake isn't enough: reach for the real thing

An in-memory fake repository can prove your application logic handles a lost read. It cannot reproduce database-level semantics — row locks, FOR UPDATE blocking, SKIP LOCKED skipping, isolation-level anomalies (phantom reads, write skew under REPEATABLE READ). Those live in Postgres, not in your fake. Testing them against H2 or a hand-rolled fake produces confident green tests that lie, because H2's locking is not Postgres's.

• Testcontainers → real Postgres. When the fix is a database mechanism — SELECT ... FOR UPDATE serializing two bookers, FOR UPDATE SKIP LOCKED letting a job queue hand distinct rows to distinct workers, or a SERIALIZABLE transaction aborting on write skew — you must test against the actual engine. Testcontainers boots a throwaway Postgres in Docker per test class; you then run the same latch-coordinated two-thread test, but now the DB itself enforces the ordering and you can assert that thread B blocked until A committed (or was skipped, for SKIP LOCKED). This is the only way to verify lock behavior with fidelity.
• jcstress → pure JVM memory-model claims. When the claim is about the Java Memory Model itself — is this volatile enough, is this lock-free counter actually safe, does this double-checked-locking publish correctly — hand-rolled latch tests are the wrong tool. The JCStress harness runs billions of iterations across real interleavings and knows the JMM's legal outcomes. Point to it for in-JVM visibility/ordering questions; do not try to reproduce it with CountDownLatch. (We do not duplicate its mechanics here — reach for the tool.)

8. Selection & trade-offs — which technique proves which claim

The escalation ladder: latch unit test for the invariant you can name (default, cheap, in CI) → Testcontainers when the guarantee is delegated to the database → stress/soak to hunt unnamed windows → jcstress for pure memory-model claims. Coverage rises left-to-right; speed falls. One rung further out, when the claim is about a distributed protocol's design rather than any single implementation — "does this two-phase hold / leader-election / commit protocol have a losing interleaving at all?" — the tool is a model checker (TLA+/TLC, Alloy): it explores the state space exhaustively at the spec level, catching design bugs before a line of code exists. That is verification of the design, not a test of the code, so it complements rather than replaces the ladder above. Most interview answers should lead with the latch test (it is the one you can write live and it is deterministic) and name Testcontainers/jcstress as the escalation when the claim moves below your code into the engine or the JMM.

Pitfalls

• Trusting a green multi-threaded test. Without a forced interleaving it passes on buggy code too. Green ≠ correct; it means the scheduler didn't pick the bad ordering. Only a test that goes red on the bug is evidence.
• Thread.sleep to coordinate. A timing guess: too short and threads don't overlap (race untested), too long and CI crawls. Replace every coordinating sleep with a latch/barrier. Sleep tests are the #1 source of "flaky, re-run it" CI rot.
• Hard-coded Instant.now() / new Date(). Makes TTL, hold-expiry, and time-boundary races untestable except by real waiting. Inject a Clock; there is no downside in production and it unlocks the whole time dimension in tests.
• Mocking away the concurrency you meant to test. If you mock the repository to "return success," you have deleted the read-modify-write that is the bug. The seam must preserve the contended operation, not stub it out.
• A fake DB that can't reproduce row-lock semantics. H2/in-memory locking ≠ Postgres. A green FOR UPDATE test against a fake is a false positive. Use Testcontainers for anything that leans on real lock or isolation behavior.
• Asserting "no exception" instead of the invariant. The oracle must be the domain truth — exactly one winner, one row, one side effect, affected-row-count of 1/0 — not the absence of a crash.

Key takeaways

• A plain multi-threaded test proves nothing — it passes nondeterministically on buggy code. Testing concurrency is two deliberate moves: carve seams so the race is reachable, and force the interleaving so it fires every run.
• Latch/barrier + a domain oracle ("exactly one winner," affected-row-count, one side effect) turns the seat-booking and parking-lot races into deterministic red-on-bug / green-on-fix tests — the answer you write on the whiteboard.
• Design for testability with DI: inject a Clock (advance time, never sleep), a repository behind an interface (pause mid-transaction to widen the window), and a controllable executor. No seam, no test.
• Match the tool to the claim: latch unit test for your logic, Testcontainers-real-Postgres for FOR UPDATE/SKIP LOCKED/isolation, stress/soak for unnamed windows, jcstress for JVM memory-model correctness — coverage vs speed vs fidelity.

Sources: Goetz et al., Java Concurrency in Practice, ch. 12 "Testing Concurrent Programs" (latch/barrier coordination, testing for safety vs liveness); Michael Feathers, Working Effectively with Legacy Code (the definition and use of seams); John Ousterhout, A Philosophy of Software Design (designing modules for testability); Testcontainers docs (throwaway real Postgres in tests); OpenJDK JCStress (Java Memory Model concurrency testing).