Pattern: Bounded Producer-Consumer & the Web Crawler (Problem 16)

Many producers, many consumers, bounded memory (Problem 16)

The multithreaded web crawler is the producer-consumer pattern at scale: URLs are discovered (produced) and fetched (consumed) concurrently, you must bound in-flight work so memory doesn't explode, and you must dedup visited URLs across threads. It's the shape of every job queue, ingestion pipeline, and task runner.

The three pieces

1. Bounded blocking queue — gives back-pressure: when full, producers block instead of piling up unbounded work (the OOM failure mode).
2. Worker pool — N consumers fetch + parse in parallel; new links feed back into the queue.
3. Concurrent visited set — putIfAbsent on a ConcurrentHashMap / ConcurrentHashMap.newKeySet() so each URL is crawled exactly once (a check-then-act done atomically).

BlockingQueue<String> q = new LinkedBlockingQueue<>(10_000);   // bounded
Set<String> seen = ConcurrentHashMap.newKeySet();
for (int i = 0; i < WORKERS; i++) pool.submit(() -> {
    while (running) {
        String url = q.take();                      // blocks if empty
        for (String link : fetchAndParse(url))
            if (seen.add(link)) q.put(link);         // add() is atomic; put blocks if full
    }
});

In Go — channels are the bounded queue

q := make(chan string, 10_000)       // buffered channel = bounded queue
var seen sync.Map
for i := 0; i < workers; i++ {
    go func() {
        for url := range q {
            for _, link := range fetchAndParse(url) {
                if _, dup := seen.LoadOrStore(link, true); !dup { q <- link }
            }
        }
    }()
}

The hard part: knowing when you're done

You can't just close the queue — workers keep adding links. Track in-flight work with an atomic counter (or a WaitGroup): increment on enqueue, decrement when a URL is fully processed; when it hits zero, the crawl is complete and you can stop the workers.

Pitfalls

• Unbounded queue → memory blow-up on a large site. Always bound + back-pressure.
• Racy visited check (if(!seen.contains) seen.add) re-crawls URLs — use one atomic add/LoadOrStore.
• Termination: naive workers block forever on an empty queue; use the in-flight counter or poison-pill sentinels to shut down.

Takeaways

• Bounded blocking queue (back-pressure) + worker pool + concurrent dedup set.
• Dedup must be a single atomic op; termination needs an in-flight counter, not a queue close.
• Go: a buffered channel IS the bounded queue; sync.Map/LoadOrStore for dedup.

---

Re-authored for this guide; pipeline diagram hand-authored as SVG. Covers Problem 16. See also: Thread Pools & Executors, Go Concurrency, Data Parallelism.

I just read a lesson on **Pattern: Bounded Producer-Consumer & the Web Crawler (Problem 16)** (Concurrency) and want to truly understand it. Explain Pattern: Bounded Producer-Consumer & the Web Crawler (Problem 16) from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Pattern: Bounded Producer-Consumer & the Web Crawler (Problem 16)** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Pattern: Bounded Producer-Consumer & the Web Crawler (Problem 16)** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Pattern: Bounded Producer-Consumer & the Web Crawler (Problem 16)** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.