The Problem Managing Complex Interactions in Distributed Systems

The Problem: Managing Complex Interactions in Distributed Systems

When you split a monolith into microservices, you also split what used to be a single, reliable function call. Inside a monolith, placeOrder() might charge a card, reserve inventory, and create a shipment as three ordinary method calls wrapped in one database transaction. Either all of it commits, or none of it does. The language runtime and the database guarantee this for free.

Now put each of those steps behind its own service, its own database, and its own network hop. The single function call becomes an interaction: Order Service calls Payment Service, which calls Inventory Service, which calls Shipping Service. Suddenly you are exposed to a whole new category of problems that simply did not exist before: the network can drop a message, a service can be slow or dead, a call can succeed on one machine but the confirmation can be lost on the way back, and there is no shared transaction to roll everything back if step 3 of 4 fails. This lesson is about naming those problems precisely, because you cannot choose the right pattern (Saga, orchestration, circuit breaker, idempotency) until you understand exactly what is breaking.

How the interaction breaks, precisely

A distributed interaction has three failure dimensions a single-process call does not:

• Partial failure. With N services in a chain, any subset can fail independently. If Payment succeeds but Inventory times out, you are left in an inconsistent intermediate state: money is captured but no stock is reserved. There is no ACID transaction spanning the services to undo the payment.
• Uncertainty of outcome. When a call times out, the caller genuinely does not know whether the operation happened. The request may have been lost on the way out (never executed), or the response may have been lost on the way back (executed, but you never heard). This is the two generals problem, and it forces you to make requests idempotent so a retry cannot double-charge.
• Coupling in time and load. Synchronous request/response means the caller is blocked and alive only as long as every downstream service is alive. A slow dependency doesn't just fail one request — it holds a thread and a connection, and under load those exhaust. This is how one slow service triggers a cascading failure.

The core tension: you want the simplicity of a single logical operation, but you're building it out of unreliable, independently-deployed parts. Every microservices interaction pattern is an answer to one of these three problems — how to keep data consistent without distributed transactions (Saga), how to coordinate a multi-step flow (orchestration vs. choreography), and how to stop failures from spreading (circuit breaker, bulkhead, timeout, retry with backoff).

A worked scenario with numbers

Consider a checkout flow at 500 QPS. The Order Service makes a synchronous call to a Fraud-check dependency whose normal p50 latency is 20 ms. Each order thread holds one connection while it waits. With a connection pool of 200, the math is comfortable: 500 QPS × 0.020 s = 10 concurrent requests in flight on average (Little's Law), well under 200.

Now the Fraud service degrades — a bad deploy pushes its latency to 2000 ms (100×). The same 500 QPS now needs 500 × 2.0 = 1000 concurrent slots, but the pool caps at 200. Within ~400 ms the pool is fully saturated; every new checkout request blocks waiting for a connection, then times out. The Order Service — which was perfectly healthy — now returns errors for 100% of checkouts, even for orders that don't need fraud checking. One slow dependency took down the whole checkout path. That is a cascading failure, and it's the canonical argument for a circuit breaker (trip open after, say, 50% errors in a 10 s window, fail fast for 5 s, then probe) plus a tight timeout (e.g. 200 ms, not the default 30 s) and a bulkhead that caps fraud-check calls to, say, 50 connections so they can never starve the rest.

Trade-offs: the ways to manage these interactions, and when to pick each

There is no single fix — you're choosing along two axes. Consistency (how do the steps stay coherent?) and coordination style (who drives the flow?).

• Synchronous request/response (REST/gRPC). Simple, strongly-consistent read-your-writes, easy to reason about. Use when the caller genuinely needs the answer now (e.g. "is this card valid?") and the chain is short. Avoid when the chain is deep or a step is slow — it maximizes temporal coupling and cascade risk.
• Saga (async, compensating transactions). Replaces the missing distributed transaction with a sequence of local transactions, each with a compensating action (refund undoes charge). Gives you eventual consistency, not atomicity. Use when a business operation spans services and must not leave money/inventory dangling. Cost: you must write and test every compensator, and expose intermediate states.
• Orchestration vs. Choreography. Orchestration = one coordinator explicitly drives each step (easy to see the flow, but the orchestrator is a coupling point). Choreography = services react to events with no central brain (loose, scalable, but the end-to-end flow is emergent and hard to debug). Orchestrate complex flows with many branches; choreograph simple fan-outs and notifications.
• Two-phase commit (2PC). The "real" distributed transaction. Almost never use it across microservices: it's a synchronous blocking protocol, the coordinator is a single point of failure, and it doesn't scale — which is precisely why Sagas exist.

Pitfalls an interviewer probes

• Retries without idempotency. "Your payment call times out — do you retry?" If yes, and the first attempt actually succeeded, you double-charge. The expected answer: retries require an idempotency key so the server dedupes, plus exponential backoff with jitter so retries don't synchronize into a retry storm that DDoSes the recovering service.
• Timeouts that are too long or absent. A default 30 s client timeout is how the pool-exhaustion cascade above happens. Interviewers want to hear that timeouts should be set relative to the dependency's p99, not left at the library default.
• Treating Saga as atomic. Candidates often say "then we roll back." You cannot roll back a committed local transaction — you compensate, and compensation itself can fail and must be retried. And there is a window where partial state is visible to other reads.
• Ignoring the dual-write problem. "You update your DB and publish an event — what if the DB commit succeeds but the publish fails?" The answer is the transactional outbox pattern (write the event to the same DB transaction, relay it separately), not two independent writes.
• Distributed tracing / observability. When a request crosses 6 services, "it's slow" is unanswerable without correlation IDs and traces. Expect to be asked how you'd even find which hop failed.

Key takeaways

• Splitting a monolith turns a reliable in-process call into a networked interaction that suffers partial failure, uncertain outcomes, and temporal coupling — none of which existed before.
• There is no distributed ACID transaction in practice; you trade atomicity for eventual consistency via Sagas and compensating actions, coordinated by orchestration (central, debuggable) or choreography (loose, emergent).
• One slow dependency can saturate connection pools and cascade; contain it with tight timeouts, circuit breakers, bulkheads, and backoff+jitter retries.
• Every retry needs an idempotency key, and every DB-plus-event write needs a transactional outbox — these are the details interviewers use to separate memorized patterns from real understanding.