Introduction to Kafka

What is Kafka?

Apache Kafka is an open-source publish-subscribe-based messaging system. It is distributed, durable, fault-tolerant, and highly scalable by design. Fundamentally, it is a system that takes streams of messages from applications known as producers, stores them reliably on a central cluster (containing a set of brokers), and allows those messages to be received by applications (known as consumers) that process the messages.

Background

Kafka was created at LinkedIn around 2010 to track various events, such as page views, messages from the messaging system, and logs from various services. Later, it was made open-source and developed into a comprehensive system which is used for:

1. Reliably storing a huge amount of data.
2. Enabling high throughput of message transfer between different entities.
3. Streaming real-time data.

At a high level, we can call Kafka a distributed Commit Log. A Commit Log (also known as a Write-Ahead log or a Transactions log) is an append-only data structure that can persistently store a sequence of records. Records are always appended to the end of the log, and once added, records cannot be deleted or modified. Reading from a commit log always happens from left to right (or old to new).

Why a log and not a queue?

The retained log is the single design choice everything else falls out of. In a classic queue, reading a message consumes it — the broker deletes it on acknowledgment, so a second reader needs a second copy of the data. In Kafka, reads are non-destructive: consuming a record is just reading a position in a file, so N independent consumer groups can read the same topic and each costs the write path nothing extra — the bytes were already going to disk once. Because each group tracks only its own offset, replay is trivial: rewinding history means moving an integer backwards, not re-publishing data. And because the log is append-only, every write is a sequential disk write that flows through the OS page cache — which, more than any exotic engineering, is what buys Kafka its throughput (see the Scalability and Performance deep dive for the batching and zero-copy mechanics that stack on top).

Kafka stores all of its messages on disk. Since all reads and writes happen in sequence, Kafka takes advantage of sequential disk reads (more on this later).

Kafka use cases

Kafka can be used for collecting big data and real-time analysis. Here are some of its top use cases:

1. Metrics: Kafka can be used to collect and aggregate monitoring data. Distributed services can push different operational metrics to Kafka servers. These metrics can then be pulled from Kafka to produce aggregated statistics.
2. Log Aggregation: Kafka can be used to collect logs from multiple sources and make them available in a standard format to multiple consumers.
3. Stream processing: Kafka is quite useful for use cases where the collected data undergoes processing at multiple stages. For example, the raw data consumed from a topic is transformed, enriched, or aggregated and pushed to a new topic for further consumption. This way of data processing is known as stream processing.
4. Commit Log: Kafka can be used as an external commit log for any distributed system. Distributed services can log their transactions to Kafka to keep track of what is happening. This transaction data can be used for replication between nodes and also becomes very useful for disaster recovery, for example, to help failed nodes to recover their states.
5. Website activity tracking: One of Kafka's original use cases was to build a user activity tracking pipeline. User activities like page clicks, searches, etc., are published to Kafka into separate topics. These topics are available for subscription for a range of use cases, including real-time processing, real-time monitoring, or loading into Hadoop or data warehousing systems for offline processing and reporting.
6. Product suggestions: Imagine an online shopping site like amazon.com, which offers a feature of 'similar products' to suggest lookalike products that a customer could be interested in buying. To make this work, we can track every consumer action, like search queries, product clicks, time spent on any product, etc., and record these activities in Kafka. Then, a consumer application can read these messages to find correlated products that can be shown to the customer in real-time. Alternatively, since all data is persistent in Kafka, a batch job can run overnight on the 'similar product' information gathered by the system, generating an email for the customer with product suggestions.

Kafka common terms

Before digging deep into Kafka's architecture, let's first go through some of its common terms.

Brokers

A Kafka server is also called a broker. Brokers are responsible for reliably storing data provided by the producers and making it available to the consumers.

Records

A record is a message or an event that gets stored in Kafka. Essentially, it is the data that travels from producer to consumer through Kafka. A record contains a key, a value, a timestamp, and optional metadata headers.

Topics

Kafka divides its messages into categories called Topics. In simple terms, a topic is like a table in a database, and the messages are the rows in that table.

• Each message that Kafka receives from a producer is associated with a topic.
• Consumers can subscribe to a topic to get notified when new messages are added to that topic.
• A topic can have multiple subscribers that read messages from it.
• In a Kafka cluster, a topic is identified by its name and must be unique.

Messages in a topic can be read as often as needed — unlike traditional messaging systems, messages are not deleted after consumption. Instead, Kafka retains messages for a configurable amount of time or until a storage size is exceeded. Kafka's performance is effectively constant with respect to data size, so storing data for a long time is perfectly fine.

Producers

Producers are applications that publish (or write) records to Kafka.

Consumers

Consumers are the applications that subscribe to (read and process) data from Kafka topics. Consumers subscribe to one or more topics and consume published messages by pulling data from the brokers.

In Kafka, producers and consumers are fully decoupled and agnostic of each other, which is a key design element to achieve the high scalability that Kafka is known for. For example, producers never need to wait for consumers.

High-level architecture

At a high level, applications (producers) send messages to a Kafka broker, and these messages are read by other applications called consumers. Messages get stored in a topic, and consumers subscribe to the topic to receive new messages.

Kafka cluster

Kafka is deployed as a cluster of one or more servers, where each server is responsible for running one Kafka broker.

Cluster coordination (KRaft, not ZooKeeper)

Every Kafka broker needs the same picture of the cluster: which topics and partitions exist, who leads each partition, and which replicas are in-sync. Historically Kafka kept that picture in Apache ZooKeeper, a separate distributed coordination service. KRaft (Kafka Raft, KIP-500) removes that dependency: a small quorum of dedicated controller nodes runs Kafka's own Raft implementation over a replicated metadata log (__cluster_metadata), and every broker follows that log the same way a consumer follows any other topic. KRaft reached general availability in Kafka 3.3; ZooKeeper mode was deprecated in 3.5 and removed entirely in Kafka 4.0. See the deep dive on Kafka Coordination: ZooKeeper → KRaft for the migration path and trade-offs.

Interactive scenario

Play with the kafka simulator and predict the outcome before each step.

Kafka use-case decision table

Not every message problem is a Kafka problem. The table below is the decision frame senior engineers use.

Worked rejection: a team needs a background-job queue at ~50 messages/second — image resizes with per-job retries, priorities, and delayed redelivery. Kafka is the wrong pick here, mechanically: it has no per-message acknowledgment (only per-partition offsets, so one stuck job blocks everything behind it), no message priorities, and no native delayed redelivery — you would end up building retry topics and delay schedulers by hand. The partition-parallelism ceiling that justifies Kafka is irrelevant at 50 msg/s, and you would still carry the operational weight of a replicated broker cluster. SQS or RabbitMQ gives per-message ack, retry with backoff, dead-lettering, and priorities out of the box.

Producer → topic → consumer trace

Trace an order_placed event:

1. Producer (the checkout service) calls producer.send("orders", key=customer_id, value=order_json). The producer picks a partition using hash(customer_id) % num_partitions so all orders for one customer land in the same partition and stay ordered.
2. Broker appends the record to that partition's log on the leader broker and replicates it to in-sync replicas. With acks=all, the producer waits until the leader and all in-sync replicas acknowledge.
3. Topic orders retains the record according to its retention policy (e.g., 7 days or until compacted).
4. Consumer group fulfillment reads the topic. Kafka assigns each partition to one consumer in the group; the consumer advances its offset after processing. A separate consumer group analytics reads the same topic independently from its own offset.

The key observation: producers and consumers are fully decoupled. The producer does not know how many consumers exist, and consumers can pause, replay, or be added without affecting producers.

What breaks first in production (operability)

The model above is the happy path. These are the first signals that tell you the cluster or consumers are sick — learn them before you need the deep dives:

SLIs worth a dashboard: consumer lag by partition, UnderReplicatedPartitions, ActiveControllerCount (=1), OfflinePartitionsCount (=0), request handler idle %, produce/fetch p99.

Capacity sketch: how big is the cluster?

A back-of-envelope Kafka sizing follows two formulas. Retention storage ≈ ingest_rate × retention_window × RF ÷ compression_ratio — e.g. 100 MB/s in, 7-day retention, RF=3, 2× compression → 100 MB/s × 604,800 s × 3 ÷ 2 ≈ 90.7 TB of raw disk across the cluster. Throughput is bounded by sequential disk and network per broker (a single broker sustains on the order of tens to a few hundred MB/s of sequential I/O, multiplied by broker count and helped by producer batching + compression), so you add brokers until aggregate sequential-write bandwidth comfortably exceeds peak ingest × RF. Partition count then follows from the parallelism you need: at least one partition per concurrent consumer in the busiest group, since a partition is the unit of consumer parallelism.

Cross-link map to Kafka deep dives

• Kafka Coordination: ZooKeeper → KRaft — why the control plane changed and how to migrate.
• Kafka Internals — producers, consumers, partitions, offsets, and replication.
• Delivery Semantics — at-most/at-least/exactly-once and idempotent producer.
• Consumer Groups & Rebalancing — partition assignment, cooperative rebalancing, static membership.
• Retention vs Compaction — when to keep events and when to keep only the latest value.
• Connect vs Streams vs Custom Consumers — choosing the integration pattern (no dedicated page yet).
• RabbitMQ vs Kafka vs ActiveMQ — ecosystem and operational trade-offs.
• Throughput / batching / zero-copy: Scalability and Performance deep page (producer batching; idempotence ≠ full EOS).