Devops Monk

REST APIs are synchronous — the caller waits for a response. Sometimes you don’t want that. An order creation shouldn’t wait for the inventory system, the notification system, and the analytics system to all respond before confirming to the user. Kafka decouples these concerns.

What Kafka Is

Kafka is a distributed event streaming platform. It stores events (messages) in an ordered, immutable log. Producers write to Kafka; consumers read from it. Producer and consumer are completely decoupled — the producer doesn’t know or care who’s consuming.

Order Service  →  [Kafka: order-events]  →  Inventory Service
                                         →  Notification Service
                                         →  Analytics Service

All three consumers get every order event. The order service fires and forgets.

Core Concepts

Topics and Partitions

A topic is a named log (e.g., order-events, payment-processed). Topics are divided into partitions for parallelism.

Topic: order-events (3 partitions)

Partition 0: [event-1] [event-4] [event-7] ...
Partition 1: [event-2] [event-5] [event-8] ...
Partition 2: [event-3] [event-6] [event-9] ...

Events in a partition are strictly ordered. Events across partitions are not. If you need all events for customer ABC to be in order, route them to the same partition using customerId as the key.

Offsets

Each event has an offset — its position in the partition. Consumers track which offset they’ve processed. If a consumer restarts, it reads from where it left off.

Partition 0:  offset 0    offset 1    offset 2    offset 3
              [event-1]   [event-4]   [event-7]   [event-10]
                                            ↑
                                    consumer committed here

Consumer Groups

Multiple consumers in the same consumer group share the work. Kafka assigns each partition to one consumer in the group at a time.

Consumer Group: inventory-service (3 consumers)

Partition 0 → Consumer A
Partition 1 → Consumer B
Partition 2 → Consumer C

Scale consumers by adding instances to the group — up to the partition count.

Running Kafka Locally

Use Docker Compose:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.7.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"

docker-compose up -d

No ZooKeeper needed — Kafka 3.3+ uses KRaft (Kafka Raft) for consensus internally.

Spring Boot Setup

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

spring:
  kafka:
    bootstrap-servers: localhost:9092

    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      acks: all                  # wait for all replicas to acknowledge
      retries: 3
      properties:
        enable.idempotence: true  # exactly-once delivery from producer

    consumer:
      group-id: order-service
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      auto-offset-reset: earliest   # start from beginning if no offset stored
      enable-auto-commit: false     # manual offset commit for reliability
      properties:
        spring.json.trusted.packages: "com.devopsmonk.*"

Topic Configuration

@Configuration
public class KafkaTopicConfig {

    @Bean
    public NewTopic orderEventsTopic() {
        return TopicBuilder.name("order-events")
            .partitions(6)
            .replicas(3)
            .config(TopicConfig.RETENTION_MS_CONFIG, String.valueOf(Duration.ofDays(30).toMillis()))
            .build();
    }

    @Bean
    public NewTopic paymentEventsTopic() {
        return TopicBuilder.name("payment-events")
            .partitions(6)
            .replicas(3)
            .build();
    }
}

Spring creates the topic if it doesn’t exist when the application starts.

Event Design

Design events as facts — things that happened, named in past tense:

// Good: event names are past-tense facts
public record OrderCreatedEvent(
    UUID orderId,
    UUID customerId,
    List<OrderItemDto> items,
    BigDecimal totalAmount,
    Instant occurredAt
) {}

public record PaymentProcessedEvent(
    UUID paymentId,
    UUID orderId,
    BigDecimal amount,
    String currency,
    Instant occurredAt
) {}

// Bad: commands or present-tense actions
// CreateOrder, ProcessPayment — those are commands, not events

Include enough context that consumers don’t need to call back to the source service:

// Bad: consumer must call Order Service to get order details
public record OrderCreatedEvent(UUID orderId) {}

// Good: consumer has everything it needs
public record OrderCreatedEvent(
    UUID orderId,
    UUID customerId,
    String customerEmail,
    String customerName,
    List<OrderItemDto> items,
    BigDecimal totalAmount,
    String shippingAddress,
    Instant occurredAt
) {}

Kafka vs REST: When to Use Each

Delivery Guarantees

Kafka offers three delivery modes:

For exactly-once end-to-end:

• Producer: enable.idempotence: true, acks: all
• Consumer: enable-auto-commit: false, manual offset commit after processing
• Processing: idempotent consumer logic (deduplicate on orderId)

What You’ve Learned

• Kafka decouples producers from consumers — producers write events, consumers read asynchronously
• Topics are divided into partitions; ordering is guaranteed within a partition
• Consumer groups share partitions across instances — add consumers to scale horizontally
• Events should be past-tense facts with enough context for consumers to act independently
• Use Kafka for background processing and multi-consumer fan-out; use REST for synchronous user-facing operations
• spring-kafka auto-configures producers and consumers from application.yml

Next: Article 43 — Producing and Consuming Kafka Messages — implement producers, consumers, error handling, and dead-letter topics in code.