Devops Monk

What Are Acknowledgments?

When a producer sends a record to a Kafka broker, it can wait for confirmation that the write was received and replicated before considering the send “complete.” The acks setting controls how much confirmation the producer requires.

flowchart LR
    Producer["Producer"]
    Leader["Partition Leader\n(Broker 1)"]
    F1["Follower\n(Broker 2)"]
    F2["Follower\n(Broker 3)"]

    Producer -->|"ProduceRequest"| Leader
    Leader -->|"replicate"| F1
    Leader -->|"replicate"| F2
    Leader -->|"ProduceResponse ✓"| Producer

    style Producer fill:#3b82f6,color:#fff
    style Leader fill:#10b981,color:#fff

The acknowledgment is the broker’s confirmation to the producer. Different acks settings give different durability guarantees — and different performance characteristics.

acks=0: Fire and Forget

The producer sends the record and does not wait for any acknowledgment.

sequenceDiagram
    participant Producer
    participant Leader as Leader\n(Broker 1)
    participant F1 as Follower 1
    participant F2 as Follower 2

    Producer->>Leader: ProduceRequest
    Note over Producer: Returns immediately\nNo waiting for ack

    Note over Leader: May crash before writing!
    Note over F1,F2: Never knows about this record

    Producer->>Producer: Considers send "complete"

• Throughput: maximum (no blocking at all)
• Durability: none — records can be lost if the leader crashes before writing to disk
• Use case: metrics, logs, or any data where loss is acceptable and throughput is critical

spring.kafka.producer.acks=0

acks=1: Leader Acknowledgment (Default)

The producer waits for the leader to write the record to its local log. Does not wait for followers.

sequenceDiagram
    participant Producer
    participant Leader as Leader\n(Broker 1)
    participant F1 as Follower 1\n(Broker 2)
    participant F2 as Follower 2\n(Broker 3)

    Producer->>Leader: ProduceRequest
    Leader->>Leader: Write to local log ✓
    Leader-->>Producer: ProduceResponse (offset=42)
    Note over Producer: Considers send complete

    Note over F1,F2: Replication happens asynchronously
    Leader->>F1: Replicate (async)
    Leader->>F2: Replicate (async)

    Note over Leader: If leader crashes HERE:\nF1 and F2 may not have the record yet\n→ record is LOST

• Throughput: high
• Durability: medium — records can be lost if the leader crashes between acknowledgment and follower replication
• Use case: default for many Spring applications; acceptable when occasional loss is tolerable

spring.kafka.producer.acks=1

acks=all (or acks=-1): All In-Sync Replicas

The producer waits for all in-sync replicas (ISR) to write the record. The leader only acknowledges after every ISR member confirms.

sequenceDiagram
    participant Producer
    participant Leader as Leader\n(Broker 1)
    participant F1 as Follower 1\n(Broker 2, ISR)
    participant F2 as Follower 2\n(Broker 3, ISR)

    Producer->>Leader: ProduceRequest (acks=all)
    Leader->>Leader: Write to local log
    Leader->>F1: Replicate
    Leader->>F2: Replicate
    F1-->>Leader: FetchResponse (confirms offset)
    F2-->>Leader: FetchResponse (confirms offset)
    Leader-->>Producer: ProduceResponse (offset=42)
    Note over Producer: Record is durable on ALL ISR members\nSurvives loss of any non-ISR broker

• Throughput: lower than acks=1 (waits for all ISR members)
• Durability: maximum — record survives as long as at least one ISR member is alive
• Use case: financial transactions, order events, any data where loss is unacceptable

spring.kafka.producer.acks=all

min.insync.replicas: The Safety Floor

acks=all waits for all current ISR members. If the ISR shrinks to just the leader (because followers fell behind), acks=all only requires acknowledgment from the leader — equivalent to acks=1.

min.insync.replicas sets the minimum number of ISR members that must be available for a produce request to succeed. It is a topic or broker-level configuration, not a producer setting.

flowchart TD
    subgraph Scenario1["Scenario 1: Healthy (ISR = 3)"]
        I1["ISR: {Leader, F1, F2}\nmin.insync.replicas=2\nacks=all\n→ Requires F1 OR F2 to also ack\n→ ✓ Write succeeds"]
    end

    subgraph Scenario2["Scenario 2: One follower down (ISR = 2)"]
        I2["ISR: {Leader, F1}\nmin.insync.replicas=2\nacks=all\n→ Leader + F1 both ack\n→ ✓ Write succeeds\n(barely meets minimum)"]
    end

    subgraph Scenario3["Scenario 3: Two followers down (ISR = 1)"]
        I3["ISR: {Leader only}\nmin.insync.replicas=2\nacks=all\n→ Only 1 ISR member\n→ ✗ NotEnoughReplicasException\n→ Producer gets error"]
    end

Setting min.insync.replicas

At the broker level (applies to all topics by default):

# server.properties or Docker environment variable
min.insync.replicas=2

At the topic level (overrides broker default for one topic):

docker exec kafka kafka-configs.sh \
  --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name orders \
  --alter \
  --add-config min.insync.replicas=2

In Spring Boot (via KafkaAdmin when creating topics programmatically):

@Bean
public NewTopic ordersTopic() {
    return TopicBuilder.name("orders")
        .partitions(3)
        .replicas(3)
        .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")
        .build();
}

The Durability Matrix

quadrantChart
    title Durability vs Throughput Trade-off
    x-axis Low Throughput --> High Throughput
    y-axis Low Durability --> High Durability
    quadrant-1 Best durability, lower throughput
    quadrant-2 Balanced
    quadrant-3 Worst durability, lower throughput
    quadrant-4 Low durability, high throughput
    acks=all + min.isr=2: [0.2, 0.9]
    acks=all + min.isr=1: [0.35, 0.7]
    acks=1: [0.75, 0.4]
    acks=0: [0.95, 0.05]

Handling NotEnoughReplicasException

When min.insync.replicas is not met, the broker rejects the produce request with NotEnoughReplicasException. Spring Kafka wraps this in a KafkaProducerException.

flowchart LR
    Producer -->|"ProduceRequest\nacks=all"| Leader
    Leader -->|"ISR count < min.insync.replicas"| Error["NotEnoughReplicasException"]
    Error -->|"wrapped in"| KPPE["KafkaProducerException"]
    KPPE -->|"delivered to"| Callback["CompletableFuture\n.whenComplete() callback"]
    Callback -->|"log + alert"| Alert["PagerDuty / Slack alert\n(cluster health issue)"]

kafkaTemplate.send("orders", event.orderId(), event)
    .whenComplete((result, ex) -> {
        if (ex != null) {
            if (ex.getCause() instanceof NotEnoughReplicasException) {
                // Cluster health issue — alert ops team
                log.error("[CRITICAL] Kafka cluster degraded: not enough ISR replicas. "
                    + "orderId={}", event.orderId(), ex);
                alertingService.sendCriticalAlert("Kafka ISR below minimum threshold");
            } else {
                log.error("Failed to publish order event: orderId={}", event.orderId(), ex);
            }
        }
    });

Recommended Production Configuration

@Bean
public ProducerFactory<String, Object> producerFactory() {
    Map<String, Object> config = new HashMap<>();
    config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);

    // Durability: wait for all ISR, retry indefinitely, no duplicates
    config.put(ProducerConfig.ACKS_CONFIG, "all");
    config.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
    config.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);  // prevents duplicate on retry

    // Timeouts: give up after 2 minutes total
    config.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120_000);
    config.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 30_000);
    config.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 100);

    // Performance
    config.put(ProducerConfig.LINGER_MS_CONFIG, 5);
    config.put(ProducerConfig.BATCH_SIZE_CONFIG, 32_768);
    config.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");

    return new DefaultKafkaProducerFactory<>(config);
}

And the topic-level config (set at topic creation):

@Bean
public NewTopic ordersTopic() {
    return TopicBuilder.name("orders")
        .partitions(3)
        .replicas(3)
        .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")
        .build();
}

Common Mistakes

flowchart TD
    M1["❌ Mistake 1:\nacks=all but replication.factor=1\n→ Only one copy exists,\nmin.isr=1 is the only option\n→ No real durability improvement"]
    M2["❌ Mistake 2:\nmin.insync.replicas=replication.factor\n(e.g. both = 3)\n→ ANY follower failure → cluster unavailable\n→ Too fragile"]
    M3["❌ Mistake 3:\nacks=1 for financial data\n→ Leader crash between ack and replication\n→ Silent data loss"]

    Fix1["✓ replication.factor=3, min.isr=2\n(tolerates 1 broker failure,\nstill requires 2 replicas for writes)"]
    Fix2["✓ min.isr = replication.factor - 1\n(tolerates 1 broker failure\nwithout blocking writes)"]
    Fix3["✓ acks=all + min.isr=2\nfor anything that matters"]

    M1 --- Fix1
    M2 --- Fix2
    M3 --- Fix3

The golden rule: min.insync.replicas = replication.factor - 1. With replication.factor=3 and min.insync.replicas=2, one broker can fail and writes continue. With both set to 3, any single failure halts all writes.

Key Takeaways

• acks=0: no waiting, maximum throughput, no durability guarantee
• acks=1: wait for leader write — leader crash after ack means data loss
• acks=all: wait for all ISR members — strongest durability guarantee
• min.insync.replicas sets the minimum ISR size required for acks=all to succeed; prevents the ISR from shrinking to just the leader
• Recommended production: acks=all + replication.factor=3 + min.insync.replicas=2
• When ISR drops below min.insync.replicas, brokers reject produce requests with NotEnoughReplicasException
• enable.idempotence=true must be paired with acks=all — prevents duplicates on retry (covered next)

Next: Producer Retries: Backoff, Timeouts, and Retry Strategies — configure how the producer retries failed sends, how long it waits between attempts, and how to set appropriate timeout boundaries.