Leader-Follower Replication

Leader-Follower Architecture

                    ┌──────────────┐
                    │   Leader     │
                    │  (Partition 0)│
                    └───────┬──────┘
                            │
                 Replication (async/sync)
                            │
          ┌─────────────────┼─────────────────┐
          │                 │                 │
          ▼                 ▼                 ▼
    ┌──────────┐      ┌──────────┐      ┌──────────┐
    │Follower 1│      │Follower 2│      │Follower 3│
    │ (Replica)│      │ (Replica)│      │ (Replica)│
    └──────────┘      └──────────┘      └──────────┘

WRITE FLOW:

1. Producer → Leader (write)
2. Leader → Append to local log
3. Leader → Replicate to Followers
4. Followers → Acknowledge replication
5. Leader → Acknowledge to Producer (after sync replicas)

READ FLOW:

- Option 1: Read from Leader (strong consistency)
- Option 2: Read from Followers (eventual consistency, higher throughput)

FAILURE SCENARIO:
Leader Fails:
┌──────────────┐
│ Leader ✕ │
└──────────────┘
↓
Election Process
↓
┌──────────────┐
│ New Leader ✓ │ ← Follower 1 promoted
│ (Follower 1) │
└──────────────┘

Single Node vs Leader-Follower

Single Node (No Replication):
┌─────────────┐
│  Server A   │
│  [DATA]     │
└─────────────┘
↓
Server crashes → DATA LOST ✕

Leader-Follower Replication:
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Leader │ ──→ │ Follower 1 │ ──→ │ Follower 2 │
│ [DATA] │ │ [DATA] │ │ [DATA] │
└─────────────┘ └─────────────┘ └─────────────┘
↓
Leader crashes → Follower promoted → DATA SAFE ✓

Synchronous Replication

1. Client → Write to Leader
2. Leader → Write to local log
3. Leader → Send to Follower 1 & 2 (parallel)
4. Follower 1 → Acknowledge
5. Follower 2 → Acknowledge
6. Leader → Acknowledge to Client (AFTER followers ACK)

Timeline:
Client Write ─┐
│
Leader Write ├──► [10ms]
│
Followers ACK ├──► [20ms] ← Wait for all followers
│
Client ACK └──► [25ms] ← Client waits 25ms total

Pros:
✓ No data loss (all replicas have data before ACK)
✓ Strong consistency (read from any replica is up-to-date)

Cons:
✕ High latency (wait for slowest follower)
✕ Availability issues (if follower down, writes block)

Asynchronous Replication

1. Client → Write to Leader
2. Leader → Write to local log
3. Leader → Acknowledge to Client (IMMEDIATE)
4. Leader → Send to Followers (async, in background)
5. Followers → Eventually apply updates

Timeline:
Client Write ─┐
│
Leader Write ├──► [10ms]
│
Client ACK └──► [12ms] ← Client doesn't wait for followers
↓
Followers ACK (background, 50ms later)

Pros:
✓ Low latency (client doesn't wait for followers)
✓ High availability (follower failures don't block writes)

Cons:
✕ Potential data loss (if leader crashes before replication)
✕ Stale reads (followers may lag behind leader)

In-Sync Replicas (ISR)

Config: replication.factor=3, min.insync.replicas=2

Replicas:

- Leader (always in ISR)
- Follower 1 (in ISR if < 10s lag)
- Follower 2 (in ISR if < 10s lag)
- Follower 3 (NOT in ISR, lagging > 10s)

Write Flow:

1. Client → Write to Leader
2. Leader → Write + send to all followers
3. Wait for min.insync.replicas=2 ACKs (Leader + 1 follower)
4. Acknowledge to Client

Guarantees:
✓ At least 2 replicas have data before ACK
✓ Fast writes (don't wait for slow follower 3)
✕ If ISR count < min.insync.replicas, writes fail (availability hit)

Best of both worlds!

Failure Detection

Heartbeat Mechanism:
Followers → Send heartbeat to Leader every 1s
Leader → Send heartbeat to Followers every 1s

Failure Scenarios:

1. Leader misses heartbeats → Followers detect leader failure
2. Follower misses heartbeats → Leader removes from ISR
3. Network partition → Split-brain prevention needed
 

Leader Election Process

LEADER ELECTION (Simplified):

1. FAILURE DETECTION
 Leader fails (no heartbeat for 10s)

2. ELECTION TRIGGER
 Followers detect failure
 Start election process

3. CANDIDATE SELECTION
 Criteria for new leader:
 - ✓ In ISR (up-to-date replica)
 - ✓ Highest offset (most data)
 - ✓ Lowest broker ID (tie-breaker)

4. NEW LEADER ANNOUNCED
 Controller broadcasts new leader
 Followers connect to new leader

5. RESUME OPERATIONS
 New leader accepts writes
 Old leader (if recovers) becomes follower

Total Failover Time: ~5-30 seconds

Kafka Controller Architecture

KAFKA ARCHITECTURE:

┌─────────────────────────────────────────┐
│ ZooKeeper / KRaft │ ← Metadata store
│ - Broker liveness │
│ - Controller election │
│ - Partition assignments │
└────────────────┬────────────────────────┘
│
┌───────┴────────┐
│ Controller │ ← ONE broker elected as controller
│ (Broker 2) │ Manages all leader elections
└───────┬────────┘
│
┌────────────┼────────────┐
│ │ │
┌───┴───┐ ┌───┴───┐ ┌───┴───┐
│Broker1│ │Broker2│ │Broker3│
└───────┘ └───────┘ └───────┘

Controller Responsibilities:

1. Monitor broker liveness
2. Elect partition leaders when failures occur
3. Update metadata in ZooKeeper/KRaft
4. Notify all brokers of leadership changes
 

Replication Lag

Leader: [msg0][msg1][msg2][msg3][msg4][msg5] ← Offset 5

Follower 1: [msg0][msg1][msg2][msg3][msg4][msg5] ← Lag: 0 ✓ IN ISR

Follower 2: [msg0][msg1][msg2][msg3] ← Lag: 2 ⚠️ IN ISR

Follower 3: [msg0][msg1] ← Lag: 4 ✕ OUT OF ISR

ISR Criteria:

- replica.lag.time.max.ms=10000 (10 seconds)
- If follower doesn't fetch within 10s → Removed from ISR

ISR Dynamics

TIMELINE OF ISR CHANGES:

T=0: All replicas in ISR
ISR = [Leader, Follower1, Follower2, Follower3]

T=15s: Follower3 network issue, can't fetch
ISR = [Leader, Follower1, Follower2]
(Follower3 removed after 10s lag)

T=30s: Follower3 recovers, catches up
ISR = [Leader, Follower1, Follower2, Follower3]
(Follower3 re-added after catching up)

T=45s: Leader crashes
Election triggered
New Leader = Follower1 (highest offset in ISR)
ISR = [Follower1(new leader), Follower2, Follower3]

Read from Leader

All reads go to Leader:

Clients → Leader (reads)
↓
[DATA v5] ← Always latest version

Pros:
✓ Strong consistency (always up-to-date)
✓ Simple (no staleness issues)

Cons:
✕ Leader bottleneck (all read traffic)
✕ Doesn't scale with more replicas

Read from Followers

Reads distributed across replicas:

Client A → Follower 1 [DATA v4] ← Slightly stale
Client B → Follower 2 [DATA v5] ← Up-to-date
Client C → Leader [DATA v5] ← Always latest

Pros:
✓ Read scalability (horizontal scaling)
✓ Lower latency (geographically closer follower)

Cons:
✕ Eventual consistency (may read stale data)
✕ Monotonic read issues (read v5, then v4)

Read-Your-Writes Consistency

Strategy: Track last write offset, read only from replicas >= that offset

1. Client writes to Leader → Receives offset 100
2. Client reads → Request includes "minOffset=100"
3. Router → Send to follower with offset >= 100
4. If no follower caught up → Read from Leader

Result: Client always reads its own writes ✓

High Availability Critical Systems

Scenario: E-commerce platform requiring 99.99% uptime
Solution: 3 replicas, auto-failover on leader crash
Result: Survive single node failure with under 30s downtime

Read-Heavy Workloads

Scenario: News site with 10:1 read/write ratio
Solution: 1 leader + 5 followers, reads from followers
Result: 6x read throughput

Geo-Distributed Reads

Scenario: Global application with users in US, EU, Asia
Solution: Leader in US, followers in EU and Asia
Result: Low-latency reads for all regions

Multi-Region Writes

Problem: Users in EU and Asia need to write locally
Issue: All writes go to single leader (high latency)
Alternative: Multi-leader replication or sharding

Need for Strong Consistency Reads

Problem: Bank balance must always be current
Issue: Follower reads may be stale
Alternative: Read from leader or use quorum reads

Extremely High Write Throughput

Problem: 100K writes/sec overwhelming single leader
Issue: Leader bottleneck
Alternative: Partition data across multiple leaders (sharding)