Distributed Database Consensus: 6 Core Modules for Raft from Theory to Implementation

数据库

Distributed consensus is the cornerstone of distributed databases—TiDB, CockroachDB, etcd, and Consul all rely on consensus protocols to guarantee data consistency. Yet, the Paxos paper is notoriously difficult to understand, and its engineering implementation is even more hellish; split-brain issues have countless teams fighting fires late at night; Leader election thrashing causes intermittent service unavailability; log replication latency makes cross-region deployments virtually useless; and linearizable consistency guarantees give developers endless headaches. The Raft protocol, designed with "understandability" as its core principle, decomposes the consensus problem into three sub-problems—Leader election, log replication, and safety—making it the de facto standard consensus protocol for distributed databases in 2026.

Core Concepts at a Glance

Concept Description
Raft Understandable distributed consensus protocol, decomposing consensus into election, replication, and safety
Leader Election Follower times out, becomes Candidate, initiates election, wins majority votes to become Leader
Log Replication Leader replicates client requests as log entries to all Followers
Heartbeat Timeout Leader sends periodic heartbeats; Followers trigger election if timeout
Term Raft's logical clock, incremented each election, used to detect stale information
Commit Index Log index confirmed by majority of nodes; entries before this index are committed
Snapshot Compress committed logs into a state snapshot to prevent unbounded log growth
Membership Change Dynamically add/remove cluster nodes without producing two Leaders
Linearizable Consistency Reads observe the most recent write; Raft implements via ReadIndex or Lease Read

Five Core Challenges

Production Raft consensus is far more than "elect a Leader and replicate logs." You must address these 5 core challenges:

1. Leader Election Stability — Network jitter causes frequent elections, making the cluster unavailable during Leader transitions. How to avoid election storms? How to set reasonable timeout parameters?

2. Log Replication Consistency — After network partitions, logs may diverge. How to truncate conflicting logs upon recovery? How to let slow nodes catch up without affecting overall throughput?

3. Network Partition Handling — The minority partition keeps failing elections while the majority partition serves normally. How to safely merge after partition recovery?

4. Membership Change Safety — Adding/removing nodes in one step may create a window where two Leaders coexist. How to implement safe membership changes?

5. Snapshot and Log Compaction — Unbounded log growth exhausts disk and memory. When to trigger snapshots? How to transfer snapshots without blocking normal replication?

Module 1: Raft State Machine and Leader Election

Raft nodes have three states: Follower, Candidate, and Leader. A Follower that doesn't receive a heartbeat within the election timeout becomes a Candidate and initiates an election.

type NodeState int

const (
    Follower NodeState = iota
    Candidate
    Leader
)

type RaftNode struct {
    mu            sync.Mutex
    id            int
    state         NodeState
    currentTerm   int
    votedFor      int
    log           []LogEntry
    commitIndex   int
    lastApplied   int
    nextIndex     map[int]int
    matchIndex    map[int]int
    heartbeatCh   chan struct{}
    electionTimer *time.Timer
    peers         []string
}

func (rn *RaftNode) startElection() {
    rn.mu.Lock()
    rn.state = Candidate
    rn.currentTerm++
    rn.votedFor = rn.id
    term := rn.currentTerm
    lastLogIndex := len(rn.log) - 1
    lastLogTerm := 0
    if lastLogIndex >= 0 {
        lastLogTerm = rn.log[lastLogIndex].Term
    }
    rn.mu.Unlock()

    votesReceived := 1
    voteCh := make(chan bool, len(rn.peers))

    for i, peer := range rn.peers {
        if i == rn.id {
            continue
        }
        go func(peerAddr string) {
            resp := rn.requestVote(peerAddr, &RequestVoteArgs{
                Term:         term,
                CandidateID:  rn.id,
                LastLogIndex: lastLogIndex,
                LastLogTerm:  lastLogTerm,
            })
            voteCh <- resp.VoteGranted
        }(peer)
    }

    for i := 0; i < len(rn.peers)-1; i++ {
        if <-voteCh {
            votesReceived++
        }
    }

    rn.mu.Lock()
    defer rn.mu.Unlock()
    if votesReceived > len(rn.peers)/2 && rn.currentTerm == term {
        rn.state = Leader
        rn.nextIndex = make(map[int]int)
        rn.matchIndex = make(map[int]int)
        for i := range rn.peers {
            rn.nextIndex[i] = len(rn.log)
            rn.matchIndex[i] = 0
        }
        go rn.heartbeatLoop()
    }
}

Key Point: Election timeouts should be randomized (150-300ms) to prevent all nodes from timing out simultaneously and splitting votes. A Candidate must win a majority of votes to become Leader.

Module 2: Log Replication and Consistency

When the Leader receives a client request, it appends the operation to its local log, then replicates it to all Followers. Once a majority confirms, the entry is committed.

type LogEntry struct {
    Term    int
    Index   int
    Command interface{}
}

func (rn *RaftNode) appendEntries(args *AppendEntriesArgs) *AppendEntriesReply {
    rn.mu.Lock()
    defer rn.mu.Unlock()

    reply := &AppendEntriesReply{Term: rn.currentTerm}

    if args.Term < rn.currentTerm {
        reply.Success = false
        return reply
    }

    if args.PrevLogIndex >= 0 {
        if args.PrevLogIndex >= len(rn.log) {
            reply.Success = false
            return reply
        }
        if rn.log[args.PrevLogIndex].Term != args.PrevLogTerm {
            rn.log = rn.log[:args.PrevLogIndex]
            reply.Success = false
            return reply
        }
    }

    for i, entry := range args.Entries {
        idx := args.PrevLogIndex + 1 + i
        if idx < len(rn.log) {
            if rn.log[idx].Term != entry.Term {
                rn.log = rn.log[:idx]
                rn.log = append(rn.log, entry)
            }
        } else {
            rn.log = append(rn.log, entry)
        }
    }

    if args.LeaderCommit > rn.commitIndex {
        lastNewIdx := args.PrevLogIndex + len(args.Entries)
        if args.LeaderCommit < lastNewIdx {
            rn.commitIndex = args.LeaderCommit
        } else {
            rn.commitIndex = lastNewIdx
        }
    }

    rn.currentTerm = max(rn.currentTerm, args.Term)
    reply.Term = rn.currentTerm
    reply.Success = true
    return reply
}

func (rn *RaftNode) replicateLog() {
    rn.mu.Lock()
    defer rn.mu.Unlock()

    if rn.state != Leader {
        return
    }

    for i, peer := range rn.peers {
        if i == rn.id {
            continue
        }
        prevIdx := rn.nextIndex[i] - 1
        prevTerm := 0
        if prevIdx >= 0 && prevIdx < len(rn.log) {
            prevTerm = rn.log[prevIdx].Term
        }
        entries := rn.log[rn.nextIndex[i]:]

        go func(peerAddr string, peerID int) {
            resp := rn.sendAppendEntries(peerAddr, &AppendEntriesArgs{
                Term:         rn.currentTerm,
                LeaderID:     rn.id,
                PrevLogIndex: prevIdx,
                PrevLogTerm:  prevTerm,
                Entries:      entries,
                LeaderCommit: rn.commitIndex,
            })
            rn.mu.Lock()
            if resp.Success {
                rn.nextIndex[peerID] = prevIdx + len(entries) + 1
                rn.matchIndex[peerID] = rn.nextIndex[peerID] - 1
            } else {
                rn.nextIndex[peerID] = max(1, rn.nextIndex[peerID]-1)
            }
            rn.mu.Unlock()
        }(peer, i)
    }
}

Key Point: Log consistency checks use PrevLogIndex and PrevLogTerm. If a Follower's log diverges from the Leader, the Leader decrements nextIndex step by step until finding the point of agreement.

Module 3: Heartbeat and Timeout Mechanism

Heartbeats are the core of Raft's operation—the Leader maintains authority through heartbeats, and Followers detect Leader liveness through heartbeats.

func (rn *RaftNode) heartbeatLoop() {
    ticker := time.NewTicker(50 * time.Millisecond)
    defer ticker.Stop()

    for {
        <-ticker.C
        rn.mu.Lock()
        if rn.state != Leader {
            rn.mu.Unlock()
            return
        }
        rn.mu.Unlock()
        rn.replicateLog()
    }
}

func (rn *RaftNode) electionTimeoutLoop() {
    for {
        rn.resetElectionTimer()
        select {
        case <-rn.electionTimer.C:
            rn.mu.Lock()
            if rn.state != Leader {
                rn.mu.Unlock()
                rn.startElection()
                continue
            }
            rn.mu.Unlock()
        case <-rn.heartbeatCh:
            continue
        }
    }
}

func (rn *RaftNode) resetElectionTimer() {
    timeout := time.Duration(150+rand.Intn(150)) * time.Millisecond
    if rn.electionTimer != nil {
        rn.electionTimer.Stop()
    }
    rn.electionTimer = time.NewTimer(timeout)
}

Key Point: The heartbeat interval should be much smaller than the election timeout (typically 1/5 to 1/10) to prevent Followers from timing out before the Leader's heartbeat arrives. Adjust timeouts based on RTT for cross-region deployments.

Module 4: Snapshot and Log Compaction

Unbounded log growth exhausts resources. The snapshot mechanism compresses committed logs into a state machine snapshot, retaining only logs after the snapshot.

type Snapshot struct {
    LastIncludedIndex int
    LastIncludedTerm  int
    Data              []byte
}

func (rn *RaftNode) takeSnapshot() {
    rn.mu.Lock()
    defer rn.mu.Unlock()

    if rn.commitIndex <= 0 {
        return
    }

    snapshotIdx := rn.commitIndex
    snapshotTerm := rn.log[snapshotIdx].Term

    stateData := rn.stateMachine.Serialize()

    snap := Snapshot{
        LastIncludedIndex: snapshotIdx,
        LastIncludedTerm:  snapshotTerm,
        Data:              stateData,
    }

    rn.log = rn.log[snapshotIdx+1:]
    rn.lastApplied = snapshotIdx
    rn.persistSnapshot(snap)
}

func (rn *RaftNode) installSnapshot(args *InstallSnapshotArgs) *InstallSnapshotReply {
    rn.mu.Lock()
    defer rn.mu.Unlock()

    reply := &InstallSnapshotReply{Term: rn.currentTerm}

    if args.Term < rn.currentTerm {
        return reply
    }

    if args.LastIncludedIndex > rn.commitIndex {
        rn.log = rn.log[args.LastIncludedIndex-rn.lastApplied:]
        rn.commitIndex = args.LastIncludedIndex
        rn.lastApplied = args.LastIncludedIndex
    }

    rn.stateMachine.Deserialize(args.Data)
    rn.currentTerm = max(rn.currentTerm, args.Term)
    reply.Term = rn.currentTerm
    return reply
}

Key Point: Snapshot size should be controlled at 10-100MB. Snapshot transfer must not block normal log replication; use a separate RPC channel.

Module 5: Membership Change and Safety

Switching directly from old to new configuration may cause split-brain. Raft implements safe changes through Joint Consensus.

type ConfigChange struct {
    Type    string // "add" or "remove"
    NodeID  int
    Address string
}

func (rn *RaftNode) proposeConfigChange(change ConfigChange) error {
    rn.mu.Lock()
    if rn.state != Leader {
        rn.mu.Unlock()
        return fmt.Errorf("not leader")
    }
    rn.mu.Unlock()

    entry := LogEntry{
        Term:    rn.currentTerm,
        Command: change,
    }

    rn.mu.Lock()
    rn.log = append(rn.log, entry)
    rn.mu.Unlock()

    rn.replicateLog()
    return nil
}

func (rn *RaftNode) applyConfigChange(change ConfigChange) {
    rn.mu.Lock()
    defer rn.mu.Unlock()

    switch change.Type {
    case "add":
        if !rn.containsPeer(change.NodeID) {
            rn.peers = append(rn.peers, change.Address)
            rn.nextIndex[change.NodeID] = len(rn.log)
            rn.matchIndex[change.NodeID] = 0
        }
    case "remove":
        newPeers := make([]string, 0)
        for i, p := range rn.peers {
            if i != change.NodeID {
                newPeers = append(newPeers, p)
            }
        }
        rn.peers = newPeers
        delete(rn.nextIndex, change.NodeID)
        delete(rn.matchIndex, change.NodeID)
    }
}

Key Point: Production environments recommend single-node changes (add/remove one node at a time) to avoid the complexity of Joint Consensus. Ensure the cluster still has a majority before removing a node.

Module 6: Client Read/Write and Linearizable Consistency

If Raft reads bypass the Leader, stale data may be returned. Linearizable reads require ReadIndex or Lease Read.

func (rn *RaftNode) linearizableRead() (interface{}, error) {
    rn.mu.Lock()
    if rn.state != Leader {
        leaderID := rn.currentLeader
        rn.mu.Unlock()
        return nil, fmt.Errorf("not leader, redirect to %d", leaderID)
    }

    readIndex := rn.commitIndex
    term := rn.currentTerm
    rn.mu.Unlock()

    confirmCh := make(chan bool, len(rn.peers))
    confirmed := 1

    for i, peer := range rn.peers {
        if i == rn.id {
            continue
        }
        go func(addr string) {
            resp := rn.sendHeartbeat(addr, term)
            confirmCh <- resp.Success
        }(peer)
    }

    for i := 0; i < len(rn.peers)-1; i++ {
        if <-confirmCh {
            confirmed++
        }
    }

    if confirmed <= len(rn.peers)/2 {
        return nil, fmt.Errorf("lost leadership")
    }

    rn.mu.Lock()
    for rn.lastApplied < readIndex {
        rn.mu.Unlock()
        time.Sleep(time.Millisecond)
        rn.mu.Lock()
    }
    result := rn.stateMachine.Read()
    rn.mu.Unlock()

    return result, nil
}

func (rn *RaftNode) leaseRead() (interface{}, error) {
    rn.mu.Lock()
    defer rn.mu.Unlock()

    if rn.state != Leader {
        return nil, fmt.Errorf("not leader")
    }

    if time.Since(rn.leaseStart) > rn.leaseDuration {
        return nil, fmt.Errorf("lease expired, fallback to ReadIndex")
    }

    return rn.stateMachine.Read(), nil
}

Key Point: ReadIndex guarantees linearizability but requires one RPC round-trip. Lease Read relies on clock assumptions—better performance but with risk. Production recommends ReadIndex + 1-second Lease optimization.

5 Common Pitfalls

# Pitfall Consequence Correct Approach
1 ❌ Fixed election timeout Multiple nodes timeout simultaneously, splitting votes ✅ Randomize election timeout (150-300ms) to avoid vote splitting
2 ❌ Followers respond to reads directly Return stale data, violating linearizable consistency ✅ Forward reads to Leader, use ReadIndex for consistency
3 ❌ One-step membership change Two Leaders may coexist during change (split-brain) ✅ Single-node change or Joint Consensus two-phase change
4 ❌ Snapshot blocks log replication Cluster writes stall during slow snapshot transfer ✅ Separate RPC channel for snapshot transfer, don't block AppendEntries
5 ❌ Not persisting currentTerm and votedFor Node may vote twice after restart, breaking election safety ✅ Synchronously persist Term/Vote to stable storage on every update

10 Error Troubleshooting

# Error Symptom Possible Cause Troubleshooting Method
1 Frequent Leader switches Election timeout too short or high network latency Increase election timeout, ensure heartbeat interval < election timeout / 5
2 High log replication latency Slow nodes dragging down overall commit Enable async replication, set maxInflight batch size limit
3 term mismatch Old Leader still trying to write after network partition Check if Leader holds latest Term; old Leader auto-downgrades after partition recovery
4 Snapshot transfer OOM Snapshot too large, loaded into memory at once Transfer snapshot in chunks, 1-4MB per chunk
5 Cluster unavailable after membership change Lost majority after change Ensure nodes still form majority after change; deploy odd number of nodes
6 Reading stale data Follower responding to reads without going through Leader Enable ReadIndex or Lease Read for linearizable reads
7 Log lost after node restart Logs not persisted to stable storage fsync after every log append; use WAL for durability
8 commitIndex not advancing Minority of nodes down, cannot reach majority Check if live nodes form majority; remove failed nodes if necessary
9 Election storms Heartbeat interval and election timeout ratio imbalanced Set heartbeat interval to 1/10 of election timeout; add PreVote phase
10 Inconsistent state after snapshot Atomicity issue between snapshot write and log truncation Complete snapshot write and log truncation in the same transaction

Advanced Optimization

1. PreVote Phase Prevents Election Storms — Add a PreVote phase before formal RequestVote. Only initiate election when logs are sufficiently up-to-date, preventing election storms after network partition recovery.

2. Batch Log Replication for Throughput — The Leader merges multiple log entries into a single AppendEntries RPC, reducing network round-trips. etcd defaults to a batch limit of 1024 entries.

3. Async Snapshot Transfer — Use a separate streaming RPC for snapshot transfer without blocking the normal log replication channel. Apply rate limiting to prevent snapshot transfer from consuming all bandwidth.

4. Learner Nodes Reduce Change Risk — New nodes join as Learners first, catch up on logs, then convert to Voters—preventing new nodes from blocking commits.

5. ReadIndex Cache Optimization — The Leader caches the latest ReadIndex after heartbeat confirmation. Subsequent reads use the cached value, reducing heartbeat confirmation frequency.

Comparison: Raft vs Multi-Paxos vs EPaxos vs ZAB

Dimension Raft Multi-Paxos EPaxos ZAB
Understandability ✅ Designed for clarity ❌ Obscure paper, complex implementation ⚠️ Moderate, relies on dependency graphs ⚠️ Moderate, similar to Raft
Leader Dependency ✅ Strong Leader model ✅ Has Leader but optimizable ❌ Leaderless, any replica can propose ✅ Strong Leader model
Cross-Region Latency ❌ Writes require Leader confirmation ⚠️ Optimizable but complex ✅ Leaderless, write to nearest replica ❌ Writes require Leader confirmation
Log Ordering ✅ Strong ordering, easy to reason ⚠️ Allows out-of-order, complex ❌ Dependency graph determines order ✅ Strong ordering
Membership Change ✅ Single-node change is simple ❌ Complex implementation ⚠️ Moderate ⚠️ Moderate
Ecosystem ✅ etcd/Consul/TiKV ⚠️ Chubby/Megastore ❌ Limited ecosystem ✅ ZooKeeper
Performance ⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐

Selection Guide: Single-region deployment → Raft (mature ecosystem, easy to understand); cross-region active-active → EPaxos (low latency with nearest-write); ZooKeeper ecosystem → ZAB; Paxos only when legacy constraints exist.

Summary and Outlook

The Raft protocol has become the de facto standard for distributed database consensus in 2026—its strong Leader model simplifies log management, randomized election timeouts prevent vote splitting, and the snapshot mechanism solves log bloat. But production deployment requires overcoming 5 core challenges: Leader election stability, log replication consistency, network partition handling, membership change safety, and snapshot/log compaction. The 6 core modules presented in this article—state machine and election, log replication, heartbeat timeout, snapshot compression, membership change, and linearizable reads—cover the complete chain from theory to production. Remember: a consensus protocol is not just about electing a Leader—it's a complete engineering system from election safety to linearizable consistency.

  • JSON Formatter — Format Raft log and configuration JSON for quick cluster state debugging
  • Hash Calculator — Generate checksum fingerprints for snapshots and logs to ensure data integrity
  • cURL to Code Converter — Convert etcd/Consul API cURL commands to Go client code
  • Base64 Encoder — Encode binary snapshot data in Raft RPC for transmission

Try these browser-local tools — no sign-up required →

#分布式共识#Raft协议#Paxos#一致性算法#Leader选举#2026#数据库