Handling Consistency Problems in Distributed Systems

Why Consistency Problems Occur

In traditional monolithic architectures, data state changes usually happen within the same service and database. Databases that follow ACID properties provide strong consistency, meaning inconsistent states are not visible to users.

For example, relational databases such as MySQL ensure strong consistency through mechanisms like MVCC (Multi-Version Concurrency Control).

However, as systems evolve:

User traffic increases
Data volume grows rapidly
Services become more complex

Many systems migrate from monolithic architectures to service-oriented or microservice architectures.

Microservices are a double-edged sword:

✅ They improve agility, scalability, and deployment efficiency
❌ They introduce network communication, which is inherently unreliable

Once services communicate over a network, partial failures become unavoidable, and consistency issues start to appear.

Consistency Models

Strong Consistency

After an update completes, all subsequent reads return the latest value.

Most user-friendly
Requires sacrificing availability (CAP theorem)

Weak Consistency

The system does not guarantee that reads will immediately reflect the latest writes.

No guarantee on when data becomes consistent

Eventual Consistency

A special form of weak consistency.

The system guarantees that, if no new updates occur, data will eventually become consistent
DNS is a classic example

The inconsistency window depends on:

Network latency
System load
Replication strategy

CAP Theorem

CAP defines three properties of distributed systems:

Consistency (C): All nodes see the same data at the same time
Availability (A): Every request receives a response
Partition Tolerance (P): The system continues to operate despite network partitions

A distributed system can only guarantee two of the three properties simultaneously.

BASE Theory

BASE is a practical approach for large-scale distributed systems that relaxes strong consistency.

Basically Available: The system remains available
Soft State: State may change over time, even without new input
Eventually Consistent: Data becomes consistent over time

BASE sacrifices strong consistency to achieve high availability and scalability.

How to Achieve Eventual Consistency

When systems grow large and data is horizontally partitioned, strong consistency becomes expensive or impossible.

The core idea behind eventual consistency is:

Query → Detect → Compensate

Query: Tracking Operation State

To detect inconsistencies, systems must:

Persist operation state
Uniquely identify each operation

Persisting Operation State

Before executing an operation (or sending a message):

Persist an operation record with status PENDING
Execute the operation
Update status to SUCCESS or FAILED

This record is stored in a reliable, independent system (database or message store).

Unique Operation Identification

Each operation must have a globally unique ID.

Common approaches:

Persistent IDs
Generated via database sequences or auto-increment fields
Time-based IDs
Combine timestamp, machine ID, and sequence number
(e.g., Twitter Snowflake)