The move from monolithic to microservices architectures presents numerous benefits, including enhanced scalability, high availability, and increased agility. However, this transition introduces complexities, particularly concerning data consistency. In a traditional monolithic application, data consistency is typically managed by a shared relational database. In contrast, a microservices architecture often employs the “database per service” pattern, where each microservice has its own data store. This distribution necessitates careful consideration of transaction management and data integrity.
The Problem: Data Consistency in Distributed Systems
Monolithic applications rely on shared relational databases to guarantee data consistency through ACID transactions. ACID is an acronym that stands for:
Atomicity: All steps of a transaction succeed or fail together, ensuring no partial states.
Consistency: All data in the database remains consistent at the end of the transaction.
Isolation: Only one transaction can access data at a time, preventing interference from other transactions.
Durability: Data is permanently persisted in the database upon transaction completion.
Relational Database Management Systems (RDBMS) support ACID properties to maintain strong data consistency.
However, in a microservices architecture, each microservice operates with its own data store, often utilizing diverse technologies. This decentralized approach eliminates the central database and single unit of work found in monoliths. Consequently, business logic spans multiple local transactions, precluding the use of a single transaction unit across databases. Despite these changes, the need for ACID properties remains.
Consider an order management system comprising stock management, payment, and order management services. If these services are designed following microservice architecture principles and the database per service pattern, completing an order involves multiple steps. First, the order service calls the stock management service to verify stock availability and reserve the products. Next, the order service invokes the payment service to process the customer’s payment. Each of these steps updates separate databases within the scope of their respective services.
The final step involves creating the order record. However, if a technical error occurs during this stage, the order record may not be created, and the customer may not receive an order number, even though the payment has been processed. This scenario exemplifies a data consistency problem.
Possible Solutions
No single solution fits every situation; different approaches can be applied depending on the specific use case. The two main strategies for addressing data consistency in microservices are:
- Distributed Transactions
- Eventual Consistency
Distributed Transactions
In a distributed transaction, operations are executed across two or more resources, such as databases and message queues. A distributed transaction manager or coordinator ensures data integrity across multiple databases. However, distributed transactions can be complex due to the involvement of multiple resources.
Two-Phase Commit (2PC) is a blocking protocol used to ensure that all transactions either succeed or fail together in a distributed transaction. The XA standard provides a specification for 2PC distributed transactions, and JTA includes a standard API for XA. However, the requirement that all resources be deployed to a single JTA platform makes 2PC unsuitable for microservices architectures.
Benefits of Distributed Transactions
- Strong data consistency
- Support for ACID features
Drawbacks of Distributed Transactions
- Complex to maintain
- High latency and low throughput due to blocking processes
- Potential for deadlocks between transactions
- Transaction coordinator as a single point of failure
Eventual Consistency
Eventual consistency is a model used in distributed systems to achieve high availability. Inconsistencies are allowed for a short period until the distributed data problem is resolved. This model does not apply to distributed ACID transactions across microservices but instead uses the BASE database model.
While the ACID model provides a consistent system, the BASE model prioritizes high availability. BASE is an acronym that stands for:
- Basically Available: Ensures data availability by replicating it across database cluster nodes.
- Soft-state: Data may change over time due to the lack of strong consistency. The responsibility for consistency is delegated to developers.
- Eventual consistency: Immediate consistency may not be possible, but consistency will be achieved eventually.
The Saga pattern is a common approach that leverages the eventual consistency model. In a Saga pattern, a distributed transaction is performed by asynchronous local transactions across related microservices. Each service updates its data in a local transaction, and a Saga manages the execution sequence of these services.
Two common implementations of Saga transactions are:
- Choreography-based SAGA: Each service produces an event after completing its task, and other services listen to these events to take action. This pattern requires a mature event-driven architecture. Event Sourcing, which stores the state of event changes using an Event Store, is often used with this pattern.The Choreography-based SAGA pattern works well for transactions with a small number of steps. However, as the number of steps increases, tracking which services listen to which events becomes challenging.
- Orchestration-based SAGA: A coordinator service (Saga Execution Orchestrator, SEG) sequences transactions according to business logic. The orchestrator determines which operation should be performed and, in the event of a failure, undoes previous steps through compensation operations. Compensations are actions taken to maintain a consistent system state when failures occur.It’s important to note that undoing changes may not always be possible if data has been modified by another transaction. Compensations must also be idempotent to handle potential retries, and they should be designed carefully. Frameworks like Camunda and Apache Camel can aid in implementing the Saga orchestration pattern.
Benefits of SAGA
- Performs non-blocking operations on local atomic transactions
- Avoids deadlocks between transactions
- Eliminates the single point of failure
Drawbacks of SAGA
- Eventual data consistency
- Lacks read isolation, requiring extra effort to prevent users from seeing operations being completed and then canceled due to a compensation transaction.
- Debugging becomes difficult as the number of participant services increases.
- Increased development costs due to the need for compensation services
- Complex design
Maintaining data consistency between distributed data stores can be extremely challenging, requiring a shift in mindset when designing new applications. The responsibility for data consistency moves from the database level to the application level.
Which Solution to Choose
The optimal solution depends on the specific use case and consistency requirements. Key design considerations include:
- Avoid distributed transactions across microservices whenever possible.
- Design systems to minimize the need for distributed consistency. Identify transaction boundaries and use strong consistency for operations within the same unit of work, while using eventual consistency for operations that can tolerate latencies.
- Consider event-driven architecture for asynchronous, non-blocking service calls.
- Design fault-tolerant systems with compensations and reconciliation processes.
- Embrace a change in mindset for design and development when using eventual consistency patterns.
Conclusion
Microservices architecture offers significant advantages, including high availability, scalability, and autonomous teams. However, achieving the full potential of this architectural style requires careful consideration of data and consistency management. By understanding the challenges and available solutions, developers can build robust and reliable microservices-based systems.
Read more such articles from our Newsletter here.
