Sagas: Managing Transactions in Distributed Systems

Written By Andrey Sydelov

Imagine building a modern e-commerce app where a single order spans multiple services: reserving stock from a warehouse microservice, processing payment through a third-party gateway, and triggering shipping via a logistics API. In a traditional relational database, ACID transactions would handle this seamlessly, ensuring everything succeeds or fails together. But in distributed systems—like those powering Amazon, Netflix, or your favorite European fintech app—these operations are spread across networks, servers, and even continents. A network glitch or server crash midway could leave your system in chaos: money deducted but no shipment sent.

This is where sagas come in. Introduced in the 1980s but revitalized in the microservices era, sagas are a design pattern for managing long-running, distributed transactions without relying on a single, all-powerful coordinator. Instead of strict ACID guarantees, sagas emphasize eventual consistency: they break operations into a sequence of local transactions, each with a compensating action to undo changes if something goes wrong. This approach aligns with the CAP theorem, trading immediate consistency for availability and fault tolerance—crucial in today's cloud-native world.

How sagas solve the pitfalls of distributed transactions, dive into their core concepts, compare choreography and orchestration styles, and provide practical examples and tips. Whether you're architecting scalable apps in the EU's GDPR-compliant environments or optimizing for high-traffic U.S. platforms, understanding sagas will help you build resilient systems that users can trust. Let's start with why traditional transactions fall short in distributed setups.

Challenges of Transactions in Distributed Systems

Limitations of ACID in Distributed Systems

Traditional ACID transactions shine in monolithic systems where a single database ensures atomicity, consistency, isolation, and durability. But in distributed systems—think microservices, cloud-native apps, or hybrid SQL/NoSQL setups—these guarantees unravel. Each service manages its own data, often on separate servers or even across continents, with no central coordinator to enforce global consistency. Network latency, partitions, or crashes can leave operations half-complete, risking data corruption or customer frustration.

Real-World Risks

Consider an e-commerce platform: reserving stock, charging a card, and scheduling delivery involve distinct services. If the payment service fails after stock is reserved, you might block inventory indefinitely or, worse, charge a customer without delivering their order. Traditional two-phase commit (2PC) protocols, which lock resources across systems to ensure consistency, are impractical here.

CAP Theorem and Trade-Offs

The CAP theorem explains why:

  • Distributed systems can’t simultaneously guarantee consistency, availability, and partition tolerance.

  • Most modern apps, from European banking systems to U.S. streaming platforms, prioritize availability and partition tolerance, accepting eventual consistency.

Sagas embrace this trade-off, replacing global locks with coordinated local transactions. Unlike ACID’s strict isolation, sagas allow temporary inconsistencies, resolved through compensating actions. And while ACID’s durability relies on Write-Ahead Logging, sagas ensure durability per service, with a distributed log tracking progress.

Specific Challenges

Distributed systems introduce unique hurdles:

  • Partial Failures: One service might succeed (e.g., stock reserved) while another fails (e.g., payment rejected).

  • Lack of Global Isolation: Services might see uncommitted changes, risking conflicts.

  • Network Issues: Latency or partitions disrupt coordination.

  • Need for Idempotency: Retries must avoid duplicating actions.

  • Long-Running Operations: Transactions spanning seconds increase conflict risks.

Why 2PC Falls Short

Traditional two-phase commit (2PC) protocols are slow, block operations during failures, and collapse under network partitions—violating the CAP theorem’s promise of availability. This makes them unsuitable for systems like Klarna’s payment processing or Spotify’s playlist updates.

Sagas as a Solution

Sagas tackle these issues by structuring workflows to tolerate failures, making them ideal for complex systems. Next, we’ll break down the core concepts of sagas and how they provide a practical solution for distributed environments.

Types of Sagas: Choreography vs. Orchestration

Two Ways to Coordinate Sagas

Sagas come in two distinct styles: choreography and orchestration. Each offers a unique approach to managing the sequence of local transactions in a distributed system, balancing control, scalability, and complexity. Choosing the right one depends on your application’s needs, whether you’re designing a high-throughput e-commerce platform or a tightly regulated financial workflow. Let’s dive into how choreography and orchestration work, their strengths, and where they shine.

Choreography: Decentralized Coordination

In a choreographed saga, each service operates independently, reacting to events published by others through a message broker like RabbitMQ or Kafka. There’s no central controller—services "dance" together by listening and responding to events. For example, in an online retail system, the inventory service might emit an "OrderReserved" event, triggering the payment service to act.

Key characteristics include:

  • Event-Driven: Services communicate via events, such as "PaymentProcessed" or "OrderFailed."

  • Loose Coupling: Services only need to understand event formats, not each other’s APIs.

  • Distributed State: Each service tracks its part of the saga, with a shared log (e.g., Kafka topic) for recovery.

Pros and Cons of Choreography

  • Advantages:

    • Highly scalable: No central bottleneck, perfect for systems with heavy traffic.

    • Fault-tolerant: No single point of failure since services operate independently.

    • Flexible: New services can subscribe to events without modifying existing ones.

  • Challenges:

    • Hard to monitor: Tracking the saga’s overall state across services can be tricky.

    • Difficult to modify: Adding new steps requires updating multiple services.

    • Debugging complexity: Event flows need robust tracing tools to diagnose issues.

Choreography excels in systems prioritizing scalability and independence, like a streaming service handling millions of users.

Orchestration: Centralized Control

In an orchestrated saga, a dedicated service or workflow engine (e.g., Camunda or Temporal) acts as the "conductor," directing each step by invoking service APIs. The orchestrator tracks the saga’s state and decides what happens next, simplifying oversight. For instance, in a travel booking system, the orchestrator might call the flight service to reserve a seat, then the payment service to charge, and finally the hotel service to confirm.

Key features include:

  • Centralized Logic: The orchestrator defines the sequence and handles compensations.

  • Explicit State: Saga progress is stored centrally, often in a database.

  • API-Based: Services expose APIs for the orchestrator to call.

Pros and Cons of Orchestration

  • Advantages:

    • Easier monitoring: Centralized state simplifies tracking and debugging.

    • Flexible updates: New steps or logic changes are managed in one place.

    • Clear error handling: The orchestrator can systematically handle retries or compensations.

  • Challenges:

    • Single point of failure: Orchestrator downtime halts sagas.

    • Tighter coupling: Services rely on the orchestrator’s commands.

    • Potential bottleneck: Heavy workloads can overwhelm the orchestrator.

Orchestration suits complex workflows with conditional logic, like a loan approval process requiring strict auditing.

Choosing the Right Approach

Choreography fits simple, high-volume systems where services need autonomy, such as an online marketplace. Orchestration is better for intricate workflows with centralized control, like a regulated payment system. Some applications blend both: choreography for scalable steps and orchestration for critical, stateful processes.

Next, we’ll explore how to implement sagas practically, with tools, code examples, and strategies for handling failures.

Implementation of Sagas in Practice

Building Sagas Step by Step

Implementing sagas in a distributed system requires careful planning to ensure reliability and fault tolerance. The process involves defining steps, managing communication, and preparing for errors, all while leveraging tools and patterns suited for distributed environments.

Here’s how to approach it:

  1. Define Saga Steps and Compensations: Identify each local transaction (e.g., reserving inventory) and its corresponding compensating action (e.g., releasing inventory). Ensure compensations are idempotent to handle retries safely.

  2. Choose a Communication Mechanism: Use a message queue (e.g., Kafka, RabbitMQ) for choreography or API calls for orchestration to coordinate services.

  3. Store Saga State: Persist the saga’s progress in a database or message log to recover from crashes.

  4. Handle Failures: Implement retries, timeouts, and dead-letter queues to manage network issues or service failures.

  5. Test Thoroughly: Simulate failures to verify compensations work as expected.

Tools for Saga Implementation

A range of tools can simplify saga development, depending on your stack and requirements:

  • Axon Framework (Java): Supports event-driven choreography and state management for sagas.

  • Eventuate: Designed for microservices, offering choreography with distributed event logs.

  • Temporal: A workflow engine for orchestration, handling retries and state persistence.

  • Kafka or RabbitMQ: Message brokers for event-driven choreography, ensuring reliable communication.

  • Custom Solutions: For simpler needs, a database table tracking saga state (e.g., saga_id, status, steps_completed) can suffice.

These tools help manage complexity, but the choice depends on your system’s scale and whether you favor choreography or orchestration.

Example: Orchestrated Saga in Pseudocode

To illustrate, consider an e-commerce order saga using orchestration. The orchestrator coordinates reserving stock, charging a payment, and shipping the order. If any step fails, it triggers compensations in reverse order. Below is a simplified Python-like pseudocode example, emphasizing idempotency to handle retries safely:

class OrderSaga:
    def start(self, order_id):
        try:
            self.reserve_stock(order_id)  # Local transaction
            self.charge_card(order_id)    # Local transaction
            self.ship_order(order_id)     # Local transaction
            self.complete_saga(order_id)  # Mark saga as done
        except Exception as e:
            self.compensate(order_id, e.step)  # Handle failure

    def reserve_stock(self, order_id):
        # Check if reservation already exists
        reservation_exists = query("SELECT 1 FROM reservations WHERE order_id = %s", order_id)
        if reservation_exists:
            return  # Idempotent: skip if already reserved
        # Reserve stock atomically
        response = query("""
            BEGIN;
            UPDATE inventory SET stock = stock - 1 WHERE product_id = 123 AND stock > 0;
            INSERT INTO reservations (order_id, product_id, quantity) VALUES (%s, 123, 1);
            COMMIT;
        """, order_id)
        if not response.ok:
            raise Exception(step="reserve")
        log_step(order_id, "reserved")

    def charge_card(self, order_id):
        # Call payment service API, commit locally
        response = api_call("payment/charge", order_id)
        if not response.ok:
            raise Exception(step="charge")
        log_step(order_id, "charged")

    def ship_order(self, order_id):
        # Call shipping service API, commit locally
        response = api_call("shipping/arrange", order_id)
        if not response.ok:
            raise Exception(step="ship")
        log_step(order_id, "shipped")

    def compensate(self, order_id, failed_step):
        # Reverse order for compensations
        if failed_step >= "ship":
            api_call("shipping/cancel", order_id)  # Idempotent
        if failed_step >= "charge":
            api_call("payment/refund", order_id)  # Idempotent
        if failed_step >= "reserve":
            # Check if reservation exists
            reservation_exists = query("SELECT 1 FROM reservations WHERE order_id = %s", order_id)
            if reservation_exists:
                query("""
                    BEGIN;
                    UPDATE inventory SET stock = stock + 1 WHERE product_id = 123;
                    DELETE FROM reservations WHERE order_id = %s;
                    COMMIT;
                """, order_id)
        log_step(order_id, "failed")

This example ensures idempotency by tracking reservations with a unique order_id in a reservations table, preventing duplicate stock decrements. Each service commits its transaction locally, ensuring durability, while the orchestrator manages the saga’s flow.

Integrating with MVCC

Within each service, sagas can leverage Multiversion Concurrency Control (MVCC), as seen in databases like PostgreSQL, to ensure local consistency. For example, the inventory service might use MVCC to manage stock updates, creating new row versions for each reservation and marking old ones as dead. The saga coordinates these local transactions globally, relying on MVCC’s snapshots to prevent conflicts within a service. This combination—MVCC for local consistency, sagas for distributed coordination—creates a robust system.

For instance, in the pseudocode above, the reserve_stock call might execute:

BEGIN;
UPDATE inventory SET stock = stock - 1 WHERE product_id = 123 AND stock > 0;
INSERT INTO reservations (order_id, product_id, quantity) VALUES (%s, 123, 1);
COMMIT;

MVCC ensures the update is isolated and durable, while the saga ensures the overall workflow (reservation, payment, shipping) completes or rolls back cleanly.

Handling Failures and Edge Cases

Failures are inevitable in distributed systems, so sagas must be resilient:

  • Timeouts: Set reasonable timeouts for service calls to avoid indefinite waits.

  • Retries: Use exponential backoff for transient failures, ensuring idempotency.

  • Dead-Letter Queues: Capture failed events in choreography for manual review.

  • Monitoring: Log saga states with unique IDs for traceability, using tools like Jaeger for distributed tracing.

By combining these strategies with robust tooling, you can implement sagas that handle the complexities of distributed systems reliably.

Advantages, Disadvantages, and Comparison with ACID

Why Sagas Shine in Distributed Systems

Sagas offer a powerful way to manage transactions across distributed systems, providing a flexible alternative to traditional ACID transactions. By breaking operations into local, reversible steps, they enable resilience and scalability in environments where services operate independently. However, like any approach, sagas come with trade-offs. Understanding their strengths and weaknesses, especially compared to ACID, helps clarify when they’re the right choice for your application.

Advantages of Sagas

Sagas are designed for the challenges of distributed systems, offering several key benefits:

  • Scalability: By avoiding global locks, sagas allow services to process transactions concurrently, supporting high-throughput systems like online marketplaces or streaming platforms.

  • Fault Tolerance: Each step commits locally, so partial failures don’t block the entire system. Compensations handle rollbacks, ensuring eventual consistency even if a service crashes.

  • Flexibility for Microservices: Sagas work well with heterogeneous data stores (e.g., SQL and NoSQL), as each service manages its own persistence, unlike ACID’s reliance on a single database.

  • Non-Blocking: Asynchronous communication (in choreography) or centralized control (in orchestration) prevents the delays inherent in locking mechanisms like two-phase commit.

These qualities make sagas ideal for cloud-native applications where availability and resilience are critical, such as a subscription service handling millions of users.

Disadvantages of Sagas

Despite their strengths, sagas introduce complexities that require careful handling:

  • Increased Complexity: Developers must implement compensating actions for each step, which adds code and testing overhead compared to ACID’s automatic rollbacks.

  • Temporary Inconsistencies: Sagas rely on eventual consistency, meaning the system may be temporarily inconsistent (e.g., stock reserved but payment pending), which can confuse users if not managed properly.

  • Monitoring Challenges: Tracking saga state across services, especially in choreography, requires robust logging and tracing tools to diagnose issues.

  • Messaging Overhead: Event-driven sagas depend on message queues, which introduce latency and potential failure points, such as lost messages or queue bottlenecks.

These drawbacks demand disciplined design, particularly for ensuring idempotency and handling edge cases, as shown in the earlier e-commerce example.

Comparing Sagas to ACID Transactions

Sagas and ACID transactions serve similar goals—ensuring reliable operations—but their approaches differ fundamentally due to their environments. Here’s how they stack up:

  • Atomicity:

    • ACID: Guarantees all operations complete as a single unit or none do, using database-level rollbacks (e.g., via MVCC in PostgreSQL).

    • Sagas: Achieves atomicity through compensations, manually undoing completed steps if a failure occurs. This is less immediate but more flexible in distributed systems.

  • Consistency:

    • ACID: Enforces strict consistency, ensuring the database adheres to constraints (e.g., foreign keys) at all times.

    • Sagas: Provides eventual consistency, allowing temporary violations resolved by compensations, aligning with the CAP theorem’s focus on availability.

  • Isolation:

    • ACID: Offers strict isolation levels (e.g., Serializable), preventing concurrent transactions from seeing partial changes.

    • Sagas: Relaxes isolation, as services may see uncommitted changes from others, relying on application logic to handle conflicts.

  • Durability:

    • ACID: Ensures committed changes are permanent via Write-Ahead Logging.

    • Sagas: Guarantees durability per service, with a saga log (e.g., in Kafka or a database) tracking progress for recovery.

Unlike ACID, which relies on a single database’s MVCC for snapshots and rollbacks, sagas use distributed coordination and event logs, trading immediate guarantees for scalability. For example, in a banking app, ACID ensures a transfer is instantly consistent, while a saga might temporarily show funds withdrawn but not deposited, resolving later via compensations.

When to Use Sagas

Sagas are the go-to choice in specific scenarios:

  • Long-Running Transactions: Operations spanning seconds or minutes, like order processing across multiple services, benefit from sagas’ asynchronous nature.

  • Distributed Data: When data lives across microservices or heterogeneous databases, sagas coordinate without requiring a single transaction manager.

  • High Availability Needs: Systems prioritizing uptime over immediate consistency, like e-commerce or streaming platforms, align with sagas’ CAP theorem trade-offs.

However, for simple operations within a single database—like updating a user profile—ACID transactions are simpler and more efficient. Sagas shine in complex, distributed workflows where flexibility outweighs the need for instant consistency.

Sagas: The Evolution of Transactions

Sagas represent a modern approach to managing transactions in distributed systems, evolving beyond the rigid constraints of ACID transactions. By breaking workflows into local, reversible steps, sagas offer a flexible, scalable solution for coordinating operations across microservices, from order processing to billing workflows. Unlike ACID’s immediate consistency, sagas prioritize availability and fault tolerance, ensuring systems remain responsive even during partial failures. This makes them indispensable for building resilient applications in today’s cloud-native world, where services operate independently across networks and data stores.

The power of sagas lies in their ability to balance reliability with scalability. As we’ve seen, choreography decentralizes coordination for high-throughput systems, while orchestration provides control for complex workflows. Tools like Temporal or Kafka, combined with idempotent designs (e.g., using a reservations table), ensure sagas handle failures gracefully, complementing local consistency mechanisms like MVCC from relational databases.

Looking Ahead

Sagas open the door to advanced patterns in distributed systems. Exploring event sourcing, where state is derived from a sequence of events, can enhance saga implementations by providing a natural log for tracking progress. Similarly, Command Query Responsibility Segregation (CQRS) pairs well with sagas, separating read and write operations for greater scalability. For systems requiring stronger consistency, distributed consensus protocols like Raft offer another layer of coordination. These topics build on sagas, addressing new challenges in large-scale architectures.

Take the Next Step

To master sagas, start by implementing a simple workflow in your project—perhaps a basic order processing system—using a tool like Temporal for orchestration or Kafka for choreography. Experiment with failure scenarios to ensure your compensations are robust, and dive into open-source projects like Axon Framework or Eventuate to see sagas in action. By understanding and applying sagas, you’ll be better equipped to build systems that stay reliable under pressure, setting the stage for tackling the next generation of distributed challenges.

Andrey Sydelov
Next

ACID, Isolation Levels, and MVCC: Architecture and Execution in Relational Databases