Event driven arch

Events

Because events are immutable and ordered:

• Replication = copy the log
• Crash recovery = replay the log
• Multi-node consensus = agree on sequence numbers

Requeuing Roulette and Event-Driven Architecture: A Comprehensive Guide

The article you shared from Oskar Dudycz’s Architecture Weekly discusses a critical anti-pattern in event-driven systems called Requeuing Roulette. Let me break down the specific issues, solutions, and related patterns to help you build more resilient event-driven systems.

The Core Problem: Out-of-Order Events

The fundamental issue arises from the nature of distributed systems: you cannot have both maximum throughput and strict ordering simultaneously. When you distribute message processing across multiple consumers for better throughput, you sacrifice the ordering guarantee that a single consumer provides.

Why Out-of-Order Processing Happens

In messaging systems, there are actually two types of order:

• Queue Order: The order messages are produced to the queue
• Processing Order: The order messages are actually consumed and processed

Out-of-order issues occur when:

• RabbitMQ queues have multiple consumers racing for messages
• You use tools like SQS or Google PubSub that only guarantee best-effort ordering
• Your outbox pattern deletes processed messages and loses sequence
• Network delays shuffle carefully ordered streams

Understanding Causal Correlation

The article introduces an essential concept: causal correlation. Messages are causally correlated when one depends on another:

• Depositing money is causally correlated to opening an account (you cannot deposit without opening first)
• Deposits to the same account are not causally correlated to each other
• Withdrawals are causally correlated to deposits (you need to check balance)

Processing uncorrelated messages in parallel is fine and increases throughput. However, processing causally correlated messages out of order creates race conditions.

What Is Requeuing Roulette?

The Requeuing Roulette anti-pattern involves putting a message back into the queue when it arrives out of order, hoping it will be processed correctly after the prerequisite message arrives. The name correctly suggests that you’re gambling on luck.

Why Requeuing Is Dangerous

RabbitMQ’s documentation states that requeued messages will be placed “to a position closer to queue head” but this is only best effort. In the worst case, you get a cascading failure:

1. Message with revision 14 arrives before revision 12
2. You requeue message 14, hoping 12 comes first
3. Message 14 might land before 13 instead
4. Now both 13 and 14 need requeueing
5. The cycle continues, creating a “roulette” of chaos

The Hidden Cost Under Load

Even when order doesn’t matter, requeueing has a hidden cost that becomes visible under load. When you reject a message with requeue set to true:

• It can be redelivered almost instantly
• Your consumer rejects it again (if conditions haven’t changed)
• This can happen hundreds of times per second
• CPU gets spent processing the same messages repeatedly
• Thousands of processable messages sit stuck behind the problem messages

Solutions and Patterns to Address These Issues

1. The Phantom Record Pattern

Instead of fighting out-of-order events, store data as it arrives and denoise on your side. This involves creating read models that can handle partial state:

type PaymentVerification = {
  paymentId: string;
  payment?: Payment;  // Optional - may arrive later
  fraudAssessment?: FraudAssessment;  // Optional
  riskEvaluation?: RiskEvaluation;  // Optional
  status: 'unknown' | 'processing' | 'approved' | 'declined';
  dataQuality: 'partial' | 'sufficient' | 'complete';
};

The key insight is that events from external systems are rumors at best—you need to interpret them to make them your own facts. Your read model aggregates data as it arrives, making decisions only when sufficient information is available.

2. Internal vs External Events

One of the most powerful techniques is separating internal and external events. Internal events can be granular, but external events should be summary events that contain complete information:

// Internal events (granular)
type ItemAddedToCart = { cartId: string; productItem: ProductItem };
type ItemRemovedFromCart = { cartId: string; productItem: ProductItem };
 
// External event (summary - published to other modules)
type CartConfirmed = {
  cartId: string;
  productItems: { productId: string; quantity: number };
  confirmedAt: Date;
};

Other modules don’t need to know about every change—they just need the final state when it matters. This dramatically reduces ordering issues between modules.

3. Using Logical Clocks (Revisions)

When you need strict ordering, timestamps are unreliable due to clock skew. Instead, use a logical clock—a monotonically increasing revision number:

1 - ItemAddedToCart (cartId: 1, name: Pizza)
2 - ItemAddedToCart (cartId: 1, name: Pizza)
3 - ItemRemovedFromCart (cartId: 1, name: Pizza)
4 - CartConfirmed (cartId: 1)

When events arrive out of order (e.g., 2, 1, 4, 3), you can:

• Detect gaps in the sequence
• Store pending events
• Process them in correct order once all arrive

4. The Outbox Pattern

The Outbox Pattern ensures reliable event delivery with at-least-once guarantees:

1. Instead of publishing directly to the queue, store the message in an outbox table within the same database transaction as your business operation
2. A background process reads the outbox and publishes messages
3. After successful publishing, mark the message as sent

This prevents the dual-write problem where your database update succeeds but message publishing fails.

5. The Inbox Pattern

Complementing the outbox, the Inbox Pattern handles incoming messages reliably:

1. Save the incoming event to an inbox table first
2. Return acknowledgment to the queue
3. Process the event from the inbox table
4. This allows for deduplication and idempotent processing

6. Idempotency Handling

Since messaging systems typically provide at-least-once delivery, you must handle duplicate messages. Strategies include:

• Unique message identifiers: Check if you’ve already processed a message ID before processing
• Business logic verification: Check if the operation was already completed (e.g., “does an invoice already exist for this reservation?“)
• Optimistic concurrency: Use version numbers to detect and reject duplicate updates

7. Message Grouping and Partitioning

Different messaging systems provide grouping mechanisms to maintain order for related messages:

System	Grouping Mechanism	Trade-off
Kafka	Partitions with message keys	Single consumer per partition limits parallelism
SQS FIFO	Message Group ID	Lower throughput than standard queues
Azure Service Bus	Sessions	Session locks add complexity
RabbitMQ	Separate queues per group	Operational complexity at scale

For Kafka specifically: messages with the same key go to the same partition, ensuring order within that partition while allowing parallelism across partitions.

8. The Saga Pattern for Distributed Transactions

When operations span multiple services, the Saga Pattern manages data consistency through compensating transactions:

• Choreography: Each service publishes events that trigger the next step
• Orchestration: A central coordinator tells services what to do

If a step fails, compensating transactions undo previous changes (e.g., cancel a reservation if payment fails).

Kafka vs Traditional Message Brokers

The article points out that Kafka handles this differently:

• Messages with the same key go to the same partition
• Order is maintained within a partition
• You don’t need to requeue—just rewind the offset to reprocess

However, Kafka has its own limitations:

• Only one consumer per partition within a consumer group
• You cannot parallelize within a single partition
• It’s a streaming/log solution, not a traditional queue

When Requeuing Might Be Acceptable

The article acknowledges that requeuing can work if:

• You want best-effort parallelism and ordering is “good enough”
• Messages are mostly not causally correlated
• Events for the same record aren’t published rapidly in succession
• Consumers are stable and don’t fail often

But these assumptions are fragile—“famous last words” as the author puts it.

Best Practices Summary

Based on Oskar Dudycz’s work and related patterns, here are key recommendations:

Design-Level Practices:

• Separate internal and external events to minimize cross-module ordering issues
• Use summary events that contain complete information
• Design for eventual consistency from the start

Implementation-Level Practices:

• Implement the Outbox Pattern for reliable publishing
• Handle idempotency in your message consumers
• Use phantom records/read models to aggregate partial data
• Add revision numbers for strict ordering requirements

Infrastructure-Level Practices:

• Use message grouping features (Kafka partitions, SQS FIFO groups)
• Route related messages to the same partition/queue
• Implement proper monitoring for processing times and queue behavior

Orchestration vs Choreography

Orchestration

Instead of deciding the full flow in one place, you break the system into independent pieces.
Each piece:

• performs its own job
• emits an event
• other pieces react to that event

So the process flow emerges from the connections between pieces.

Each step triggers the next via events:

1. Place Order
   API Gateway → Lambda → write to DynamoDB
   DynamoDB stream triggers next Lambda → publish “OrderPlaced” → SNS topic
2. Package Order
   Lambda subscribed to “OrderPlaced”
   → notify warehouse via IoT → worker packages → publish “OrderPackaged”
3. Charge Customer
   Lambda subscribed to “OrderPackaged”
   → call credit card API → store result → publish “OrderCharged”
4. Ship Order
   Lambda subscribed to “OrderCharged”
   → notify driver → publish “OrderShipped”

The event chain itself acts as the workflow.
The system moves forward because each component “fires” the next step.

Strengths

1. No central coordinator → high performance, low overhead
2. Teams can independently own each step
3. Loosely coupled → easy to replace/modify a step

Weaknesses

1. Hard to know the current state of an order
   You must inspect multiple services, databases, logs.
   Flow is not visible anywhere as a whole.

2. Hard failure handling
   If step 3 fails (charging customer), you must manually emit compensation events and have all earlier services listen for them.
   No built-in reversal/rollback control.

3. Business flow becomes scattered across the system
   Logic is everywhere, not in one place.

Cherography

Instead of spreading the flow across services, you centralize the flow in one workflow definition.

A workflow engine (like AWS Step Functions):

• Manages state transitions
• Knows which step comes next
• Models success/failure paths
• Stores execution history

How the e-commerce flow becomes orchestrated

The workflow tool explicitly defines:

1. Step: Place Order
2. Step: Save to DB
3. Step: Notify Warehouse
4. Wait for a success/timeout
5. Step: Charge Customer
6. Step: Save charge result
7. Step: Notify driver
8. Step: Mark completed/shipped

The workflow engine enforces the entire sequence.

Strengths

1. Perfect visibility
   At any moment you can see:
   • Which step an order is on
   • Whether it failed
   • Why it failed
   • Historical timeline
2. Explicit failure handling
   Each branch can say exactly what to do on error (retry, compensate, cancel).
3. Easier reasoning
   The entire business process is in one place.

Weaknesses

1. More expensive
   The orchestrator runs the workflow and charges per state transition.
2. Harder to distribute across multiple teams
   The workflow “owns” everything, so services can’t fully own their own step.
3. Tighter coupling
   Steps become dependent on the central workflow.

Pitfall and tips

• Event Notification (thin events): publish minimal information (what changed) and let subscribers fetch full state.

  • Pro: avoids stale-data acting; Con: increases follow-up API calls.

• Event-Carried State Transfer (fat events): include full snapshot in the event.

  • Pro: good for synchronization, replays, and reducing downstream API calls; Con: larger messages and potential stale snapshots.

• Hypermedia links in events: embed links to related APIs/resources so consumers can discover the authoritative data source. Mitigates confusion about where to fetch details.

• Event batching: group many events together for delivery (useful when receivers catch up after downtime). Be mindful of receiver UX and processing model.

Pros

• Design microservices without distributed transactions
• Build systems that can fully rebuild themselves
• Make changes observable and traceable