Multi-region implementation review
This page helps you review your application to handle regional replication, repeated processing, and regional outages. Use this checklist before adding a region to your project.
Start with Multi-region operations for primary regions, asynchronous replication, and primary selection modes.
Review Consumers and external side effects
-
Choose where each Consumer processes events: in every receiving region, only in the origin region, or in one designated region. Check whether external destinations are regional or shared.
-
Make external actions safe to repeat, including concurrent attempts from different regions when applicable.
-
Define how to recover unfinished actions after an outage, including events acknowledged and skipped in other regions.
Choose read consistency for each handler
Identify reads that tolerate stale replicated state and those that require primary-region consistency.
Check flows that write and immediately read, such as submitting an order and displaying its status.
Review each handler’s ReadOnlyEffect or Effect return type using Event Sourced Entity replication and Key Value Entity replication.
In request-region mode, an Effect read received in a region other than the entity’s primary moves the primary to that region, and needs every region to be reachable.
Review Workflow retries and failover
Make external calls safe to retry after a failure or lost response. When a Consumer starts a Workflow, use a deterministic Workflow identifier and make repeated start commands safe.
A step that is running when its region is marked down can run again in another region while the first attempt still runs. Make step side effects safe for concurrent attempts. Conflicting Workflow states are resolved by timestamp, with the newer state retained. An external action may have completed even when the retained Workflow state does not record it. Reconcile uncertain outcomes with external systems after failover.
A Workflow handles its commands in its primary region. Commands received in other regions are forwarded there, so they fail while that region cannot be reached and has not been marked down. See Implementing Workflows for execution, retries, and replication.
Check payload sizes and replication filters
Review the serialized size of Key Value Entity state, Event Sourced Entity events, and Workflow events carrying state against the 1 MiB replication limit. Leave room for message metadata. An oversized Workflow event can fail during replication, so successful local persistence does not establish successful replication. Keep large assets in external storage and persist references to them. See Payload and state size.
Inventory replication filters and record which regions can receive each entity’s data. Do not rely on excluded regions having an up-to-date recovery copy; previously replicated data can remain there but become stale.
Check agent session memory placement separately. Do not assume that every session has a copy in every project region. See Session memory replication.
Remove dependencies on process-local state
These checks also apply to multiple instances in a single region:
-
Keep authoritative business state out of mutable static fields and local caches.
-
Define how caches tolerate stale data and recover after a restart.
-
Review Java timers and scheduled executors that run independently in each instance. Use Akka timers for durable scheduled calls, and handle repeated execution.
-
Akka timers are stored in the region where they are scheduled and are not replicated. A timer scheduled from a Consumer is scheduled in every region that processes the event, unless the Consumer guards it.
Align request routing with primary selection
In request-region mode:
-
Creating an Event Sourced Entity or Key Value Entity, and transferring its primary, requires every region to respond. During deployments, restarts, region additions, and regional outages, these writes can fail with an
Unexpected error. Make clients retry them, or use pinned-region mode if these entities must remain creatable while a region is unreachable. -
Starting a Workflow does not require every region to respond. Its first persisted event selects the local region as primary; later commands from other regions are forwarded there.
-
Keep writes for a given entity in a consistent region during normal operation. Concurrent writes to one entity from several regions transfer the primary back and forth, and can fail.
-
A region marked down rejects writes instead of forwarding them. Route clients to another region.
In pinned-region mode:
-
Writes received in any region are forwarded to the project’s primary region, including writes that create entities and Workflows. Changing incoming traffic does not change the primary region.
-
After a failed region is brought back, writes stay with the new primary region until you change the project’s primary region again.
down-region changes the runtime primary without restarting services. Changing the project primary with set-primary, or changing primarySelectionMode, takes effect through a rolling update.
Review routing for clients and background jobs, and define a failover route.
Follow the failover procedure in Multi-region setup.
Before a failover, check that every service in the region is running: a paused or deleted service makes down-region and bring-up-region stop before they reach the remaining services.
Prepare for adding a region
When you add a region to a project with running services:
-
Consumers in the new region process the full event history of the entities they consume. A side effect that runs in every receiving region runs again for every past event. Use origin-only or designated-region processing for such side effects.
-
The new region can report ready before replicated data arrives. For a short time, reads there can return empty state.
-
In request-region mode, writes that create Event Sourced Entities or Key Value Entities can fail in all regions for several minutes.
-
Routes created before the addition keep the hostname of the original region. Check that a route serves the new region.
Prepare for a region returning
Follow Bring up region for disaster recovery. Account for the following when restoring traffic:
-
Consumers resume from their stored offsets and process outstanding events. External actions must still tolerate duplicate delivery.
-
Timers scheduled in the failed region do not run elsewhere during the outage. Overdue timers fire when that region returns, so their handlers must account for late execution.
-
Local reads can remain stale until replication catches up.
-
In pinned-region mode, the replacement primary remains primary. In request-region mode, a subsequent write in the returning region can move an entity’s primary back there.
-
Reconcile Workflow outcomes with external systems. An action completed in the failed region can be absent from the retained Workflow state.
Testing
Use a local cluster to check behavior with multiple service instances in one region. Validate regional routing, primary transfers, and recovery in a deployed multi-region project.
-
Write in one region and read from another. Check both local reads and reads routed to the primary.
-
Deliver the same business operation more than once and verify that external actions are deduplicated, including attempts from different regions when applicable.
-
In request-region mode, write to one entity from two regions at the same time, and create entities during a rolling restart. Check that clients retry failed writes.
-
Exercise regional failover and recover unfinished external actions. For designated-region processing, include events already acknowledged and skipped in the replacement region.
-
Mark a region down while a Workflow step runs there, and check that the repeated step is safe.
-
Bring the failed region back after failover. Check Consumer catch-up, overdue timers, Workflow outcomes, primary selection, and read freshness before restoring traffic.
-
Add a region to a project that already has data. Check Consumer side effects and reads in the new region.
-
Verify which data is available in the recovery region, including agent session memory.
-
Verify that external databases, object stores, and APIs remain reachable from the failover region, with the required data and credentials.
Record the data loss and interruption your application can tolerate. Asynchronous replication means that a surviving region may not contain the latest writes from a failed region. See Availability and recovery for recovery scopes.