History of the question
Enterprises expanding globally face strict data residency laws like GDPR and CCPA. Traditional monolithic databases centralize data in one region, violating sovereignty or causing high latency. Early distributed systems used active-passive replication, but that creates single points of failure and write latency issues. Modern architectures must support multi-active regions where users in EU, US, and APAC can write locally while respecting data locality constraints.
The problem
The core challenge lies in the CAP theorem trade-offs. You cannot have strong consistency across regions with low latency and partition tolerance simultaneously. Additionally, foreign key relationships spanning regions become impossible when data cannot cross borders. Cross-region transactions risk violating compliance if PII leaks during coordination. Maintaining sub-100ms reads requires caching, but cache invalidation across sovereign boundaries is complex.
The solution
Implement a Cell-Based Architecture using database-native geo-partitioning (e.g., CockroachDB or Google Cloud Spanner). Partition tables by region column, ensuring PII never leaves its physical cell. Use Change Data Capture (CDC) via Apache Kafka to replicate only non-sensitive metadata globally. For cross-region transactions, implement the Saga pattern with local compensations to avoid distributed locks. Deploy Redis clusters at the edge with Cache-Aside pattern for read-heavy workloads, using TTL-based invalidation to avoid cross-region cache coordination.
The Context
A global payment processor needed to launch in Germany and Singapore while maintaining their US data center. Regulatory requirements mandated that EU user transaction histories remain physically in Frankfurt, while APAC data stayed in Singapore. However, cross-border transfers required deducting funds from a US account and crediting an EU account within the same logical transaction, all while keeping balance lookups under 100ms.
Solution 1: Centralized Database with Regional Read Replicas
This approach would host the primary database in US-East with read replicas in EU and APAC, offering a simple consistency model and straightforward ACID guarantees without complex synchronization. However, this violates data sovereignty laws because write traffic routes to US-East, potentially persisting EU PII on US soil, while writes from Singapore incur 200ms+ latency that fails user experience requirements. The architecture also creates a single point of failure in US-East, making it unacceptable for a global payment platform requiring regional autonomy.
Solution 2: Fully Isolated Regional Silos with Nightly ETL
This design operates independent PostgreSQL clusters in each region, processing cross-region transfers during nightly maintenance windows to ensure perfect compliance isolation and simple regional autonomy. This approach fails to support real-time international payments, creating poor user experience and making reconciliation errors difficult to unwind during batch processing. Additionally, the architecture cannot support global account balance aggregations without significant delay, rendering it unsuitable for a modern fintech platform.
Solution 3: Geo-Partitioned Database with Saga Orchestration
This strategy deploys CockroachDB with geo-partitioned tables using partition_key mapping to user home regions, implementing a Temporal workflow to manage cross-region transfers as local transactions with compensating actions. This design enforces native data residency through partition constraints while achieving sub-50ms local reads via leaseholders pinned to regional nodes, though it introduces operational complexity requiring distributed SQL expertise. The solution handles eventual consistency for cross-region metadata via Kafka CDC streams and manages temporary inconsistency during saga execution through TTL-based pending state visibility.
Chosen Approach
The team selected Solution 3 because it uniquely satisfied both compliance and latency constraints without sacrificing transactional semantics or requiring destructive data migrations. They configured CockroachDB REGIONAL BY ROW tables pinning EU rows to Frankfurt nodes, deployed Redis Cluster at edge locations with 5-second TTL for metadata caching, and implemented Temporal sagas to orchestrate cross-region transfers with automatic compensations on failure.
Result
The system passed GDPR audits with zero cross-border PII leakage while processing 50,000 daily cross-region transactions with 99th percentile read latency of 45ms. Customer support teams could query pending saga states via API endpoints to resolve transient inconsistencies during regional outages. The architecture now supports expansion into new markets by simply adding new cells to the CockroachDB cluster without application changes.
How do you maintain referential integrity when a foreign key relationship spans two data sovereignty zones?
You cannot enforce database-level foreign key constraints across regions when data cannot physically leave its zone. Implement application-level referential integrity using UUID references and asynchronous validation via the Outbox Pattern publishing to Kafka; consumers verify references and publish acknowledgments, with orphan detection after timeouts. This sacrifices immediate consistency for compliance but ensures eventual integrity without data migration, using Saga compensation to rollback transactions referencing invalid foreign keys.
What happens to in-flight transactions when a region fails during a cross-region saga?
Saga patterns do not automatically handle failures; you must design for idempotency using idempotency keys stored in Redis or Etcd local to each region to prevent duplicate processing during retries. If Region B fails during a credit operation, the orchestrator timeouts trigger compensating transactions in Region A to refund deducted amounts, utilizing PostgreSQL Advisory Locks or ZooKeeper Distributed Locks to prevent race conditions during orchestrator failover. The system must expose pending transaction states via API endpoints for customer support intervention, ensuring that partial failure states remain queryable and resolvable without data corruption.
How do you perform zero-downtime schema migrations across geo-partitioned cells with different maintenance windows?
Employ the Expand-Contract pattern combined with Feature Flags managed by LaunchDarkly, first deploying additive DDL changes (new columns, tables) across all regions during their respective windows using Flyway or Liquibase while keeping applications backward-compatible. Migrate data asynchronously using Debezium CDC pipelines, then enable new code paths via feature flags only after confirming schema propagation through health checks, ensuring no region serves stale data. Never perform destructive DDL (dropping columns) until all regions confirm migration completion, utilizing Blue-Green deployments within each cell to rollback instantly if replication lag exceeds thresholds.