At 5 AM, after our second all-nighter, we watched the migration complete. But the real moment came the next morning. As actual traffic started flowing through the new infrastructure, we saw Redis latency drop from double-digit milliseconds to single-digit—some workloads hitting below 1ms. That’s when exhaustion turned into relief. Two months of planning and three weeks of execution, validated in a single metric.
When your customer base shifts dramatically, infrastructure location becomes a strategic business decision, not just a technical one. Recently, my team undertook one of the most significant engineering projects for our business: migrating our entire production workload from one GCP region to another.
This wasn’t optional. Data residency compliance requirements demanded it, and our business had made a strategic decision to focus entirely on customers in the new region—nearly 100% of our traffic now originated there, compared to our earlier US presence. The mandate was clear: move everything with minimal downtime, adhere to strict timelines, and stay within budget.
The challenge was amplified by our operational reality—we had workflows running throughout the day. Business disruption wasn’t just inconvenient; it could be costly.
Planning: The Foundation of Execution
Migration of this scale doesn’t start with servers—it starts with understanding constraints and making deliberate tradeoffs. Our ecosystem was complex:
- Self-hosted databases: MongoDB clusters, 2 PostgreSQL instances, Cassandra (4-node cluster), Kafka (3 brokers), Elasticsearch cluster, and 6 different Redis instances
- GKE cluster running 13 microservices in Java, Python, and Node.js
- Cloud Storage buckets, monitoring infrastructure, and Cloud Tasks
We spent two months on planning, starting in July. This wasn’t bureaucracy—it was risk management. We had a limited budget to work with and a hard deadline. Every decision had to balance technical feasibility against business constraints.
Mapping dependencies and costs. Our first task was creating a detailed inventory of every database with its daily network I/O bandwidth. This revealed critical insights: one Redis instance consumed far more bandwidth than our PostgreSQL databases. This finding shaped our entire migration strategy—we now knew which databases we could migrate first without burning through budget on cross-region traffic if we needed to run dual infrastructure.
Creating detailed migration plans for each component. Not all our microservices were truly independent. Multiple services shared the same databases. Some produced to Kafka while others consumed. We had to sequence the migration carefully, understanding which dependencies could break and when.
Running proof-of-concepts where clarity was lacking. For Cassandra’s multi-DC setup, we built a complete POC. The team executed it successfully in our test environment. But I ultimately decided against using it in production—a decision I’ll explain later.
Building the runbook. We documented every step, no matter how small. Because migrations happen at odd hours when people are tired, and clarity becomes the difference between control and chaos.
Execution: Decision-Making Under Pressure
We prepared the new region’s GKE cluster and started parallel releases in both regions before migration began. This allowed us to validate behavior under real conditions. But the architecture made simple sequential migration impossible—shared databases and Kafka dependencies meant we had to orchestrate carefully.
PostgreSQL: The Model Migration
PostgreSQL was our smoothest experience, and it became the template we wished we could apply everywhere.
We started by ensuring both instances had read replicas. One already had it; the other didn’t, so we added one in the old region first. Then we created Slave A in the new region from the master. We created Slave B from Slave A, also in the new region, and Slave C in the old region for backup.
The warmup phase was critical. We migrated our reads to Slave A gradually—first the Logstash pipelines, then background tasks. This ensured the slave was battle-tested under real load before we made it primary.
The actual cutover was just a DNS switch. After validation, we brought down the old cluster, keeping Slave C at a smaller size for cost control while maintaining a backup in the old region.
This went like butter because we did all the preparation beforehand.
MongoDB: Smooth but with a Learning
For MongoDB, we added nodes to the cluster one by one in the new region. Once they were healthy, we marked them as primary and removed old region nodes, keeping one running in the old region for safety.
This went smoothly—until we discovered one consumer had significant latency issues. We switched back to the old region nodes by making them primary again, and migrated that consumer later when we moved its application. This taught us that gradual migration requires monitoring not just infrastructure health, but application behavior.
Kafka: Parallel Infrastructure
For Kafka, we built a new cluster in the new region and created parallel consumers in both regions. As applications migrated, we moved producers service by service. This approach gave us confidence—if something broke, we could rollback individual services without affecting the entire Kafka infrastructure.
Redis: Different Strategies for Different Instances
Redis presented different challenges across our 6 instances. For some, we created live replicas and failed over. For others, we encountered memory ballooning issues during replication, forcing us to take the snapshot approach—cutover and recreate from AMI snapshots.
The one with the highest network bandwidth, we saved for the final migration alongside our monolith and Cassandra, accepting a planned downtime window.
Elasticsearch: Dual Writes for Safety
We created a new cluster from snapshots and implemented dual writes from the application to both clusters. This gave us rollback readiness while our Logstash pipeline in the new region consumed from the PostgreSQL slave we’d already migrated.
Cassandra: The Hardest Decision
Cassandra was our dragon, and it taught me the most about leadership in engineering.
We had planned to keep parallel infrastructure running in both regions for two weeks as a safety net. But Cassandra’s high network bandwidth consumption made this financially unfeasible—it would blow through our budget.
We evaluated three approaches:
- Dual writes from applications – Impossible. The load would add unacceptable latency to our services.
- Multi-DC setup – The team executed a successful POC. Technically, it worked. But I made the call to not use it in production. Why? Two reasons: First, it was overshooting our budget. Second, and more importantly, I didn’t want to introduce new cluster configurations under migration pressure. We didn’t know how the cluster would behave with these changes under production load, and the risk wasn’t worth the reward.
- Planned downtime with node-by-node migration – We chose this. The team was confident because we’d successfully performed a cluster upgrade a few months prior. We had the operational muscle memory.
We scheduled a 2-hour downtime window during the lowest traffic period, alongside our monolith migration and the highest-bandwidth Redis instance. Business stakeholders were informed well in advance. We brought down all consumers first to prevent data loss, then migrated the cluster node by node.
This decision reflects a key principle: sometimes pragmatism beats elegance. The “perfect” multi-DC solution would have been technically impressive, but the business-aligned solution was 2 hours of planned downtime with clear communication.
The Roadblocks Nobody Plans For
The OpenVPN Ghost: After migrating our Python application, we found certain pods consistently failing with Redis connectivity timeout errors. They were all on the same node, so we cordoned it off. Then another node showed the same problem. We lost more than 1.5 weeks on this.
The pods could connect to everything else—even other Redis instances—but only had problems with this one Redis. We recreated the Redis multiple times, scrutinized route tables, compared configurations with our Taiwan region where everything worked fine. The node itself had connectivity to Redis, just not the pods running on it.
The culprit? Years ago, someone had installed OpenVPN on the Redis machines. It was still running with its default IP ranges. Whenever pods came up with a subnet in that range, they couldn’t connect. A tiny configuration fossil from the past, costing us a week and a half.
The DNS Caching Surprise: After updating DNS for our databases, we discovered a Java Play application still connecting to old database IPs. It had cached the DNS settings. We caught this within an hour and redeployed, but it was a reminder to verify DNS refresh behavior in all application frameworks.
These incidents reinforced a truth about migrations: they expose every hidden corner of your infrastructure—the decisions made years ago that nobody documented, the configurations nobody remembers enabling.
Leadership Lessons: What I’d Tell My Past Self
1. Document everything, especially the “obvious” steps. Our detailed runbook saved us during those 5 AM migrations. When you’re exhausted and under pressure, having every step written down—however simple it seems—keeps the team moving confidently. This isn’t about trusting your team’s competence; it’s about respecting human limitations at 3 AM.
2. Align business stakeholders early and often. The Cassandra downtime decision worked because we communicated it three weeks in advance, explained the tradeoffs clearly, and scheduled around product launches. Leadership isn’t about avoiding difficult conversations—it’s about having them at the right time with the right context.
3. Budget time for the unknown. The OpenVPN issue wasn’t in any planning document. But we had built buffer time into our schedule specifically for “things we don’t know we don’t know.” That buffer saved our deadline.
4. Make pragmatic decisions, not perfect ones. The multi-DC Cassandra setup was technically appealing. But given our budget constraints, timeline pressure, and risk tolerance, planned downtime was the right call. Perfect is the enemy of shipped, and shipped is the enemy of missed compliance deadlines.
5. Build on past operational wins. We were confident migrating Cassandra because we’d upgraded the cluster months before. That operational experience gave us muscle memory when it mattered. Invest in operational maturity even when there’s no immediate crisis—you’re building organizational capability for future challenges.
6. Trust your team, then verify the details. The team executed the multi-DC POC successfully. But as a leader, I had to look at the broader picture—budget, risk, and business impact—and make a different call for production. This tension between technical capability and business context is where leadership happens.
The Results That Mattered
The migration wasn’t just about compliance—it delivered tangible performance improvements that affected our customer experience:
Latency improvements were dramatic. Redis, Cassandra, Kafka, and MongoDB all dropped from double-digit milliseconds to single-digit, with some workloads achieving sub-1ms response times. For real-time workflows, this was transformative.
Infrastructure clarity improved. Our dependency mapping and bandwidth analysis became living documents that informed capacity planning and architectural decisions for months afterward. We discovered services that weren’t actually critical, workflows that could be optimized, and dependencies that could be simplified.
Team confidence grew. Engineers who had never touched certain databases became comfortable with cluster operations. We learned to plan deeply, communicate clearly, and execute under pressure. This organizational capability is more valuable than any single migration.
Closing Thoughts
Migrations are rarely about technology alone—they’re about leadership under constraints. Budget, compliance, risk, timeline, and customer impact all pull in different directions. The real win wasn’t just moving infrastructure across regions. It was building organizational confidence that our team, our processes, and our decision-making can handle change at scale.
As leaders, we often celebrate velocity and moving fast. But sometimes the most important thing you can do is slow down, plan deeply, align stakeholders clearly, and then execute decisively when it matters.
That morning after our second 5 AM session, watching latency drop to microseconds—it wasn’t just a technical achievement. It was validation that thoughtful preparation, pragmatic decision-making, and clear communication can guide teams through complex challenges.
And it was proof that sometimes, the best engineering decision is the one that considers the business, not just the technology.