Summary

On Thursday, December 11th, 2025, at approximately 19:00 PST, a migration began to move an account from the prod-2 Looker spoke to a new, isolated spoke. The migration involved two deployments: one to add files to the new spoke and one to remove them from prod-2. Around 20:00 PST, reports emerged that prod-2 account users were seeing error messages in dashboard tiles, as the second deployment to remove the files had appeared to have failed without notice. Ultimately, by 20:30, access was fully restored to prod-2 accounts once the second commit was successfully deployed.

Root Cause

A specific account on the prod-2 spoke held niche logic and unique requirements that were incompatible with updated aspects of Looker and as such, the decision to migrate them to a different spoke was made. To ensure strict continuity of existing logic, the new spoke began as a direct copy of prod-2 which then had been pruned to specific files. Due to the shared base, the first deployment inadvertently applied the targeted account's configuration to both spokes, resulting in the prod-2 spoke only containing a single account’s files. The second deployment that was thought to have been installed earlier was either overwritten by the first or failed without immediate notification to the development team. After the reports of lost access had surfaced, the development team reviewed the original spoke and observed that the second commit was available, as if awaiting to be deployed.

Impact

The change affected the entire prod-2 spoke, impacting modules beyond CCM and consequently all users with accounts hosted in prod-2. Due to the mirrored first deployment, dashboard tiles displayed "Trouble loading data." Downtime was from 19:00 PST to 20:30 PST, with full functionality restored after the successful second update.

‌

Remediation

Immediate 

We expedited the execution the final step of migration

Permanent

Stricter guidelines related to testing and monitoring immediately after deployment have been put into place

Action Items

1. Alternative migration processes - Future full account migrations, while rare, will use Client API/SDK tools instead of copying the original spoke to avoid potential repository conflicts.
2. Requirement to immediately monitor - Post-deployment, the team will use support tools to monitor immediate outcomes on representative customer accounts.
3. Increased observability requirements - To ensure varied perspectives and insights, all deployments now require a specific number and variance of team members so that this will not occur again.