The migration that runs while both versions are live

The failure is always the same. The migration ran, the deploy went out, and for ninety seconds half the fleet was old code meeting a new schema. Those ninety seconds are the whole problem, and no amount of care inside the migration script addresses them.

The constraint

Old code and new code will run at the same time, against the same database. This is true during a rolling deploy, true during a canary, and true again during a rollback — which is the case people forget, because a rollback puts old code against a schema that has already moved.

Every change must therefore be compatible in both directions, or the deploy is a coordinated outage.

Expand, migrate, contract

Three releases, not one. It feels slow and it is the only shape that works.

Expand. Add the new structure without touching the old. A new nullable column, a new table, a new field the old code ignores. Both versions work: old code does not know it exists, new code writes to both.

Migrate. Backfill existing rows, and switch reads to the new structure once the backfill is complete. Writes still go to both. This release is reversible: rolling back returns to reading the old structure, which is still correct because it is still being written.

Contract. In a later release — days or weeks later, not minutes — stop writing the old structure and drop it. Only once nothing reads it, which you should verify rather than assume.

The temptation is to compress this into one deploy. The compression is exactly what removes the ability to roll back.

The specific operations that bite

Renaming a column. Never rename. Add the new one, write both, backfill, move reads, drop the old. A rename is an atomic break for every version that does not have it.

Adding NOT NULL. Add the column nullable with a default, backfill, then add the constraint. Adding a non-nullable column in one step breaks every old insert in flight.

Dropping a column. Stop reading it in one release, stop writing it in the next, drop it in a third. A dropped column that old code still selects is an immediate error on every request that touches it.

Changing a type. A new column of the new type, dual writes, backfill, move reads, drop. In-place type changes rewrite the table and take a lock proportional to its size.

Adding an index. Use the concurrent form your database offers. A plain index build takes a write lock for the duration, which on a large table is an outage nobody attributed to a migration.

Foreign keys and check constraints. Add as NOT VALID first and validate separately, so the initial statement does not scan the whole table under a lock.

Locks are the other half

A migration that is logically correct can still cause an incident by holding a lock. Two habits prevent most of it:

• Set a short lock timeout for the migration session. It is far better for a migration to fail fast and be retried than to queue every query behind it. A statement waiting on a lock blocks everything behind it, and the outage looks like a database problem rather than a deploy.
• Run long backfills in batches, outside the migration, with a pause between them. A single UPDATE across ten million rows holds locks and generates replication lag that arrives as a separate incident on your replicas.

Verifying rather than assuming

Before the contract step, prove nothing uses the old shape:

1. Search the codebase, including generated queries, ORM models and analytics jobs that nobody deploys with the application.
2. Instrument it. Log a warning whenever the old field is read, ship that, and wait. A week of silence is evidence; a code search is a hypothesis.
3. Check the consumers you do not own — reporting tools, exports, another team's service reading your database directly. This is where the surprise usually lives.

The question is never "is this migration correct?" It is "is this migration correct while the previous version is still running, and again if we roll back?"

What to write down

For each migration, three lines in the pull request:

• What runs before the deploy, and what runs after.
• What happens if we roll back at each point in between.
• When the contract step happens, and what has to be true first.

That is the whole discipline. It costs a few minutes per change and it converts the most common source of self-inflicted downtime into a routine one.