Brands post paid campaigns on a mobile marketplace API, and creators apply from the app, backed by a NestJS and TypeORM service on Google Cloud Run with Cloud SQL Postgres. When a deploy adds a column the list endpoint reads, a creator opening GET /campaigns can suddenly see a generic 500 whose body mentions column "x" does not exist, even though the deploy looked healthy minutes earlier.

That symptom is what this setup produces when a database migration (an ordered SQL change to Postgres) fails during container boot yet the process still starts. The container entrypoint is the shell script that runs before Node; ours used to call migrations there, retry five times, print that migration failed, and continue. Cloud Run then marked the revision green because the health check only hits routes that never touch the new column, shifted traffic to the new revision (an immutable deployed snapshot of the service), and left the database on the old schema while code expected the new one.

Why boot-time migration feels resilient

On a serverless platform there is no single "app start." Every scaled instance runs the same entrypoint, so several copies can hammer the same migration at once. Fail open means a failed step still returns success and the main process starts; fail closed would stop the deploy. Starting anyway reads as graceful degradation. Terraform in the same pipeline could also change the service image field, so infrastructure apply and gcloud run deploy both had a path to move traffic before anyone proved the schema matched the binary.

Pick Migrate at boot (start anyway), turn on Migration fails, press Deploy, and watch the health check stay green while GET /campaigns turns red. Switch to Migrate job first (fail closed), deploy again with the same failure toggled, and the pipeline stops before traffic moves.

Gate the schema before the revision

The reusable rule: treat a schema migration as a deploy gate. Run it exactly once, before any traffic moves to new code, and fail closed. Let only one automation change the running image so nothing skips that gate.

The entrypoint now only writes credentials and execs Node. Migrations moved to a Cloud Run Job (a run-to-completion task with no HTTP traffic), named ${service}-migrate, with database env and secrets only, no job-level retries, and a ten minute timeout. The script exits non-zero after the last retry:

attempt=1
until node ./node_modules/typeorm/cli.js migration:run -d ./dist/src/data-source.js; do
  if [ "$attempt" -ge "$MAX_ATTEMPTS" ]; then
    echo "Migration failed after $MAX_ATTEMPTS attempts."
    exit 1
  fi
  attempt=$((attempt + 1)); sleep "$RETRY_INTERVAL"
done

The GitHub Actions deploy resolves the image, updates the job to that image, runs gcloud run jobs execute … --wait, and only then runs gcloud run deploy. Terraform uses ignore_changes on the container image so apply cannot roll the service to :latest ahead of the migrate step; only the workflow moves images. Per-environment concurrency groups with cancel-in-progress: false keep two deploys from migrating and releasing at the same time.

Costs and when this is not enough

Migrations must stay backward compatible with the code still serving traffic during the gate (expand then contract: add nullable columns first, deploy code that reads them, then remove old shapes in a later migration). Rollback is redeploy the previous image or shift traffic to an older revision; TypeORM will not automatically revert SQL. Terraform can still create a new revision when env vars change on the current image, so strict ordering covers image bumps, not every infra tweak.

I found this while hardening the pipeline, not after a production outage. The trap is quiet: green health, red business routes, migration errors buried in instance logs.

Happy coding!

Sander