Skip to content

Rollback

Use this page when cutover fails before system-app-pre has succeeded on 2.16. Until then, embedded Deployments stay at 0 replicas and their PVCs stay. Clone-friendly copy: docs/runbooks/en/07-rollback.md (Spanish: docs/runbooks/07-rollback.md).

After system-app-pre succeeds, new writes exist only on the external database. Scaling the embedded databases back up drops those writes. Rollback is a restore of the fresh dump taken immediately before /values-216-external.yaml. There is no OLM downgrade.

Ownership and rollback paths

Who owns what

Namespace Operator manages You manage
3scale system-app, APIcast, Zync, zync-database, system-storage Connection secrets (system-database, system-redis, backend-redis)
3scale-db Nothing from the 3scale operator after externalComponents PostgreSQL 15, both Redis 7, PVCs, images, backups, NetworkPolicy

See Why this repo for the full external ≠ off-cluster explanation and what the 3scale operator stops reconciling.

PostgreSQL restore fails

  1. Set externalComponents.system.database back to false on the APIManager.
  2. Reapply embedded PostgreSQL in the APIManager spec if needed: spec.system.database.postgresql: {}.
  3. Restore secret system-database from the backup you took (system-database-secret.yaml).
  4. Scale up the 3scale operator and embedded system-postgresql.
  5. Validate Admin Portal and APIs against embedded data.
  6. Do not delete the new PostgreSQL PVC in 3scale-db until embedded PostgreSQL is confirmed healthy.

Redis RDB empty or restore fails

  1. Set externalComponents.system.redis and externalComponents.backend.redis back to false.
  2. Restore secrets system-redis and backend-redis from backup.
  3. Scale embedded backend-redis and system-redis back up on 3scale 2.15.
  4. Validate portals and API traffic.
  5. Do not delete new Redis PVCs in 3scale-db until embedded Redis is confirmed healthy.

If dump.rdb was empty on Git Bash, use docs/runbooks/02-bis-externalize-redis-windows.md (MSYS_NO_PATHCONV=1, oc exec + cat) and re-run Externalize Redis.

3scale operator 2.16 preflight loop

Symptoms: operator 2.16 installs but the instance upgrade never completes; version checks fail every ~10 minutes.

  1. Confirm PostgreSQL ≥ 15.0 and Redis ≥ 7.2 on the external pods (Upgrade operator to 2.16).
  2. Apply the PostgreSQL 15 grant if system-app-pre reports permission denied for schema public — see Grant CREATE on schema public.
  3. Delete the system-app-pre Job and let the operator recreate it.
  4. Do not combine an OpenShift upgrade with the 3scale upgrade in the same window.
  5. If versions cannot be fixed in time, stay on channel threescale-2.15 until external databases meet preflight.

GitOps rollback trap (Redis)

If the Pattern stays on values-cutover.yaml, a pod restart can drop Redis data. Switch extraValueFiles to /values-216-external.yaml before you treat the migration as done. See GitOps pitfalls and Day 2 operations.

Next steps