Design and implement a resiliency strategy for deployment

A deployment resiliency strategy limits exposure, detects degradation, and provides tested recovery paths. Automate detection and reversal where the requirements call for it.

Automatic reversal

yaml

strategy:
  runOnce:
    deploy:
      steps: [ { script: ./deploy.sh } ]
    postRouteTraffic:
      steps: [ { script: ./smoke-and-metrics.sh } ]
    on:
      failure:
        steps: [ { script: ./rollback.sh } ]
      success:
        steps: [ { script: ./cleanup.sh } ]

The important pairing is in the authored steps. postRouteTraffic runs steps after traffic is routed, where a health-monitoring step can fail the lifecycle. on: failure then runs authored rollback or cleanup steps after any lifecycle-step failure. Neither hook detects or reverses by itself.

Layers

Layer	Mechanism
Detect	Health-monitoring steps under `postRouteTraffic`; Query Azure Monitor Alerts succeeds when no configured alert rule is active
Reverse	Authored recovery steps under `on: failure`; swap the same slots back; disable the affected flagged feature
Contain	Canary and rolling — limit exposure before you detect
Serialise	Exclusive lock — allow one run or stage at a time to use the protected resource until that stage completes

Exclusive lock defaults to runLatest, which retains only the latest contender. Use sequential when every queued run must acquire the protected resource in turn. The lock does not remain held merely because an incident continues after the stage completes.

Idempotency

Retries and partial failures mean a deployment step may run twice. Scripts must be safe to re-run — the resiliency plan is only as good as the least idempotent step in it.

Primary sources

Deployment resiliency

Design and implement a resiliency strategy for deployment

Automatic reversal

Layers

Idempotency