Skip to content
Back to Deployment

Canary Release and Progressive Rollout

Metric-driven traffic ramp (1% → 10% → 50% → 100%) with automatic rollback on error budget breach.

MonitoringAdvanced

Why

Canary limits blast radius: a bad deploy affects 1% of users, not 100%. Progressive rollout with automatic metric gates (error rate, latency p99, business KPIs) catches regressions that pass staging but fail at production scale, rolling back before customers notice.

The CNCF 2024 survey shows Prometheus and Grafana are used by 70%+ of cloud-native organizations. OpenTelemetry became the standard for instrumentation in 2024, reducing vendor lock-in for metrics and traces.

When

Use canary for high-risk changes (payment flow, auth, schema migrations), high-traffic services (>10k RPS), and continuous deployment. Skip for low-traffic internal tools where blue-green suffices and the canary signal is too noisy at 1%.

How

1. Define metrics — error rate (5xx/total), latency p99, business KPI (conversion, checkout success) 2. Set thresholds — error rate < 0.1%, p99 < 500ms, KPI within 2% of baseline; breach = rollback 3. Choose tool — Argo Rollouts (K8s), Flagger, AWS CodeDeploy, Spinnaker; or LB weight manipulation 4. Deploy canary — 1% traffic to new version, 99% to stable; run for 5-10 min analysis window 5. Metric gate — compare canary vs stable; if within threshold, ramp to 10%; else rollback to 0% 6. Progressive ramp — 1% → 10% → 25% → 50% → 100%, with metric gate at each step (5-10 min each) 7. Auto-rollback — on threshold breach at any step, shift 100% back to stable, alert on-call 8. Post-deploy — keep stable version for 1h (rollback window), then decommission

Example

# rollout.yaml — Argo Rollouts canary with metric analysis apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: myapp spec: replicas: 10 strategy: canary: trafficRouting: istio: host: myapp.example.com steps: - setWeight: 5 - analysis: templates: - templateName: success-rate args: - name: service-name value: myapp - setWeight: 25 - analysis: templates: - templateName: success-rate args: - name: service-name value: myapp - setWeight: 50 - setWeight: 100 template: spec: containers: - name: myapp image: myapp:v2.0 ports: [{ containerPort: 3000 }] --- # analysis-template.yaml — Prometheus success-rate gate apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: args: - name: service-name metrics: - name: success-rate interval: 1m successCondition: result >= 0.99 failureLimit: 3 provider: prometheus: address: http://prometheus:9090 query: | sum(rate(http_requests_total{service="{{args.service-name}}",code!~"5.."}[2m])) / sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))

Common Problems

  • ⚠1% traffic too noisy — at 100 RPS, 1% = 1 req/s, error rate swings wildly, use longer analysis window
  • ⚠Metric gate too loose — p99 < 1s passes but p99.9 is 10s, 0.1% of users get terrible latency
  • ⚠Business KPI not measured — error rate fine but conversion drops 20%, canary 'passes' and ships the bug
  • ⚠No baseline comparison — canary metrics checked against absolute threshold, not vs stable version

Best Practices

  • ✓Compare canary vs stable (relative), not absolute thresholds — catches 'both bad but canary worse'
  • ✓Include a business KPI metric — error rate + latency + conversion, not just infra metrics
  • ✓Set failureLimit to 2-3 — one bad sample shouldn't rollback, but a trend should
  • ✓Mirror traffic before canary — shadow real requests to canary, compare responses without user impact

FAQ

What percentage should I start my canary at?

Start at 1-5% for 15-30 min, then ramp to 10%, 25%, 50%, 100% if metrics are clean. Starting at 50% defeats the purpose — you affect half your users before catching issues. Use automated progressive delivery (Flagger, Argo Rollouts).

What metrics should I watch during a canary?

Error rate (5xx), latency p99, and business metrics (conversion, signup). Set thresholds before deploying — e.g., rollback if error rate exceeds 0.5% or p99 latency increases over 20%. Don't watch CPU/memory; they don't indicate user impact.

How is canary different from A/B testing?

Canary tests for errors/performance (does the new version break?), A/B tests for business outcomes (does it convert better?). Canary is a safety mechanism; A/B is an experiment. You can run canary within an A/B test.

How do I route only some users to the canary?

Use a weighted load balancer (Istio, ALB weighted target groups) or a feature flag with a percentage rollout. Sticky sessions ensure a user stays on the same version — otherwise they'll see flapping behavior.

How long should a canary run before full rollout?

Long enough to see meaningful traffic — at least 10,000 requests or 30 min, whichever is longer. Too short and you miss rare errors; too long and you delay the release. Automate the ramp so you don't babysit it.