Hotfix Process

Ship a critical fix outside the normal release cadence — fast, safe, and with a post-mortem.

Why

A hotfix is an exception to the release process, and exceptions are where process breaks down — a fix rushed out without tests, without review, without rollback, becomes a bigger incident than the vuln it fixed. A defined hotfix process keeps the speed while keeping the safety: branch from the release tag (not main, which may have unreleased changes), write the fix + a regression test, get a fast-track review (1 reviewer, 30-min SLA), deploy with a rollback warm, and schedule a post-mortem within 48h. The process also makes hotfixes rare-by-design: if you're hotfixing every week, your release cadence is too slow, and the fix is to speed up the normal process, not to normalize hotfixes.

When

For P0 vulns only — exploitable in production, no workaround available. Never for features or non-critical bugs.

How

1. Branch from the production release tag (not main): git checkout -b hotfix/<vuln-id> v1.2.2
2. Write the minimal fix + a regression test that reproduces the vuln; commit
3. Fast-track review: 1 reviewer, 30-min SLA, security lead notified (not the normal 2-reviewer, 24h gate)
4. Run the full test suite + e2e; if any non-vuln test fails, fix it or scope the hotfix narrower
5. Tag as a patch: v1.2.3; deploy to staging, smoke test, then prod with the previous image tagged :prev for rollback
6. Monitor for 1h; if regressions, kubectl rollout undo (instant rollback to :prev)
7. Merge the hotfix branch back to main (so the fix isn't lost in the next release)
8. Schedule a post-mortem within 48h: what broke, how was it found, how to prevent recurrence
9. Track a hotfix log: date, vuln, time-to-fix, did rollback work, post-mortem link

Example

# Hotfix for CVE-2024-9999 (auth bypass, P0)
# 1. Branch from the production tag (NOT main)
git checkout -b hotfix/cve-2024-9999 v1.2.2

# 2. Minimal fix + regression test
# src/auth.ts: add the missing token verification
# test/auth-bypass.test.ts:
it('rejects forged tokens (CVE-2024-9999)', () => {
  expect(() => auth(forgedToken)).toThrow('invalid token');
});
git commit -m "fix(security): auth bypass [CVE-2024-9999, P0 hotfix]"

# 3. Fast-track review (1 reviewer, 30-min SLA)
gh pr create --reviewer security-lead --label hotfix
# security-lead reviews within 30 min

# 4. Tag + deploy with rollback warm
git tag v1.2.3
docker tag registry/app:current registry/app:prev  # rollback warm
docker build -t registry/app:v1.2.3 . && docker push registry/app:v1.2.3
kubectl set image deployment/app app=registry/app:v1.2.3
# monitor 1h; if error rate spikes: kubectl rollout undo

# 5. Merge back to main + post-mortem
git checkout main && git merge hotfix/cve-2024-9999
# schedule post-mortem within 48h
# log: hotfixes/2024-04-15-cve-2024-9999.md

Common Problems

⚠Branching from main instead of the release tag — the hotfix includes unreleased changes, so the patch release has unintended features.
⚠Skipping the regression test to save time — the fix ships, but a future refactor silently reverts it because nothing catches it.
⚠No post-mortem — the hotfix is treated as over when it deploys, but the root cause (how the vuln got in) is never investigated.

Best Practices

✓Always branch from the production release tag, not main — a hotfix must contain only the fix, not unreleased work.
✓Require a regression test even for a hotfix — the test proves the fix and prevents silent reversion in a future refactor.
✓Schedule a post-mortem within 48h of every hotfix — the deploy is not the end; understanding the root cause prevents recurrence.