Spot & Preemptible
Use 60-90% discounted spare capacity for workloads that tolerate interruption.
Why
Cloud providers sell their spare capacity at 60-90% discount, reclaimable with 2-min notice. For batch jobs, CI runners, and stateless web tiers that can checkpoint and restart, this is free money — a 100-instance CI fleet drops from $30k/mo to $6k/mo. The catch is interruption: spot instances get reclaimed during capacity crunches, so they're wrong for anything stateful or latency-sensitive. The discipline is to identify which workloads are interruption-tolerant and move them to spot, leaving on-demand/reserved only for the parts that can't be killed.
When
For any batch, CI, or stateless workload. Takes 1-2 hours to migrate. Skip for databases, queues with messages in-flight, or user-facing latency-critical paths.
How
1. List workloads; mark each as interruption-tolerant or -sensitive 2. For tolerant ones (CI, batch, stateless web, ML training with checkpointing): a. Switch the autoscaling group to 90% spot + 10% on-demand (fallback) b. Add a graceful-shutdown handler that checkpoints on the 2-min warning c. Set up a spot-interruption metric; alert if rate > 5%/hour (capacity crunch) 3. For sensitive ones, keep on-demand/reserved — do NOT chase the discount 4. Track the savings: (on-demand cost − spot cost) × hours, report monthly 5. Have a fallback plan: if spot becomes unavailable, auto-scale on-demand
Example
CI runner fleet: 50 instances, 24/7, xlarge ($0.19/hr) On-demand: 50 × $0.19 × 730 = $6,935/mo Spot (70%): 35 × $0.057 × 730 = $1,456/mo (spot price 70% off) On-demand fallback (30%): 15 × $0.19 × 730 = $2,079/mo Total: $3,535/mo — 49% savings, $40k/year Interruption handling: CI jobs are idempotent + re-queued on reclaim. Avg interruption: 2%/day → negligible re-run cost.
Common Problems
- ⚠Moving a stateful workload to spot to save money — it gets reclaimed mid-write and you lose data; spot is for stateless only.
Best Practices
- ✓Always keep 10-20% on-demand capacity in a spot fleet so the service stays up during a capacity crunch that reclaims all spot.
- ✓Make jobs idempotent before moving to spot — a job that can't safely re-run will cause data corruption on interruption, not just delay.