Log Retention

Set tiered retention (hot/warm/cold) that balances query speed, cost, and compliance needs.

Why

Keeping 90 days of logs hot (fast SSD, fully indexed) costs 10x more than warm (cheap disk, less index) and 100x more than cold (object storage, no index). But you need hot for the last 7-30 days (incident diagnosis, real-time dashboards), warm for 90 days (postmortem forensics), and cold for 1-7 years (compliance, rare investigations). A tiered policy puts each log in the right tier based on age: hot for 30 days, warm for 60 more, cold for 2+ years. This cuts log storage costs 70-80% while keeping fast queries for recent incidents and compliance coverage for the long tail. The trap is keeping everything hot 'just in case' — a 10,000-service org keeping 90 days hot spends $500k/year on log storage, vs. $100k with tiering, for the same queryability on recent data.

When

When you set up log aggregation. Review the policy quarterly as volume and compliance needs change.

How

1. Define the tiers and their query patterns
   - Hot (0-30d): fast queries (<1s), real-time dashboards, incident diagnosis
   - Warm (30-90d): slower queries (<10s), postmortem forensics, trend analysis
   - Cold (90d-7y): slow queries (<60s) or restore-to-hot, compliance, rare investigations

2. Set the retention per tier
   - Hot: 30 days (or 7 if budget is tight; 30 is the sweet spot for incident diagnosis)
   - Warm: 60 days (30-90 total from ingest)
   - Cold: 2-7 years (driven by compliance: GDPR, HIPAA, SOC2)

3. Move logs between tiers automatically
   - Hot → warm at 30d: change storage class, reduce index
   - Warm → cold at 90d: move to object storage (S3), drop index
   - Cold → delete at 7y: compliance retention met

4. Query the right tier for the job
   - Incident diagnosis (last 24h): query hot → <1s
   - Postmortem (last 30d): query hot → <1s
   - Trend analysis (last 90d): query warm → <10s
   - Compliance audit (last 3y): restore cold to hot, then query → <60s

5. Track cost per tier and adjust
   - If hot is 60% of cost but 5% of queries → shorten hot to 14d
   - If cold is never queried → shorten cold to 2y (or compliance minimum)
   - The policy is a dial, not a constant; adjust based on actual usage

Example

# Retention policy (Loki + S3):
retention:
  hot:
    duration: 30d
    storage: ssd-local          # fast queries, expensive
    index: full                 # all fields indexed
  warm:
    duration: 60d               # 30-90d from ingest
    storage: hdd-shared         # slower queries, cheaper
    index: reduced              # only service, level, trace_id
  cold:
    duration: 2555d             # 7 years (GDPR)
    storage: s3-glacier         # object storage, no index
    index: none                 # restore-to-hot to query

# Cost comparison (10k services, 1TB/day):
#   All hot (90d):  90 TB × $23/TB/mo = $2,070/mo = $24.8k/year
#   Tiered:         30 TB hot ($690) + 60 TB warm ($360) + 2555 TB cold ($25) = $1,075/mo = $12.9k/year
#   Savings: 48% for the same recent-data queryability

# Query routing (automatic):
#   "last 24h checkout errors" → hot → 0.3s
#   "last 90d error trends"    → warm → 8s
#   "last 3y for compliance"   → cold → restore 2m + query 60s = 2m60s
#   The user doesn't pick the tier; the query age determines it

Common Problems

⚠All hot, no tiering — 10x the cost for the same recent-data queryability; 'just in case' isn't a retention strategy.
⚠No cold tier — compliance requires 7-year retention but you delete at 90d, creating a compliance gap.
⚠Manual tier moves — someone moves logs to cold by hand each month, so it doesn't happen; automate hot→warm→cold by age.
⚠Querying cold without restoring — a 7-year compliance query takes 60s on cold (no index) vs. 1s on hot; restore first for investigations.

Best Practices

✓Set tiered retention: hot 30d (fast queries, full index), warm 60d (slower, reduced index), cold 2-7y (object storage, compliance).
✓Automate tier transitions by age (hot→warm at 30d, warm→cold at 90d); manual moves don't happen.
✓Route queries by age automatically — recent to hot, 30-90d to warm, compliance to cold-with-restore; the user doesn't pick the tier.
✓Track cost per tier and adjust; if hot is 60% of cost but 5% of queries, shorten hot — the policy is a dial, not a constant.