Race Condition
Find and fix concurrency bugs with deterministic reproduction, not luck.
Why
Race conditions are the hardest bugs because they're non-deterministic: the test passes 99 times, fails on the 100th, and you can't reproduce it. They happen when two async operations access shared state and the order matters — 'check then act' (read balance, then write balance minus amount) is the classic. Without a method, developers add a sleep 'to let it settle' and ship — the bug returns in prod under load. The fix is always one of: make the operation atomic (DB transaction, mutex), make the check and act a single operation (compare-and-swap), or serialize the operations (queue). The method is: reproduce deterministically (inject delays, use a deterministic scheduler), find the interleaving, then apply the right fix. Concurrency bugs that ship to prod cause data corruption that's far more expensive than the week you'd spend finding them.
When
When a bug is intermittent (passes sometimes, fails sometimes) under the same input. Takes 2-4 h. Skip never for shared-state code.
How
1. Reproduce deterministically — force the interleaving you suspect - Inject artificial delays at the boundary: await sleep(100) between check and act - Use a deterministic async scheduler (sinon fake timers, @effect/Trie) to control ordering 2. Identify the shared state and the two operations that race - "balance is read by both withdraw() and deposit(); the read-then-write isn't atomic" - Draw the interleaving on paper: T1 reads 100, T2 reads 100, T1 writes 50, T2 writes 200 (should be 150) 3. Pick the fix pattern that matches the problem - Atomic operation: DB transaction (BEGIN; ... COMMIT) wraps check + act - Compare-and-swap: UPDATE accounts SET balance = balance - 50 WHERE id = 1 AND balance >= 50 - Serialize: push operations to a queue, process one at a time per account 4. Add a stress test that runs the operation 1000x concurrently - With the fix, the invariant (sum of balances = constant) holds every run - Without the fix, the test fails ~30% of the time — that's the proof 5. Audit sibling code for the same pattern — races love company - Every "check then act" on shared state is suspect; fix them all in this PR
Example
// BUG: classic check-then-act race on balance
async function withdraw(accountId, amount) {
const balance = await db.getBalance(accountId); // T1: reads 100
if (balance < amount) throw new Error("insufficient funds");
await db.setBalance(accountId, balance - amount); // T2: writes 50, but T2 also read 100!
}
// 100 concurrent withdrawals of 10 from a balance of 100 → balance can go negative
// FIX 1: atomic compare-and-swap (best — one query, no transaction)
await db.query(
"UPDATE accounts SET balance = balance - $1 WHERE id = $2 AND balance >= $1",
[amount, accountId],
);
// if 0 rows affected → insufficient funds (someone else won the race)
// FIX 2: transaction with row lock (when you need to read-then-write more logic)
await db.transaction(async (tx) => {
const balance = await tx.query("SELECT balance FROM accounts WHERE id = $1 FOR UPDATE", [accountId]);
if (balance < amount) throw new Error("insufficient funds");
await tx.query("UPDATE accounts SET balance = balance - $1 WHERE id = $2", [amount, accountId]);
});
// Stress test — proves the fix
test("100 concurrent withdrawals never overdraft", async () => {
await db.setBalance(1, 1000);
const results = await Promise.allSettled(Array.from({ length: 100 }, () => withdraw(1, 10)));
const finalBalance = await db.getBalance(1);
expect(finalBalance).toBeGreaterThanOrEqual(0); // never negative
expect(finalBalance + results.filter(r => r.status === "fulfilled").length * 10).toBe(1000);
});Common Problems
- ⚠Adding a sleep() to 'fix' the race — it hides the bug under low load and returns under high load.
- ⚠Using a JS mutex on a DB race — the mutex is per-process; two server instances still race at the DB.
- ⚠Wrapping the whole function in a transaction but forgetting FOR UPDATE, so the read still isn't locked.
- ⚠Fixing one race and not auditing siblings — the same check-then-act pattern is in 5 other places.
Best Practices
- ✓Prefer compare-and-swap (UPDATE ... WHERE condition) over read-then-write — it's atomic at the DB level and needs no transaction.
- ✓Add a concurrent stress test for every shared-state operation; run it 1000x in CI so races fail the build, not prod.
- ✓Use SELECT ... FOR UPDATE inside transactions to lock rows you'll update — without it, the read is non-locking and races.
- ✓Document the concurrency model at the top of shared-state modules (e.g. 'all mutations go through a single actor') so future edits don't break the invariant.