Debugging a Race Condition That Took Three Weeks to Find

Every engineer has one bug that haunts them. This is mine.

The symptom

Roughly once every few thousand transactions, a customer's account balance would briefly display an incorrect value — one that self-corrected within a second or two on refresh. No data was actually lost or corrupted, but customers were, understandably, alarmed. Support tickets trickled in for weeks before anyone connected them.

The maddening part was reproducibility — or the total lack of it. It happened in production, never in staging, and never on demand. Our logs showed nothing unusual at the time of each incident.

The wrong turns

My first theory was a caching bug — maybe a stale cache entry occasionally served before invalidation. I spent four days adding cache-hit logging and instrumenting our Redis calls. Nothing. Cache behavior was completely correct.

My second theory was clock skew between services affecting our optimistic locking. I built a whole dashboard tracking NTP drift across our fleet. Also nothing — drift was consistently under 5ms, far too small to explain what we were seeing.

The breakthrough

The fix came from a single log line I added almost as an afterthought: logging the thread ID alongside every balance read and write. That log line revealed that two requests — a balance read and a concurrent balance update — were occasionally interleaving inside what we'd assumed was an atomic operation.

// the bug: read-modify-write without a lock
let balance = db.get_balance(user_id).await?;
let new_balance = balance + amount;
db.set_balance(user_id, new_balance).await?;

The read and write were two separate database calls, not a single atomic transaction. Under low load, the window between them was small enough to never matter. Under the concurrency we'd grown into, two requests for the same user could occasionally race through that window at once, and whichever write landed second would silently overwrite the other.

The fix, and the lesson

The fix itself was almost embarrassingly small — wrapping the read-modify-write in a single database transaction with a row-level lock. Three lines of code, three weeks of searching.

The real lesson wasn't technical, it was procedural: we didn't have per-thread or per-request tracing on a code path we'd assumed was "simple enough" not to need it. Now, any function that reads and writes shared state gets that instrumentation from day one, whether we think we need it or not.