Concurrency & Shared State
Concurrency bugs are the worst kind: rare, unpredictable, invisible in testing, and severe in production. They appear when several things touch shared state at once without coordination. The most reliable defence is not clever locking. It is not sharing mutable state in the first place.
A web service is concurrent by nature: many requests run at once, across threads and instances. Most of the time that is fine, because each request works on its own data. Trouble starts wherever they share something mutable — a static field, a cached object, a row two requests both update, a counter. There the order of operations becomes unpredictable, and 'works on my machine' tells you nothing.
There are two failure modes to design against. The first is a race: interleaved operations produce a state neither one intended. The second is a lost update: two read-modify-write cycles run, and one silently overwrites the other. In a financial system these are not glitches. They are double-spends, miscounted balances, and duplicated effects. Prefer designs that avoid shared mutable state. Where state must be shared, coordinate access explicitly.
Avoid shared mutable state
- DoPrefer stateless, share-nothing designs. Each request works on its own data, so most concurrency problems simply cannot happen.
- DoFavour immutable data and pure transformations. Data that never changes can be shared freely without coordination.
- DoKeep per-request state per-request. Never store request- or user-specific data in static or shared fields.
- ConsiderPushing coordination down to the database (transactions, atomic updates, constraints), which is built to manage concurrent access correctly.
- Do notUse mutable static state, singletons with mutable fields, or shared in-memory collections for convenience. They are concurrency bugs waiting for load.
- NeverAssume single-threaded or single-instance execution. The service runs many requests at once across multiple instances.
Coordinate when you must share
- DoProtect genuinely shared mutable state with the right primitive — a lock, a concurrent collection, or atomic operations — chosen on purpose, not by guesswork.
- AlwaysHandle read-modify-write with optimistic concurrency (row version / ETag) or suitable locking, so a concurrent update cannot be silently lost.
- DoMake operations idempotent and use idempotency keys, so retries and redeliveries under concurrency do not apply twice (see Data Integrity & Transactions).
- ConsiderDistributed coordination (a distributed lock or a single-owner queue) when correctness must hold across multiple instances, not just threads.
- Do notHold a lock across I/O or an external call, or take multiple locks in inconsistent order. That is how you get stalls and deadlocks.
- NeverDo a money- or balance-changing read-modify-write without concurrency control. Last-writer-wins on a balance loses money.
var acct = Load(id);
acct.Balance -= amount; // two requests both read the old balance
Save(acct); // the second Save overwrites the first
Under concurrent requests, one debit silently disappears: both read the same starting balance, and the later write wins. Money vanishes with no error.
var rows = conn.Execute(@"UPDATE Accounts SET Balance = Balance - @amt, Version = Version + 1
WHERE Id=@id AND Balance >= @amt AND Version=@expected", p);
if (rows == 0) throw new ConcurrencyConflict();
The update is a single atomic statement, guarded by the expected version and a sufficient-funds check. A concurrent change is detected, not silently lost.
Self-review checklist
- AskWhat mutable state does this share between concurrent requests — and could I avoid sharing it at all?
- AskIf two requests run this at the same instant, is the result still correct?
- AskOn a read-modify-write, is a concurrent update detected, or silently overwritten?
- AskAm I assuming one thread or one instance anywhere I should not be?