Incidents & Operations
The Failures Worth Writing Down
Section 4 of 5
These systems run unattended against live markets, which means their operational history is not a list of green test runs — it is a list of things that went wrong in production, were diagnosed from logs and exchange state, and were fixed in a way that made the class of failure harder to repeat.
Each incident below is written up in the repository itself, with the root cause separated from the symptom and the fix separated from the mitigation. The most useful of them changed how the system treats uncertainty, which is a different and more durable outcome than fixing a bug.
~24h
a real filled position went untracked
3
outcomes the fill check now distinguishes
298MB
a write-ahead log reached before being capped
2
ledger episodes reconciled from exchange fills
Incident: A Real Position Nobody Knew About
A buy order filled on the exchange. The very next call — the one that confirms the fill — hit a transient broker glitch and raised. The exception reached a generic handler that logged it and moved to the next tick, and a real position sat untracked for about a day until it was spotted by eye in the broker app.
The retry absorbs the transient glitch and on its own would probably have prevented this specific incident. It is not the interesting part.
The interesting part is that the fill check now returns three distinguishable outcomes rather than two, and the third one is explicitly uncertain. A system that only models filled and not-filled must map every ambiguous case onto one of them, and both choices are wrong in a way that costs money: treating an unknown as not-filled invites a retry that doubles a position that already exists, and treating it as filled creates a phantom the ledger will never reconcile.
So the unknown case raises an error type that exists for no other purpose, with the contract written at the point of definition: callers must never treat it as safe to retry, and never as safe to treat as closed. Giving uncertainty its own name is what stops it being silently rounded to whichever neighbour is more convenient.
The reconciliation itself carries a subtlety worth keeping. A buy is confirmed against the position's own average price, but a sell cannot be — once a position is reduced or closed, its average price describes the remaining entry cost, not the exit that just happened. The exit uses the order's own price instead. Getting that backwards would produce a ledger that reconciles in quantity and lies about profit.