Incidents & Operations

The Failures Worth Writing Down

Section 4 of 5

These systems run unattended against live markets, which means their operational history is not a list of green test runs — it is a list of things that went wrong in production, were diagnosed from logs and exchange state, and were fixed in a way that made the class of failure harder to repeat.

Each incident below is written up in the repository itself, with the root cause separated from the symptom and the fix separated from the mitigation. The most useful of them changed how the system treats uncertainty, which is a different and more durable outcome than fixing a bug.

~24h

a real filled position went untracked

3

outcomes the fill check now distinguishes

298MB

a write-ahead log reached before being capped

2

ledger episodes reconciled from exchange fills

Incident: A Real Position Nobody Knew About

A buy order filled on the exchange. The very next call — the one that confirms the fill — hit a transient broker glitch and raised. The exception reached a generic handler that logged it and moved to the next tick, and a real position sat untracked for about a day until it was spotted by eye in the broker app.

Real order placed — confirmation call failstransient broker lag, order history momentarily emptyRetry the status check — three attempts, two seconds apartabsorbs exactly the lag that caused the incidentStill unknown — reconcile against ground truthreal position quantity versus a baseline captured before the orderQuantity moved by the full sizetreat as filled, using the correct price for the directionQuantity unchangeddid not fill — actively cancel it, never leave it resting unseenMatches neither, or the reconciliation itself failedraise a distinctly named unknown-status errornever “safe to retry” — could double a position that already filledThe valuable outcome was not the retry. It was making “I do not know” a state the system can represent.

The retry absorbs the transient glitch and on its own would probably have prevented this specific incident. It is not the interesting part.

The interesting part is that the fill check now returns three distinguishable outcomes rather than two, and the third one is explicitly uncertain. A system that only models filled and not-filled must map every ambiguous case onto one of them, and both choices are wrong in a way that costs money: treating an unknown as not-filled invites a retry that doubles a position that already exists, and treating it as filled creates a phantom the ledger will never reconcile.

So the unknown case raises an error type that exists for no other purpose, with the contract written at the point of definition: callers must never treat it as safe to retry, and never as safe to treat as closed. Giving uncertainty its own name is what stops it being silently rounded to whichever neighbour is more convenient.

The reconciliation itself carries a subtlety worth keeping. A buy is confirmed against the position's own average price, but a sell cannot be — once a position is reduced or closed, its average price describes the remaining entry cost, not the exit that just happened. The exit uses the order's own price instead. Getting that backwards would produce a ledger that reconciles in quantity and lies about profit.