Autonomous bot fleet and backtest engine: reliability engineering

Bash cannot detach a daemon on macOS — tested, not assumed

Restarting a process from a script looks trivial right up to the first hang. Five backgrounding patterns were tested empirically, and all five left the parent shell stuck in __wait4 on the supposedly disowned child — one process hung that way for 1 h 32 min:

• nohup … & disown — the classic that doesn't work
• set -m + disown, subshell launch, nested fork variants — same result
• macOS ships no setsid(1), so there is no "correct" shell answer at all

The fix is one shared Python launcher: fork → setsid → fork → execvp, with the grandchild guaranteed to end up at PPID=1. Every restart in the project goes through it; launchd jobs additionally set AbandonProcessGroup, or the OS reaps the children along with the script that spawned them.

Liveness is measured per event type, not per process

The baseline watchdog tracks the timestamp of the last event of any type and kills the process if silence exceeds 360 seconds — a threshold found empirically, since 180 s produced false positives on normal cycle gaps. The supervisor brings the service back, so the watchdog can afford to simply exit with a distinct return code.

But that only cures a fully dead connection. The real incident was subtler: one event type stopped arriving while everything else flowed as usual — and the generic silence counter never fired. Hence the rule now applied to every service: liveness is not "is the process up", it's "when did I last receive the thing I exist for". Every critical event type gets its own counter, the health report shows "minutes since last" per stream, and there's a preventive re-subscribe on a timer.

A liveness check has to be honest

pgrep -f patterns must be anchored to the end of argv (…/book_recorder\.py$). Without the anchor the check matches any neighbouring shell that happens to carry the string in its command line — including the calling script itself. Monitoring that counts itself as the live process is worse than no monitoring: it reports green on a dead system.

SQL-first, with provenance on every row

The rule for all new writers: data goes to SQL, and JSONL is allowed only as a rolling buffer with ≤48 h retention and daily rotation. Every dataset gets a row in data_provenance (source, data status, audit notes, whether it's safe to delete); every record carries ingested_at and a link back to its source dataset. The trigger was mundane: an audit found tens of gigabytes of duplicates that nobody could delete, because nobody could prove where they came from.

• live writers open the connection on flush and close it immediately — the lock is held for milliseconds instead of the writer's entire lifetime
• a continuously written database is single-writer; analyst connections are read_only=True only
• a new live writer gets its own database file rather than sharing one with an existing writer

Decision records are never deleted

The postmortem that produced a rule of its own: roughly five weeks of decision history vanished under disk pressure — routine archive rotation removed files that cannot be reconstructed. The answer is append-only storage: harvest every entry and close record from live runs and from archives, dedup by line hash (idempotent, so it can run every 15 minutes), sync to an offsite copy. The key rule is baked into the pruning code: harvesting runs before any deletion, and if it fails, the deletion is cancelled entirely. Disk is cheaper than data, always.

Event-driven reaction instead of timer polling

A 30-second poll is a perfectly good design when events are rare. Here the decision windows live for seconds, so reaction is driven by the incoming WS event and REST stays a catch-up scan on a 5 s interval. The full path is measured component by component, so latency arguments can be settled with numbers rather than opinions:

• connect and subscribe — ~40 ms; detail request — ~26 ms; external state read — ~35 ms; submit — ~41 ms
• cryptographic signing — 0.24 ms after swapping the pure-Python backend for a native one; before that it was three orders of magnitude more expensive and ate the entire budget
• background caches and parallel requests instead of sequential waits — tens of milliseconds more
• fast path end to end — 250–350 ms; you can't go lower without an architecture change, the rest is network-bound

An invariants log instead of oral tradition

Every rule that gets discovered is written down as a numbered invariant: the date it was proven, the incident that produced it, a "how to apply" section, and — as a separate item — the conditions under which the rule stops being true. Rules don't come from opinion: they come from incidents and carry a way to reproduce them. That's what stops the same mistake from being rediscovered six months later on a new service.

A backtest that doesn't flatter you

Validating a fix on a live run means waiting days and still having no statistics. Instead, each fix is replayed over historical ledgers using excursion replay: every closed record stores the real extremes of the move, so the counterfactual is computed from how things actually moved rather than from a re-simulation. The method is fixed as a set of non-negotiable rules:

• assumption-free metrics first — distributions and hit rates at each level; if there's no movement at all, no exit tuning will help
• dedup replicas: the same signal is duplicated across configurations roughly 12 times — without dedup the sample is fake-inflated
• segment by market regime and by signal type; a blended number hides a system that wins in one regime and loses in another
• bounds for ambiguous cases: when the intra-bar ordering is unknown, compute the optimistic and pessimistic estimate and accept the conclusion only if both agree
• random-entry control: the same exit logic on random entries must lose — otherwise the "effect" came from the exit mechanics, not the signal
• n ≥ 30 per cell, out-of-sample and walk-forward folds, and a multiple-testing correction

The practical effect is unpleasant and useful at the same time: most ideas that looked like they worked don't survive this method. Of eight mechanical strategies tested, two survived. That is the result — knowing the other six don't work before paying for them.

The risk loop halts before a human would

A multi-layer monitor runs on a schedule and checks several independent halt conditions. A halt is expressed as a marker file that every process can see — a simple, dependency-free kill-switch. After that it's discipline: halt markers are never hand-edited, and resuming goes only through the dedicated script that checks why the halt happened. New filters get their own rule: audit mode first, where the filter only logs its decision, and only after enough data does it earn the right to block.