Autonomous bots and backtesting: how to make them reliable

Why background jobs on macOS start through Python

Restarting a process from a script looks simple until the first hang. I tested five ways of starting a process in the background. In all five the parent shell stayed stuck in __wait4 on the child it had supposedly let go. One process hung like that for 1 h 32 min:

• nohup … & disown: the classic way, which didn't work here
• set -m with disown, a subshell launch, nested forks: the same result
• macOS has no setsid(1) command, so one standard shell command can't solve this

The fix is one shared Python launcher: fork → setsid → fork → execvp. The grandchild process always ends up with PPID=1 and runs independently of the shell. Every restart in the project goes through it. For macOS jobs (launchd) I also set AbandonProcessGroup; otherwise the system kills the children together with the script that started them.

Each event type has its own sign of life

The basic watchdog tracks the time of the last event of any type. It stops the service if nothing has arrived for more than 360 seconds. I found that threshold in practice: at 180 s it fired on normal pauses. The service manager brings the service back, so the watchdog only needs to exit with a special return code.

That catches only a fully dead connection. The real incident was subtler: one event type stopped arriving while everything else flowed as usual, and the general silence counter never fired. Hence the rule now used for every service: a service is alive if it recently received the thing it exists for. Each important event type has its own counter. The health report shows how many minutes ago the last event arrived in each stream, and the service re-subscribes to the stream on a timer as a precaution.

A "process is alive" check has to be honest

pgrep -f patterns must be anchored to the end of the command line (…/book_recorder\.py$). Without the anchor the check matches any neighbouring shell whose command happens to contain the string, including the calling script itself. Such monitoring counts itself as the live process and shows green on a dead system.

Data goes to a database, and every row shows where it came from

The rule for all new code: data goes into a database. JSONL files are allowed only as a temporary buffer, kept for no more than 48 hours, with a new file each day. Every dataset has a row in data_provenance: source, status, review notes and whether it's safe to delete. Every record carries ingested_at and a link to its source dataset. The reason was mundane: a review found tens of gigabytes of duplicates that nobody could delete, because nobody could prove where they came from.

• processes that write live data open the connection only to save and close it right away, so the lock lasts milliseconds
• a database that is written to all the time has one writer; analysis connects only with read_only=True
• each new process that writes live data gets its own database file

Decision records are never deleted

One incident review produced a rule of its own. About five weeks of decision history disappeared when the disk filled up: routine archive cleanup removed files that can't be rebuilt. The answer is storage that is only ever added to. Every entry and exit record is collected from live runs and from archives, duplicates are removed by line hash (so the job is safe to run every 15 minutes), and a copy goes to a separate location. The key rule is built into the cleanup code: collection runs before any deletion, and if collection fails, the deletion is cancelled.

The reaction starts from the event itself

Polling every 30 seconds is a perfectly good design when events are rare. Here a decision window lasts seconds, so the reaction starts from the incoming event, and a regular request every 5 s only picks up what was missed. The full path is measured step by step, so arguments about speed are settled with numbers:

• connect and subscribe: ~40 ms; details request: ~26 ms; reading external state: ~35 ms; sending: ~41 ms
• cryptographic signing: 0.24 ms after switching from a pure-Python library to a native one; before that it took about a thousand times longer and used up the whole time budget
• background caches and parallel requests in place of waiting in sequence saved tens of milliseconds more
• the whole fast path: 250–350 ms. Going lower would take a different architecture; the rest is network time

Every rule goes into a log of invariants

Every rule I discover is written down as a numbered invariant: the date it was proven, the incident that produced it, how to apply it and, as a separate item, when the rule stops being true. Each rule comes from an incident and includes a way to reproduce it. That keeps the same mistake from being rediscovered six months later on a new service.

An honest backtest

Checking a fix on live runs means waiting for days and still having too little data. So each fix is replayed over the historical records. Every closed record stores the real highs and lows of the price move, and the "what if" result is calculated from how prices actually moved, with no new simulation. The method has fixed rules:

• first, measures that need no assumptions: distributions and how often each level is reached. If the price doesn't move, no exit setting will help
• remove copies: the same signal appears in about 12 configurations, and without removing the copies the sample looks bigger than it is
• split results by market regime and by signal type: one overall number hides a system that wins in one regime and loses in another
• bounds for unclear cases: when the order of prices inside a candle is unknown, calculate the optimistic and the pessimistic result and accept a conclusion only if both agree
• a random-entry check: the same exit rules applied to random entries must lose money. If they win, the "effect" came from the exit rules themselves
• at least 30 cases per group, tests on data the idea wasn't built on, rolling forward tests, and a correction for testing many ideas at once

In practice this is both unpleasant and useful: most ideas that seemed to work don't pass. Of eight rule-based strategies tested, two passed. That is the result: knowing that the other six don't work before paying for them.

The risk check stops the system before a person would

A multi-level check runs on a schedule and looks at several independent stop conditions. A stop is a marker file that every process can see: a simple emergency switch with no dependencies. The rest is discipline. Stop markers are never edited by hand, and the system restarts only through a dedicated script that checks why it stopped. New filters follow their own rule: first a trial mode, where the filter only logs its decision, and only after enough data does it get the right to block.