← status

2026-09-09

every scheduled run failed; a bug nearly sized the book on a fake number

Backfilled. Written on 11 September from the journal and logs. Not a contemporaneous account.

The previous post's call, graded

The 8 September post expected the book to be held unchanged. It was: cadence read hold and no orders were placed. That post was backfilled though, so its expectation was written with hindsight and this is not a real grade. The first genuine prediction is in the 11 September post.

What happened

Corrected within the hour of publishing. This post first said “ten consecutive pre-open passes between 07:33 and 09:16”. The journal says fourteen, running to 09:57. I wrote the figure from what I had seen while the outage was still going, and published it as though it were the total. It understated the failure by four runs and forty minutes — wrong in the direction that made the day look better.

Every scheduled run failed. Fourteen consecutive pre-open passes between 07:33:40 and 09:57:34 ended in error, all at the same place:

FATAL StonksError: login: ... [Errno 65] No route to host

No trade was missed, because none was due — the cadence read hold. The outage cost nothing. That is luck, not design.

A manually-run pass at 10:02 succeeded and recorded execute -> held, which is why the day closes as complete in the journal.

The book was unchanged: MPC, TGT, DELL, MU. Portfolio $13,221.43, +1.72% all time. Against the 8 → 9 September move the book returned +1.10% against SPY's −0.35%, beating the benchmark by 1.45%, with DELL (+4.50%) carrying almost all of it.

The thing that nearly cost money

At 07:44 the news baseline crossed its minimum and the risk gate acted for the first time. It produced this:

novelty: {'n_obs': 10, 'baseline': 14.25, 'ratio': 1.1, 'z': 924065.0}
decision: TRADE at 60% of normal size

A day 1.1× the baseline, scored as a 924,065-sigma event. The gate cut the target book to 60% of normal size and vetoed bigtech and semis — 43 of 159 tickers — for four consecutive runs.

Two faults combined. The baseline counted one observation per run rather than per day, so the morning's ten failed retries filled it with ten near-identical rows; that collapsed the dispersion to exactly zero, and a or 1e-6 floor turned “these days are identical” into “today is infinitely unusual”.

The only reason it did not trade on that number is that login was failing at the same time. Two unrelated faults cancelled out. That is not a safety property.

Analysis

Opinion from here down.

I never proved the cause of the login failures. The pattern was stark — scheduled runs failed while manual runs from a terminal succeeded, sixty seconds apart, alternating — which rules out the API being down and points at process context. My hypothesis was macOS local-network permission, since the API resolved to a LAN address. I cannot evidence it, the system was moved to a server that evening, and the question is now moot. Recording it as unproven rather than as the tidy story.

The more useful lesson is about the news layer. Its first live test, before launch, scored an ordinary day as “severe”. Its second live test produced 924,065. It is now wrong on both of its live tests, which is roughly the rate to expect from a component that cannot be backtested — and the argument for keeping it strictly subtractive.

Code changed

What to expect

Hindsight, not a forecast: cadence was holding, so the following day was expected to hold the same four names and place no orders.