The Engine's Day
One long-lived process wakes at 06:45 ET, verifies its book before it decides anything, computes the world once, and then spends the rest of the day reacting to it — with a ledger that lets it be killed at any stage and resume there.
The morning cycle
The daemon is a BackgroundService that never shuts down; the IB Gateway session is why. A timer fires the morning cycle, and the phases run in a fixed order. Stage names are the same strings written to DaemonRunProgressSteps.Stage, which matters later.
| Phase | Stage | When (ET) | What it does |
|---|---|---|---|
| 0 | reconcile | cycle start | Re-read configuration from the database, then check every Pending/Submitted order against the broker. An intent-versus-fill mismatch halts that account; the fleet proceeds. The system never decides on a book it has not verified. |
| 1 | pull | 06:30 → 06:45 | Bounded async fan-out over every registered source — GMI instruments, FMP bars and quotes, macro series, news. Freshness is recorded per source. Stale inputs mean do nothing, never guess; if a major print is due or data is late, the cycle records the hold and stops. |
| 2 | probabilities | ~06:50 | Assemble the day's MarketState once: source values, per-symbol technicals, the resolver stances, the probability vector. Freeze it, snapshot it to blob. This is the last database read most of the cycle will do. |
| 3 | decide | 07:00 | Bulk-read account state, then run accounts in parallel. Per account: each strategy's decide(), the hold-review pipeline, the entry gauntlet, allocation, caps and sizing. Pure compute, no database access inside the task. An exception halts that account and the fleet continues. |
| 4 | execute | 09:28–09:30 | Paper accounts fill through the simulator in parallel. Live orders are written to TradeOrders first, then queued to a single-consumer channel that owns the broker connection. |
| 6 | idle | ~16:15 | Mark to close, write AccountEquityDays against same-start SPY, close-outs into ClosedTrades, the daily digest, and the idle flag the deploy pipeline gates on. |
Morning or Hourly: Decision 4d
The first draft said hourly windows "react, they do not re-plan." That was close but not precise enough, and on 2026-09-02 Allan replaced it with a rule per resolver rather than per window. Every resolver and estimator carries a RecomputeCadenceTypes value: Morning or Hourly.
Technical, price-driven, news and stress-gauge resolvers are Hourly; they rerun in every window against fresh bars and news. Macro, fundamental and regime resolvers are Morning; they are computed once at 06:45 and carried forward, so the regime cannot flip at 11:00 on a partial data print. The assembler rebuilds MarketState each window from the Hourly set plus the carried Morning set, and MarketState.Provenance stamps which is which.
The precise wording now: hourly windows re-plan what moves intraday and nothing else. Call budgets scale with the Hourly set alone.
The hourly windows
From 10:00 to 15:00 on the hour, plus a close window, the engine runs a reduced cycle: tactical sources for held and watchlist names only, not the full 520-symbol universe; the Hourly resolvers and the stress gauge; the exit-evaluation pipeline over open positions; the entry gauntlet for tactical candidates reacting to news. The full fan-out and the allocation rebalance wait for the next morning.
The main prize is risk responsiveness: a drawdown stop or a stress escalation at 11:15 is acted on within the hour instead of at a 14:00 checkpoint. What hourly cadence does not buy is day trading; holding horizons start at daily.
A sweep that finds nothing writes nothing. Without that rule, 40 accounts times roughly 25 positions times 7 windows would add about 7,000 rows a day recording that nothing happened.
Killed at stage four, restart at stage four
DaemonRunProgressSteps is the resume ledger. One row per stage as it completes, keyed on (CycleDate, Window, Stage, FkTradingAccountsKey). That unique key is also the idempotency key: each account checkpoints on its own as it finishes a stage, so a restart re-reads the ledger and skips finished work, including work finished by a parallel task that outlived its siblings.
A stage is checkpointed after its writes commit. Re-running decide for an account that already checkpointed is a no-op, not a duplicate order. Nothing that matters exists only in memory.
Catch-up: collapse, never replay
On every startup the daemon reconciles first, then computes the windows the market calendar says should have run since its last completed one and diffs that against the ledger. The missed set is collapsed: all missed intraday windows become one catch-up run at current prices; a missed morning cycle whose deadline has not passed runs now; a missed morning cycle past its deadline is a failure day; missed computations (end-of-day equity, outcome resolution) are backfilled in order.
Why collapse rather than replay: each window recomputes from current market state, so running seven missed windows in sequence would generate seven rounds of orders against one set of prices — turnover manufactured out of downtime. One run reaches the same target book with one round of orders.
Every stimulus class carries a staleness budget. Past it, a would-be entry is journaled as StandAside with the reason STALE_INTENT rather than executed late. Risk exits have no budget: a stop that should have fired at 11:00 still fires at 15:00, because the position is still wrong. Catch-up decisions carry IntendedWindowUtc beside DecidedUtc, so attribution never credits a late decision with on-time execution.
The deploy handshake
Deployments happen through the trading day. The pipeline writes a DrainForDeploy command to DaemonCommands before it touches the service. The daemon sees it at its next poll point — between phases, and before each account in the fan-out — and finishes the smallest safe unit: the current account's decide, or the current order through broker acknowledgement. It never stops between submitting a live order and recording the ack. Then it checkpoints, writes an OpsEvents row, sets AcknowledgedUtc on the command, and exits 0 so systemd records a clean stop.
The ack is the pipeline's green light. The pipeline polls for it with a 120-second timeout and proceeds regardless when it expires: a deploy must never hang on a wedged daemon, because catch-up makes a hard kill recoverable. SIGTERM is the fallback path, same drain logic inside systemd's 90-second stop window.
A deploy never writes engine state. An account Started before the deploy is Started after it and gets caught up; an account a human stopped stays stopped. Policy still avoids the open-execution window, 09:25–09:45; the protocol survives it anyway.
- 06:30 — data snapshot the pull works from
- 06:45 — the daemon wakes; reconcile, then pull
- ~06:50 — the world is computed once and frozen
- 07:00 — decide, in parallel across accounts
- 09:28–09:30 — execute
- 10:00–15:00 — hourly windows, plus a close window
- ~16:15 — end of day, then
idle
- Reconcile before deciding — never act on an unverified book
- Compute the world once; everything downstream reads the frozen copy
- Never stop between submitting a live order and recording its ack