The News Archiver

The probe of 2026-08-28 corrected a wrong belief. The news archive was thought to reach 2019; it reaches 365 rolling days, and the far end evaporates as time passes. An item reachable this morning is gone twelve months after it was published, whether or not we ever fetched it. Every other realistic vendor at the same $20–60 a month reaches further — Tiingo to 2014, Benzinga to 2010 — but price was never the decision variable.

The consequence changes what is buildable. Three of the four layers in the news pipeline — the lexicon reader, the event classifier, and the market's own reaction — cost nothing and exist precisely because they can be replayed over history. On a one-year window there is no history to replay them over: twelve months of one regime, shrinking from the back.

The way out costs nothing. The window is a retrieval limit on the vendor's side, not a retention limit on ours. Pull the feed daily, keep our own copy, and seed it with the year that is still reachable. Do that and the archive problem solves itself over time instead of getting worse — but only if it starts before the belt carries off another month. That is why it went first, ahead of the backtester the design says to build first: it was the single most time-sensitive item in the whole plan.

Two limits, measured by binary search on the from parameter, and they are different from each other. Paging without a date range bottoms out at about 90 days. A from/to window reaches exactly 365 days and not one more.

Two rules fell out of that and both are in the code. Always pass from/to: the parameter is the difference between a quarter and a year of reach. And a from older than 364 days is a hard 402, not an empty result. A backfill loop that walks backwards until it finds nothing will error instead of stopping, so the service carries an explicit floor — FloorFor(today) = today − 364 — and a day below it is recorded as BeyondWindow without spending a call.

Each feed-day is one blob at news/raw/{yyyy-MM-dd}/{feed}.jsonl.gz: one line per item, the full payload as FMP returned it, gzipped. The service pages through a day at 250 items a page and stops on the first short page. Items are sorted into a canonical order — published date, then URL, then title — before serialization, so the bytes do not depend on FMP's paging order; the serializer is fixed to no indentation and the default encoder; and .NET's GZipStream writes a header without a timestamp. Identical input yields identical bytes, and that is not an aesthetic preference. It is how a re-run proves itself.

Every run writes a manifest: which days, how many items, how many requests, and every outcome. Gaps are recorded, never tolerated silently, and a weekday that came back empty is flagged as suspicious rather than accepted — an empty weekday is a question, not an answer.

Every blob Plutus writes goes through one thin interface, IArtifactStore, and a put has four possible outcomes:

Created      the path did not exist; it does now
Unchanged    the path existed with byte-identical content — nothing written. The idempotency proof.
Overwritten  the path existed with different content and Overwrite was set — a deliberate, logged repair
Conflict     the path existed with different content and Overwrite was not set — nothing written, never a silent replace

The default refuses to overwrite, and the refusal is not a client-side exists-check that could race. The Azure implementation sends the request with IfNoneMatch = *, which means "only create, never replace" at the service; immutability is enforced by the storage account itself. A second put of the same bytes compares hashes and reports Unchanged; a second put of different bytes is a Conflict the manifest marks as incomplete.

The console takes --mode skip | verify | overwrite, and the three map onto the outcomes above.

• skip, the default: a day that already exists costs no calls. This is what makes the backfill resumable — kill it, restart it, and it picks up where it stopped.
• verify: re-fetch and compare, never overwrite. A day that comes back Unchanged proves the pipeline is deterministic end to end, vendor included; four random days did on 2026-09-01. A day that comes back Conflict means the vendor's answer changed, and that is worth knowing.
• overwrite: a deliberate repair, logged as Overwritten. Normal code never sets it.

The build surfaced one plan fact the original probe had not reached. grades-latest-news accepts limit ≤ 100 and page = 0 only, and ignores from/to entirely. It is a latest-100 feed and cannot be backfilled at any price on this plan. So it is archived as a snapshot per pull, named by the instant it was taken, and downstream dedups by symbol, published date and URL across snapshots. One snapshot did not cover a full day of grade changes, which argues for taking one every hourly window rather than once a morning.

Since 2026-09-01 the window has stopped costing history. The daily pull is a service the daemon will call, not daemon code, so Phase 4 consumes it without rewriting it.

• stock — news/stock-latest, dated pages, one blob per day
• general — news/general-latest, the same
• grades — grades-latest-news, a snapshot per pull
• manifests — one per run, every outcome recorded

• No MySQL — blob only, so no migrations
• No orders, no broker
• Licensing is the one open risk: archive now, read the terms in parallel, derive-and-discard bodies later if required. Discarding later is possible; recovering days never fetched is not.