What this is. We're a two-person operation — a human builder in Newton, Massachusetts, and his AI partner — in the middle of a project: taking a brand-new open model and assimilating it into the working stack we produce books and client work with. These are our project notes, published as we go.
Plainly, about our context: we are not benchmarkers, and we did not walk in with a map. We picked a real job from our own queue, pointed the model at it, and went in basically blind, expecting to learn a few things along the way. We did — mostly from what didn't work, and then from the one thing that did. Everything below is exactly what happened, with the numbers attached.
Recursive released Flint this week — an open-weight coding model, Apache 2.0, free to try on OpenRouter. The model card gives you benchmarks and a one-line warning. What it doesn't give you is what happens when you hand it a real job. We did. Here are the receipts.
Our test job was real work, not a benchmark: a continuity audit of a 32,000-word book manuscript — check the cast descriptions against the series' locked design sheet, check terminology against a house rule, quote every violation exactly. A 262K-context model should eat this. Cost of the whole experiment: $0. Answers delivered under our harness settings: zero. Under the vendor's own sampling defaults: a completed audit.
The runs
All runs on the free OpenRouter endpoint. Runs 1–5 carried our harness setting, temperature 0.2 — a routine choice for audit work, where you want the machine deterministic.
- Whole book, standard budget — 48.5K tokens in, 12,000 out. Empty: the budget was spent thinking.
- Whole book, bigger budget — 48.5K in, 32,000 out. Empty: 29,537 reasoning tokens, no answer.
- One mission only — 5.5K in, 16,000 out. Empty: 18,164 reasoning tokens, no answer.
- One mission, reasoning dialed to “low” — 5.5K in, 16,000 out. Empty: 18,153 reasoning tokens. Effectively unchanged.
- One mission, huge budget — 5.5K in, 64,000 out. Empty: 56,059 reasoning tokens across nine minutes, and still no answer. The deliberation did not terminate inside a budget twelve times the size of the task itself.
- One mission, vendor's defaults — same chapter, 64,000 budget, but temperature omitted, so the server applied the model card's recommended sampling settings. Delivered in 3 minutes 35 seconds: the mission checked clean against the lock, every check itemized. The audit completed.
Three things the model card doesn't tell you
- The recommended sampling settings are load-bearing. The card publishes a sampling table and applies those settings when a request omits them. What it doesn't say is what happens when you supply your own: our temperature 0.2 — as ordinary a harness choice as exists — starved delivery at every budget we tried, up to a deliberation past 56,000 tokens that never terminated. Remove that one setting, and the identical task completes in under four minutes. One variable changed; one observation, not a law. But nobody prints “your routine settings may silently break delivery” on the card, and the card's own warning — long reasoning can exhaust the output budget — doesn't hint at it either.
- The reasoning dial is decorative on the free endpoint. The API accepts
effort: low. The burn was effectively unchanged — 18,153 reasoning tokens against 18,164 without the dial. A control that accepts your input and ignores it is worse than no control — it teaches you a wrong model of the machine. - Nobody will tell you whose hardware you're on. OpenRouter's own public listing for this model returns an empty provider list. Your text goes somewhere. The listing declines to say where.
What this is not
This is not a verdict on Flint's weights — in the end, the weights did the job. Run-to-run variance isn't excluded by a single flipped variable; the record is five non-deliveries under our settings and one delivery under the vendor's, and we report exactly that. The same weights on a paid endpoint, or running locally under a different runtime, may behave differently again. We claim only what we measured, and we measured the free endpoint, because that's the door every curious person will walk through first. We're learning this machine the way we'd learn any new tool in the shop: by using it on real work and writing down what happens.
The point
Model cards are vendor copy — benchmarks in the frame the vendor chose, warnings without magnitudes, and defaults whose importance nobody states. The missing layer in open AI is operational receipts — what actually happens, in numbers, when someone runs the thing on a real job. Vendors rarely publish them; the failures aren't in their interest. Users mostly publish complaints, which carry no numbers. So the receipts have to come from the people doing the work.
We'll keep publishing ours. Field Notes #2: the same audit, the same weights, running on our own hardware — where the dials, and the defaults, are ours.
— Frank Kurka & Muse, Newton, Massachusetts