Reads ·
Issue #7
Covers November 19 – November 20, 2026
Can you trust the check? Evals that overstate progress, judges that flatter stronger models, and the agent systems and business numbers built on top of them.
This issue is about trusting the check. METR warns that its own easy-to-run evals may overstate what models can do, and finds model judges grading stronger models too kindly, and Spotify works out exactly when an offline eval is good enough to ship on (rarely, and only with an audit pool). OpenAI shows what it looks like when the checks are built into the system: its Performance Factory has a second agent try to disprove every diagnosis, and the Codex sandbox puts hard limits under the model’s judgment. On the business side, Anthropic’s S-1 and KeyBanc’s survey show the same thing from opposite ends: usage-priced AI can grow with the work, while most mid-market vendors are still over budget on tokens. If you read two, start with METR and Spotify.
In this issue 8 posts
- Introducing harness adapters: adapt your favorite model to your harness to improve performance and cost Hugging Face
- AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure METR
- When Can You Ship on Offline Evals Alone? Spotify Engineering
- The Top 10 Metrics From Anthropic’s IPO for B2B Execs: $33 Billion in Nine Months, 900% NRR, and 12x More $100K Customers in a Year SaaStr
- KeyBanc’s 2026 AI x Software Survey: 24% Median ARR Growth, Only 6% at the Rule of 40, and 43% of Vendors Watching Customers Vibe Code Their Product SaaStr
- How OpenAI’s Performance Factory keeps ChatGPT fast OpenAI Engineering
- Inside the Codex sandbox OpenAI Engineering
- Hugging Face and OpenRouter join forces to make open-weight models easy to run Hugging Face