Albacel Reads Issue #7

Reads ·

Issue #7

Covers November 19 – November 20, 2026

Can you trust the check? Evals that overstate progress, judges that flatter stronger models, and the agent systems and business numbers built on top of them.

This issue is about trusting the check. METR warns that its own easy-to-run evals may overstate what models can do, and finds model judges grading stronger models too kindly, and Spotify works out exactly when an offline eval is good enough to ship on (rarely, and only with an audit pool). OpenAI shows what it looks like when the checks are built into the system: its Performance Factory has a second agent try to disprove every diagnosis, and the Codex sandbox puts hard limits under the model’s judgment. On the business side, Anthropic’s S-1 and KeyBanc’s survey show the same thing from opposite ends: usage-priced AI can grow with the work, while most mid-market vendors are still over budget on tokens. If you read two, start with METR and Spotify.

The posts

Hugging Face

Introducing harness adapters: adapt your favorite model to your harness to improve performance and cost

Key number 39% → 55% Terminal-Bench 2.1 score in the pi harness, with tokens per task nearly halved

Hugging Face trained a small LoRA adapter for Qwen3.8-27B inside each of three coding-agent harnesses (pi, OpenCode and Hermes Agent), using reinforcement learning on the harness exactly as it ships. Under pi, the model’s Terminal-Bench 2.1 score rose from 39% to 55% while average tokens per task fell from 2.9K to 1.5K, and on the hardest SWE-bench Pro tasks it went from 39.2% to 64.7%. On tasks both versions solved, the adapters used 53–72% fewer tool calls. Much of the post is the data pipeline: of 1,127 candidate tasks, 858 passed validation, which also caught a Terminal-Bench verifier that accepted wrong answers, and training kept only tasks the base model solved some of the time. A merged adapter works under all three harnesses, with smaller gains. On business-automation tasks in a different harness, the adapters were no better than the base model.

Choosing modelsBuilding agentsInference stack Harness and multi-agent design What agents cost

Read at Hugging Face (opens in a new tab)

METR

AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure

Key number 3.6 months time-horizon doubling across METR's old and new task suites

METR’s quarterly update tests frontier models up to Opus 5.5 and GPT-6 Astra on its older time-horizon suite and a new, harder set of 36 tasks. Across both, the length of task a model can finish half the time keeps doubling about every 3.6 months, and models reach up to 27 hours on the new tasks that can be scored automatically. They typically beat the skilled humans METR recruited and place around the 96th–98th percentile in long optimization contests, but still trail the very best people. They do worse on tasks that are costly to build and score, which METR suspects look less like the environments models are trained in. On judgment tasks, model judges scored work by equally strong or stronger models about 0.2 points (out of 1) too high. Swarms of agents beat best-of-k only when there was a clear metric and fast feedback, and hurt when agents had to invent their own measure of progress.

AI ecosystemChoosing modelsBuilding agents Can you trust the eval?

Read at METR (opens in a new tab)

Spotify Engineering

When Can You Ship on Offline Evals Alone?

Mårten Schultzberg

Key number 24% vs 60% how much of the online result the eval seems to explain, naively vs truly

Spotify asks when an offline eval can replace an A/B test while keeping the risk of shipping something harmful low. In its worked example, noise in both measurements makes the eval look as if it explains 24% of the variation in online results when the true figure is 60%. The proposed decision gate ships changes with strongly positive eval results, kills those with strongly negative ones and sends the rest to an A/B test, with thresholds that depend on how well the eval predicts online results. The assumptions are strong: past experiments must all be of one kind, or pooling them can show a relationship that doesn’t exist. An LLM judge whose bias depends on which version it is grading, as when comparing two underlying models, can flip the result. Once the gate is live, a random sample of its decisions still needs A/B tests to catch drift. Spotify still pairs its evals with online experiments.

Building agentsAdvising Can you trust the eval?

Read at Spotify Engineering (opens in a new tab)

SaaStr

The Top 10 Metrics From Anthropic’s IPO for B2B Execs: $33 Billion in Nine Months, 900% NRR, and 12x More $100K Customers in a Year

Jason Lemkin

Key number ~900% Anthropic's net revenue retention, against 120–130% for seat-priced products

Lemkin pulls ten operating metrics from Anthropic’s IPO filing for founders and B2B executives. Revenue was $33.24 billion for the nine months to September, and net revenue retention about 900%, which he attributes mainly to usage pricing: consumption was 79% of revenue. Customers spending $100K or more a year grew from more than 950 to more than 11,900. Gross margin, derived from footnotes and excluding model development, rose from 29.4% to about 85.5%, but compute as a share of revenue went from 48% in Q2 to 58% in Q3. Nearly half of first-half revenue came through cloud marketplaces, and sales and marketing was about 23% of revenue. The filing also shows the cost of usage pricing: spending fell about 20% over the 2025 holidays.

AI businessMarkets & safe use

Read at SaaStr (opens in a new tab)

SaaStr

KeyBanc’s 2026 AI x Software Survey: 24% Median ARR Growth, Only 6% at the Rule of 40, and 43% of Vendors Watching Customers Vibe Code Their Product

Jason Lemkin

Key number 30 of 68 software vendors over budget on AI spend, mostly from token volume

Lemkin summarizes KeyBanc’s private software survey, which this year added a large section on AI. It covers 81 companies with a median ARR of $22M, and some findings rest on only 4 to 15 answers. 43% of vendors say customers are already using AI coding tools to rebuild their product or features. Only 41% charge for AI, at a median premium of 20%. 30 of 68 companies are over budget on AI spend; 25 of the 33 that explained why cited higher token usage, while unit costs were flat for most. Companies spending more than 5% of revenue on AI ran 66% gross margins against 78% for light spenders. 65% have cut no headcount because of AI, 32% have no formal way to measure AI productivity, and 43% of vendors require human approval for their agents’ actions.

AdvisingAI businessMarkets & safe use What agents cost Work and adoption

Read at SaaStr (opens in a new tab)

OpenAI Engineering

How OpenAI’s Performance Factory keeps ChatGPT fast

Okay Zed, Cyrus Forbes

Key number 70% of 100 agent root-cause diagnoses graded accurate on manual review

OpenAI describes the Performance Factory, which uses Codex agents to investigate and fix performance regressions in ChatGPT. It has four parts: a task tracker whose workflows are defined in configuration and Markdown prompts, a dispatch loop that hands ready tasks to agents, an API that runs Codex in the cloud under a scoped service identity, and subscriptions that wake a task when a review comment or deployment arrives. In a little over three months the first workflow ran more than 1,700 investigations and found the root cause of more than 600. Early agents stopped at the first plausible cause, so a separate Codex session now tries to disprove each diagnosis and, later, each proposed fix; in a manual review of 100 diagnoses, 70% were accurate. Engineers still review and merge every fix, and OpenAI says writing good fixes has been harder than finding causes. Changes to the agents are tested against past incidents, with the answers hidden.

Building agentsAdvising Harness and multi-agent design Can you trust the eval?

Read at OpenAI Engineering (opens in a new tab)

OpenAI Engineering

Inside the Codex sandbox

Michael Bolin

OpenAI explains how Codex limits the commands it runs on a user’s own machine. Instead of a container or VM, it uses each operating system’s own sandboxing: Seatbelt on macOS, bubblewrap with seccomp filters on Linux, and Microsoft’s new MXC library on Windows, all behind one API. Permission profiles set which paths are writable, read-only or denied, and the default lets the agent write inside the workspace but not in .git or its own config, read broadly, and use no network. Network access, when allowed, goes through a proxy that checks each domain before resolving it, because a DNS lookup can itself leak data. Enterprises can layer profiles and lock which ones users may choose, and the codex sandbox command can wrap other tools, such as an MCP server that Codex otherwise runs unsandboxed. The post is clear about the limits: an agent can still plant code in an ignored dependency that runs later outside the sandbox.

Building agentsMarkets & safe use Agents acting in the world

Read at OpenAI Engineering (opens in a new tab)

Hugging Face

Hugging Face and OpenRouter join forces to make open-weight models easy to run

Key number 26.8% → 71.7% open-weight share of OpenRouter tokens, October 2025 to September 2026

Hugging Face is replacing its own router for language-model inference with OpenRouter’s, which picks a provider for each request on live price, speed and reliability and retries on another provider if the first fails. The API stays the same, supports the OpenAI and Anthropic request formats, and keeps a conversation on the provider that holds its prompt cache. OpenRouter says the new routing cut capacity errors by 46% and improved time to first token by 16.9%. Hugging Face passes provider prices through without markup, but users can no longer bring their own provider keys for these models. The post also shares market figures: open-weight models went from 26.8% to 71.7% of OpenRouter’s tokens in a year and cost about a quarter as much on average. The best open-weight model scores 46 against 65 for Claude Fable 5.2 on one intelligence index, at about a seventh of the cost per task.

Inference stackChoosing models What agents cost

Read at Hugging Face (opens in a new tab)