Lead · 7 chapters
Staying current sustainably
A small curated source set rather than the firehose, and a personal evaluation suite as a stable benchmark across model releases
Without a stable evaluation habit, model updates turn into hype-driven migrations or a costly refusal to change.
Read these first
What you will know
Maintain a personal eval suite that survives model releases
Versioned cases, expected outcomes, and scoring rules expose regressions even when prompts, APIs, or model names change.
Answer "should we switch models?" in an afternoon
A fixed comparison slice covering quality, latency, cost, and failure cases keeps the decision bounded rather than anecdotal.
Keep a curated source list rather than following everything
A few trusted feeds with explicit inclusion criteria preserve useful signal while limiting attention and duplicate reporting.
Reject a hyped release when it fails your predefined eval thresholds
Precommitted gates reduce novelty bias and make a defensible no-change decision possible before enthusiasm distorts the sample.
Time-box a recurring review of your sources and record which updates change your backlog
A dated decision log separates actionable developments from interesting noise and reveals whether the information diet is paying off.
Chapters
- 1
Inclusion criteria, not interest, decide what stays on the list
The firehose is largely the same six items repeated with commentary attached. A source list worth maintaining stays small and admits things by a stated rule: primary sources, model cards and release notes ahead of threads about them, and people whose past claims you have actually checked. Writing the criterion down is what lets you drop a feed later without the feeling that something important is being missed.
- 2
An eval suite outlives prompts, APIs and model names, or it is not one
A suite welded to one prompt string and one provider's response format dies at the next rewrite. Store cases as inputs plus expected outcomes, version them, and score with something independent of the system under test, so the same case runs against a model that does not exist yet. That property is what turns evaluation from a project you did once into a standing asset you keep.
- 3
The afternoon answer needs its comparison slice to already exist
A model ships on Tuesday and the CEO asks on Thursday. What makes Friday possible is a fixed slice built long before: representative quality cases, latency measured at your prompt lengths, cost per task at your token profile, and the failure cases that have bitten you before. Assembled after the question arrives, the same work takes a fortnight and yields an anecdote rather than a comparison.
- 4
Thresholds set while excited are not really thresholds
A no that gets relitigated weekly costs more than a yes. Committing the gates in advance, the quality delta, the latency ceiling and the cost per task you would accept, turns the decision into arithmetic instead of a contest of enthusiasm. It also lets a rejection be stated in one sentence naming the number that was missed, which is the thing that actually stops the conversation from reopening.
- 5
When engineers disagree with your suite, the suite is the suspect
Three engineers who used a model for a week and call it obviously better are sampling something your cases do not contain: instruction following, tone, tool use in messy context, recovery after a bad turn. Treat the disagreement as a bug report filed against the suite, pull the sessions, turn them into cases, and rerun. One side is measuring the wrong thing, and it is not automatically the humans.
- 6
A year-old suite tests the problems you had a year ago
Cases frozen for comparability quietly stop representing production, so the suite keeps passing while the failures users actually hit go unmeasured. Feeding it from incidents, sampled recent traffic and support escalations keeps it current, which is why new cases are added and versioned rather than swapped in silently. Three quarters of rejections count as discipline only if the bar still describes today's product.
- 7
A dated log shows whether your reading is paying for itself
A timeboxed review, an hour a week rather than continuous attention, produces a list of things that happened. The log records which of them changed a decision, a backlog item or a threshold, and which were merely interesting. Two quarters in it answers two questions at once: whether the source list earns its time, and whether your gates rejected everything because nothing was good or because nobody revisited them.
External resources
- LLM Evals FAQ: Everything You Need to Knowhamel.dev
Reference for building the personal eval suite you re-run on every model release
- What Makes an Evaluation Useful? Common Pitfalls and ...arxiv.org
Practical criteria and common pitfalls for designing evaluations that meaningfully compare model releases.
- How to Build AI Benchmarks That Evolve with Your Modelslabelstud.io
This is a substantive explainer on building and evolving personal AI benchmarks, covering stable task sets, scoring methods, and comparing model versions for regressions and improvements.
- 2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon WillisonAI Engineer
A working model of tracking the field via one stable test across every release
- Scott and Mark learn to Vibe Check with Simon Willison | LIVE113Microsoft Developer
How to design personal vibe-check prompts that stay meaningful over time