AI-specific technical leadership

Read these first

• Prompt versioning, storage and review
• Labeling and annotation operations

What you will know

• Set a review standard for prompts, evals, skills, and config

  Versioned criteria and required evidence prevent a harmless-looking change from silently altering behaviour or invalidating results.

• Assign ownership of each to a person

  Named decision-makers make escalations, maintenance, and incident follow-up possible when several disciplines are affected.

• Fund annotation as real work in a real plan

  Sample size, reviewer calibration, and turnaround time determine whether evaluation data is trustworthy rather than merely abundant.

• Onboard an engineer onto the team's AI practices

  A new contributor should be able to apply the gates and produce acceptable evaluation or configuration changes without relying on tribal knowledge.

• Explain what is different from leading any other engineering work

  Probabilistic behaviour, shifting model dependencies, and subjective quality signals make evidence and operating boundaries more important than code ownership alone.

Chapters

1. 1

   Behaviour lives outside the code and changes without a commit

   The thing that genuinely differs about leading AI work is that your dependency ships changes you did not review, and quality is a distribution rather than a pass or fail. When complaints spike on Thursday after Monday's prompt change, the question is whether it was the prompt, a silent model update or a shift in input mix, and you can only answer it if every request recorded the prompt version, model version and config in force.

   Read chapter
2. 2

   A prompt change needs evidence, and so does retrieval_k

   Rewriting one sentence in a system prompt is a behaviour change, so the gate is evaluation results on the affected suite before and after, attached to the pull request. The harder point is that moving retrieval_k from 5 to 8, nudging temperature, or accepting a new model version are the same kind of change wearing config clothing, and they are the ones that habitually ship with no behaviour review at all.

   Read chapter
3. 3

   The eval set is your definition of correct, so it needs an owner

   Prompts, evaluation suites, skills and operational config each need a named person with authority to reject a change, not a team label. The evaluation set matters most: it is the operative definition of correct, and an unowned one drifts as people quietly delete the cases that keep failing. Ownership is also what makes escalation and incident follow-up land on someone rather than circulating.

   Read chapter
4. 4

   Annotation is funded work or your metrics are decoration

   Trustworthiness in evaluation data comes from reviewer calibration and measured agreement between annotators, not from volume. Ten thousand labels produced by engineers between tickets, with no guidelines and no agreement measurement, describe the labellers rather than the system. That is the argument for spending a headcount on someone who owns evaluation data instead of a seventh engineer whose output nobody can verify.

   Read chapter
5. 5

   Tribal knowledge fails the new-engineer test

   The honest test of whether your AI practices are written down is whether someone who joined last week can land a prompt change with the required evidence, unaided, in their first few days. If they need to ask which suite to run, where the eval set lives, or who signs off on a config change, the gates exist in people's heads. Onboarding is where you find that out cheaply.

   Read chapter