AI features that survive contact with production

The gap between an impressive prototype and a feature people rely on is mostly unglamorous engineering. The model is rarely the hard part.

Measure before you build

Decide what “working” means numerically before writing the prompt. For document extraction, that is field-level accuracy against a labelled set. For a support assistant, it is deflection rate and escalation quality. Teams that skip this end up arguing about vibes in month two, with no way to tell whether a change helped.

Build the evaluation set from real inputs, including the ugly ones — scanned faxes, mixed languages, half-filled forms. A hundred honest examples beat a thousand synthetic ones.

Design for being wrong

Assume the model will be confidently incorrect on some fraction of requests, and decide in advance what happens then:

• Show confidence and let a human confirm anything below threshold
• Keep the source document visible next to every extracted value
• Log every input, output, and model version for later audit
• Make the fallback path a first-class feature, not an error page

An AI feature with a good correction workflow beats a more accurate one that gives users no way to intervene.

Control cost and latency early

Token spend and p95 latency are product constraints, not infrastructure details. Cache aggressively, route easy cases to smaller models, and set a hard budget per request. Measure both from day one, because a feature that costs more per call than it saves in labour will not survive its first finance review.

Own the exit

Model providers change, deprecate, and reprice. Keep prompts, evaluation sets, and the orchestration layer in your own repository so switching is a configuration change rather than a rebuild.