LLM applications and evaluation¶

Building applications on large language models, and measuring whether they work: choosing models, managing prompts, evaluating answers systematically, and tracing and monitoring them in production. The practices together are often called LLMOps.

Before reading

Embeddings, vector search and RAG

Why it matters¶

A demo of a language model application takes an afternoon; knowing whether it answers correctly for real users takes much longer. Outputs are free text, they vary between runs, and small changes to a prompt or a model can improve one case and break five others. Without evaluation, every change is a guess. Much of this work is data work: building evaluation sets, logging traces, and measuring quality over time.

How it works¶

Model choice comes first. Hosted models from providers are called through an API and are usually the most capable, with costs per token and data leaving the organisation under the provider's terms. Self-hosted open-weight models run on the organisation's own infrastructure, which gives control over data and cost at scale but needs GPUs and operations work. Many applications use several models: a strong one for hard steps, a cheaper one for simple steps such as classification.

Prompts are code. The instructions, examples and templates that drive a model are kept in version control, reviewed, and tied to evaluation results, rather than edited in a console. A change to a prompt is tested like a change to a model.

Evaluation sets are the core asset. An evaluation set is a collection of realistic inputs with what a good output must contain: questions from real users with reference answers, the documents that should be retrieved, or criteria an answer must meet. It is built from real traffic, expert input and known failure cases, and kept as a versioned table. Each change to prompt, model, retrieval or tools is run against the whole set before release.

Scoring outputs uses several methods. Deterministic checks verify format, citations, required fields or forbidden content. Reference comparison checks overlap with a known answer. Model-graded evaluation, often called LLM-as-a-judge, asks a model to score an answer against criteria such as correctness, grounding in the retrieved sources, or helpfulness; it scales well but must itself be checked against human judgements, because judges have biases, such as preferring longer answers. Human review of samples remains the reference. For retrieval applications, retrieval and generation are scored separately: did search find the right chunks, and did the answer use them faithfully.

flowchart TB
  CH[Change: prompt,<br/>model or retrieval] --> RUN[Run against<br/>evaluation set]
  RUN --> SC[Score: checks,<br/>references,<br/>model-graded]
  SC --> CMP{Better than<br/>current version?}
  CMP -->|yes| REL[Release]
  CMP -->|no| CH
  REL --> PROD[Production traces:<br/>inputs, retrieved docs,<br/>outputs, feedback]
  PROD -->|new cases| EVS[(Evaluation set)]
  EVS --> RUN

In production, every request is traced: the input, the retrieved documents, each model call with its prompt and output, tool calls, tokens, latency and cost, and any user feedback. Traces are stored as tables, sampled for review, scored automatically for groundedness or policy violations, and turned into new evaluation cases when something goes wrong. Tools such as MLflow, Langfuse and others provide tracing and evaluation, and write to stores the platform can query.

Guardrails check inputs and outputs: blocking personal data from leaving, detecting prompt injection, refusing out-of-scope requests, and validating structured outputs. They are evaluated like everything else, including how often they block legitimate requests.

Key terms¶

LLMOps: practices for building, evaluating and operating LLM applications.
Open-weight model: a model whose weights can be downloaded and run on own infrastructure.
Evaluation set: versioned inputs with criteria or references for good outputs.
LLM-as-a-judge: using a model to score outputs against criteria.
Groundedness: whether an answer is supported by the retrieved sources.
Trace: the full record of one request through an LLM application.
Guardrail: a check that blocks or corrects unsafe inputs or outputs.

Example¶

The shop's support assistant answers staff questions from manuals and articles.

Evaluation set: 400 real questions from staff, each with the article that answers it and the facts a correct answer must contain, stored as shop.eval_support_v7.
Scoring: retrieval recall (was the right article in the top five), a judge model scoring correctness and groundedness against the facts, and a check that every answer cites a source. The judge's scores were validated on 100 questions: the judge agreed with the two staff members' consensus rating on 91, and the two staff members agreed with each other on 94.
A prompt change to make answers shorter raises correctness from 84% to 86% but drops citations to 70%; it is reworked before release. A 2-point change on 400 questions is within run-to-run noise, so the team reruns each version three times before reading a difference.
In production, answers marked unhelpful by staff are reviewed weekly; 30 of them became new evaluation cases last month.

Pitfalls¶

Testing with a handful of examples. Quality judged by a demo does not hold for real traffic.
Prompts edited in production. Changes are untracked and untested.
Trusting judge models blindly. Check them against human ratings.
Scoring only the final answer. Retrieval failures look like model failures.
No traces. Bad answers cannot be explained, reproduced or fixed.

Learn more¶

Hamel Husain, Your AI product needs evals: https://hamel.dev/blog/posts/evals/
MLflow documentation, GenAI evaluation and tracing: https://mlflow.org/docs/latest/genai/
OWASP, Top 10 for LLM applications: https://genai.owasp.org/llm-top-10/

Agents and MCP · Fine-tuning or RAG · Responsible AI and AI governance · Observability and SLOs