Skip to main content
SI
TheBestSI
Model Engineering8 min read

How to Evaluate AI Models: A Practical Engineering Framework

ByTheBestSI Editorial Board
•
Published:

1. Constructing a Golden Evaluation Dataset

Public benchmarks (like MMLU or GSM8K) are irrelevant for evaluating whether an LLM can classify your proprietary customer support tickets or generate valid SQL queries for your internal Postgres schema.

Build an internal “golden dataset” containing 100 to 500 representative historical examples, including tricky edge cases, malformed user inputs, and adversarial prompts. Grade models deterministically against known unit tests whenever possible.

2. Latency Metrics: TTFT vs. Generation Speed

Evaluate API latency along two distinct vectors:

  • Time-to-First-Token (TTFT): How long before the user or service receives the first character? High TTFT destroys conversational interactivity.
  • Tokens-Per-Second (TPS): How rapidly does the model stream the remaining generation? Essential for high-throughput background processing.

3. Cost Modeling: Input Tokens vs. Output Tokens

Output tokens are typically priced 3x to 5x higher than input tokens across frontier providers. When designing agent workflows, measure the total token amplification ratio (how many output tokens are generated per input prompt token) to forecast operational budgets accurately.