LLM evaluation metrics for model selection: quality, cost and latency

1.The dimensions you compare candidate models on

Databricks Foundation Model APIs are built so you can efficiently compare LLMs to find the best candidate for your use case, or swap a production model for one that performs better. To make that comparison you need numbers, not impressions. For a RAG application the Databricks guidance splits those numbers into three groups. Retrieval quality measures whether the app found relevant supporting data. Response quality measures whether the answer is accurate against the ground truth, grounded in the retrieved context (or hallucinated), and safe (no toxicity). System performance covers cost and latency, such as overall latency and token consumption.

There are two ways to compute these metrics. Deterministic measurement covers cost and latency, which come straight from the application's outputs, and some retrieval metrics when your evaluation set lists the documents that contain each answer. LLM judge-based measurement uses a separate LLM to grade retrieval and response quality. The table below lists the metrics Databricks recommends. Put cost and latency next to the quality scores when you compare candidates, and re-measure all of them whenever you change the model.

Model size is a selection variable, and Databricks guidance is to start small and scale up as needed. Smaller open-source models can often give satisfactory results at lower cost and with faster inference, while a larger model is worth it only if the smaller one struggles on your queries. Then monitor the effect of each change on response quality, latency and cost. Measure latency rather than guessing it. Databricks defines total latency as time to first token plus time per output token multiplied by the number of tokens generated, so output length matters, and latency also changes with the number of concurrent requests. Databricks provides a benchmarking notebook to load-test an LLM endpoint for this.

Sources1234

2.Judges that need ground truth and judges that don't

The last column of the metrics table matters when you plan a comparison. Some judges compare the app's output with a human-labeled answer, so they only work if your evaluation data includes one. Others grade the output against the request or the retrieved context, so they can score any input. The metrics table above uses Agent Evaluation metric names. MLflow has its own built-in judges, listed below, with similar but differently named checks (for example RetrievalRelevance and RetrievalGroundedness). In MLflow, built-in judges are predefined scorers that use Databricks-hosted LLMs. Each one takes inputs and outputs, and the ground-truth judges also take expectations.

In practice you don't have to wait for a fully labeled dataset before comparing models. Agent Evaluation can score quality without ground truth, and once ground truth is available it adds metrics such as answer correctness. A sensible order is to compare candidates first on groundedness, relevance, safety, cost and latency, then add correctness as labels arrive. If no built-in judge fits your use case, scorers also come as custom LLM judges, code-based scorers for deterministic checks such as exact matching or format validation, and third-party scorers.

Sources56

3.A fair comparison: one evaluation set, one harness

Metrics only let you compare models if every candidate is scored on the same inputs. Databricks recommends a human-labeled evaluation set: a curated, representative set of queries with ground-truth answers and, optionally, the supporting documents that should be retrieved. It should be representative of production traffic, challenging (including adversarial prompts such as prompt-injection attempts), and updated over time. Aim for at least 30 questions, ideally 100–200. To avoid overfitting while you try many configurations, split the set into training (~70%), test (~20%) and validation (~10%) portions.

The harness that runs the comparison is mlflow.genai.evaluate(). It runs your app over the evaluation data, applies the scorers you choose, and returns an EvaluationResult. Databricks lists validating prompt or model changes across app versions as one of its main uses. To compare two LLMs, keep data and scorers the same and change only the app wrapper, using the optional model_id to track which version produced which results.

def mlflow.genai.evaluate(
    data: Union[pd.DataFrame, List[Dict], mlflow.genai.datasets.EvaluationDataset],  # Test data.
    scorers: list[mlflow.genai.scorers.Scorer],  # Quality metrics, built-in or custom.
    predict_fn: Optional[Callable[..., Any]] = None,  # App wrapper. Used for direct evaluation only.
    model_id: Optional[str] = None,  # Optional version tracking.
) -> mlflow.models.evaluation.base.EvaluationResult:

results = mlflow.genai.evaluate(
    data=[
        {"inputs": {"question": "What is MLflow?"}},
        {"inputs": {"question": "How do I get started?"}}
    ],
     ? =my_chatbot_app,
    scorers=[RelevanceToQuery(), Safety()]
)

Sources57

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.A smaller model is always faster, so you can pick it on parameter count without measuring latency.Why is that wrong?

   Databricks says to measure latency: it depends on time to first token and on how many output tokens are generated, and it changes with concurrent load. Smaller models are a good place to start, but you confirm with metrics on your own workload.

   Covered in The dimensions you compare candidate models on

2. 2.You can't compare candidate LLMs until you have a fully ground-truth-labeled evaluation set.Why is that wrong?

   Agent Evaluation can assess quality without ground truth, using reference-free judges such as groundedness. Ground truth only adds metrics such as answer correctness.

   Covered in Judges that need ground truth and judges that don't