Most evaluation tables look orderly. Each row contains a prompt, an input, a document, a question, a case description, or a benchmark item. The system produces a response. A score is assigned, and the rows are aggregated. The structure is familiar, but in semantic domains it can be wrong at the foundation. The row may be only one expression of the thing being measured, not the measurement unit itself.
This matters because many AI evaluations are not really about strings of text. They are about intents, policy boundaries, decision-relevant situations, safety conditions, obligations, concepts, or user needs. Those objects can appear through many surface forms. If each surface form is treated as an independent unit, the evaluation may count rows while losing track of meaning.
What gets collapsed
Row-level evaluation often collapses three different layers: what is being measured, how it is expressed, and what behavior is observed. Those layers are related, but they are not interchangeable. The thing being measured might be a policy-relevant situation. The expression might be a phrasing, translation, format, order, or contextual frame. The observed behavior might be a refusal, answer, classification, escalation, or action.
When those layers are collapsed into a single row, the evaluation can appear more precise than it is. A dataset may look large because it contains many rows, while several of those rows are actually repeated measurements of the same underlying semantic case. A split may look statistically clean while related realizations of the same meaning appear in both training and test partitions. Aggregate metrics may look stable while hiding instability within specific semantic units.
The problem is not that rows are useless. Rows are necessary for storage, execution, and scoring. The problem is treating the row as the conceptual unit when the evaluation question is semantic.
Canonical semantic units
Canonical Semantic Realization starts by naming a different unit: the canonical semantic unit. A canonical semantic unit is the meaning-bearing condition under study, defined independently of any single observable expression. It is the target the evaluator is trying to measure. The wording, wrapper, language, ordering, or context is a realization of that target, not the target itself.
This distinction turns repeated surface forms into repeated measurements. A paraphrase, translation, format shift, contextual wrapper, or role framing can all be valid realizations of the same semantic unit if they preserve the relevant meaning. In that case, ten rows may not be ten independent semantic cases. They may be ten ways of observing one case through different representation channels.
That structure gives disagreement a place to live. If several realizations of the same semantic unit produce different outcomes, the evaluation can ask whether the variation was valid, whether the expected behavior was clear, and which representation channel exposed the instability. Without the semantic unit, the disagreement is harder to interpret.
Why this matters for deployment
In real AI deployments, users do not interact with semantic units directly. They express them through language, documents, interfaces, and context. A system may appear reliable when evaluated row by row while still behaving inconsistently across equivalent expressions of the same case. That is precisely the kind of problem organizations need to see before relying on the system in a workflow.
The better frame is simple: meaning is the unit, realization is controlled variation, and outcome is empirical measurement. This does not solve every problem in semantic evaluation, but it restores structure. It separates semantic identity from surface expression and makes repeated measurements visible as repeated measurements.
At Invarra, this is why we do not treat prompt strings as the final unit of analysis. The row is a useful artifact. The semantic case is the thing that matters.