How to read a model card without getting fooled

Every model launch now comes with a card: a document that is part technical specification, part safety disclosure and part marketing. Knowing which part you are reading at any moment is the whole skill.

This guide goes through a typical card top to bottom and flags what to trust, what to verify and what to ignore.

The benchmark table

The centrepiece of most cards is a table comparing the new model to its competitors on a dozen benchmarks. Treat it as a claim, not a measurement, and read the footnotes before the numbers.

Things that legitimately move scores by several points and are routinely applied unevenly:

• Sampling settings. Greedy decoding versus temperature sampling, and how many samples are drawn. “pass@1” with majority voting over 64 samples is not the same as one attempt.
• Prompting. Zero-shot versus few-shot, and whether a chain-of-thought instruction was used. Cards frequently report their own model with the favourable setting and competitors with whatever number was published.
• Tool access. Coding benchmarks in particular vary enormously depending on whether the model can run code, browse or retry.
• Reasoning budget. For models that scale test-time compute, the score depends on how much thinking was allowed. A “high” setting can cost 20x the tokens of a “low” one.

A good card states all four for every row. A great card runs competitors under the same settings. If neither is true, the table tells you about the vendor’s marketing team, not the model.

For the “why” behind this, read our editorial on why benchmarks have become a marketing channel.

Figure 1: Four settings have to match before two rows in a table can be compared. If they do not, the row is marketing.

Context window

The headline context length is a hard limit, not a capability. What you want to know is how much of that window the model can actually use. Look for:

• a needle-in-a-haystack result across positions and depths, ideally with multiple needles;
• a long-context reasoning benchmark rather than pure retrieval; and
• any statement about quality degradation beyond a certain length.

If the card only lists the maximum, assume usable context is materially smaller and test it yourself before designing around it.

Training data and cutoff

Two questions matter for builders: when does the model’s knowledge stop, and what kinds of data was it trained on? The knowledge cutoff should be a specific month. “Training data” sections have become more detailed under EU AI Act documentation requirements, but they still rarely tell you whether a benchmark’s test set was in the training mix. Contamination is the elephant in every benchmark table.

Safety and evaluations

The safety section usually contains the most careful measurement in the whole document, because it is the part regulators read. It is worth reading for a different reason: the refusal behaviour it describes is what your users will hit. Over-refusal rates, jailbreak resistance and the categories of content the model declines all shape product behaviour directly.

Limitations

This is the section to read first if you read only one. A card that says “the model may hallucinate” and nothing else is not disclosing anything. A card that says “accuracy on numerical reasoning over tables drops sharply beyond roughly 40 rows” is giving you an engineering constraint. The specificity of the limitations section is the best single proxy for how honest the rest of the card is.

What is not in the card

Three things decide whether a model is viable for you and none of them lives in the card:

1. Price, including cached input, batch discounts and long-context surcharges. See who benefits from falling inference prices.
2. Rate limits and availability, especially at launch, when the best models are often effectively unavailable at production volume.
3. Data terms: retention, training on your inputs, regional hosting.

Read the pricing page and the terms of service alongside the card. Together they are the specification. Alone, the card is a brochure with citations.

A checklist

Before adopting a model on the strength of its card, you should be able to answer:

• Under what settings were the benchmark numbers produced, and were competitors run the same way?
• What is the usable context length, as opposed to the maximum?
• What is the knowledge cutoff, to the month?
• What specific failure modes does the vendor admit to?
• What does it cost at your expected token mix, and can you get the rate limit you need?

If the card and its companion pages answer all five, you have enough to run your own evaluation. If they do not, that is the finding.