Five questions to ask before a statistic leaves the report it came from

Content

AI has made content easy to produce. Reports, decks, and briefings that used to take a week can now take as little as minutes, and most of them include statistics drawn from market research, analyst reports, or whatever the model found. What has not sped up is claim verification. Generation now outpaces review in most organisations, but accountability has not moved: whoever publishes a document still answers for every claim in it, whether a person wrote it or a model did.

We ran into this ourselves. A completed document offers no clue as to which of its claims will hold up. The citations look tidy, but there’s rarely time to open every reference, and no record of what the last reader checked. So each person who picks up the document has to decide again how much to trust it.

So we built EMBER (Evidence Mapping and Benchmarking of Existing Research). It takes an existing document, extracts every verifiable factual claim, tests each one against evidence outside the document, and returns the document word-for-word with a trust marker on every claim and a note on what to do about it. It runs five checks to get there, and we think the same five checks are worth asking of any statistic before reusing it, whether or not you use the tool.

1. What is this document actually claiming?

Identify the claim before you check anything. Most sentences in a report are framing, opinion or context, and none of that can be tested. What can be tested is the number, the date, and the comparison. Write those down separately, using the document’s exact wording, so you know precisely what you are about to carry into your own work.

This is EMBER’s first step too: it extracts every verifiable claim in the author’s exact words and classifies it by type, ranging from past events, which are easiest to check, to forecasts, which are hardest. Opinions and framing are left unmarked, so sentences without a marker make no checkable claims.

2. Does credible evidence back this up?

A claim supported only by its own document isn’t yet a fact. Look for evidence the author did not choose. Open the source the document cites and see whether it says what the document says it does. Then look for a second, independent source, preferring primary documents and official data over commentary on them. If you cannot find anything outside the document supporting the claim, treat the claim as the document’s own assertion rather than something established.

That’s the three-level check EMBER applies to every claim, along with one rule worth borrowing: a source counts only if you can point to the exact words on the page that support the claim. If you can’t find that sentence, you haven’t found a source.

3. How much weight can this claim take?

Not every claim deserves the same confidence, and it helps to know which kind you are holding. Ask three things. How strong is the best source behind it? Do independent sources agree or disagree? And how verifiable is the claim by its nature, given that a past event can be confirmed, while a forecast cannot?

EMBER sorts every claim into one of four tiers on exactly those questions. Verified means multiple credible independent sources confirm it as written. Caution means support exists but is thin, usually a single source. Contested means credible sources disagree with the claim, or with each other. Unverified means nothing outside the document could be found. A document full of internal figures and forward-looking statements will have a large Unverified share, and that is normal since non-publicly available information is not independently verifiable. The tier to take seriously is Contested because external evidence actively contradicts the claim.

4. Does the cited source say what the document says it does?

This is the check almost nobody runs, and the one that catches the most. A reference can be real, the paper genuine, the link live, and the figure attributed to it still not be in there. The footnote looks authoritative, so it gets accepted at face value. The fix is simple and tedious: open the reference and read the sentence it is supposed to support.

EMBER flags this separately from the trust tier, because a claim can be true and still be miscited, broken or weakly sourced. It also distinguishes a dead link from a gated one that is alive but behind a paywall, so a valid source is not mistaken for a fabricated one.

5. What do I do about it?

A verdict without an action leaves you where you started. The right action depends on what went wrong: for example, a figure contradicted by stronger contradicting external evidence gets clarification on how the numbers and sources should change and why, while disputed framing gets a breakdown of where the source of confusion might originate and how it can be cleared up.

EMBER writes this up as a remediation brief, one note per claim with a priority and an effort level, so the person fixing the document can see what to tackle first and roughly how long it will take.

Why the five are worth the trouble

Once a figure is repeated as the organisation’s own, a mistake in it reflects on the organisation, not on whoever first wrote it, and pointing back to the original rarely undoes the damage. EMBER asks these five questions of every claim in a document at once.

Learn more about how EMBER runs these checks on a real document, and what the report looks like when it comes back.

Recommended articles