| Goal | Reproducible observations, not ranking promises |
|---|---|
| Unit | Prompt, model, date and cited answer |
| Comparison | Same questions across repeated runs |
| Real dataset | AI visibility report |
This page describes how to measure whether an AI answer service mentions, accurately identifies and appropriately cites a person or organisation. It turns lessons from the five search laboratories experiment and the two-day Claude comparison into a general method. The published AI visibility report contains one specific prompt set and its historical results; this reference explains how to design a comparable test for a different subject.
Design the fixed prompt set
Include several distinct intent types. Branded questions name the subject directly and ask about their identity, credentials or products. Nonbranded discovery questions ask who can solve the relevant problem without mentioning the subject. Comparative questions name a class of providers and ask for reasons to choose among them. Fact-check questions ask for specific projects, dates or affiliations whose correct answers are known from records.
Freeze wording, order and classification during a measurement series. A wording change can alter which people are named, making before-and-after numbers incomparable. Publish the prompt text where permission and confidentiality allow it.
Capture each run
| Field | Meaning |
|---|---|
| Date and model | Run time, provider, model or product surface, region when relevant. |
| Prompt ID | Stable key and exact question used. |
| Subject named? | Explicit name present, absent or unclear. |
| Position | Order among named alternatives, when an ordered list exists. |
| Citations | Exact linked sources and whether they support the specific claim. |
| Claims | Factual statements extracted for verification against reliable evidence. |
| Notes | Refusals, errors, ambiguities, namesake confusion and cost. |
Save the original answer where privacy and terms permit. A second AI model can extract structured names and claims, but spot-check the extraction manually. Extractors can miss names and treat confident speculation as a fact. Compare models and platforms separately. A web-search answer and an ordinary chat completion may rely on different source access.
Interpreting the metrics
A branded mention rate asks whether an engine can identify a specifically named subject. A nonbranded mention rate asks whether the subject appears when a relevant service or category is requested. A citation count asks which sources were used; it is not a count of endorsements. A higher number of mentions need not mean stronger evidence, and an accurate answer that does not name the subject may be appropriate.
A single run is a snapshot. Repeated runs with stable wording can reveal changes, but outputs are stochastic and products change their retrieval systems and ranking logic. Test several times if a small difference would otherwise drive a costly decision. Keep error analysis separate from promotional visibility.
What to do with false claims
Trace the claim to its cited page when possible. Check original records and the official website. Correct an outdated biography or inaccurate public directory at the source. Add a dated correction to the primary website. Document misattribution to namesakes with occupation and organisational context. Do not try to control a model by publishing fabricated testimonials, impersonating sources or asserting facts that cannot be independently checked.
Publishing the results responsibly
Publish method, prompt count, questions, dates, models, important limitations and checks performed. Aggregate others' names where individual inclusion would disclose unnecessary personal information. Separate facts from interpretation and explain what cannot be inferred from the sample. Record API costs if relevant. Do not claim a causal ranking effect from a change observed in one run.
Related references: AI visibility report for an implementation with sixteen questions, Connections for service access, Online perception management field guide for ongoing identity maintenance, and Five live sites, five search labs for the original context.