Citelytics · Blog

For brands and networks

What to log so an AI visibility claim can be checked six months later

A defensible AI visibility result needs more than a score: preserve the question set, run conditions, outputs, coding rules and limits of the claim.

Published: 2026-11-11

Six months after a report goes out, someone may ask what “visibility increased” actually means. Can they recover the questions, see which locations were included, inspect the answers, and reproduce the calculation? If not, the number is difficult to assess.

Treat each measurement run as a dated research record, not just a dashboard snapshot. Store enough to explain what was tested, how it was tested, what was observed, how observations became metrics, and what the result does not establish. The NIST AI Risk Management Framework offers guidance for managing AI risks and evaluating systems; use it as a reference, not as a prescribed checklist for this measurement.

Start with the claim and its unit

Write the claim in a form that could be checked: for example, “Across this defined set of non-branded local-service questions, the brand appeared in 18 of 60 recorded answers in the March run.” Specify the numerator, denominator, inclusion rule for “appeared,” and comparison period. If the claim is about citations, recommendations, or a particular location, define that separately; they are not interchangeable outcomes.

Also log the unit of analysis. Is one observation a prompt, a prompt-model pair, a location-query pair, or an individual answer? A multi-location program should preserve the location identifier and the rule for assigning a question to a location. Otherwise, a future analyst cannot tell whether a change reflects brand visibility, a changed location mix, or a changed sample.

Preserve the sample, not just the summary

Save the full prompt set as run, including wording, language, market, intent, brand-name status and location context. Record who selected the prompts, when, and under what inclusion or exclusion rules. Keep a versioned master set and a run-specific copy: editing a shared prompt list later should not rewrite the historical sample.

Log the sampling method and the planned sample size before the run, then record the number actually completed and any exclusions. If you report a rate, show the denominator and the number of observations behind it. A result from a small or deliberately curated set can describe that set; it does not, by itself, establish how often a brand appears across all users’ AI experiences.

Capture the conditions that shaped the answer

For every observation, log the date and time, product or interface, access method, model name or version when disclosed, and relevant settings. Record geography, language, device or account context where applicable, and whether browsing, search, or other tools were enabled. If a platform does not expose a model version or setting, mark it “not available”; do not infer it.

Save the complete answer and any visible citations or links, not only a manually coded “yes.” Preserve screenshots or exports when the interface matters, and retain the raw text in a searchable format. Keep a record of collection failures, retries and changes to the procedure. These details help distinguish a shift in the observed sample from a changed product, configuration, or collection process.

OpenAI’s API documentation notes that prompting behavior can differ between model snapshots. Record the model version when available, along with run conditions; a log cannot make a third-party interface deterministic. It documents what your team observed, not a promise that the same query will produce the same answer later.

Make the coding reproducible

Define the labels before reviewing the run. For example, separate an unlinked brand mention from a cited source, a recommendation from a neutral reference, and the correct location from another branch. State how ambiguous answers, misspellings, parent-company names and competitor comparisons are handled.

Store the coded record alongside the raw answer: coder, coding date, label values, and any adjudication or override. For subjective labels, double-code a prespecified subset and retain disagreements and the resolution rule. Report the share double-coded and the agreement method you used. A metric derived from human interpretation should not be presented as if it were a direct platform counter.

Keep the calculation and its boundaries

Archive the calculation logic, spreadsheet or code version, filters, and final data export. A future analyst should be able to trace the published figure to individual observations and regenerate it. Preserve the report version and the date it was issued, plus any corrections made afterward.

Separate observed facts from interpretation. “The brand appeared in 18 of 60 sampled answers” is an observation. “Visibility improved because of the campaign” is a causal claim requiring evidence beyond a before-and-after count. Likewise, a prompt sample does not establish the share of real user conversations, traffic, or sales influenced by AI. State these limits beside the result, not only in a methodology appendix.

A six-month check should take minutes, not guesswork

An analyst should be able to select a reported metric, follow it to its denominator, inspect the underlying answers, verify the labels and rerun the calculation. If any link is missing, narrow the claim to what the surviving evidence supports. The goal is not to freeze a changing AI product; it is to make your measurement choices visible, reviewable and honest.

Sources

Measuring a brand or a network?

Every scan keeps its raw answers and sources, so a number can be re-checked months later instead of taken on trust.

See plans

More on this

← All articles