Compare AI observations over time
AI observations can be compared over time only when the relevant population and collection conditions are compatible. Record the question, entity rules, interface, retrieval mode, geography, language, period and method. A measured change can reveal persistence, variation or a shift in representation within that scope. It does not by itself show why the change occurred or prove that an intervention worked.
Choose the comparable set
Start with the claim you want to test. Comparing the same buyer questions across two periods requires known question versions and matching rules. Comparing source composition requires compatible citation capture and normalization. An organization’s name, domains or group structure may also have changed; retain the historical mapping and explain any reconciled comparison.
Model identity, tools, account context, location, language and collection method can affect the reading. Known differences need explicit treatment. Unknown versions remain unknown; the comparison may describe two observed experiences without isolating change in the underlying model.
The matched subset may be smaller than the full collection. Report what was included, excluded or missing and why. A larger mixed panel can conceal the very condition the comparison was supposed to hold constant. The surface reference explains which system dimensions should be retained.
Report change with its population and denominator
Illustrative arithmetic, not a research result: an organization appears in 6 of 10 eligible answers in one period and 9 of 15 in another. The count increases, but the rate remains 60%. If the added questions address different needs, even equal rates do not establish stable positioning in a matched population.
Use counts, rates and coverage together when each answers a necessary question. State whether a period figure pools eligible observations or summarizes per-run rates. Those operations are not interchangeable, particularly when runs have different sizes or source counts.
Confirmed nonappearance in a valid answer, a valid search capture with no AI Overview and a failed capture are distinct states. A confirmed empty surface can count as zero for a metric whose declared population includes it; unavailable data cannot. Report missingness separately so a change in collection coverage is not mistaken for a change in representation.
Measurement and uncertainty makes the eligible population, missing observations and denominator explicit before a reported movement is interpreted.
Separate before and after from intervention effect
A website revision, a model update, a source change and ordinary response variation can occur during the same interval. Temporal sequence alone cannot identify which factor caused an observed shift. A before-and-after comparison can be informative while remaining descriptive.
A causal study would need an appropriate design, such as controlled variation of the candidate factor, comparable cells, repetitions and assessment of alternative explanations. A controlled assistant experiment would still support a claim only within its experimental conditions, not automatically across public interfaces or business outcomes.
Preserve stable cases and counterexamples alongside changes. Selecting only dramatic movements can exaggerate variability. The evidence standard distinguishes temporal observation, behavioral sensitivity and causal support.
Plan a follow-up that can answer the next question
Define the baseline and the change of interest before collecting the follow-up. Keep the question and classification rules stable where possible, document unavoidable changes and record what business event or intervention is being examined. Specify how unknown configurations and failed observations will be handled.
A follow-up can test whether a description persists, whether a clarified fact appears under the same conditions or whether a role differs across modes. Choose the relevant object rather than rechecking a headline score alone. Retain old evidence so the new result can be compared with its actual predecessor.
A change may justify investigating a source or protecting a stable strength; it need not trigger immediate implementation. The prioritization guide connects the observed difference to business relevance and a responsible next step.
When that movement affects a material decision, the organizational research pathway can frame the follow-up around what the organization actually needs to resolve.
Common questions
- Can a post-change improvement prove the change worked?
- No. Comparable observations can show temporal change without isolating its cause.
- What if the model version or query changes?
- Keep the original method labels and identify which observations remain compatible. Do not relabel older results retrospectively.
06 Where this leads
A useful follow-up begins before the change is made. The website and communications brief separates acceptance of the revised asset from assessment of later AI answers. The commissioning guide helps specify the observation scope and useful output when answering the change question requires further research.