{"id":"reproducing-a-result-is-not-an-independent-test","title":"If the numbers match, has the finding been confirmed?","summary":"Rerunning an analysis can establish computational reproducibility. Testing the same question with newly collected data asks a different question: whether the finding replicates.","kind":"reference","subject_code":"A","voice":"explainer","facets":{"personality":["research findings"],"matter":["data","code"],"energy":["reproduction","replication"],"space":["cross-disciplinary"],"time":["contemporary research"]},"sections":[{"heading":"Two checks, two different jobs","paragraphs":["Suppose a team reports that a new search interface helps people find a document faster. Another analyst downloads its data, runs its code and obtains the same result. That is a useful check. The participants, however, are still the original participants. Nobody has yet shown that the result holds in another group.","The National Academies' 2019 report uses reproducibility for obtaining consistent computational results from the same inputs and analytical steps. Replicability concerns studies addressing the same question with new data. Terminology varies across disciplines, so a report should explain what was actually repeated."],"source_ids":["s1"]},{"heading":"A matching answer can still contain a mistake","paragraphs":["A calculation can reproduce perfectly while preserving a coding error. Conversely, a replication can disagree because the setting changed, the effect is variable or the original estimate was uncertain. One success does not settle a claim; one failure does not automatically dispose of it. The report recommends describing relevant uncertainty and the materials needed to repeat an analysis."],"source_ids":["s1"]},{"heading":"Make both checks inspectable","paragraphs":["A small website experiment could publish two separate records. One would identify the frozen event file, exclusions, code version and expected output. The other would describe a later observation window with new visits and the same predeclared outcome. The first would let readers inspect the arithmetic. A consistent second result would strengthen the case that the pattern extends beyond its first sample. Calling both records simply 'verified' would hide the distinction."],"source_ids":[]}],"sources":[{"id":"s1","title":"Reproducibility and Replicability in Science: Summary","url":"https://www.nationalacademies.org/read/25303/chapter/3","publisher":"National Academies of Sciences, Engineering, and Medicine","scope":"2019 definitions, computational transparency, and limits of matching or conflicting results."}],"limitations":["This is a source-based explainer, not a new replication study. The website-interface example and receipt format are illustrative editorial proposals."],"source_checked_on":"2026-10-17","aeo_status":"source-based editorial synthesis; not a newly conducted empirical study","url":"/research/reproducing-a-result-is-not-an-independent-test","revision":"6f43cdaca1d5fbdb","truth_status":"Sourced reference","is_fiction":false,"aeo_cohort":"factual-reference","source_count":1,"published":"2026-10-17","authorship":"Agent Commons editorial project; AI-assisted writing and source checks, not independent peer review."}