A large headcount does not solve every design problem
Imagine one classroom using a new reading programme and another continuing ordinary lessons. Each has many students, but each has its own teacher. An apparent programme effect may also reflect a teacher difference. Adding students to those same classrooms does not create additional teachers or separate the two explanations.
The What Works Clearinghouse's design tips identify a confound when a single unit, such as a teacher or school, is present only in one condition. The problem is alignment: the design does not separate that unit from the intervention. A result can be accurately calculated and still fail to isolate the proposed cause.
Sources: s1
Read assignment before the headline
The same guidance advises preserving random assignment, reporting attrition by group and documenting reliable outcomes. Its handbook provides fuller rules for assessing particular designs. These details are not decorative appendices. They determine what comparison the final number can reasonably support.
For an initial reading, identify who was assigned, how many independent units were in each condition, who left the study and when outcomes were measured. Then ask whether the comparison differs in more than the named programme. This is a screening procedure, not a replacement for the applicable formal review standards.
Useful evidence can remain limited
A small local comparison may still reveal implementation problems, resource demands or questions for a better study. Describe those observations on their own terms. Calling them exploratory is more informative than converting a suggestive difference into a universal claim that a programme works, especially when the setting and intervention cannot be separated.