Statistics notes: Absence of evidence is not evidence of absence

Abstract

Randomized controlled clinical trials are conducted to determine whether differences of clinical importance exist between selected treatment regimens. When statistical analysis of the study data finds a P value greater than 5%, it is convention to deem the assessed difference nonsignificant. Just because convention dictates that such study findings be termed nonsignificant, or negative, however, it does not necessarily follow that the study found nothing of clinical importance. Subject samples used in controlled trials tend to be too small. The studies therefore lack the necessary power to detect real, and clinically worthwhile, differences in treatment. Freiman et al. found that only 30% of a sample of 71 trials published in the New England Journal of Medicine in 1978-79 with a P value greater than 10% were large enough to have a 90% chance of detecting even a 50% difference in the effectiveness of the treatments being compared, and they found no improvement in a similar sample of trials published in 1988. It is therefore wrong and unwise to interpret so many negative trials as providing evidence of the ineffectiveness of new treatments. One must instead seriously question whether the absence of evidence is a valid justification for inaction. Efforts must be made to look for quantification of an association rather than just a P value, especially when the risks under investigation are small. The authors cite a recent trial comparing octreotide and sclerotherapy in patients with variceal bleeding, as well as the overview of clinical trials evaluating fibrinolytic treatment for preventing reinfarction after acute myocardial infarction as examples.

Keywords

MeSH Terms

Affiliated Institutions

• St George's, University of London GB

Related Publications