Traditionally, data collected in a research study is submitted to a
significance test to assess the viability of the null hypothesis. The
p-value provided by the significance test, and used to reject the null
hypothesis, is a function of three factors: The larger the observed effect,
the larger the sample size, and/or the more liberal the criterion required
for significance (alpha ), the more likely it is that the test will yield a
A power analysis, executed when the study is being planned, is used to
anticipate the likelihood that the study will yield a significant effect and
is based on the same factors as the significance test itself. Specifically,
the larger the effect size used in the power analysis, the larger the sample
size, and/or the more liberal the criterion required for significance
(alpha), the higher the expectation that the study will yield a
statistically significant effect.
These three factors, together with power, form a closed system - once any
three are established, the fourth is completely determined. The goal of a
power analysis is to find an appropriate balance among these factors by
taking into account the substantive goals of the study, and the resources
available to the researcher.
Role of Effect Size in Power
The discussion to
this point has focused on power analysis, which is the logical precursor to
a test of significance. If the researcher designing a study to test the null
hypothesis, then the study design should ensure, to a high degree of
certainty, that the study will be able to provide an adequate (i.e.
powerful) testing of the null hypothesis.
The study may be designed with another goal as well. In addition to (or
instead of) testing the null hypothesis the researcher might use the study
to estimate the magnitude of the effect - to report, for example that the
treatment increases the cure rate by 10 points, or by 20 points, or by 30
points. In this case, study planning would focus not on the study's ability
to reject the null hypothesis but rather on the precision with which it will
allow us to estimate the magnitude of the effect.
Assume, for example, that we are planning to compare the response rates for
treatments, and anticipate that these rates will differ from each other by
20 percentage points. We would like to be able to report the rate difference
with a precision of plus/minus 10 points.
The precision with which we will be able to report the rate difference is a
function of the confidence level required, the sample size, and the variance
of the outcome index. Except in the indirect manner discussed below, it is
not affected by the effect size.