Interpreting Negative Studies
                    =============================

(summary of a conversation between Geoff Rutledge and John Schoffstal,
regarding a study in which 100 of 100 patients with cocaine-associated CP were
found not to have MI)

What's the beta error or power of a study with 100 patients when the baseline
incidence of the disease is thought to be about 5 - 6%?

One way to answer this is to compute the confidence interval for the rate
of non MI in the population of these patients.  In other words, if we
examine 100 patients, and find 0 have MI, how high could the underlying
rate of MI be?

For large studies, we can calculate a confidence interval using the
normal approximation to the Binomial theorem, which says that the
expected number of observed cases is n*r, and the variance is n*r(1-r),
where n is the number of trials (in this case 100) and r is the rate (in
this case 1). This approximation only works well for nr and n(1-r)
greater than about 5. For our negative study, n(1-r) = 0, and the
approximation does not work.

In this case, as for other studies that have a small number of occurences
of the event we are looking for, we have to use the Binomial distribution
directly, computing the chance that, if the rate were whatever, we would
have observed all 100 non MI cases.

That is, we are interested in the probability that the rate of non MI
could have been less than some rate X. To compute this, we can integrate
the probability that the rate was X for all possible rates from 0 to X.

In this example, the integral of rates from 0 to 0.9708 sums to a
probability of 0.05, so the 95% confidence interval on the rate is
(0.9708,1.0), and there is only a 5% chance that the rate of MI in this
population is greater than (1-0.9708) ~= 0.0292.

This assumes that you do not want to do a Bayesian analysis and assert a
strong prior belief that the rate is greater than that (to do so you
would have to figure out the distribution of your prior estimate for the
rate).
------------------------------------------
     The quick and dirty way to calculate a 95% confidence interval when
the number of adverse event is zero, of course, is to divide 3 by the
number of subjects. I.e., an approximation of the upper bound of the 95%
c.i. in this case is 3/100, or .03, which isn't too far from .0292.

     You probably wouldn't want to use this method for a published paper,
but it's handy for what-to-do-while-waiting-for-the-statistician, or doing
in your head while reading papers, or listening to presentations at SAEM.
<g>

     Source: Hanley JA, Lippman-Hand A. If nothing goes wrong, is
everything all right? Interpreting zero numerators. JAMA 1983;249:1743.