It’s about ratios with low numerators.
The problem you have with interpreting clinical trial results sometimes is that sometimes the event you are looking for either didn’t happen, or only happened once or twice. This means that it would seem to be quite difficult to calculate a reliable risk for that event – particularly if it didn’t happen at all.
Luckily, help is at hand, and there is a simple statistical method to obtain an upper 95% confidence limit for a zero numerator (when the event doesn’t happen at all), or 95% confidence intervals for when you have a numerator between 1 and 4 (i.e., the event happened between 1 and 4 times).
This enables you to judge how reliable your study results are – or, at least, what’s the worst that could happen.
Numerator is zero
“We reviewed 14,455 eye examinations done with the drug fluorescein, and nobody died.”(1) Surely, a death rate of 0/14,455 means that either nobody ever dies after a fluorescein eye examination – or, alternatively, that the risk of death is uncalculatable?
Well, the former is difficult to believe, and the latter is unacceptable. So does this mean that in order to calculate a risk of death, you have to keep doing whatever it is, until somebody dies?
Well, we could probably make some estimates about the maximum risk.
Obviously, we are not going to be able to calculate an accurate chance-of-death if nobody has died yet. However, if 14,455 patients in a row had the examination and they all survived, then the risk can’t be too high – for instance, it couldn’t be 1/100, or even 1/1000, because probably we wouldn’t be so lucky as to get all the way to 14,455 without somebody dying if that was the case. On the other hand, we couldn’t be quite so confident about saying that the risk of dying must be less than 1/10,000 – because our first death might just be a bit late. We certainly couldn’t say that the risk must be less than 1/15,000. So we know there must be an upper limit where we can say “we’re pretty sure that the chance of dying isn’t any more than X”.
So, if we can work it out like that, there must be a proper way of doing it. Fortunately, Hanley and Lippman-Hand(2) come to our rescue.
In medicine, we tend to deal with 95% confidence intervals a lot. Basically, your 95% confidence interval is where you can say “I’m 95% confident that the real result – if we checked the whole population and not just a sample – would be within this range.” (It’s a bit more complicated than that, but this is a useful way of thinking of it.)
We use 95% because it’s a convention that a 5% chance that the results of your study are completely due to chance, or otherwise unrepresentative of reality, is low enough that we can live with it. Hanley and Lippman-Hand report a simple way of finding out where the upper limit of your 95% confidence interval is (i.e., the point at which you can say “There’s only a 5% chance that the real number is beyond this point”).
All you have to do it:
Upper limit of 95% confidence interval = 3/n, where n is the number of people in your group.
So, for the fluorescein patients above, we do 3/14,455 = 1/4818. So, we can be 95% sure that the risk of death after an eye examination with fluorescein is less than 1/4818. It might be a lot less – but it probably won’t be any more than that.
And that’s a very comforting thing. Now we have some real numbers.
This is important, because there’s a very real difference between a risk of approximately 1/5000, and a risk of zero.
But what if we tested a lot more patients, and one died? Would our problems be over at that point?
Yannuzzi et al(3) said “We looked at 221,781 eye examinations done with the drug fluorescein, and only one patient died.” So, that gives a chance-of-death of 1:220,000. Fantastic!
However, what if the 221,782nd patient (who didn’t quite make it into the study), also died?
That would be a chance-of-death of 2:221,782, or round about 1:110,000. Twice as often. Just with one more patient. This makes those numbers seem suddenly less comforting.
But fortunately, there is a mathematical workaround for this as well.(4) The following table gives a “fudge factor” numerator to use for different sizes of observed numerator and different sizes of denominator group.
|Upper Limit of Exact 95% Confidence Intervals From the Binomial Distribution|
|*for zero numerators, it’s a single upper confidence limit. The others are confidence intervals.|
So, for a group of 221,781 patients, of whom 1 died… well, the table doesn’t quite go that far. But the “fudge factor” numerator to use for an observed numerator of 1 and a denominator of >1000 is going to be at least 5.6.
So, if we use 5.6: 5.6/221,781 = 1/39,604.
So the worst it could possibly be is a risk of death of approximately 1/40,000.
And how accurate is that?
Well, in 1983, Zografos(5) reported on 594,687 angiographies with fluorescein. In his study, he found that 12 patients had died. And that results in a risk of death of 1/49,557.
Zografos didn’t give us any confidence intervals, either, but his observed frequency of death after fluorescein angiography is very close to (and on the right side of) our estimated upper confidence interval from the smaller study by Yannuzzi.
- Beleña JM, Núñez M, Rodríguez M. Adverse Reactions Due to Fluorescein during Retinal Angiography. JSM Ophthalmol. 1:1004.
- Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all right?: Interpreting zero numerators. JAMA. 1983 Apr 1;249(13):1743–5.
- Yannuzzi LA, Rohrer KT, Tindel LJ, Sobel RS, Costanza MA, Shields W, et al. Fluorescein angiography complication survey. Ophthalmology. 1986 May;93(5):611–7.
- Newman TB. IF almost nothing goes wrong, is almost everything all right? interpreting small numerators. JAMA. 1995 Oct 4;274(13):1013–1013.
- Zografos L. [International survey on the incidence of severe or fatal complications which may occur during fluorescein angiography]. J Fr Ophtalmol. 1983;6(5):495–506.