Fun with statistics: Low numerators

statistics-706382_640Last week, I came across a great statistical trick that I had to rush out and share with everyone, because it was so incredibly cool.

It’s about ratios with low numerators.

The problem you have with interpreting clinical trial results sometimes is that sometimes the event you are looking for either didn’t happen, or only happened once or twice. This means that it would seem to be quite difficult to calculate a reliable risk for that event – particularly if it didn’t happen at all.

Luckily, help is at hand, and there is a simple statistical method to obtain an upper 95% confidence limit for a zero numerator (when the event doesn’t happen at all), or 95% confidence intervals for when you have a numerator between 1 and 4 (i.e., the event happened between 1 and 4 times).

This enables you to judge how reliable your study results are – or, at least, what’s the worst that could happen.

Numerator is zero

For example,

“We reviewed 14,455 eye examinations done with the drug fluorescein, and nobody died.”(1) Surely, a death rate of 0/14,455 means that either nobody ever dies after a fluorescein eye examination – or, alternatively, that the risk of death is uncalculatable?

Well, the former is difficult to believe, and the latter is unacceptable. So does this mean that in order to calculate a risk of death, you have to keep doing whatever it is, until somebody dies?

Well, we could probably make some estimates about the maximum risk.

Obviously, we are not going to be able to calculate an accurate chance-of-death if nobody has died yet. However, if 14,455 patients in a row had the examination and they all survived, then the risk can’t be too high – for instance, it couldn’t be 1/100, or even 1/1000, because probably we wouldn’t be so lucky as to get all the way to 14,455 without somebody dying if that was the case. On the other hand, we couldn’t be quite so confident about saying that the risk of dying must be less than 1/10,000 – because our first death might just be a bit late. We certainly couldn’t say that the risk must be less than 1/15,000. So we know there must be an upper limit where we can say “we’re pretty sure that the chance of dying isn’t any more than X”.

So, if we can work it out like that, there must be a proper way of doing it. Fortunately, Hanley and Lippman-Hand(2) come to our rescue.

In medicine, we tend to deal with 95% confidence intervals a lot. Basically, your 95% confidence interval is where you can say “I’m 95% confident that the real result – if we checked the whole population and not just a sample – would be within this range.” (It’s a bit more complicated than that, but this is a useful way of thinking of it.)

We use 95% because it’s a convention that a 5% chance that the results of your study are completely due to chance, or otherwise unrepresentative of reality, is low enough that we can live with it. Hanley and Lippman-Hand report a simple way of finding out where the upper limit of your 95% confidence interval is (i.e., the point at which you can say “There’s only a 5% chance that the real number is beyond this point”).

All you have to do it:

Upper limit of 95% confidence interval = 3/n, where n is the number of people in your group.

So, for the fluorescein patients above, we do 3/14,455 = 1/4818. So, we can be 95% sure that the risk of death after an eye examination with fluorescein is less than 1/4818. It might be a lot less – but it probably won’t be any more than that.

And that’s a very comforting thing. Now we have some real numbers.

This is important, because there’s a very real difference between a risk of approximately 1/5000, and a risk of zero.

Low numerator

But what if we tested a lot more patients, and one died? Would our problems be over at that point?

Yannuzzi et al(3) said “We looked at 221,781 eye examinations done with the drug fluorescein, and only one patient died.” So, that gives a chance-of-death of 1:220,000. Fantastic!

However, what if the 221,782nd patient (who didn’t quite make it into the study), also died?

That would be a chance-of-death of 2:221,782, or round about 1:110,000. Twice as often. Just with one more patient. This makes those numbers seem suddenly less comforting.

But fortunately, there is a mathematical workaround for this as well.(4) The following table gives a “fudge factor” numerator to use  for different sizes of observed numerator and different sizes of denominator group.

Upper Limit of Exact 95% Confidence Intervals From the Binomial Distribution
Observed Numerator*
Denominator 0 1 2 3 4
10 2.6 4.5 5.6 6.5 7.4
20 2.8 5.0 6.3 7.6 8.7
50 2.9 5.3 6.9 8.3 9.6
100 3.0 5.4 7.0 8.5 9.9
200 3.0 5.5 7.1 8.7 10.1
500 3.0 5.5 7.2 8.7 10.2
1000 3.0 5.6 7.2 8.8 10.2
*for zero numerators, it’s a single upper confidence limit. The others are confidence intervals.

So, for a group of 221,781 patients, of whom 1 died… well, the table doesn’t quite go that far. But the “fudge factor” numerator to use for an observed numerator of 1 and a denominator of >1000 is going to be at least 5.6.

So, if we use 5.6: 5.6/221,781 = 1/39,604.

So the worst it could possibly be is a risk of death of approximately 1/40,000.

And how accurate is that?

Well, in 1983, Zografos(5) reported on 594,687 angiographies with fluorescein. In his study, he found that 12 patients had died. And that results in a risk of death of 1/49,557.

Zografos didn’t give us any confidence intervals, either, but his observed frequency of death after fluorescein angiography is very close to (and on the right side of) our estimated upper confidence interval from the smaller study by Yannuzzi.


  1. Beleña JM, Núñez M, Rodríguez M. Adverse Reactions Due to Fluorescein during Retinal Angiography. JSM Ophthalmol. 1:1004.
  2. Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all right?: Interpreting zero numerators. JAMA. 1983 Apr 1;249(13):1743–5.
  3. Yannuzzi LA, Rohrer KT, Tindel LJ, Sobel RS, Costanza MA, Shields W, et al. Fluorescein angiography complication survey. Ophthalmology. 1986 May;93(5):611–7.
  4. Newman TB. IF almost nothing goes wrong, is almost everything all right? interpreting small numerators. JAMA. 1995 Oct 4;274(13):1013–1013.
  5. Zografos L. [International survey on the incidence of severe or fatal complications which may occur during fluorescein angiography]. J Fr Ophtalmol. 1983;6(5):495–506.



Ubble-Bubble, toil and trouble…

The UK Biobank, which is running a long-term observational study on nearly half a million middle-aged and elderly people in the UK, has produced a cross between an online game and a fortune teller.

It’s at and it asks you a series of innocuous questions: how old are you (only works for people who are 40+), how fast you walk, how you rate your own health, did you ever smoke, and so on. Then you press the button, and it gives you two pieces of information:

  1. Your chance of dying in the next five years and
  2. The average age of people who have the same chance of dying (which they call your ‘Ubble Age’.

I did the test – and had to lie about my age because I’m not forty yet – and it came out that (if I had actually been forty) I would have had a 0.2% chance of dying within the next five years. My ‘Ubble Age’ is 27.

This doesn’t mean that Ubble thinks I’m actually 27, or that I, personally, have a 0.2% chance of dying within 5 years. What it means is that

  • the average 27-year-old has a 0.2% chance of dying within 5 years and
  • If you had a crowd of 1000 27-year-olds, and you had a party this year, when you tried to have a reunion party in five years’ time, two of them wouldn’t be able to attend because they’d be dead.

A journalist in the Guardian has sneered at the test, because it’s easy to change the outcome: all you have to do, for example, say that you rate your health as ‘good’ instead of ‘fair’, and you get a better outcome. Also, the journalist who did the test got an ‘Ubble Age’ of 54 instead of her real age of 40. She concluded that she “would rather concentrate on living”, thus demonstrating that she has completely missed the point.

The point of Ubble is not to tell you when you are going to die, so you know whether to either make a will or borrow lots of money from people you don’t like.

The point of the Ubble test is to give people a chance to evaluate their own health, and what that means.

If a person’s ‘Ubble Age’ is older than their physical age, then that tells the  person something – it means that something about them or the way they’re living gives them a greater chance of dying than the average person their age. Maybe that’s due to factors outside their control, like having a past diagnosis of cancer, which is the greatest predictor of dying-in-five-years for women, apparently. On the other hand, maybe it’s because they smoke, or because they’re generally unfit. These things can be changed.

Playing with the questions gives you some interesting points: if I tell the quiz that I’m a current smoker, then my risk of death immediately doubles (to 0.4%), and my Ubble Age climbs 10 years, to 37.

If I stopped smoking (making me a ‘past smoker’), my risk of dying goes back to 0.2%, and my Ubble Age falls to 29 (not quite as good as being a never-smoker, but still a lot better than being a current smoker).

So I can use the quiz to estimate what effect making simple changes in my lifestyle would have on my health, and therefore my chance of dying. Of course, I still might get run over by a bus tomorrow, and even smokers sometimes live into their 90s, but that’s not the point. The point is giving myself the best chance of living until I’m 102.

Another point the journalist missed is that lying to the quiz isn’t a useful thing to do, unless you’re doing it to see what the results would be if you made a lifestyle change. The results don’t go anywhere: you’re only doing it for your own interest. So why lie?

The journalist also complained about the inaccuracy of people’s own self-made “health ratings”, saying that people often mis-estimate their own health. Well, that’s his/her opinion. The statistics, apparently, say differently – especially in the case of men. In looking at the questionnaires completed by nearly half a million people, the researchers found that self-reported health status (e.g. poor, fair, good, excellent) was the best predictor of future mortality for men. So whether all those people were lying or deluded or not, the results they put on the study questionnaires were gold (or maybe the reason self-reported health status isn’t the top predictor for women is that women – like the journalist – are more likely to be lying or deluded about their health, rather than that factor being taken over by a more powerful one: past cancer diagnosis).

The major point to take away from this is that Ubble is an exercise in statistics. On a personal level, it’s most useful for figuring out whether you’re likely to die earlier or later than other people your age – and thus, what your general health is like. If you come out as likely to die earlier (i.e., you get an older Ubble Age), then maybe you should think about making some lifestyle changes.

But for healthcare professionals, it’s far more exciting. The more data we have on people, as they grow older and die, the more we can predict where we need to direct resources for healthcare.

For instance, we know we can test Prostate Specific Antigen (PSA) in men, and we could potentially screen for prostate cancer. The trouble is, at the moment, the tests throw up a lot of false positives: where men get a result that says “Yep, you’re going to get prostate cancer”, and then they never do. So you end up with a lot of testing and following-up and worry, all for nothing. But with more data, we could pull all the factors together, and make screening more specific. Maybe it should go something like “If you get a high PSA result and you smoke and you rate your health as fair or below then you should be followed up.”

That’s the sort of thing that the UK Biobank might be able to tell us, as they get more data and evaluate it.

An even more interesting thing is that when they were evaluating the data, the researchers found that doing the questionnaire was a better predictor of when people were going to die than any physical tests (like blood tests). Which would you rather have – a short questionnaire, or a blood test?

So, in short, researchers have not yet discovered how to predict the future – but they may have started to do something nearly as good: help everyone figure out what lifestyle changes could have the biggest impact on their health, and help professionals figure out how to use our health resources best.

There’s a good article on Medscape here.