Guenter Stertenbrink has spent years urging experts on infectious disease outbreaks to estimate the probability of an H5N1 influenza pandemic. In December 2005 he got around to me. The resulting entry in my website Guestbook was entitled “Likelihood of a severe pandemic – the hunger for a number.” It didn’t satisfy Guenter, who sent me this detailed point-by-point response. My original Guestbook comments are indented and in italics; Guenter’s replies are in roman.
Guenter: Thanks for your long reply. Well, I wrote “one number is better than 1000 words” but 1000 words are better than nothing.
Peter: I have followed with interest your search for a quantitative estimate of how likely a severe flu pandemic is in the next few years.
It started in March 2004, then it slept for a while but now with increasing common interest and signs of increasing danger I feel the urge to continue.
The Flu Wiki dialogue you provoked has been enlightening in many ways. But as you know all too well, it didn’t yield the number you sought.
Not yet. But we mustn’t give up.
It is incredibly frustrating not to have such a number. (I’m not going to have one for you either.
Alas. So I’ll be walking through your mail and will try to reconvert your words into a number. It will be my estimate of your estimate.
I’m just a communication expert, not a flu expert.)
That could be even better. Flu experts are often fixed to their area of research. The task of the communication expert is to collect and coordinate the different results and sub-estimates and convert them into a (subjective) overall estimate.
We can calculate the odds that a severe hurricane will hit any particular location within any specified period of time. Why can’t we do the same thing for a pandemic?
In a very limited sense, we can. Over the past 300 years there have been roughly three influenza pandemics per century (not evenly spaced), so in any randomly chosen year the odds of a pandemic are about one-in-30.
OK, when you discard any other information which we might have.
The most severe influenza pandemic known to history was the 1918 Spanish Flu pandemic. We have pretty decent history back around 500 years. So based on this extremely limited data set, I suppose the odds of a pandemic at least as bad as 1918 are about one-in-500 per year.
OK.
The problem is that this ignores what we know about the influenza virus of the moment, H5N1. How does what we know about H5N1 change the odds? We don’t know.
Ahh, there it is again, this “we don’t know.” We do know a lot. But what we know is not so easy to interpret.
Some very well informed and diligent examiner might come up with (nearly) the best possible estimate given the available data. In practice however many competing estimates will exist, which are roughly normally distributed with more or less big variation depending on the nature of the problem. We can just take the mean of the estimates as our individual estimate or weight the estimates according to our feeling for the reputation of the corresponding estimators.
Some virologists say H5N1 looks alarmingly like the Spanish Flu; they think the current odds of a severe pandemic are a lot higher than one-in-500 per year, maybe even higher than one-in-30. Other virologists say H5N1 has been around since 1997 without learning the trick of efficient human-to-human transmission, so it probably never will – suggesting that the probability of a severe pandemic today is no greater than it was before 1997, roughly one-in-500.
And others (presumably the majority) are between these two extremes.
Both groups are guessing.
Call it whatever you want. As long as you don’t define the terms it can’t be wrong. I prefer “estimate” over guessing. Guessing would be IMO when absolutely no data were available to base the guess upon – e.g. lotteries.
For nearly all of the past 500 years, we lacked the ability to monitor a novel flu virus before it did or didn’t go pandemic. In recent decades, scientists have monitored a handful of novel flu viruses in other species. Other than H5N1, only H7N7 and H1N1 (the 1976 swine flu virus) caused any human deaths. Neither one caused a pandemic. H5N1 is the only flu virus so far that we have watched become widely endemic in birds and jump species to humans scores of times. If it starts a pandemic, that will be the first time ever that we have seen a flu virus follow this pattern and then start a pandemic; we’ll be one for one.
?
If it doesn’t start a pandemic, that will be the first time ever that we have seen a flu virus follow this pattern and then not start a pandemic; we’ll be zero for one.
Zero out of one? … Now I see, you mean: Zero pandemics resulted out of the one event of a virus occurrence with that pattern.
There have been pandemics before, but we didn’t know about them until they were well launched. There have been false alarms before, too, but we didn’t know about them either until people started (and then stopped) dying. How many times has an influenza virus followed a course like that of H5N1 (high infectiousness and high virulence among some bird species, low infectiousness and high virulence among humans) and then mutated into a human pandemic? We don’t know.
We know that it’s at most [number of pandemics ever]. That’s better than nothing. We have some data for the last 500 years, as you say. If there had been prior severe pandemics, then archaeologists might have found some hints.
How many times has an influenza virus followed that course and then not mutated into a human pandemic? We don’t know that either. This is the first time we’ve been able to watch.
In the absence of data, then, what you’re asking for is a collection of expert guesses.
Not absense. Just maybe a (more or less insufficient?) small amount of data.
This isn’t a foolish thing to ask for. There is a lot of research, and even more argument, on the value of expert guesses as a stand-in for actual evidence. There are even formal procedures (the best known is called Delphi) for gathering and tabulating the guesses.
I don’t know about this. Maybe I can look up Delphi later.
Most such procedures seek a compromise between isolated individual judgments (too little opportunity to learn from each other) and roundtable discussions (too much pressure to conform). Their results are reported not as a single numerical estimate but as a distribution of estimates. The variance of the distribution – how much consensus the experts ended up reaching – is at least as important as its mean or median.
I think the mean is more important than the variance – at least when the main purpose is to find or improve my own estimate.
Also of interest is the shape of the distribution. For the question you’re asking, I would predict two humps. (Warning: I am now guessing about what flu experts might guess.)
No need to warn about this. It’s clear that you are doing this. What else (using your definition of guessing)?
The larger of the two humps, I think, would be those who go with precedent and stick to the view that severe pandemics are very rare; the smaller hump would be those whose guts are telling them H5N1 is different. If you could get the experts to guess, and if the expert guesses turned out the way I’m guessing, what would that say about the real risk?
Let’s define the real risk as the guess of a very qualified super-guesser, who has the same available data as we have but the most time and expertise and honesty to analyze them.
I don’t know.
This “I don’t know” I interpret as: “although I might have some idea, I don’t want to say this now, since I’m less certain about it than in the average of my assertions.”
It would probably say more about the psychology of risk estimation in the midst of uncertainty, the sort of thing Daniel Kahneman and Amos Tversky studied so effectively. And of course it would say something about the common tendency to get more confident and more extreme in your judgment when you’re immersed in a public controversy.
Your expectation of the two heaps could also indicate IMO that those people are biased or dishonest. They don’t tell what they really mean, but rather what they think would be best to say for them, for their company, their government, their career, their appearance.
To their credit, most experts know they’re guessing – which is why they work hard not to get nailed to a specific number.
If guessing or “guessing” were more common, then they wouldn’t have to worry so much about it. “Not guessing” should also be generally considered bad behavior – worse were wrongly guessing, though. But you can stand some wrong guesses when you also have lots of correct guesses.
And I suspect most experts would agree that the “true” probability distribution has three humps.
In mathematical terms here you probably mean the probability distribution of the random variable: “number of pandemic deaths in the next 5 years.”
The random variable “value given by some expert to my question” I’d expect were normally distributed. Remember, I’m not asking for the most likely scenario, but for the expectation value of deaths, which is the average over all possible scenarios weighted by their likelihood.
There’s a good chance that nothing will happen in the next five years; there’s presumably a good chance that H5N1, if it does go pandemic, will be fairly mild, killing fewer than ten million; and there’s also a non-trivial chance (given its current virulence and other factors) of a real catastrophe similar to or even worse than 1918.
“Good chance” and “non-trivial” are other interesting vocabularies introduced to avoid giving more informative numbers. One of my other favorites is: “we cannot exclude that….” – which is almost as informative as “we don’t know.”
I don’t see this 3rd hump. Are you suggesting that e.g. 100 million deaths are more likely than 50 million? Or that there is some other number of deaths which is more likely than any other, smaller number of deaths? I can’t see why.
It’s hard for an expert to address all three humps at once – especially when talking to the media. So most experts tend to focus their public remarks on just one of the three humps.
They use the opportunity to cut their information content….
There are worst case scenario experts, mild (“typical”) scenario experts, and it-might-not-happen-for-many-years experts.
And those why lie between these. But these usually get less attention in the media so we are not so much aware of them. But I think they do exist. The common trick is to make journalists think you have some spectacular opinion but then somehow hide it in words rather than giving numbers. It must just produce a good headline to draw attention.
And too often, at least as they’re quoted in the media, the experts end up less than clear that they’re talking about only one of three possible futures. The early media coverage of H5N1 (that is, up until the fall of 2005) tended to emphasize the mild scenario, often without saying it could be a lot more severe than that.
Maybe, but there were also some who gave 300 million deaths. They got not so much attention, but that was how I became first interested in March 2004.
The current media coverage tends to emphasize the severe scenario, often without saying it could be a lot milder than that.
Yes. I wonder whether this is just some man-made trend or whether there is indeed scientific background. Well, probably both. The media strengthen the trend. I can’t see this trend over here in Europe, though.
By the way, the experts’ public guesses (if you could get them to guess publicly) are probably not as good a predictor as their private behavior. I’d like to know how many virologists now have antiviral stockpiles for themselves and their families.
Yes. Although that is assuming that they are lying or not saying what they really think. If the whole practice of giving numbers for probability estimates became more common, then we could develop statistics about which predictions were more successful than others. Experts would be trying to be more honest so not to lose their reputations on bad statistical estimates.
Another hotly debated question is the relative accuracy of global guesses about big questions versus narrower guesses about sub-questions.
I haven’t noticed this debate. But big questions are rare and sub-questions more often occur. So to build up the guessing-reputation of a person, these small questions are more useful.
It’s not hard to divide your pandemic question into components: What is the probability that H5N1 will develop the capacity for efficient human-to-human transmission? If it does, what change in its virulence would be expected to accompany the increase in transmissibility? How likely are antivirals to work against the mutated virus? How long will it take to develop how much vaccine? You can compile the answers to these sub-questions into an answer to the big question.
Yes. But other people might prefer to choose other subdivisions.
What we don’t know is whether it’s better to ask the experts the big question or to ask them the little questions and then do the math. The two procedures typically yield very different answers.
Well, “we don’t know,” but we can estimate. I have had the experience that experts often answer (if they answer at all) one question but then don’t reply to followups. So I ask the big question first.
Opinions differ on which sort of answer is likelier to be on target.
Perhaps the hardest thing for us all to come to terms with – harder even than realizing nobody can give us a number – is realizing we don’t need a number.
Huh?
Regardless of the probability of a severe pandemic in the next five years, pandemic preparedness is a good investment.
Yes, but how much preparedness? Should we spend $7.1 billion or $71 billion or $70 million? So I disagree. I think we do need a number. It’s important.
This is true for three reasons.
First, just about everybody agrees that there will be more pandemics in the future.
Do you mean: more pandemics than in the past? Then I can’t see why. To the contrary, I’d assume we will have fewer. But presumably you mean: There will be some more pandemics, even if H5N1 will remain harmless (?). Then I think, yes with probability maybe 80% there will be at least one more pandemic.
I have no epidemiological expertise, but I think there is some chance that we will have virus diseases completely under control in some decades or centuries. We can get the virus-code now; we’ll probably have nanotechnology and improved genetics.
There may or may not ever be one as bad as 1918 was. But milder pandemics are inevitable, sooner or later.
I’m surprised. You seem to give a probability of 100% here! How careless….
Getting ready for a mild pandemic is a no-brainer.
Second, except for vaccines and antivirals, much of what we should do to get ready for a severe pandemic will also prepare us for other sorts of catastrophes – terrorist attacks, earthquakes, etc. Psychologically as well as logistically, disaster preparedness is largely generic.
Yes, but it’s more effective if the preparation is specific. Useless preparations have some positive side-effects, yet 70% or such of the effort spent on them is in vain.
Third and most important, the cost of preparedness is a very tiny fraction of the cost of being unprepared.
That depends on how much preparedness you do, of course. (Give a number!)
Assume a 30% for a mild pandemic with $1e11 economic loss, 10% for a 1918-like pandemic with $8e11 loss. (The Asian Development Bank assumes that preparing halves the impact. Then $5.5e10 is justified worldwide. Bush wants to spend $7e9 now.)
Nobody (well, almost nobody) is urging governments, businesses, or households to turn their priorities upside down getting ready for a severe pandemic that may never come.
Depends on the likelihood. In the USA you have the example of hurricanes. Should you evacuate when the hitting-probability is 30%? 50%? 80%?
Most are urging moderate, commonsense precautions, easily integrated into normal living. Figure it this way. Forgetting everything we know about H5N1, there has been one severe influenza pandemic in the past 500 years. Estimate how much damage a severe pandemic could do. Then plan on spending one-500th as much each year on preparedness.
Now we are where we were at the beginning of your post: “The problem is that this ignores what we know about the influenza virus of the moment, H5N1.”
My estimate of your estimate is: 1e7 H5N1 deaths in the next 5 years. (Projected 1918: 2e8 × 5 years / 500 years history would give 2e6. For the special current situation I multiply by 5.) Feel free to correct this.
Copyright © 2005 by by Guenter Stertenbrink and Peter M. Sandman
