Uppyn's Blog

The world through Dutch eyes in Sydney

Monthly Archives: September 2013

What does 95% certainty mean?

When you read the news from the IPCC about the latest climate predictions it is hard to not be impressed by the confidence that the climate scientists have in their models. They make statements such as:

It is extremely likely that more than half of the observed increase in global average surface temperature from 1951 to 2010 was caused by the anthropogenic increase in greenhouse gas concentrations and other anthropogenic forcings together. The best estimate of the human induced contribution to warming is similar to the observed warming over this period.

And extremely likely in this case means 95% certainty. But I ask myself, what does this mean?

The IPCC has just published its fifth summary for policymakers (SPM) and a couple of weeks ago they have leaked the report from the so-called working group 1 (WG1 which looks at the physics of climate change). One would expect that these documents would contain an explanation of how this level of confidence was determined. And that there would be a section in the report that would provide insight in the calculation of these confidence intervals. But you would be disappointed.

Confidence interval

So what is the meaning of a confidence interval in statistics? Well, your gut feeling would tell you it must be something like this: suppose I am interested in the average weight of men older than 40 years in my suburb. I would then go out and collect data by weighing some of my neighbours and calculate their average weight. Some of my neighbours may think this is a bit strange so I may only get ten neighbours to agree with me. But in the end I have calculated an average weight of 75kg and I hope that this would tell me something about the weight of all men in my area. Off course, I did not measure all the men in my area and because of that, I am not certain of the average. If I would have measured all men, I would be absolutely certain. So I need to make an estimate of my confidence. In some way I have calculated a 95% confidence interval around this average and it turns out to be 71 to 78kg. Now what does that mean?

Well, it depends on which statistical theory you believe in. If you are a ‘frequentist’ then a confidence interval means something like this: if you repeat your experiment an infinite number of times, which is pretty hard to do, and you would collect all the data and calculate all infinite averages, the real and true average of the weight of men in my area would be between 71kg and 78kg, 95% of the time. This says nothing about the one experiment that I just did. Not very helpful, is it?

On the other hand, if you are a Bayesian statistician, the conclusion is a bit closer to home. These guys call this interval a credible interval and it means what we really want: the chance that the average from my experiment will be between 71kg and 78kg is 95% percent. This makes sense but unfortunately, the math is now a lot harder!

But for both interpretations, the confidence interval tells me something really important: I can never be totally confident about the average weight of the men in my area until I have measured them all. In other words, my confidence comes from the data that I collected!


But things get really tricky when it comes to climate. Unlike other disciplines, climate scientist cannot easily do experiments with the climate. They have to rely on computer models instead to make predictions. That means that there is no observational data to work with and to calculate averages from. So how do they do it?

The most common way goes like this: you get a bunch of very smart physicists and let them make a computer program that simulates the climate of the planet. This program will predict the average temperature of the planet. You let them run the computer models many times and every time you run it, you change some parameter of the model. For instance, you let it run with slightly different starting conditions. Or you let it run with some physical parameters set to a different value. As a result, every model run will be a little different. Then you ask your colleagues to make their own version of the models. And you let them run it many times with different parameters as well. If you are the IPCC you do this for more than 100 models. In the end, you pile all those climate model runs on top of each other and you see a kind of envelope emerging. This is called ‘ensemble runs’. The average of all runs then becomes the ‘most likely’ value of your prediction. (in reality, this is much more complex, e.g. the IPCC runs so-called scenarios for how they think carbon dioxide emission will be in the future). In this way, climate scientist work out how much variation there is to be expected in their predictions.

Because of the physics that goes into the models, and some of it is not at all well understood at the fundamental level, climate scientist claim they are able to separate natural variation (what the climate would do if we were not there) from man’s influence. They model, and publish in the reports, graphs that show how the world would have warmed (or not)  with and without man’s carbon dioxide emissions. In my humble view, there are big problems in this area. For example, the influence of clouds and aerosols (soot and dust) is not well understood. Climate sensitivity (how much will the planet warm if CO2 doubles) is another area of heated debate.

We are now getting closer to what we want. But first ask yourself, is this way of coming up with an estimate of variation a good way? It may be the only way, but suppose your model has a problem? For instance, it does not get the clouds right? And as most models are based on the same basic physics, it may well be that they all get the clouds wrong. That would mean that all your runs are suddenly suspect. What does that do you’re your estimate of variation?

What about the data?

But we have to move on! Now, what about the data? Yes, this is where the data comes in. There are 4 main data sets available for global temperatures and they are all very close. I now have my model runs and my data so I can start comparing. How does one do this? The math is somewhat involved but you could do a statistical test and ask yourself if the model predictions and the observations are the same. Actually, the temperature trends of the models and observations are compared because the trend is not so sensitive to year on year variation. And the trend is actually what we are interested in. For such a test you have to define the confidence level and it is customary to choose 95%.

Independent statisticians do such work (e.g. Lucia Liljegren or Steve McIntyre) and what they find is remarkable: the average of all those climate model runs is not consistent with observations with 95% confidence! In fact, they find that most models predict temperature trends that are much higher than what we observe. See the figure below.

GLB Temperature Trends 1979-2013

Figure 1: On the horizontal axis we see climate model runs and on the vertical axis we see the temperature trend. The red line is the average trend between 1979-2013 from HadCrut4 data (image source).

The box plots show the computer model ensemble runs. Note that the observed trend is well below the vast majority of the climate model predictions for the period mentioned!

Careful conclusion

To return to the IPCC statement, they are 95% confident that at least half of the warming between 1951 and 2010 was caused by humans. That is a lot of confidence given the challenges that are still out there.

What should we make of this? Off course, the IPCC speaks about the period from 1951 to 2010. And the data is clear, there has been warming in this period.  Probably about 0.8 of a degree. But when we look at the last 15 years it becomes clear that the tool of choice for the climatologists, computer simulations, makes predictions that are well above what is observed. If we would follow the scientific method, we are getting ourselves into serious problems. The scientific method requires us to test, through experiment or observation, if the real world behaves as predicted by our theory.  If our tests agree with the theory, our confidence increases, otherwise it decreases.

Now ask yourself again, how certain are you about this 95%?


Reading suggestions

Confidence intervals

Skeptical view:

Judith Curry

Climate Audit

Consensus view:

Real Climate


I am not a climate scientist, but an engineer who started to read up on climate change a couple of years ago after a friend started to ask some uncomfortable questions about the consensus view. Disturbingly, I found that there is a lot that cannot be explained by the consensus. My views on climate change can be characterized as follows: the basic physics of radiative transfer of heat through the atmosphere is correct and more CO2 will warm the planet, all else being the same. Where I disagree with the consensus is that this warming will be catastrophic (as far as we can tell today) and that it does not warrant draconian measures on the world’s economy. As Australian film maker Topher Field says about climate change mitigation measures in his 50 to 1 project: if they are expensive enough to be effective, they are unaffordable. If they are affordable, they are not going to be effective.