Free Republic Browse · Search General/Chat Topics · Post Article

Hell and High Histogramming – Mastering an Interesting Heat Wave Puzzle
Watts up with That? ^ | July 10, 2012 | Guest Post by Willis Eschenbach

Posted on 07/11/2012 11:22:45 AM PDT by Ernest_at_the_Beach

Anthony Watts, Lucia Liljegren , and Michael Tobis have all done a good job blogging about Jeff Masters’ egregious math error. His error was that he claimed that a run of high US temperatures had only a chance of 1 in 1.6 million of being a natural occurrence. Here’s his claim:

U.S. heat over the past 13 months: a one in 1.6 million event

Each of the 13 months from June 2011 through June 2012 ranked among the warmest third of their historical distribution for the first time in the 1895 – present record. According to NCDC, the odds of this occurring randomly during any particular month are 1 in 1,594,323. Thus, we should only see one more 13-month period so warm between now and 124,652 AD–assuming the climate is staying the same as it did during the past 118 years. These are ridiculously long odds, and it is highly unlikely that the extremity of the heat during the past 13 months could have occurred without a warming climate.

All of the other commenters pointed out reasons why he was wrong … but they didn’t get to what is right.

Let me propose a different way of analyzing the situation … the old-fashioned way, by actually looking at the observations themselves. There are a couple of oddities to be found there. To analyze this, I calculated, for each year of the record, how many of the months from June to June inclusive were in the top third of the historical record. Figure 1 shows the histogram of that data, that is to say, it shows how many June-to-June periods had one month in the top third, two months in the top third, and so on.

Figure 1. Histogram of the number of June-to-June months with temperatures in the top third (tercile) of the historical record, for each of the past 116 years. Red line shows the expected number if they have a Poisson distribution with lambda = 5.206, and N (number of 13-month intervals) = 116. The value of lambda has been fit to give the best results. Photo Source.

The first thing I noticed when I plotted the histogram is that it looked like a Poisson distribution. This is a very common distribution for data which represents discrete occurrences, as in this case. Poisson distributions cover things like how many people you’ll find in line in a bank at any given instant, for example. So I overlaid the data with a Poisson distribution, and I got a good match

Now, looking at that histogram, the finding of one period in which all thirteen were in the warmest third doesn’t seem so unusual. In fact, with the number of years that we are investigating, the Poisson distribution gives an expected value of 0.2 occurrences. In this case, we find one occurrence where all thirteen were in the warmest third, so that’s not unusual at all.

Once I did that analysis, though, I thought “Wait a minute. Why June to June? Why not August to August, or April to April?” I realized I wasn’t looking at the full universe from which we were selecting the 13-month periods. I needed to look at all of the 13 month periods, from January-to-January to December-to-December.

So I took a second look, and this time I looked at all of the possible contiguous 13-month periods in the historical data. Figure 2 shows a histogram of all of the results, along with the corresponding Poisson distribution.

Figure 2. Histogram of the number of months with temperatures in the top third (tercile) of the historical record for all possible contiguous 13-month periods. Red line shows the expected number if they have a Poisson distribution with lambda = 5.213, and N (number of 13-month intervals) = 1374. Once again, the value of lambda has been fit to give the best results. Photo Source

Note that the total number of periods is much larger (1374 instead of 116) because we are looking, not just at June-to-June, but at all possible 13-month periods. Note also that the fit to the theoretical Poisson distribution is better, with Figure 2 showing only about 2/3 of the RMS error of the first dataset.

The most interesting thing to me is that in both cases, I used an iterative fit (Excel solver) to calculate the value for lambda. And despite there being 12 times as much data in the second analysis, the values of the two lambdas agreed to two decimal places. I see this as strong confirmation that indeed we are looking at a Poisson distribution.

Finally, the sting in the end of the tale. With 1374 contiguous 13-month periods and a Poisson distribution, the number of periods with 13 winners that we would expect to find is 2.6 … so in fact, far from Jeff Masters claim that finding 13 in the top third is a one in a million chance, my results show finding only one case with all thirteen in the top third is actually below the number that we would expect given the size and the nature of the dataset …

w.

Data Source, NOAA US Temperatures, thanks to Lucia for the link.

TOPICS: Conspiracy; Science; Weather
KEYWORDS: climatechange; globalwarming; globalwarminghoax; heatwave

1 posted on 07/11/2012 11:22:51 AM PDT by Ernest_at_the_Beach

To: All
CNN ran with this story and quoted the US National Weather Service....FR thread:
2 posted on 07/11/2012 11:38:51 AM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: All
89 Responses to Hell and High Histogramming – Mastering an Interesting Heat Wave Puzzle

****************************************************************

I will look thru them and post a few.

3 posted on 07/11/2012 11:50:34 AM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: All

**************************************EXCERPT*******************************************

• As always, astounding reasoning, Willis. I find your conclusion flawless. I find that it also supports something that I have long suspected — few people are actually qualified to work with statistics or make statistical pronouncements. From what I recall, Jeff was only quoting some ass at NOAA, so perhaps it isn’t his fault. However, you really should communicate your reasoning to him. I think that there is absolutely no question that you have demonstrated that it is a Poisson process with significant autocorrelation — indeed, from the histogram (exactly as one would expect) and that as you say if anything it suggests that there have (probably) been other thirteen month stretches. It is also interesting to note that the distribution peaks at 5 months. That is, the most likely number of months in a year to be in the top 1/3 is between 1/3 and 1/2 of them!

Yet according the reasoning of the unknown statistician at NOAA, the odds of having any interval of 5 months in the top third are $(1/3)^5 \approx 4/1000$. They seem to think that every month is an independent trial or something.

Sigh.

rgb

4 posted on 07/11/2012 11:53:07 AM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: All
More:

*****************************EXCERPT***********************************************

Murray Grainger says:

July 10, 2012 at 10:28 pm

1 in 2.6 is close enough to 1 in 1.6 million for the average climate alarmist; what’s your beef? Nothing that a little data adjustment won’t fix.

5 posted on 07/11/2012 11:57:43 AM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: All
And a compliment to Willis:

**************************************EXCERPT********************************************

• captainfish says:

Willis, Can I please borrow your brain for a few days.. I could make a gazillion dollars with all that extra smarts and speed of thought. I can’t even comprehend the amount of work and effort that it even took to come up with the line of analysis, let alone sit down with the data. But then, I still don’t have your brain. But then, unfortunately, not many scientists do either.
Thank you, Sir.

6 posted on 07/11/2012 12:00:21 PM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: All
Chuckle....more:

**************************************EXCERPT******************************************

Steve R says:

July 10, 2012 at 11:14 pm

This whole 1 in 1.6 million issue has been great entertainment. It’s also been an eye opener, to see so many climate scientists struggling with a fairly basic statistical concept.

7 posted on 07/11/2012 12:03:43 PM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: All
Education tangent ....for me:

*************************************************************

**************************************EXCERPT*****************************************

In probability theory and statistics, the Poisson distribution (pronounced [pwasɔ̃]) is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time and/or space if these events occur with a known average rate and independently of the time since the last event.[1] (The Poisson distribution can also be used for the number of events in other specified intervals such as distance, area or volume.)

Suppose someone typically gets 4 pieces of mail per day. That becomes the expectation, but there will be a certain spread: sometimes a little more, sometimes a little less, once in a while nothing at all.[2] Given only the average rate, for a certain period of observation (pieces of mail per day, phonecalls per hour, etc.), and assuming that the process, or mix of processes, that produce the event flow are essentially random, the Poisson distribution specifies how likely it is that the count will be 3, or 5, or 11, or any other number, during one period of observation. That is, it predicts the degree of spread around a known average rate of occurrence.[2]

The distribution's practical usefulness has been explained by the Poisson law of small numbers.[3]

8 posted on 07/11/2012 12:10:29 PM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: All
Another chuckle from the comments:

**************************************EXCERPT************************************

Dave Wendt says:

July 10, 2012 at 11:44 pm

It would appear that inadvertently Mr. Masters, or whoever provided him with his numbers, has arrived at a ratio that is quite correct, the only problem being the ratio is applied to the wrong query. If you ask “what are the odds of a story about a human caused plague of horrendous heatwaves, which appears in any Lamestream Media source, NOT being complete BS?” the ratio of 1 in 1.6 million appears, to my eye at least, to be just about spot on.

9 posted on 07/11/2012 12:15:04 PM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: All
10 posted on 07/11/2012 12:29:20 PM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: Inyo-Mono; Sub-Driver; cripplecreek

fyi

11 posted on 07/11/2012 12:32:06 PM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: TigerLikesRooster; landsbaum; Signalman; NormsRevenge; steelyourfaith; Lancey Howard; ...
Not sure if anyone is checking this thread out....Statistics might be a BIG TURNOFF.

Anyway just letting you know about it.

12 posted on 07/11/2012 2:43:04 PM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: Ernest_at_the_Beach
Meanwhile, last week, in Peru the government was making a concerted effort to issue blankets out to the masses as they are experiencing lower then average temperatures in their early pangs of winter.
I forget where I read that article. Perhaps I should have posted it.
13 posted on 07/11/2012 3:44:46 PM PDT by Marine_Uncle (Honor must be earned.)

To: Ernest_at_the_Beach
RE: "Meanwhile, last week, in Peru the government was making a concerted effort to issue blankets out to the masses as they are experiencing lower then average temperatures in their early pangs of winter. I forget where I read that article. Perhaps I should have posted it."

TWANG!!!!! Now I remember where I read it.....
Here: Extreme winter weather in Peru: Government sending blankets to freezing people
14 posted on 07/11/2012 6:55:52 PM PDT by Marine_Uncle (Honor must be earned.)

To: Marine_Uncle
Got to have that posted here.

May I?... or do you want to?

15 posted on 07/11/2012 7:57:36 PM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: Marine_Uncle
I am waiting for a call back from my daughter on some urgent family business so it would be a while before I could do it....just as soon have you do it ....but we ought to get on here.

I think a lot of eyes see stuff here.

16 posted on 07/11/2012 8:02:49 PM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: Ernest_at_the_Beach

OK Ernest. I’ll post it to the general news forums.

17 posted on 07/11/2012 8:06:23 PM PDT by Marine_Uncle (Honor must be earned.)

To: Ernest_at_the_Beach

I can’t post it. It is a blog site. I tried. FR bounced me.

18 posted on 07/11/2012 8:17:52 PM PDT by Marine_Uncle (Honor must be earned.)

To: Marine_Uncle
Should go in this topic area....its why I post so much in it.

Just saves the headache.

19 posted on 07/11/2012 9:27:09 PM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: Ernest_at_the_Beach
I suppose one could extract the content of the article then post it as a vanity or something. Bottom line is I went through the whole front end process providing all required fields and FR bounced me due to recognizing it as a blog site.
I'm a bit rusty on posting tricks at this point.
20 posted on 07/11/2012 9:39:09 PM PDT by Marine_Uncle (Honor must be earned.)

To: Marine_Uncle
I can take alook at it in the morning.

Some websites are really making it difficult .

21 posted on 07/11/2012 9:56:05 PM PDT by Ernest_at_the_Beach (The Global Warming Hoax was a Criminal Act....where is Al Gore?)

To: Ernest_at_the_Beach

Good enough Ernest.

22 posted on 07/11/2012 10:14:41 PM PDT by Marine_Uncle (Honor must be earned.)

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

 Free Republic Browse · Search General/Chat Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794