CRU's Source Code: Climategate Uncovered

American Thinker ^ | November 25, 2009 | Marc Sheppard

[neverdem]

As the evidence of climate fraud at the University of East Anglia’s prestigious Climactic Research Unit (CRU) continues to mount, those who’ve been caught green-handed continue to parry their due opprobrium and comeuppance, thanks primarily to a dead-silent mainstream media.  But should the hubris and duplicity evident in the emails of those whose millennial temperature charts literally fuel the warming alarmism movement somehow fail to convince the world of the scam that’s been perpetrated upon it, certainly these revelations of the fraud cooked into the computer programs that create such charts will.

First -- Let’s briefly review a few pertinent details.   

We reported on Saturday that among the most revealing of the “hacked” emails released last week was one dated November 1999, in which CRU chief PD Jones wrote these words to Hockey-Stick-Team leaders Michael Mann, Raymond Bradley and Malcolm Hughes:

“I’ve just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) amd (sic) from 1961 for Keith’s to hide the decline.”

Predictably, the suggestion of a climate-related data-adjusting “trick” being employed by such alarmist bellwethers 10 years ago instantly raised more than a few eyebrows.  And with similar alacrity, the Big Green Scare Machine shifted into CYA gear. 

Almost immediately after the news hit on Friday, Jones told Investigative Magazine’s TGIF Edition [PDF] that he “had no idea” what he might have meant by the words “hide the decline” a decade prior:

“They’re talking about the instrumental data which is unaltered – but they’re talking about proxy data going further back in time, a thousand years, and it’s just about how you add on the last few years, because when you get proxy data you sample things like tree rings and ice cores, and they don’t always have the last few years. So one way is to add on the instrumental data for the last few years.”

Baloney.

Mere hours later, Jones’s warmist soul mates at RealClimate offered an entirely different explanation:

“The paper in question is the Mann, Bradley and Hughes (1998) Nature paper on the original multiproxy temperature reconstruction, and the ‘trick’ is just to plot the instrumental records along with reconstruction so that the context of the recent warming is clear. Scientists often use the term “trick” to refer to “a good way to deal with a problem”, rather than something that is “secret”, and so there is nothing problematic in this at all. As for the ‘decline’, it is well known that Keith Briffa’s maximum latewood tree ring density proxy diverges from the temperature records after 1960 (this is more commonly known as the “divergence problem”–see e.g. the recent discussion in this paper) and has been discussed in the literature since Briffa et al in Nature in 1998 (Nature, 391, 678-682). Those authors have always recommend not using the post 1960 part of their reconstruction, and so while ‘hiding’ is probably a poor choice of words (since it is ‘hidden’ in plain sight), not using the data in the plot is completely appropriate, as is further research to understand why this happens.”

And later that day, Jean S at Climate Audit explained the reality of the quandary. In order to smooth a timed series, it’s necessary to pad it beyond the end-time. But it seems however hard they tried, when MBH plotted instrumental data against their tree ring reconstructions, no smoothing method would ever undo the fact that after 1960, the tree ring series pointed downward while the instrumental series pointed upward – hence the divergence:

“So Mann’s solution [Mike’s Nature Trick] was to use the instrumental record for padding [both], which changes the smoothed series to point upwards.”

So the author of the email claimed the “trick” was adding instrumental measurements for years beyond available proxy data, his co-conspirators at Real Climate admitted it was actually a replacement of proxy data due to a known yet inexplicable post-1960 “divergence” anomaly, and CA called it what it was – a cheat.

The next day, the UEA spoke out for the first time on the subject when its first related press-release was posted to its homepage.  And Jones demonstrated to the world the benefits a good night’s sleep imparts to one’s memory, though not one’s integrity:

“The word 'trick' was used here colloquially as in a clever thing to do. It is ludicrous to suggest that it refers to anything untoward.”

Tick Tock.

Of course, RealClimate also avowed there was “no evidence of the falsifying of data” in the emails.  But as Jones chose not to walk back his statement that the “tricks” were rarely exercised, and even assured us that he was “refer[ring] to one diagram – not a scientific paper,” his explanation remained at–odds with that of his virtual-confederates at RC.

And as Jones must have known at the time -- such would prove to be the very least of CRU’s problems.

Getting with the Green Program(s)

One can only imagine the angst suffered daily by the co-conspirators, who knew full well that the “Documents” sub-folder of the CRU FOI2009 file contained more than enough probative program source code to unmask CRU’s phantom methodology. 

In fact, there are hundreds of IDL and FORTRAN source files buried in dozens of subordinate sub-folders.  And many do properly analyze and chart maximum latewood density (MXD), the growth parameter commonly utilized by CRU scientists as a temperature proxy, from raw or legitimately normalized data.  Ah, but many do so much more. 

Skimming through the often spaghetti-like code, the number of programs which subject the data to a mixed-bag of transformative and filtering routines is simply staggering.  Granted, many of these “alterations” run from benign smoothing algorithms (e.g. omitting rogue outliers) to moderate infilling mechanisms (e.g. estimating missing station data from that of those closely surrounding).  But many others fall into the precarious range between highly questionable (removing MXD data which demonstrate poor correlations with local temperature) to downright fraudulent (replacing MXD data entirely with measured data to reverse a disorderly trend-line).

In fact, workarounds for the post-1960 “divergence problem”, as described by both RealClimate and Climate Audit, can be found throughout the source code.  So much so that perhaps the most ubiquitous programmer’s comment (REM) I ran across warns that the particular module “Uses ‘corrected’ MXD - but shouldn't usually plot past 1960 because these will be artificially adjusted to look closer to the real temperatures.”

What exactly is meant by “corrected” MXD, you ask?  Outstanding question -- and the answer appears amorphous from program to program.  Indeed, while some employ one or two of the aforementioned “corrections,” others throw everything but the kitchen sink at the raw data prior to output.

For instance, in subfolder “osborn-tree6mannoldprog” there’s a program (Calibrate_mxd.pro) that calibrates the MXD data against available local instrumental summer (growing season) temperatures between 1911-1990, then merges that data into a new file.  That file is then digested and further modified by another program (Pl_calibmxd1.pro) which creates calibration statistics for the MXD against the stored temperature and “estimates” (infills) figures where such temperature readings were not available.  The file created by that program is modified once again by Pl_Decline.pro, which “corrects it” – as described by the author -- by “identifying and “artificially” removing “the decline.” 

But oddly enough – the series doesn’t begin its “decline adjustment” in 1960 -- the supposed year of the enigmatic “divergence.”  In fact, all data between 1930 and 1994 are subject to “correction.”

And such games are by no means unique to the folder attributed to Michael Mann.

A Clear and Present Rearranger  

In 2 other programs, briffa_Sep98_d.pro and briffa_Sep98_e.pro, the “correction” is bolder by far. The programmer (Keith Briffa?) entitled the “adjustment” routine “Apply a VERY ARTIFICAL correction for decline!!” And he/she wasn’t kidding. Now, IDL is not a native language of mine, but its syntax is similar enough to others I’m familiar with, so please bear with me while I get a tad techie on you.

Here’s the “fudge factor” (notice the brash SOB actually called it that in his REM statement):

yrloc=[1400,findgen(19)*5.+1904]

valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,2.6,2.6,2.6]*0.75         ; fudge factor

These 2 lines of code establish a 20 element array (yrloc) comprised of the year 1400 (base year but not sure why needed here) and 19 years between 1904 and 1994 in half-decade increments.  Then the corresponding “fudge factor” (from the valadj matrix) is applied to each interval.  As you can see, not only are temperatures biased to the upside later in the century (though certainly prior to 1964) but a few mid-century intervals are being biased slightly lower.  That, coupled with the post-1930 restatement we encountered earlier, would imply that in addition to an embarrassing false decline experienced with their MXD after 1960 (or earlier), CRU’s “divergence problem” also includes a minor false incline after 1930.

And the former apparently wasn’t a particularly well-guarded secret, although the actual adjustment period remained buried beneath the surface.

Plotting programs such as data4alps.pro print this reminder to the user prior to rendering the chart:

“IMPORTANT NOTE: The data after 1960 should not be used.  The tree-ring density records tend to show a decline after 1960 relative to the summer temperature in many high-latitude locations.  In this data set this ‘decline’ has been artificially removed in an ad-hoc way, and this means that data after 1960 no longer represent tree-ring density variations, but have been modified to look more like the observed temperatures.”

Others, such as mxdgrid2ascii.pro, issue this warning:

“NOTE: recent decline in tree-ring density has been ARTIFICIALLY REMOVED to facilitate calibration.  THEREFORE, post-1960 values will be much closer to observed temperatures then (sic) they should be which will incorrectly imply the reconstruction is more skilful than it actually is.  See Osborn et al. (2004).'

Care to offer another explanation, Dr. Jones?

Gotcha

Clamoring alarmists can and will spin this until they’re dizzy.   The ever-clueless mainstream media can and will ignore this until it’s forced upon them as front-page news, and then most will join the alarmists on the denial merry-go-round.

But here’s what’s undeniable:  If a divergence exists between measured temperatures and those derived from dendrochronological data after (circa) 1960 then discarding only the post-1960 figures is disingenuous to say the least. The very existence of a divergence betrays a potential serious flaw in the process by which temperatures are reconstructed from tree-ring density.  If it’s bogus beyond a set threshold, then any honest men of science would instinctively question its integrity prior to that boundary.  And only the lowliest would apply a hack in order to produce a desired result.

And to do so without declaring as such in a footnote on every chart in every report in every study in every book in every classroom on every website that such a corrupt process is relied upon is not just a crime against science, it’s a crime against mankind.

Indeed, miners of the CRU folder have unearthed dozens of email threads and supporting documents revealing much to loathe about this cadre of hucksters and their vile intentions.  This veritable goldmine has given us tales ranging from evidence destruction to spitting on the Freedom of Information Act on both sides of the Atlantic. But the now irrefutable evidence that alarmists have indeed been cooking the data for at least a decade may just be the most important strike in human history.

Advocates of the global governance/financial redistribution sought by the United Nations at Copenhagen in two weeks and the expanded domestic governance/financial redistribution sought by Liberal politicians both substantiate their drastic proposals with the pending climate emergency predicted in the reports of the Intergovernmental Panel on Climate Change (IPCC).  Kyoto, Waxman-Markey, Kerry-Boxer, EPA regulation of the very substances of life – all bad policy concepts enabled solely by IPCC reports.  And the IPCC, in turn, bases those reports largely on the data and charts provided by the research scientists as CRU – largely from tree ring data -- who just happen to be editors and lead authors of that same U.N. panel.

Bottom line:  CRU’s evidence is now irrevocably tainted.  As such -- all assumptions based on that evidence must now be reevaluated and readjudicated. And all policy based on those counterfeit assumptions must also be re-examined.

Gotcha.  We’ve known they’ve been lying all along, and now we can prove it.  It’s time to bring sanity back to this debate. 

It’s time for the First IPCC Reassessment Report.

[steelyourfaith]

 

Beam me to Planet Gore !

[airborne]

So......does this mean we’re not all gonna die?? ;^)

[Just mythoughts]

bttt

[Berlin_Freeper]

Germany: Hamburg Swans Take Winter Retreat(Video)

[Swans get boat ride]

[Berlin_Freeper]

Germany: Hamburg Swans Take Winter Retreat(Video)

[airborne]

Document Not Found
Sorry, the requested document does not exist on this server.

[Revel]

A Very good article. I like it because it is one of the very few that I have seen that really deals with the code. I hope there are many more.

[airborne]

Now I see it.

City officials keep the winter pond from freezing over just for the swans and make sure they have enough to eat

WTF?

[Berlin_Freeper]

They got to keep the pond from freezing over.

(So they can run their swan concentration camp)

[airborne]

More like swan welfare!

[Rocky]

So the tree ring data after 1960 doesn’t match real temperature measurements. But apparently that didn’t stop these guys from using tree ring data as a source of global temperatures for before and after 1960. A reasonable person would question the usefulness of tree rings in determining global temperatures, since the period for which we have the most real data does not correlate well.

But if you’re a global warming fanatic, the tactic is to continue to use the tree ring data, and just falsify the post-1960 tree ring data by substituting real temperature measurements. Obviously, the older tree ring data is useful to them in making some other point. Perhaps making the medieval warming period disappear?

If I understand correctly what’s going on here, these guys need to be drummed out of the science community. And they should go to jail. The social impact of these liars has been tremendous. Hopefully, Copenhagen is a dead meeting now.

[steelyourfaith]

Agreed.

If only MSM would will not give this revelation of the hoax the props it deserves. Still, and in spite of the disingenuous MSM, more & more folks in the middle are becoming aware of the truth.

[Rocky]

It’s time for the First IPCC Reassessment Report.

No. It's time to dissolve the IPCC and stop funding every organization that is promoting AGW. The game's over.

[TChad]

I especially enjoyed the “fudge factor.” That’s a surefire way to make a graph look the way it should when the data won’t cooperate.

[The Magical Mischief Tour]

This is by far the best analysis of the data so far and the most damaging argument for the case that these shysters have taken part in one of the greatest conspiracies of all mankind.

[dila813]

Enough already, has this been reported to the FBI?

Who is the correct federal agency if not?

They have had Federal Grants and Contracts with the Federal Government and lied to Congress and others.

Has this been reported?

I have dropped this in the hopper for the FBI, but I would hope the authors would go down to the FBI and sit down with an agent.

This is getting beyond news stories.

[DB]

Tree rings have to be significantly affected by rain fall, temperature and CO2 levels. I suspect they don’t have a good way of untangling the various factors into accurate individual components.

So they just make stuff up...

[count-your-change]

“The word ‘trick’ was used here colloquially as in a clever thing to do. It is ludicrous to suggest that it refers to anything untoward.”

Yeah, I think I can understand that. It's like when a Mafia loan shark comes to collect the “vig” and says he's going to break legs. What he really means is that a strongly worded letter is going to be sent to the credit bureau.
Nothing untoward, eh?

[myknowledge]

[PA Engineer]

Thank you very much for the post. I finally get to resurrect all of my years of F77 education. There is much more in this code that I am slugging through. BTTT!

[Marie]

btt

[backhoe]

Mark my words, here:

http://www.freedominion.com.pa/phpBB2/viewtopic.php?p=1433905#1433905

Watch this space:

This?

"Does this mean we can have our incandescent light bulbs back?"

...is going to become the rallying cry that helps drag this swindle into the light of day.

And yes, it is a swindle- a fraud-- a deliberately designed construct intended to cheat you out of things you hold dear-- money, prosperity, freedom... and a few other things, as well.

...Watch this space...

---

I'm keeping a running file ( always updated ) on this fraud and scandal:

"Hadley CRU has apparently been hacked –[epic fraud?]"

Click the picture:

[SueRae]

They’re had. Even though there will be die hard warmers, including that SOB in office, the electorate is catching on. I want every school who insisted on showing the Goreacle’s crockumentary to review this material. Spread the word.

[Wonder Warthog]

"So the tree ring data after 1960 doesn’t match real temperature measurements."

It depends on which "real temperature measurements" they chose. Remember, the balloon and satellite temperature data also show either no warming or only a very slight warming. The ONLY contemporary temperature data that actually shows warming are those from the land-based weather stations, and that has already been show by McIntyre to have significant problems.

So it may actually be the case that the tree-ring data really DOES match real temperature measurements, and both sets show no warming.

[browardchad]

Excellent overview. Thanks for posting.

Perhaps CRU's next major publication should be entitled "The CRU Cookbook: 100 Fudged Recipes That Will Warm Your World."

[steveab]

Petition the Law Makers to Stop this Scam

It’s time to put this rest
The IPCC did their best
But no one is buying their BS
CO2 Lags Temperature
The ice has shown
The Hot Spot in the Atmosphere ran off with the Heat in the Oceans
And can’t be found
Hockey Stick is Broken and the MWP is UP Side Down
The 60’s were warmer than 2009. The trend line is against us The Team thundered
Then they sliced and diced until it’s under.
They said the debate was closed.
And they control the “Journals” you know
They hid the data with a frown.
If found out, The Team goes down
The Team’s goose cooked like they cooked the Books.
They’re no better than the Eron Crooks.
They’ll say they did it for prestige and pride
And, of course, a big slice of the money pie

So, make your case CO2, that “Evil Gas”, Rules the Climate if you can
Or declare CO2 a non pollutant
And take it out of EPA’s Hand

Petition the Law Makers to Stop this Scam

[Hardastarboard]

I’ve read on this very website, within the last two years, that the computer models are flawed. Now we have incontrovertible proof.

[Red Boots]

“The word ‘trick’ was used here colloquially as in a clever thing to do. It is ludicrous to suggest that it refers to anything untoward.”

Yada,yada, yada. One thing you can always count on the left to do is pretend that their words don't mean what they say. Its so childish !

[ml/nj]

Very important article!

(You who search for my stuff, and you know who you are: Be sure to read this.)

ML/NJ

[Rocky]

So it may actually be the case that the tree-ring data really DOES match real temperature measurements, and both sets show no warming.

That is another perspective. You make a good point.

Among the e-mails released in the recent scandal are a couple from a plant specialist expressing his concern to, I think it was Phil Jones, about the use of tree rings to measure temperatures. He was saying that tree rings are affected by several factors and he doubted that you could separate those effects to determine how much was due to temperature. Apparently, Jones never responded to his e-mails.

[ScottinVA]

What did Al Gore know, and when did he know it?

[ml/nj]

BTW, I am thinking of bringing this article to the attention of a left-thinking friend. I actually expect that he might know nothing of Climategate. So I actually looked at today's printed New York Times to get a sense of what someone who thinks himself well-read might know about this. (I haven't been looking at the printed NYT for a couple of years now after about 50 years of daily reading.) The answer is apparently nothing. I turned every page in the main news section. There was something about "7 Charged for False Building Documents," but nothing about anyone falsifying climate data. Noooo.

The things in this American Thinker piece are either facts or they are not. It's pretty easy to see why so many people are able to live in an alternate reality with their smug attitudes of superiority when they continue to trust the NYT to deliver the news to them.

I did a search just at nytimes.com for CRU. Several articles from past days do show up. All except two appear to be associated with "blogs." Maybe these other two were in the printed paper during past days, but if they were my guess is that they were buried someplace were a newsreader might likely miss them.

ML/NJ

[vamoose]

Has anyone noticed that wikipedia is pretty much devoid of mention of the CRU file release? No mention of it on the current “Hadley Centre” page. After being put in with a reference it was edited out. Ditto for Michael Mann’s page. The page “Hadley Centre for Climate Prediction and Research File Release Incident” has been deleted altogether. Phil Jones’ page does mention it, but downplays it at the same time.

hide the decline, hide the incident, hide the truth.

[ml/nj]

What did Al Gore know

Probably nothing. In Gore's defense, he appears to be exceptionally stupid.

ML/NJ

[TStro]

The tree ring data (cherry picked from tree rings that didn’t support their hockey stick, from trees in the Ural Mountains) was used to dispose of the “inconvenient” Medieval Warm Period and Little Ice Age.

[jdogbearhunter]

The man behind CRU, Phil Jones, FBA (FakeButAccurate)

[Wonder Warthog]

"He was saying that tree rings are affected by several factors and he doubted that you could separate those effects to determine how much was due to temperature."

Well, I've never studied up on how they determine temps using tree rings. I thought it was something like measuring the O16/O18 isotope ratio, which ought to be pretty independent of other factors.

[Unrepentant VN Vet]

Enter “Climate fraud” in Google.

Get 2,950,000 hits.

Methinks it’s a bit late for somebody out there. Dam’s plumb broke on this deal, and the enviroweenies are all running for the boats.

[neverdem]

Matthew Dowd at WaPo : Yes, she can: Palin has a shot at the presidency (and she could win too)

1919: Betrayal and the Birth of Modern Liberalism - Disillusionment with Woodrow Wilson changed the American Left forever.

This Day in History November 24,1871 National Rifle Association organized in New York City

NRA Fights Back Against Bloomberg's Gun Control Coalition

Some noteworthy articles about politics, foreign or military affairs, IMHO, FReepmail me if you want on or off my list.

[xcamel]

FReepmail me to get on or off

Ping me if you find one I've missed.

---

[Rocky]

I found the reference which discussed the tree rings:

http://www.powerlineblog.com/archives/2009/11/024995.php

Near the bottom of the article, there are two e-mails from a “environmental plant physiologist” named Don Keiller.

Here is one of them:

“Dear Professor Briffa, my apologies for contacting you directly, particularly since I hear that you are unwell. However the recent release of tree ring data by CRU has prompted much discussion and indeed disquiet about the methodology and conclusions of a number of key papers by you and co-workers.

“As an environmental plant physiologist, I have followed the long debate starting with Mann et al (1998) and through to Kaufman et al (2009). As time has progressed I have found myself more concerned with the whole scientific basis of dendroclimatology. In particular;

1) The appropriateness of the statistical analyses employed
2) The reliance on the same small datasets in these multiple studies
3) The concept of “teleconnection” by which certain trees respond to the “Global Temperature Field”, rather than local climate
4) The assumption that tree ring width and density are related to temperature in a linear manner.

“Whilst I would not describe myself as an expert statistician, I do use inferential statistics routinely for both research and teaching and find difficulty in understanding the statistical rationale in these papers. As a plant physiologist I can say without hesitation that points 3 and 4 do not agree with the accepted science.

“There is a saying that “extraordinary claims require extraordinary proof”. Given the scientific, political and economic importance of these papers, further detailed explanation is urgently required.”

[rightwingextremist1776]

bookmark

[JPG]

Can I use a ‘fudge factor’ when I balance my checkbook? If so, I’ll never be overdrawn.

[Wonder Warthog]

"4) The assumption that tree ring width and density are related to temperature in a linear manner."

Ye flippin' gods! Any grade-school educated farm boy could tell these guys that this is a ridiculous assumption. If this is what they're using, then I would trust their data not at all.

[Sally'sConcerns]

Might be better to contact Scotland Yard rather than the FBI since these emails originated out of a university in England. I don’t think the FBI has much jurisdiction there.

[AFPhys]

Agreed.

The EMails showed what I was already certain about.

This analysis of the code is the wooden stake in the heart of this vampire.

What concerns me is the tons and tons of data that has NOT yet been made public. There clearly is much code and data that is MISSING from this 198Mb data dump.

What other smoking guns and article massagings have been done by these “elite” Climatati??????

[dila813]

They had government contracts and grants, I am pretty sure they do or another federal agency.

[PeaceBeWithYou]

[editor-surveyor]

Back in the ‘70s I used to do this kind of manipulation with water quality data; they ain’t foolin’ me!