Posted on 11/29/2009 7:58:10 AM PST by joinedafterattack
It has been much more difficult than you apparently comprehend to maintain long term data bases. I personally have data on written journals, punched paper tape, punched cards, 9-track (~12”) mag tape, 7-track tape, three different types of data tape cartidges, 8” floppy disk, 5-1/4 floppy (2 formats), 3 1/2 floppy, 100M zip drive, 50Mb hard drive “cartridge”, dozens of Winchester hard drives ranging from 10Mb to 500Gb, CD (3 formats), DVD (2 formats), several “solid state” formats. This does not by any means exhaust the possible formats.
The lab I last worked at has a large warehouse in which data from the past 25 years is stored on at least 8 of these types of formats. The manpower has never been available to transfer the older data to newer formats. Fortunately, people seldom ask for the older datasets since the new experimental results supersede older results in my field, and they are not dependent on those older results.
I AM NOT ABSOLVING CRU for dumping the tapes and records. What they have done here is unforgivable scientifically. They were given an international trust to maintain historical records of temperature, and were funded to do this. They should never have dumped this data, because they KNEW they were manipulating it in various ways for publication. They were TASKED to maintain this data! Many countries completely turned over all sorts of historical data they had collected trusting that CRU could and would do a better job of maintaining it than they could! That trust has been violated.
The preservation of historical data is a huge huge problem in science. The formats change so rapidly. Many are fairly volatile, especially those that depend on magnetics. The databases really ought to be rewritten periodically, but manpower is seldom available.
Consider the problem that Hubble researchers have: 5years, or 50 years, or possibly even 250 years from now, there is a supernova 20,000 light years from Earth. That star had never gained attention. However, over the years, that section of sky has been imaged thousands of times. Suddenly, astronomers want to look at its history. Very important questions depend on this type of history! There have been millions and millions of terabytes of information gathered, but now we want to sort through it to look at how a few dozen pixels have changed from time to time. What format is it being kept in? How do we maintain the data in reliable archives? How do we index it and find it and read it when needed? The money has been spent to do the research over the years. Has it all been wasted? Yes- if those archives are not available. However, good choices of assembling and maintaining that data is a very tricky problem, and I don't believe that problem has been adequately solved.
Your saying “that 20Tb of data would only take up one cu.ft.” is a horrible simplification of the problems we have with data storage.
Again: CRU grievously violated a trust they were tasked with, and I'll never forgive their horrible judgment. The FACT that they manipulated that data in ways that have never been well vetted or understood by the science and mathematics community SHOULD have made saving the raw historical data infinitely more important, and GOOD SCIENTISTS in charge at CRU would have made it clear how critical it was to save it.
AS SOON as good scientists had realized that raw data was gone, they would have NOTIFIED THE WORLD it had been lost. Efforts would immediately have been made to gather as primitive data as was still available. ALL papers that depend on manipulated data would contain cautionary notes of the uncertainty built in as a result of the data loss.
The way that Bible Scholars have had to work over the years is a good example of the way GOOD scholars would work when they KNOW that the original manuscripts are not available.
Dumping the data is unforgivable, but hiding that it was dumped, and pretending that loss of the original data is unimportant and the massaged data has higher intrinsic value is even worse.
Why 'climategate' won't stop greens
This is very surprising because Canadians are total moonbats where GW and man killing "nature" is concerned. I know people whose kids are so gone on the notion (thanks to brain washing in school) that they cry when someone steps on a bug. One neighbor's 11 yr. old daughter made her father participate in a funeral for a bug he stepped on 2 summers ago. And he went along with it. So the fact that this is in a Canadian newspaper is amazing. I am going to go read some of the comments and see what the reactions have been.
Mine are in a rented storage building because the IRS will demand proof of your investments from the date of purchase if they ever audit you.
I guess the IRS requirements are more strict than science which will change our lives forever?
About 13 years ago I worked on a study involving data from the late 70s that was on tapes. There was extensive data on about 20,000 individuals. The guy who authored the study kept the original tapes in his basement all that time and called in a favor to have it converted into ascii data. It was relatively trivial for me to convert it and load it up for comparison to a more recent data set.
It can be done if you’re even remotely responsible, let alone a government agency.
I can’t get into the comment section. Either the paper has disabled it or my refusal to accept cookies is the problem. If anyone can see them, please let me know.
Has anyone seen where it was mentioned how many sites and for how long data was collected?
This story just keeps gettin’ better and better...LOL!
And yet, the elitist smarter-than-us-average-bears POTUS still plans to participate in this foolishness in Europe...
< scoffs >
To: “Michael E. Mann”
Subject: IPCC & FOI
Date: Thu May 29 11:04:11 2008
Mike,
Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He’s not in at the moment – minor family crisis. Can you also email Gene and get him to do the same? I don’t have his new email address. We will be getting Caspar to do likewise. I see that CA claim they discovered the 1945 problem in the Nature paper!! Cheers Phil Prof. Phil Jones Climatic Research Unit
Maintaining databases of information during the last 30 years has been incredibly difficult, and often requires reformatting the data and re-saving to a different media every 5 years or so at great expense.
Some here have said “it's not expensive to store 20Tb of info.” Well, not today, but no more than 20 years ago, it cost $10/Mb for Disk Storage - so $2Million/20Tb - and that does not include the space for the 200,000 Winchester drives, the air conditioning to keep them running, cost of translating the paper, paper tape and mag tape records to format for the drives, or maintaining the indexing to keep it from being useless. It isn't the case that this wasn't ever converted to digital, either, but that much of that conversion would have been originally put on paper-tape or 9-track or 7-track mag tapes, and may never have been put on hard drives after that.
Again - I'm not forgiving the CRU people on this basis- I've addressed what I think of them in post#121.
I've often wondered why Harvard is so expensive if only smart people to go there. Logically it should be less expensive to train someone as smart as our POTUS.
“Doesn’t matter that they’ve violated every part of the scientific method.”
****
And how ironic is it that these “secular progressives” rely on science in order to reject religious morality and education.
They are completely shameful. If they had a conscience, they’d never show their face in public again. All of ‘em.
It is very suspicious, because Jones himself, in one of the released emails, said that he would destroy the data rather than release it on an FOI request. He should be in Jail.
Thanks for the reminder. I have to double check this: I believe that East Anglia (CRU) was tasked by the international community to gather and archive ALL country’s climate data, and that many of those countries turned all their raw records to CRU. IF this understanding of mine is correct, the raw data is gone gone gone, and can not be recovered, unless somebody somewhere has a 9-track or disk file that they copied a particular country or data station to while they were examining that particular part of the data.
Charlatans!
Estimated number of sites and data collected please. Then we can argue storage requirements. Strange it isn’t mentioned. It may not justify their claims.
How hard would it have been to record the original values right next to the revised values? If these were “inputs” to some process of revision then they certainly could have been recorded as easily as the outputs which we have now.
These folks at the CRU need to get into vote counting or something where their methods would be more respected.
lol
I understand the issues. I worked at a university where my duties included capital equipment inventory. I had regular conversations with researchers on why they were keeping equipment that was no longer actively in use - & it was usually because of compatibility issues & ability to retrieve old data.
CRU should certainly have put a call out to other researchers for assistance in maintaining this data. It should be treated as a scientific crime & every one of them should lose their positions!
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.