CRU E-mails and data

[PersianPaladin]

What none of these "skeptic" idiots say is the simple fact that the acceleration in stratospheric cooling at our poles has been correllated with CO2 increases. The same correllation has been noted in the planet Venus given its very high atmospheric content of CO2 and high temperatures, and notably Venus' upper atmosphere is over 4 to 5 times colder than the upper atmosphere of Earth.

These emails, are a storm in a tea-cup and POSSIBLY orchestrated and over-hyped by the big capital-intensive special interests.

Edited November 24, 2009 by PersianPaladin

[Methuselah]

What none of these "skeptic" idiots say is the simple fact that the acceleration in stratospheric cooling at our poles has been correllated with CO2 increases. The same correllation has been noted in the planet Venus given its very high atmospheric content of CO2 and high temperatures, and notably Venus' upper atmosphere is over 4 to 5 times colder than the upper atmosphere of Earth.

These emails, are a storm in a tea-cup and POSSIBLY orchestrated and over-hyped by the big capital-intensive special interests.

Your last paragraph I agree with PP...But (IMO) all sceptics are not necessarily idiots. There are very many well-justified reasons for scepticism - not least, because it is inherent in the scientific method?

[PersianPaladin]

Your last paragraph I agree with PP...But (IMO) all sceptics are not necessarily idiots. There are very many well-justified reasons for scepticism - not least, because it is inherent in the scientific method?

Agreed. I apologise.

[Methuselah]

Agreed. I apologise.

No probs, mate...

[Thundery wintry showers]

I think that his post is ill-founded, to be honest. He starts off by describing the hack as a 'serious blow'. A what? In my view this is a tacit admission that he is, effectively, at 'war', or, at least in 'battle' with all those evil miscreants who dare to either question what he says, or, the worst thing of all: disagree with his views.

That doesn't strike me as someone searching for the truth - that strikes me (sorry - keeping with the 'blow' metaphor, here) as someone disseminating propaganda on the basis of an immutable dogma.

In my view, his writings court no favour for those who genuinely have concluded that CO₂ is dangerously high. And whatever anyone wants to say - it is a falsifiable hypothesis given that it is based on empirical and experimental findings (Arrhenius, 1896 & Langley, 1884) so such a conclusion deserves respect even though I think that the extent of the relationship to be an order of magnitude of overstatement.

Yes, I certainly agree.

[Chris Knight]

The "leak" or "hack" has nothing to do with AGW, the mechanism of greenhouse gas forcing, or of the role of forcing feedbacks. Thus the disclosure does not affect AGW theory per se.

Both the emails and the documents focus on the temperature record, particularly the instrumental station records and tree-ring proxies. Exactly the material that ClimateAudit has been after.

The station temperature record was what the FOI requests were about, and someone has gone through a lot of trouble to copy both the relevant documents and the emails, and to filter out much irrelevant stuff*.

It is almost as though someone was covering their back in case they were ordered to fall on their sword if the FOI request was granted (The request was rejected on the day after the last email in the file was sent). Someone had been putting this datafile together for some time, someone who does not fear the consequences of illegal disclosure as much as the consequences of withholding or destroying information requested from the FOI. In fact they were probably rather disappointed that the information was not going to see the light of day, and released it anyway. I wonder how they feel now? Exhilarated, or sick?

The emails are nothing unusual for an academic department, backbiting, snark and jealousy is daily fare in all the academic workplaces I have been employed in - but it is still normal to consider that emails are like letters or faxes, but they can come back to bite you, since they are inevitably archived somewhere. If they had just archived their emails in their CRUTEM database, they probably would have been safe forever.

The state of the databases is possibly why there has been a certain amount of reticence to publish raw station data - it is just not available in anything like a publishable form, since the code that has accrued over the years is not up to reconstructing the data, and the appalling state of documentation means that do do so will require major forensic reconstruction.

Gavin Schmidt states on RealClimate that he mostly does his own coding (in FORTRAN). It is the way within academia that without formal training, computers have become tools that academics use to produce their own data. Sometimes that goes beyond Excel, or Filemaker Pro on a Mac (or god forbid, Hypercard) and the old mainframe languages like FORTRAN or COBOL (in science or economics, in medicine, M (MUMPS)code is an old favourite, found throughout the NHS) are the only way to access earlier databases, the data often hard coded so that the data is not portable.

That is the problem facing "Harry" the programmer who has been trying to port the legacy CRUtem 2.1 data over to version 3.0, so it will run on Linux and Sun Alpha systems, with some code segments in FORTRAN77, other code in FORTRAN90. Here's an example of his frustration from HARRY_READ_ME.txt

I am seriously close to giving up, again. The history of this is so complex that I can't get far enough into it before by head hurts and I have to stop. Each parameter has a tortuous history of manual and semi-automated interventions that I simply cannot just go back to early versions and run the update prog.

I could be throwing away all kinds of corrections - to lat/lons, to WMOs (yes!), and more.

So what the hell can I do about all these duplicate stations? Well, how about fixdupes.for? That would be perfect - except that I never finished it, I was diverted off to fight some other fire. Aarrgghhh.

I - need - a - database - cleaner.

What about the ones I used for the CRUTEM3 work with Phil Brohan? Can't find the bugger!! Looked everywhere, Matlab scripts aplenty but not the one that produced the plots I used in my CRU presentation in 2005. Oh, FXXK IT. Sorry. I will have to WRITE a program to find potential duplicates. It can show me pairs of headers,

and correlations between the data, and I can say 'yay' or 'nay'. There is the finddupes.for program, though I think the comment for *this* program sums it up nicely:

' program postprocdupes2

c Further post-processing of the duplicates file - just to show how crap the

c program that produced it was! Well - not so much that but that once it was

c running, it took 2 days to finish so I couldn't really reset it to improve

c things. Anyway, *this* version does the following useful stuff:

c (1) Removes and squirrels away all segments where dates don't match;

c (2) Marks segments >5 where dates don't match;

c (3) Groups segments from the same pair of stations;

c (4) Sorts based on total segment length for each station pair'

You see how messy it gets when you actually examine the problem?

This time around, (dedupedb.for), I took as simple an approach as possible - and almost immediately hit a problem that's generic but which doesn't seem to get much attention: what's the minimum n for a reliable standard deviation?

I wrote a quick Matlab proglet, stdevtest2.m, which takes a 12-column matrix of values and, for each month, calculates standard deviations using sliding windows of increasing size - finishing with the whole vector and what's taken to be *the* standard deviation.

The results are depressing. For Paris, with 237 years, +/- 20% of the real value was possible with even 40 values. Windter months were more variable than Summer ones of course. What we really need, and I don't think it'll happen of course, is a set of metrics (by latitude band perhaps) so that we have a broad measure of

the acceptable minimum value count for a given month and location. Even better, a confidence figure that allowed the actual standard deviation comparison to be made with a looseness proportional to the sample size.

All that's beyond me - statistically and in terms of time. I'm going to have to say '30'.. it's pretty good apart from DJF. For the one station I've looked at.

Back to the actual database issues - I need a day or two to think about the duplicate finder.

That is one unhappy tale, and the 700Kb textfile describes three years of grief for "Harry". It also indicates what a mess the data is really in, and however much the CRU have spent on computer systems and model making, they need to spend more on teams of data analysts who can clean up the mess they have allowed to build up, instead of leaving it up to one individual.

They also need to honestly reassess the confidence they have in the long term record, including the instrumental station record, before making proclamations on how much we may expect to warm in the future. If the data that feeds the climate models is only slightly flawed, the output is absolutely worthless.

*For example, on the TWO forum Prof. Tom Choularton seemed almost to be disappointed that he had no emails represented, since he has been an external examiner for UEA CRU for many years, and certainly had exchanged many emails over the years with people there.

[Devonian]

That is one unhappy tale, and the 700Kb textfile describes three years of grief for "Harry". It also indicates what a mess the data is really in, and however much the CRU have spent on computer systems and model making, they need to spend more on teams of data analysts who can clean up the mess they have allowed to build up, instead of leaving it up to one individual.

Spend more? Haven't you read all the goings on about gravy trains and research grants? . Spend too little one lot of sceptics say they need to spend more on computer power, spend more and another lot start blathering on about gravy trains, grants 'our taxes'...

They also need to honestly reassess the confidence they have in the long term record, including the instrumental station record, before making proclamations on how much we may expect to warm in the future. If the data that feeds the climate models is only slightly flawed, the output is absolutely worthless.

You're speaking as a?

*For example, on the TWO forum Prof. Tom Choularton seemed almost to be disappointed that he had no emails represented, since he has been an external examiner for UEA CRU for many years, and certainly had exchanged many emails over the years with people there.

I didn't see that, but nice to know you're there. Under another guise or just lurking?

Edited November 24, 2009 by Devonian

[Chris Knight]

Spend more? Haven't you read all the goings on about gravy trains and research grants? . Spend too little one lot of sceptics say they need to spend more on computer power, spend more and another lot start blathering on about gravy trains, grants 'our taxes'...

You're speaking as a?

I didn't see that, but nice to know you're there. Under another guise or just lurking?

I don't believe all that you read, Dev. If Prof Jones can get £14,492.00 for "CRU involvement in the development of an improved global historic surface temperature dataset" for a 9-month project in 2004, I am sure he had the clout to get say £120,000.00 for "Essential upgrade to core surface station temperature database" for 12 months - covering the salaries of 3 data analysts. Especially after reading his programmer's reports. Has the Prof no heart at all?

So I think they should spend more on cleaning up the existing core data, speaking as one who has spent most of my working life supporting academic research, either doing the practical lab work or as an analyst of research data (but not at UEA).

It's nice to see you there too! I don't post much there - it is somehow like when once walking into a backstreet pub in Mortlake - the conversation stops and all eyes follow you, and you get the feeling that they all suspect you of being a policeman in plain clothes. :unsure:

[Devonian]

.

It's nice to see you there too! I don't post much there - it is somehow like when once walking into a backstreet pub in Mortlake - the conversation stops and all eyes follow you, and you get the feeling that they all suspect you of being a policeman in plain clothes.

LOL, I know what you mean, but it's better than that, if somewhat surreal at times...

Edited November 24, 2009 by Devonian

[shoreham]

The state of the databases is possibly why there has been a certain amount of reticence to publish raw station data - it is just not available in anything like a publishable form, since the code that has accrued over the years is not up to reconstructing the data, and the appalling state of documentation means that do do so will require major forensic reconstruction.

Gavin Schmidt states on RealClimate that he mostly does his own coding (in FORTRAN). It is the way within academia that without formal training, computers have become tools that academics use to produce their own data. Sometimes that goes beyond Excel, or Filemaker Pro on a Mac (or god forbid, Hypercard) and the old mainframe languages like FORTRAN or COBOL (in science or economics, in medicine, M (MUMPS)code is an old favourite, found throughout the NHS) are the only way to access earlier databases, the data often hard coded so that the data is not portable.

That is the problem facing "Harry" the programmer who has been trying to port the legacy CRUtem 2.1 data over to version 3.0, so it will run on Linux and Sun Alpha systems, with some code segments in FORTRAN77, other code in FORTRAN90. Here's an example of his frustration from HARRY_READ_ME.txt

That is one unhappy tale, and the 700Kb textfile describes three years of grief for "Harry". It also indicates what a mess the data is really in, and however much the CRU have spent on computer systems and model making, they need to spend more on teams of data analysts who can clean up the mess they have allowed to build up, instead of leaving it up to one individual.

They also need to honestly reassess the confidence they have in the long term record, including the instrumental station record, before making proclamations on how much we may expect to warm in the future. If the data that feeds the climate models is only slightly flawed, the output is absolutely worthless.

As I pointed out in an earlier post this is the crux of the whole matter

There is nothing intrinsically wrong with legacy architecture or writing programs in the old programming languages such as FORTRAN so long as proper attention is paid to system design, code reviews, testing, documentration etc. Throughout the cold war much of the nuclear defence of the Western world depended on the early warning systems that largely ran on mainframes running COBOL and old CODAYSL IDMS databases all of which are extremely reliable if used properly. The real issue here is how the raw data was vetted, evaluated. stored and modified over time. It is clear that this has been done in a haphazard and uncontrolled manner with almost no configuration management or audit trails. This means that unless the original source readings or data files have been retained there must be doubts over its reliabilty, particulary as it seems from 'Harry's' comments that the basic structure of the original database was not underpinned by the proper use of referential integrity or primary and foreign key constraints. As a consequence it is quite easy for data in different tables or files to become corrupted or out of sync. If the data is shot then it does not matter how brilliant are the rest of the coding alogorithms since once GIGO takes effect all results must be regarded as suspect.

Gordon Manley must be spinning in his grave.

Edited November 24, 2009 by shoreham

[VillagePlank]

Wow - I missed the 'Harry' emails.

Certainly seems like someone, somewhere, needs to get on the fund-raising bandwagon and move all the data into a relational database. At least that way, there is also scientific papers backing up the theory of design and recording of data (Codd, 1970) and the design of writing and reading data is not left to some non-expert who hasn't got a clue what two-phase commit and ACID actually means.The business world rely heavily on ACID and have done for a very long time, now.

OK - a bit strong, but it's late and the Old Speckled Hen is kicking in - and I can't be arsed to re-write it so that it is less strong. So - apologies in advance!

EDIT: I feel very very sorry for 'Harry'

Edited November 24, 2009 by VillagePlank

[Chris Knight]

As I pointed out in an earlier post this is the crux of the whole matter

There is nothing intrinsically wrong with legacy architecture or writing programs in the old programming languages such as FORTRAN so long as proper attention is paid to system design, code reviews, testing, documentration etc. Throughout the cold war much of the nuclear defence of the Western world depended on the early warning systems that largely ran on mainframes running COBOL and old CODAYSL IDMS databases all of which are extremely reliable if used properly. The real issue here is how the raw data was vetted, evaluated. stored and modified over time. It is clear that this has been done in a haphazard and uncontrolled manner with almost no configuration management or audit trails. This means that unless the original source readings or data files have been retained there must be doubts over its reliabilty, particulary as it seems from 'Harry's' comments that the basic structure of the original database was not underpinned by the proper use of referential integrity constraints. As a consequence it is quite easy for data in different tables or files to become corrupted or out of sync. If the data is shot then it does not matter how brilliant are the rest of the coding alogorithms since once GIGO takes effect all results must be regarded as suspect.

Gordon Manley must be spinning in his grave.

There is nothing wrong with properly maintained legacy systems that have been designed and properly managed, as you say Shoreham (just down the road from me?), but they do need to be upgraded to run seamlessly on current hardware and modern OSs, etc. It is possible to get to the state where one Monday morning, all you have is a set of backup tapes, and nothing to read them with.

It seems that the UEA CRU CRUTEM so-called "database" has grown organically with less than ideal input from a variety of idiosyncratic academic coding sources over time, and it is now a nigh impossible task to salvage a definitive clean dataset.

[VillagePlank]

There is nothing wrong with properly maintained legacy systems that have been designed and properly managed, as you say Shoreham (just down the road from me?), but they do need to be upgraded to run seamlessly on current hardware and modern OSs, etc. It is possible to get to the state where one Monday morning, all you have is a set of backup tapes, and nothing to read them with.

It seems that the UEA CRU CRUTEM so-called "database" has grown organically with less than ideal input from a variety of idiosyncratic academic coding sources over time, and it is now a nigh impossible task to salvage a definitive clean dataset.

Yup - redesign using the relational model (which shouldn't be too hard since the domain experts should be to hand) and then the biggest task is writing the data import/export routine and verifying the run. Abstract what you actually want from the data; so some fun long weekends figuring out linear and non-linear algebra and expressing them in SQL. A fun and interesting project, I reckon.Most RDBMS' also come with a StdDev, and StdDevP functions already built in, too - it might save Harry some angst!

select site, stdev(mytempseries) from mydataset group by site

.... and, furthermore, most RDBMS's will run on commodity hardware!

Edited November 24, 2009 by VillagePlank

[AtlanticFlamethrower]

Now a petition that some people might like to sign

http://petitions.number10.gov.uk/UEACRU/

[osmposm]

Yup - redesign using the relational model (which shouldn't be too hard since the domain experts should be to hand) and then the biggest task is writing the data import/export routine and verifying the run. Abstract what you actually want from the data; so some fun long weekends figuring out linear and non-linear algebra and expressing them in SQL. A fun and interesting project, I reckon.Most RDBMS' also come with a StdDev, and StdDevP functions already built in, too....

This is not a complaint, VP...Chris...Shoreham - lord knows, I am glad there are people out there who understand it - but I have to confess that you could all be talking Assyro-Babylonian underwater for all the sense I can make of those last few posts :o .

What worries me is that there are probably plenty of people on here who find my verbose, vocabulary-vomiting, literary-referenced, subordinate-clause-ridden essays just as inpenetrable....

Edited November 25, 2009 by osmposm

[osmposm]

Ossie,your prodigious talents as a wordsmith are well-noted,certainly by myself,and I freely admit to being a little envious at your seemingly effortless flow. I guess the reason many of us (though I note you generally single out the sceptics/deniers) link to commentary by others who have achieved the position they are in,is because most of us lack the word-skills they have,though they encapsulate neatly what we could never articulate. I see nothing wrong with it,it is afterall no more than introducing others to articles/commentaries and opinion that may otherwise have gone overlooked. Perhaps you are concerned that readers might be 'turned'? I don't object to how much pro-AGW stuff is linked to - it's up to me what I make of it. I appreciate those who wish to study purely the science aspect,but it can't be escaped that AGW/politics are so closely entwined as to be inseparable wrt 'what has to be done'. Hence,all the political content.

I partly take your point, Barrie, and I think what I wrote was unfair. I'm sorry. I am also very flattered (I think - see my post above!) by the first bit, though, believe me, effortless it ain't...I often spend a couple of hours or more on a long (overlong?) post, and when I'm backing it up with stats analysis it can take me several days, on and off. I am lucky to have the time to spend on it, though I often wish I didn't - both having the time, that is, and choosing to spend it on Netweather rather than on some more sensible activity like earning a living (or sleeping).

But the key to the complaint was my phrase "vehement and entrenched opinion pieces". It's not the reasonably-well-argued-with-substantial-scientific-input stuff I have a problem with - ClimateAudit, say or even WattsUp - it's the almost purely political rants, often on hard-right websites, that don't seem to add much to the discussion.

The impression I have - and even as I write this I know that the impression may well be biased - is that convinced "antis" (and it would be foolish not to include you here) tend to link more to non- or pseudo-science blogs, editorial and Op-ed pieces than convinced (or broadly) "pros". Personally I don't like it when either side does, but I don't often see on here links to (in my opinion) patently leftist AGW-supporting political pieces - and the few that do appear are ironically usually posted by "antis" to show how ridiculous the "pros" are getting!

You will probably reply that the "broadly or totally pros" (I hate these terms, sorry, they paint an unreal picture) don't need to post such links, as almost everything written by the press and government is already hopelessly biased in their direction. There would be much truth in that.

All I will say is that I am not in the least worried that somebody with an open and even slightly scientific mind would be "turned" by such pieces, because I think they are so obviously political/partisan, and have little or nothing persuasive in the way of even elementary layman's science to support them. But then I would say that, wouldn't I?

Edited November 25, 2009 by osmposm

[winterfreak]

Hi guy's, long time lurker....Very interesting this last week eh? As i'm not nearly as knowledgeable as most of you on here, i thought i would lighten the mood just a little.....

Saw this on YOUtube....

http://www.youtube.com/watch?v=nEiLgbBGKVk&feature=player_embedded

Enjoy!!!

[VillagePlank]

This is not a complaint, VP...Chris...Shoreham - lord knows, I am glad there are people out there who understand it - but I have to confess that you could all be talking Assyro-Babylonian underwater for all the sense I can make of those last few posts .

I guess the only help I'd be able to give is to start here: Codd, 1970; but you may need a book on set-theory to get the most out of it.

Edited November 25, 2009 by VillagePlank

[Chris Knight]

This is not a complaint, VP...Chris...Shoreham - lord knows, I am glad there are people out there who understand it - but I have to confess that you could all be talking Assyro-Babylonian underwater for all the sense I can make of those last few posts .

What worries me is that there are probably plenty of people on here who find my verbose, vocabulary-vomiting, literary-referenced, subordinate-clause-ridden essays just as inpenetrable....

I try not to technobabble, Osmposm, honest I do, but there are times when you have to use the jargon for precision and brevity. I will try to provide a glossary in future, but there is always Google.

"Legacy systems" means old software running on old computers and accessories - software that is likely not to run properly on current systems - many modern computers have replaced serial and parallel ports with USB (Universal Serial Bus) ports, so old accessories like backup tapes may not be readily attached to new computers. The drivers and backup installation software may be on floppy disks - when can you find floppy disk drives on new computers nowadays? Unix-type operating systems (like Linux distributions) may be more immune to these situations than Windows Operating Systems(OSs), which seem to exclude some legacy software and hardware with each version change.

VP, last night, admitted to running on OS H*, which by name implies that it is a legacy system and is only fully understood by others running on the same OS! I could make out a few phrases he wrote, that is all.

*Old Speckled Hen

edited for missed parenthesis

Edited November 25, 2009 by Chris Knight

[Essan]

Now a petition that some people might like to sign

http://petitions.number10.gov.uk/UEACRU/

Certainly not! I don't believe in hysterical mob rule, disinformation or the Inquistion!

[jethro]

This is not a complaint, VP...Chris...Shoreham - lord knows, I am glad there are people out there who understand it - but I have to confess that you could all be talking Assyro-Babylonian underwater for all the sense I can make of those last few posts .

You're not alone Ossie....

What worries me is the "fun long weekends" to sort it out. Fun?????? Really??

[VillagePlank]

You're not alone Ossie....

What worries me is the "fun long weekends" to sort it out. Fun?????? Really??

Consider it like cleaning out an old room that you haven't been in for years; going through old photos, books, trashing some of the stuff, finding the good stuff - and then redecorating the room with a complete new approach. Still, the same room, still the same capacity - but it's clean and tidy, and you're not falling over old junk to open the window to let some fresh air in.

*Note to self - don't talk work

[jethro]

Just teasing VP.

I admire anyone who can understand all that stuff, I've tried and failed miserably - there's something deep inside my brain which screams "Noooooooo, get me out of here". I blame my old Maths Master, a very clever man indeed who used to wander silently through class waiting for an opportunity to catch you day dreaming out the window; bringing his yard stick down heavily and silently from behind on your desk, missing fingers by millimetres. Computer babble, logarithms, algebra...my brain's on the lookout for Mr. Easterbrook.

[johnholmes]

logarithms, algebra

goodness me are you old enough to have been taught those things chuck?

remember the days when some of us could remember what tan, sine, cosine 30, 45, and 60 were without looking it up in log tables-log tables-grief--I must have a lie down!

[jethro]

goodness me are you old enough to have been taught those things chuck?

remember the days when some of us could remember what tan, sine, cosine 30, 45, and 60 were without looking it up in log tables-log tables-grief--I must have a lie down!

Yup.

Had a slide rule too, remember those?