Y2K Bug in NASA Climate Data

[The PIT]

Actually I was joking about the conspiracy theory however it makes you wonder accurate the data is world wide as well.

[pottyprof]

Sorry P3. My views remain unchanged. Interesting read though.

[Essan]

Who has claimed the mistake was intentional?

Effectively everyone who has tried to make an issue of this. Otherwise why even mention it?

Bearing in mind the combined difference is only 0.02c ........

[gmoran]

The data is about as good as they can get it. Doesn't mean there aren't sometimes problems. Rarely are these significant. This adjustment makes very little - sorry - no difference to the climate trends of the past hundred years.

:)P

Parmiendes

while AF is over egging the significance, I think you and Stratos are missing some context here when you reply to him. ClimateAudit were going after the GISS stats because there were concerns about weather stations and adjustment methodologies. There was the predictable response of not sharing methodologies. CA carried on analysing regardless and discovered the error. There are quite a few conclusions that can be drawn from this, and none of them are particularly good for AGW activists ( and I don't mean AGW isn't real).

AGW science is driving policy decisions. The fact that climate scientists are frequently secretive with their data sets and methodologies is a scandal. The issues are too important.

[AtlanticFlamethrower]

Effectively everyone who has tried to make an issue of this. Otherwise why even mention it?

Bearing in mind the combined difference is only 0.02c ........

Which planet's news do you read?

Steve McIntyre, who discovered the bug, did not say it was intentional. Those who believe that must be very few and far between which is why I'm incredulous you have apparently read so much about them.

[Gray-Wolf]

With plunges and plumes doing their best (through the mid latitudes) to facilitate the slow warming of the north I'd be very easy to arrive at an 'average temp' which did not stray too far from the 'norm' though the climate could have been as extremely cold as it was extremely warm (or just a few 'extremes' to skew things) yet we're still quibbling over nth's of a degree! It isn't about averages guys, it's about extremes.

You would do as well to check out both the extinction (local) and population migration for the cold weather critters that rely upon the temp regimes they evolved to exploit.

No argument about what has occurred there (is there?).

[AtlanticFlamethrower]

What was the last ice age if it wasn't extreme?

The US temperature anomaly record shows ups and downs, no global warming. Sorry.

Despite this I do not rule out the possibility the world could get warmer, or for that matter colder.

[parmenides3]

Parmiendes

while AF is over egging the significance, I think you and Stratos are missing some context here when you reply to him. ClimateAudit were going after the GISS stats because there were concerns about weather stations and adjustment methodologies. There was the predictable response of not sharing methodologies. CA carried on analysing regardless and discovered the error. There are quite a few conclusions that can be drawn from this, and none of them are particularly good for AGW activists ( and I don't mean AGW isn't real).

AGW science is driving policy decisions. The fact that climate scientists are frequently secretive with their data sets and methodologies is a scandal. The issues are too important.

There are two possible responses to why scientists might be reluctant to share data: firstly, because they spent a long time getting it and want to keep hold of it so they can do more work, and secondly, because the institutions they work for often 'own' the copyright to the data anyway, and won't sanction its release.

But TBH, I think this is a misrepresentation. In almost all the papers I read, methodology is set out very precisely; the papers wouldn't be accepted by a journal if it wasn't, therefore, this is readily available to anyone who studies the papers. Secondly, as far as I can tell, most of the key data used in climate modelling or measurement is publically available; you can download much of it yourself from the NASA or NOAA-affiliated organisation websites. Sometimes, the LBL codes or programming protocols are kept under wraps, but this is no more than any programmer does. The claim that material is being 'kept secret' is seen as significant only if you believe in the first place that some kind of fraud or conspiracy is taking place to deceive us; otherwise, it is of no significance anyway.

Have you actually read the technical pages supporting the WG1 SPM? The content is filled with detail. There is no evidence of anything being 'kept back'. And this is the simple version; on the PCMDI site there are links to 600+ papers, all filled with details, which were used in the collation of the latest summaries. Where's the secrecy?

Respectfully,

:)P

[gmoran]

There are two possible responses to why scientists might be reluctant to share data: firstly, because they spent a long time getting it and want to keep hold of it so they can do more work, and secondly, because the institutions they work for often 'own' the copyright to the data anyway, and won't sanction its release.

But TBH, I think this is a misrepresentation. In almost all the papers I read, methodology is set out very precisely; the papers wouldn't be accepted by a journal if it wasn't, therefore, this is readily available to anyone who studies the papers. Secondly, as far as I can tell, most of the key data used in climate modelling or measurement is publically available; you can download much of it yourself from the NASA or NOAA-affiliated organisation websites. Sometimes, the LBL codes or programming protocols are kept under wraps, but this is no more than any programmer does. The claim that material is being 'kept secret' is seen as significant only if you believe in the first place that some kind of fraud or conspiracy is taking place to deceive us; otherwise, it is of no significance anyway.

Have you actually read the technical pages supporting the WG1 SPM? The content is filled with detail. There is no evidence of anything being 'kept back'. And this is the simple version; on the PCMDI site there are links to 600+ papers, all filled with details, which were used in the collation of the latest summaries. Where's the secrecy?

Respectfully,

:)P

The whole point of climateaudit is that climate scientists are not sharing datasets and methodologies.

The whole point about the GISS revisions is that CA asked for the data and were told no, they asked for the methodology for adjusting for weather stations to account for urbanisation effects and were toild no. So CA did their own digging and found the discrepencies that have now caused the GISS record to be changed.

This quote from Judith Curry at real climate highlights the issue:

Last week, Bush signed a bill on “America COMPETES”. There is a relevant part on the open exchange of data and metadata:

“SEC. 1009. RELEASE OF SCIENTIFIC RESEARCH RESULTS.

(a) Principles- Not later than 90 days after the date of the enactment of this Act, the Director of the Office of Science and Technology Policy, in consultation with the Director of the Office of Management and Budget and the heads of all Federal civilian agencies that conduct scientific research, shall develop and issue an overarching set of principles to ensure the communication and open exchange of data and results to other agencies, policymakers, and the public of research conducted by a scientist employed by a Federal civilian agency and to prevent the intentional or unintentional suppression or distortion of such research findings. The principles shall encourage the open exchange of data and results of research undertaken by a scientist employed by such an agency and shall be consistent with existing Federal laws, including chapter 18 of title 35, United States Code (commonly known as the `Bayh-Dole Act’). The principles shall also take into consideration the policies of peer-reviewed scientific journals in which Federal scientists may currently publish results.”

As a climate researcher, I wholeheartedly support the above principles. In my opinion research scientists (and particularly government research scientists) should not be given any “choice” in this matter if they wish to receive government research funding, publish their research in the peer reviewed journals of the major professional societies, and have their data used in assessment reports.

Yes all this adds to the cost of doing research, and even the COMPETE bill is apparently an unfunded mandate. But it’s a cost we need to accommodate in some way. I have seen too many examples in the climate field where scientists do not want to make their data and metadata available to skeptics such as Steve McIntyre since they don’t want to see their research attacked (and this has even been condoned by a funding agency). Well, in the world of science, if you want your hypotheses and theories to be accepted, they must be able to survive attacks by skeptics. Because of its policy importance, climate research at times seems like “blood sport.” But in the long run, the credibility of climate research will suffer if climate researchers don’t “take the high ground” in engaging skeptics.

With regards to Steve McIntyre and climateaudit. In the early days of McIntyre’s attacks on the “hockey stick”, it was relatively easy to dismiss him as an industry “stooge.” Well, given his lengthy track record in actually doing work to audit climate data, it is absolutely inappropriate in my opinion to dismiss him. Climateaudit has attracted a dedicated community of climateauditors, a few of whom are knowledgeable about statistics and are interesting thinkers (the site also attracts “denialists”). For all the auditing activity at climateaudit, they have found relatively little in the way of bonafide issues that actually change something, but this is not to say that they have found nothing. So taking the high ground, lets thank Steve and climateauditors if they actually find something useful, assess it and assimilate it, and move on. Such actions by climate researchers would provide less fodder for the denialists, in my opinion.

Comment by Judith Curry — 11 August 2007 @ 8:44 AM

[parmenides3]

I don't think it's quite right to say that this is 'the whole point' of CA. It was originally started to work as a mouthpiece for Steve McIntyre's critique of the 'Hockey Stick', in the same way that RC exists to some extent because Mann wanted a mouthpiece to respond to the criticisms of his work.

I also think you need to distinguish McIntyre himself, and maybe two or three others who 'inhabit' the site, from the majority of posters/commenters. Curry's comments about the few doing some important but ultimately not very revealing analysis of a small number of datasets or calculations; this sounds about right. The vast majority of what you get on CA is mutual hand-holding by uninformed denialists and sceptics bleating about how everything proves that they are right and everyone else is wrong. As Curry observes, though, this doesn't mean that CA has found nothing.

I'm not defending GISS's actions above, only trying to understand myself why some scientists might want to hold on to their own work, in particular, if they think that someone is going to come along a peck holes in it. But Curry is probably right on two key points; publically funded work probably should be publically available, but this cuts both ways; the US administration can be very recalcitrant when it comes to publishing material which reaches politically unfavourable conclusions, some of which may well be of more import than the GISS dataset which, after all, is not the be-all and end-all of climate research.

To say that CA were 'concerned about weather stations and adjustment methodologies' might be a bit naive. It wasn't McIntyre who first pointed to the fact that some observations might contain biases - credit here should go to Roger Pielke Sr. - and, in the end, this wasn't what they have been working on; the alteration was to a previously missed mathematical error which had resulted in a small discrepancy in the comparative records. And McIntyre should be applauded for finding this and drawing it to NASA's attention. CA's track record is that it seeks actively for numerical or statistical imperfections in a small number of datasets or analyses which support the conclusion og AGW, then seek to undermine the conclusions by attempting to demonstrate that this invalidates the work. The purpose of this is to discredit AGW science and provide people with a reason to believe that somehow the whole global warming thing is a fake problem and will all go away.

Steve himself does not deny GW, nor does he (in private, at least) challenge the vast majority of climate science. CA's work, when it is worthwhile, which is only occasionally, has not, in spite of much effort, actually resulted in any findings which substantively change the scientific picture of what is going on, or what is likely to happen. Therefore, whilst it can be a useful resource for someone who wishes to seek evidence for their own belief that AGW is not as much of a problem as most scientists seem to think it is, in scientific terms, it is a side player.

When I attempted to give an explanation why GISS might be reluctant to pass over its data, I was really responding to your comment that for it not to do so was a 'scandal'; I think this is too strong. To argue instead that it is not, in principle, an acceptable response, is a fair point, on which Curry's comment speaks eloquently. To suggest that it is all that important is, perhaps, to overstate the significance of the work.

I hope this helps explain my POV.

:)P

[gmoran]

CA's track record is that it seeks actively for numerical or statistical imperfections in a small number of datasets or analyses which support the conclusion og AGW, then seek to undermine the conclusions by attempting to demonstrate that this invalidates the work. The purpose of this is to discredit AGW science and provide people with a reason to believe that somehow the whole global warming thing is a fake problem and will all go away.

...

CA's work, when it is worthwhile, which is only occasionally, has not, in spite of much effort, actually resulted in any findings which substantively change the scientific picture of what is going on, or what is likely to happen.

CA's work now has two significant scalps: Mann's hockey stick; GISS's Temperature record. Both are important, more for demonstrating that some of the perceived flaws in the alarmist AGW view really might be worth pursuing.

[Devonian]

CA's work now has two significant scalps: Mann's hockey stick; GISS's Temperature record. Both are important, more for demonstrating that some of the perceived flaws in the alarmist AGW view really might be worth pursuing.

This is a graphic (from this report) showing the changes between the old and new global temperature data sets. There is a difference, but you need good eyes a magnifying glass to see it. Whether it can be described as a 'scalp' is somewhat debatable......

Oh, and the 'hockey stick' still stands - but it's probably best not to start that debate again

[Essan]

Which planet's news do you read?

LA Times?

"The re-ranking occurred Aug. 7 with little fanfare, but it touched off a firestorm among climate bloggers and commentators, who took the new rankings as evidence that global warming was a hoax."

Steve McIntyre, who discovered the bug, did not say it was intentional. Those who believe that must be very few and far between which is why I'm incredulous you have apparently read so much about them.

Having seen comments on numerous different forums, both with regards this and other climate issues, I can assure you that there are a lot of people - including a number of prominent 'sceptics' - out there who think NASA are deliberately manipulating figures to support AGW theory. And as for the accusations made against Mann!

Beleive me, places like Net-wx are amongst the most civil and politest of forums for discussing such issues

[AtlanticFlamethrower]

That's the global chart, subject to flaws of statistic collection worldwide. Then there's the 1980s+ chart that shows warming - few people deny that, you can see that on the chart below.

Here is the US temperature anomaly chart, the "jewel in the crown" of climate records because it is the most reliable.

"The re-ranking occurred Aug. 7 with little fanfare, but it touched off a firestorm among climate bloggers and commentators, who took the new rankings as evidence that global warming was a hoax."

Having seen comments on numerous different forums, both with regards this and other climate issues, I can assure you that there are a lot of people - including a number of prominent 'sceptics' - out there who think NASA are deliberately manipulating figures to support AGW theory. And as for the accusations made against Mann!

Beleive me, places like Net-wx are amongst the most civil and politest of forums for discussing such issues

Why don't you just ignore them like most rational people? There is only one person in this thread spreading the "GW is a stitch-up" conspiracy meme and it's YOU. Nice going, Darwin.

Edited August 17, 2007 by AtlanticFlamethrower

[Essan]

Why don't you just ignore them like most rational people? There is only one person in this thread spreading the "GW is a stitch-up" conspiracy meme and it's YOU. Nice going, Darwin.

I prefer to keep up with what people are saying, rather than bury my head in the sands and ignore all I don't want to hear. There's an old adage about 'knowing one's enemy'

Oh, and btw, read back through this thread: a number of posters have questioned NASA's reliability in this matter And besides, my point remains: why make such a fuss over nothing? The reason this issue had received so much prominence worldwide is because it casts doubt on the trustworthiness of temp data..... or, at least, according to some it does

[Devonian]

That's the global chart, subject to flaws of statistic collection worldwide. Then there's the 1980s+ chart that shows warming - few people deny that, you can see that on the chart below.

Here is the US temperature anomaly chart, the "jewel in the crown" of climate records because it is the most reliable.

So you think only Americans can properly place and read thermometers?

[AtlanticFlamethrower]

So you think only Americans can properly place and read thermometers?

Not saying the British cannot - of course they can, perhaps better - but 48 states of America is a much larger sample area than tiny Britain.

When we refer to "global temperature" this includes not just the US but Europe, Asia, Africa, South America... etc - these global stations do not in fact cover the whole globe rather much less than 50% of it.

It would be a brave individual who would say statistics from each of these areas are equally reliable. Arguably US statistics are the most reliable for the largest sample area we have - continuous record for 48 states of USA since at least the beginning of the 20th century.

The best sample area we have shows a recent global warming but one that is not quite as anomalously warm than the previous which was at its height in the 1930s.

That has to be significant.

Essan

I prefer to keep up with what people are saying, rather than bury my head in the sands and ignore all I don't want to hear. There's an old adage about 'knowing one's enemy'

Do you take David Ike seriously?

Oh, and btw, read back through this thread: a number of posters have questioned NASA's reliability in this matter tongue.gif And besides, my point remains: why make such a fuss over nothing? The reason this issue had received so much prominence worldwide is because it casts doubt on the trustworthiness of temp data..... or, at least, according to some it does wink.gif

I stand corrected. The Pit for one mentioned, but did not endorse, the conspiracy angle.

You don't think there is any significance in this chart?

I'd be interested to have your interpretation of what is happening here.

[Mr Sleet]

Not saying the British cannot - of course they can, perhaps better - but 48 states of America is a much larger sample area than tiny Britain.

When we refer to "global temperature" this includes not just the US but Europe, Asia, Africa, South America... etc - these global stations do not in fact cover the whole globe rather much less than 50% of it.

It would be a brave individual who would say statistics from each of these areas are equally reliable. Arguably US statistics are the most reliable for the largest sample area we have - continuous record for 48 states of USA since at least the beginning of the 20th century.

The best sample area we have shows a recent global warming but one that is not quite as anomalously warm than the previous which was at its height in the 1930s.

That has to be significant.

Essan

Do you take David Ike seriously?

I stand corrected. The Pit for one mentioned, but did not endorse, the conspiracy angle.

You don't think there is any significance in this chart?

I'd be interested to have your interpretation of what is happening here.

I'd say that the US temperature is about the same as it was 100 years ago.

And I wouldn't trust NASA 100% either, same as I wouldn't trust any scientist 100% ( and I earn a living as one ! ).

[Devonian]

I'd say that the US temperature is about the same as it was 100 years ago.

Would you? Crikey.

Would you say then CET is about what it was 100 years ago?

The question being, where does 'about' stop and higher than begin?

[AtlanticFlamethrower]

I'd say that the US temperature is about the same as it was 100 years ago.

And I wouldn't trust NASA 100% either, same as I wouldn't trust any scientist 100% ( and I earn a living as one ! ).

And I wouldn't trust you with GCSE line graph analysis... 100 years? Surely you mean't 70...

[Devonian]

And I wouldn't trust you with GCSE line graph analysis... 100 years? Surely you mean't 70...

Well, if so my question is withdrawn

[gmoran]

This is a graphic (from this report) showing the changes between the old and new global temperature data sets. There is a difference, but you need good eyes a magnifying glass to see it. Whether it can be described as a 'scalp' is somewhat debatable......

It isn't the global temps that is important in this regard. It's that there was concern about datasets and methodologies, access to which were requested and denied, and it was then found that there was a significant error. Does the error show AGW isn't happening? no; does it show US temps aren't showing a warming trend since the 90's? no; but the change is still interesting and significant.

Oh, and the 'hockey stick' still stands - but it's probably best not to start that debate again

The "blade" still "stands", the flattening of the stick does not; and only some sort of denialist would claim that MBh98's statistical methods, use of bristle-cone data and HS graphic are anything other than dis-credited. Storch web site

[Devonian]

The "blade" still "stands", the flattening of the stick does not; and only some sort of denialist would claim that MBh98's statistical methods, use of bristle-cone data and HS graphic are anything other than dis-credited. Storch web site

Nice example of poisoning the well that!

[parmenides3]

It isn't the global temps that is important in this regard. It's that there was concern about datasets and methodologies, access to which were requested and denied, and it was then found that there was a significant error. Does the error show AGW isn't happening? no; does it show US temps aren't showing a warming trend since the 90's? no; but the change is still interesting and significant.

The "blade" still "stands", the flattening of the stick does not; and only some sort of denialist would claim that MBh98's statistical methods, use of bristle-cone data and HS graphic are anything other than dis-credited. Storch web site

Nice work, qm: Von Storch is a decent chap & a respected scientist. For a slightly different perspective on the 2004 paper, it's worth comparing your post with this: http://www.realclimate.org/index.php/archi...-repercussions/

I'll state one thing for certain now: there is no point in arguing about the hockey stick. People who want it to be wrong will insist on this regardless, and people who don't want it to be wrong will insist on their opinion, regardless. No way is there going to be resolution on NW about this, whatever anyone thinks. All that really matters is that there are no palaeoclimatic reconstructions which make the past look anything like the present.

On the question of the availability of data, I think most people would agree that science should be as transparent as possible, and anything which contributes to this is probably a good thing. But people who jump from corrections or reanalysis of data to doubt about AGW are barking up the wrong tree. I also have my doubts about anything which resembles a conspiracy theory, but maybe that's just me.

:)P

[gmoran]

Nice example of poisoning the well that!

Not at all; there is a warming at the end of the 20th/beginning of the 21st that is distinct in the last millennium. But I don't like seeing the last 1000 years revised for idealogical purposes. AGW? evidence looks reasonable. Alarmist AG and I'd have to say I'm naturally skeptical of such an anthropocentric and catastrophic perspective, especially given the number of elephants standing in the room.