Buy

Books
Click images for more details

Twitter
Support

 

Recent comments
Recent posts
Links

A few sites I've stumbled across recently....

Powered by Squarespace
« Mann overboard - Josh 157 | Main | Hopeful fudging »
Wednesday
Mar212012

Climatologists go open

John Graham-Cumming notes that CRU and the Met Office have come over all open, as these excerpts from the CRUTEM4 paper make clear.

we are able to make the station data for all the series in the CRUTEM4 network freely available, together with software to produce the gridded data (http://www.cru.uea.ac.uk/cru/data/temperature/ and http://www.metoffice.gov.uk/hadobs/).

..and...

The code required to produce the CRUTEM4 fields and timeseries will soon be made available. Note that code previously released for CRUTEM3 cannot be used to exactly reproduce CRUTEM4 temperature series due to changes in processing methodology.

PrintView Printer Friendly Version

Reader Comments (48)

Beware of Greeks bearing gifts.

Mar 21, 2012 at 11:16 AM | Unregistered Commenterpesadia

bewear of geeks bearing gifs

Mar 21, 2012 at 11:32 AM | Unregistered CommenterAnoneumouse

Could Richard Betts clarify whether we now have the digital originally recorded raw station data and the 'value added' adjusted data with the station selection and adjustment justifications. I thought some or all of the history on the station adjustments was not kept and the metoffice were to do a rework of the whole thing?

Mar 21, 2012 at 11:56 AM | Registered CommenterPharos

bewear of geeks bearing gifs
Quote of the Week! (I'd do something about that typo, though!

Mar 21, 2012 at 12:01 PM | Registered CommenterMike Jackson

Currently, all the individual station data and the CRUTEM4 gridded product is available at the Met Office Hadobs website, also both of the recent papers, on CRUTEM4 and HADCRUT4. The code itself does not seem to be there yet but I expect it will be soon.


JGC says that he finds the statement
"our preference is that the underlying station data, and software to produce the gridded data, be made openly available" slightly surprising.

I wonder what could possibly have brought about this remarkable change of attitude? Perhaps Jonathan Jones or Don Keiller might have some insights into this question :)

Mar 21, 2012 at 12:40 PM | Registered CommenterPaul Matthews

Hurray! If anyone inthe future asks what good came of climategate then this is one simple answer.

Ironically though if they'd done this five years ago there probably wouldn't have been a climategate because the driver for that leak appears to have been the continual stonewalling of FOIA efforts to get precisely this information.

Mar 21, 2012 at 12:46 PM | Unregistered CommenterFrancisT

"changes in processing methodology"

Settled science.

Andrew

Mar 21, 2012 at 12:47 PM | Unregistered CommenterBad Andrew

Looking at http://www.cru.uea.ac.uk/cru/data/temperature/ there are two sets of input station data, one for CRUTEM3 and one for CRUTEM4. That seems strange to me, shouldn't there be a consistent single set of input data, regardless of the algorithm used to process it?

Mar 21, 2012 at 1:55 PM | Unregistered Commenterclimatebeagle

mm...

I downloaded some of the station data a few days ago, just out of idle curiousity. It's not representative I know, but opening up a few of the files, I was horified but just how much of data was -99C (i.e blank).

Presumably they've used the best records available in the various regions - makes you wonder though, how much reliance you can put on the end result.

Mar 21, 2012 at 1:59 PM | Unregistered CommenterGSW

May be worth going to this - Gleick the forger in Oxford.

http://www.bsg.ox.ac.uk/events/peter-gleick-oxford-amnesty-lectures

Mar 21, 2012 at 2:21 PM | Unregistered CommenterFrederick Bloggsworth

I'm with Paul Matthews in giving praise to Jonathan and Don for this surprising and positive turn of phrase.

Mar 21, 2012 at 2:34 PM | Unregistered CommenterRichard Drake

May be worth going to this - Gleick the forger in Oxford.

http://www.bsg.ox.ac.uk/events/peter-gleick-oxford-amnesty-lectures

Mar 21, 2012 at 2:34 PM | Unregistered CommenterFrederick Bloggsworth

"due to changes in processing methodology": well, of course. Otherwise comparisons would be straightforward.

Mar 21, 2012 at 2:37 PM | Unregistered Commenterdearieme

And wow, Fred: Gleick hosted by the Oxford Amnesty Lectures:

2012 Lectures Protect the Human/Protect the Planet

Amnesty’s watchword, ‘Protect the Human’, is a call to respect existing human life. But in putting humans at the centre of our moral universe, do we imperil future generations; is a call to ‘human’ rights also an imperious claim against our place in the natural order? What of other animals and life forms, the planet itself?

Both climate change and the fight against it have immense human consequences. Do human rights get in the way of an effective response? Or might they form the basis of a new environmental ethics? Can we refashion human rights to acknowledge our interconnectedness with nature? Or should human rights move aside to make room for less human-centred system of values?

Can we protect the human and save the planet? Join an Economics Nobel Laureate, an environmental activist, a UN Special Rapporteur, a political theorist, and others to find out in this year’s series of Oxford Amnesty Lectures.

I haven't really been tracking the subversion of human rights groups like Amnesty by our New Unhappy Lords (GK Chesterton's phrase). Index on Censorship's take on climate openness at the Data debate Josh and I attended in December and in publishing Fred Pearce's forthright views was a great deal more healthy than one might have expected. Amnesty I have less idea about. Presumably Gleick was signed up for this before admitting his deception and identity fraud. What should be done to persuade the organisers that the invitation can't now be right? Do Oxford or Amnesty people have ideas on that?

Mar 21, 2012 at 2:51 PM | Unregistered CommenterRichard Drake

"May be worth going to this - Gleick the forger in Oxford."

thanks, Frederick, I have a message for Oxford University from Leo Hickman:

"Beware of Gleicks bearing gifts."

Mar 21, 2012 at 2:52 PM | Unregistered Commentergeronimo

Re: climatebeagle

CRUTEM3 uses a single file with all the different stations in it.
CRUTEM4 uses a separate file for each station.

Mar 21, 2012 at 3:00 PM | Unregistered CommenterTerryS

Frederick Bloggsworth

Good to see Gleick's career is forging ahead through forging new contacts at Oxford.
Since it's just up the road from my Hampshire Riviera I've half a mind to squander £8 on a ticket and £2 on popcorn.

Mar 21, 2012 at 3:10 PM | Unregistered CommenterAlan Reed

Alan, I don't think popcorn is adequate for this one. What about someone investing in face masks with Gleick's face printed on them? If a few people wore those throughout the lecture and had some photocopies of an well-worded explanation to give to people I think that could have an impact. But as I've said in an earlier post trapped in moderation:

I haven't really been tracking the subversion of human rights groups like Amnesty by our New Unhappy Lords (GK Chesterton's phrase). Index on Censorship's take on climate openness at the Data debate Josh and I attended in December and in publishing Fred Pearce's forthright views was a great deal more healthy than one might have expected. Amnesty I have less idea about. Presumably Gleick was signed up for this before admitting his deception and identity fraud. What should be done to persuade the organisers that the invitation can't now be right? Do Oxford or Amnesty people have ideas on that?

Links left out this time. But to get an idea of the history of the lectures try

http://www.oxford-amnesty-lectures.org/index.php?p=Archives

I think this is a fine opportunity but it needs to be done really well. Best to persuade the organisers to dis-invite the guy. The face mask approach or some other stunt might be handy if they refuse to listen. But I'd be open to anything.

Mar 21, 2012 at 3:38 PM | Unregistered CommenterRichard Drake

At least one member of the faculty hosting Gleick talks sense .............. I wonder if they will meet?

http://www.dieterhelm.co.uk/sites/default/files/Huhne_hype_060212.pdf

Mar 21, 2012 at 3:46 PM | Unregistered CommenterRobert Thomson

A further thought about Amnesty and Index on Censorship. They naturally support dissidents. Why was nobody from a sceptical viewpoint invited to the Data Debate? Has anyone from a sceptical viewpoint - economics, politics or science - been included in the seven luminaries this time?

http://www.oxford-amnesty-lectures.org/index.php?p=Lectures

Best guess would be that Olivier de Schutter is nearest:

http://news.bbc.co.uk/1/hi/7381392.stm

That links back to one of the most important Bishop Hill posts of all time, The Entrepreneur:

http://www.bishop-hill.net/blog/2012/2/20/the-entrepreneur.html

But look at the dreadful title he's been given or accepted. That's just two days after Gleick.

I think the dissident question is really worth asking the organisers. But there are masses of issues raised. We should organise and do some real good :)

Mar 21, 2012 at 3:49 PM | Unregistered CommenterRichard Drake

FB

"May be worth going to this - Gleick the forger in Oxford."

Oxford coma I suspect.

Mar 21, 2012 at 3:58 PM | Unregistered Commentersimpleseekeraftertruth

@ pesadia

Beware of Greeks bearing gifts.

This is supposed to be a sceptic blog, not a cynical one. If the announcement is sincere, and we might as well take it at face value because if it is not sincere it will do more damage to the reputation of the Met Office than simply maintaining the status quo would, then the Biblical quotation below is far more appropriate.

There will be more joy in heaven over one sinner who repents than over ninety-nine righteous persons who need no repentance.” Luke 15:1-7.

Mar 21, 2012 at 4:00 PM | Unregistered CommenterRoy

Roy:

This is supposed to be a sceptic blog, not a cynical one.

Very well put. We're going to have to up our game if we have a go at Amnesty and their ilk. Amnesty - what the prodigal son got. I need to think about that. Luke 15 rules.

Mar 21, 2012 at 4:06 PM | Unregistered CommenterRichard Drake

I downloaded all the new station data for Hadcrut4 and then compared the results with the old Hadcrut3 data. There are 5549 stations in the set compared to 5097 in Hadcrut3. 738 new stations have been added while 286 stations have been discarded. Those added are all mainly in northern Russia. Quite a lot of stations from North America have been discarded. There are none added in the southern hemisphere but others have been lost, despite sparse coverage. The changes to the global anomalies are small and statistically insignificant. However they do psychologically change the impression of "warming" over the last 15 years - moving 2010 and 2005 up a bit and 1998 down a bit. Also the 19th century data has got just a tiny bit cooler. You can read more about this analysis at my blog

Mar 21, 2012 at 4:41 PM | Unregistered CommenterClive Best

Clive

Are there objective criteria for station selection?

Mar 21, 2012 at 4:49 PM | Registered CommenterBishop Hill

This approach is welcome.

It will be interesting to review the scale of, and the reasons behind adjustments to raw data.

As I understand matters, they have adjusted sea temperatures to take account differences between bucket measurements and engine room measurements. I must read their papers which presumably explain some of the genral approaches.

I think that it is often overlooked that ships are in effect measuring sea temperature at a depth in the region of about 10m below the surface (this is a typical depth of the water intake) and are not actually measuring surface temperature.

Mar 21, 2012 at 4:50 PM | Unregistered Commenterrichard verney

No need to waste cynicism on Gleick:

"Believing that people are motivated by self-interest; distrustful of human sincerity or integrity".

He only did it because he honestly believed he would never get caught.

Mar 21, 2012 at 5:02 PM | Unregistered CommenterAlan Reed

If I wanted to accurately plot the temperature history of your car as a function of time with sensors attached to the the car's surface, that I periodically moved around every couple of years, and claimed I could 'munge' all the measurements into a single temperature using an averaging program....and every few years I reported that you car was actually colder in the past than I originally claimed...would you:
a) assume I was a statistical genius and award me a large grant? or
b) treat everything I said with suspicion?
Your choice!

Mar 21, 2012 at 5:16 PM | Unregistered CommenterZT

@ Mike Jackson

Its not a typo

giff (hard g) stands for graphics interchange format

Mar 21, 2012 at 5:28 PM | Unregistered CommenterAnoneumouse

Read the small print on the notice advertising the Gleick event in Oxford: "Please note, the lectures are not part of Amnesty International and they are not University events".

Mar 21, 2012 at 5:53 PM | Unregistered CommenterMike Post

Paul, CRU have not suddenly turned over a new leaf.
They are still fighting a rearguard action, through their solicitors, to prevent the release of requested emails.

I will provide more details when (legally) possible.

Mar 21, 2012 at 6:03 PM | Unregistered CommenterDon Keiller

Clive, do you not think it suspicious that some stations in the southern hemisphere have been lost, despite the already sparse coverage?

Could this be because the SH is not "on message" temperature wise?

Mar 21, 2012 at 6:06 PM | Unregistered CommenterDon Keiller

I can't imagine how they can justify dropping stations from the Southern Hemisphere. It certainly is true that SH temperatures (not anomalies) show no warming at all since about 1950. The arctic is warming and most new stations are in the far north. A couple of Arctic stations have even been dropped for example this one which appears not to be warming much. Perhaps CRU have a perfectly rational explanation - I don't know. Antarctica is certainly not warming and there are no new stations added there.

In reality the changes to the global anomaly are statistically insignificant. However the front page headline for "policy makers" is that 2010 is the hottest year. This is a meaningless statement since statistically there has been no observed warming for nearly 15 years, and this remains true.

Mar 21, 2012 at 6:24 PM | Unregistered CommenterClive Best

Peter G ained
L ittle
E vidence
I llegally
C aught
K lutzy

Mar 21, 2012 at 6:38 PM | Unregistered Commenterpesadia

I can find no explanation by CRU or by Hadley as to their station selection criteria. They really need to be open and transparent about this.

However their software requires the following:
Anomalies are defined for each station by subtracting the monthly values for a particular month from these normal values. Stations without normals for 1961-1990 or where any anomaly is > 5 standard deviations are excluded. The standard deviation is calculated from the 1961-190 monthly normals.

The large majority of the dropped stations pass the above criteria !

Mar 21, 2012 at 6:39 PM | Unregistered CommenterClive Best

Something went wrong there.Hope u can work it out.

Mar 21, 2012 at 6:41 PM | Unregistered Commenterpesadia

Anoneumouse
I'm well aware what a gif is. I'm less sure about "bewear", however. (|:-|)

Mar 21, 2012 at 7:20 PM | Registered CommenterMike Jackson

Mike Post:

Read the small print on the notice advertising the Gleick event in Oxford: "Please note, the lectures are not part of Amnesty International and they are not University events".

Thanks for this. They raise money for Amnesty - if anyone buys the resulting book - from what I can tell and they use the name. They've attracted a stellar list of speakers over the years (to a Guardianista more than to me I guess) and there's this:

The 2012 series is organised in collaboration with the The Faculty of Law, University of Oxford.

Time to contact that Law Professor at All Souls I met at a family do in 2008? Could be. I don't really think either organisation can credibly distance themselves from this one. Gleick should be stood down.

Mar 21, 2012 at 7:48 PM | Unregistered CommenterRichard Drake

believe it when I see it , I will and watch out for that where they hide the thimble .
But you have to say given there us already a lack of data the reasons for drooping any station must be very good indeed , Anyone got any idea why these stations were drooped or are we back to the 'trust us' approach to science?

Mar 21, 2012 at 7:52 PM | Unregistered CommenterKnR

I'm an Amnesty member and I don't remember seeing much if any mention of CAGW or sustainability in their recent campaigns.

Although I don't by any means pretend to represent the membership as a whole, I see AI aligning most closely with social libertarianism, which doesn't sit well with many of the ideas and "solutions" touted by those of a more alarmist nature.

Mar 22, 2012 at 12:27 AM | Unregistered CommenterThrog

"our preference is that the underlying station data, and software to produce the gridded data, be made openly available"

I certainly welcome that. But it does not necessarily mean that they have done so, or are going to do so. Wishing is not having, and the statement leaves wiggle room for later saying "So and so wouldn't let us do that" or "we've lost it". I note, without prejudice, that the Met Office makes no mention of the two biggest countries in the world, Russia and Canada, at
http://www.metoffice.gov.uk/hadobs/crutem4/data/download.html

Mar 22, 2012 at 2:09 AM | Unregistered Commentermichael hart

It all sounds too much like fudge fudge, wink wink.

Mar 22, 2012 at 7:26 AM | Unregistered CommenterHuhneToTheSlammer

Richard said;

"I think that it is often overlooked that ships are in effect measuring sea temperature at a depth in the region of about 10m below the surface (this is a typical depth of the water intake) and are not actually measuring surface temperature."

More accurately, I believe;

At a variable depth depending on the size of the ship and its lading on that voyage but typically in the range 2 to 8 metres.

So a tanker, departing in ballast from the Gulf of Mexico (for example), will produce completely different data from that gained on its inbound voyage, since it is now measuring temperature at a completely different depth.

Mar 22, 2012 at 7:58 AM | Unregistered CommenterNicL

Clive says station 710820 (Alert, Canada) has been dropped.
In fact it is there in the new CRUTEM4 dataset, but with a new number, 713550.

Mar 22, 2012 at 9:37 AM | Registered CommenterPaul Matthews

Just a quick look so far but the "raw data" seems to be much the same in Western Australia.

E.G. The Kalgoorlie P.O. is ignored prior to 1942 and the Grid is filled with the last 4 years of the 1800's (1896-1889) fromSouthern Cross which global warming caused to cool ~ 1.2 degrees C between 1990 and 1999..

The Geraldton Town is ignored before 1942 and the grid is filled with the last 4 years of the 1800's from Hamelin Pool.

I shall check the rest in detail when I get some free time as the big oil cheque is well overdue.

Mar 22, 2012 at 2:15 PM | Unregistered CommenterRipper

Mar 22, 2012 at 7:58 AM | NicL
///////////////////////////////////////////
Fully agree that it is dependent upon the design, size, configuration of the vessel and whether the vessel is sailing fully laden or not.

In general, since vessels' earnings are usually made on laden voyages, voyage planning seeks to minimize ballast legs (ie., non revenue earning legs). Some trades do not permit this, and they are essentially round trip voyages having a laden out and ballast back leg. But all of that said, trade, as a world wide whole, will be biased towards the laden ship. As such shup's are not recording SST.

Mar 22, 2012 at 5:41 PM | Unregistered Commenterrichard verney

Richard Drake,

"Best to persuade the organisers to dis-invite the guy."

Isn't that the sort of thing that Gleick's chums do? I think it's an awful idea in about seventeen ways. For a start, giving Gleick a platform now can surely only be a good idea. Let the web get more tangled, I say.

Mar 23, 2012 at 7:00 AM | Unregistered CommenterJames Evans

The mystery deepens. CRU have changed the numbering of 110 stations between CRUTEM3 and CRUTEM4. Another 175 stations have been discarded.

Paul wrote: Clive says station 710820 (Alert, Canada) has been dropped.
In fact it is there in the new CRUTEM4 dataset, but with a new number, 713550.

This was one of the stations which has changed numbering. 710820 -> 710823. This is where the mystery starts because in CRUTEM3 the start date was 1950 whereas in CRUTEM4 the start date is now 1987 ! This causes the station to be excluded from the anomaly averages because it has no normals. So in reality it too has been discarded. You can see this by comparing these graphs:

ALERT (crutem3) and this new one ALERT (crutem4)

Now there are no anomalies defined for this station and so it is discarded anyway from the global average.
However I just noticed that Paul is also right - because it then appears a 2rd time in CRUTEM4 as 713550.

I hope there are not too many other duplicates like this !

Mar 23, 2012 at 10:43 AM | Unregistered CommenterClive Best

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>