Gfs Accuracy Or Not?

[johnholmes]

hi

I thought it might be a good idea to put some fact on what its accuracy shows rather than anecdotal comments which abound in the model thread.

1) Remember in these checks they are always taken using the same time start and finish time. That is 12Z. No special reason when I first started, it was just the most convenient time for me to use.

Over the past 17 months I have obviously looked at times at the other runs and my SUBJECTIVE analysis is that the 12z is overall at least as good in terms of both accuracy and consistency as any other run.

2) Any comments I have made through the whole period have been as OBJECTIVE as I can be. believe it or not I also like snow, frost, thunder and gales!!

In January I wrote this

'The statistics are these

39 out of 45 were rated good from T+120, and around 30 were rated as quite good to good from T+144. Even at T+168 some 28 were rated good.

Now to have those sort of statistics to my mind, the ability to say 7 days ahead, in something like 62% at 7 days would give a largely correct forecast for the air mass, wind direction, weather, max and min temp and rainfall along with pressure being within about 5mb of the actual, is quite something to an old ex forecaster.

At T+120 hours, 5 days, for all this to be in place with a % of 87% is even more astonishing.'

One other point. Up until early January T+168 was about as accurate as the T+120 to T+144 had been 12 months previously. In other words the change of grid length has made it more accurate further out. Like I said until the 'blocking' really started in January.

Since then two things have happened

.1) We have had more blocking situations, or at least a much less active Atlantic, than I can remember in many years. Its only the past 4-5 years that I have got involved with Internet use, so there is a gap from about 1995 until then. But, I spent hours every day prior to 1995 as a professional forecaster watching charts, and I cannot recall anything like this winter post about 1991-2. This is a subjective assessment only

2) I have changed the presentation format but I hope my objectivity has remained the same..

So given that, judging by many comments on the forum, that this winter, whilst not delivering feet of the white stuff over the whole Uk at about the same time, has been a very different late autumn into early spring for a long time. So how has GFS coped.

Okay, okay, I know some of you will say instantly, and usually without any proof, its rubbish. I don't think that is quite true. yes, my checking does suggest its not been as good since early January. However it did accurately predict at about T+120 the easterly just after Christmas. So what has happened since?

Since that period I have done 6 checks, 5 with a B mark, varying from B+ to B-, and 1 with a straight C.

This does indicate to me that it is struggling with the synoptics we are having. But how are the other models coping. Sadly we have no objective checks on them. However, that has very recently changed with TWS comparing all the major models from T+96 out to T+144, and giving them each a mark.

I commend his thread to all of you who want to genuinely see how the models are coping rather than making negative comments.If, and its a big If, TWS can continue doing this we should build up a first class idea of how they perform. We may even be able to start having sub headings on the overall state of the North Atlantic/Europe area as being mobile(in its loosest sense) or blocked. Then comparing how each model copes with each situation. Like I said its a big job and I hope that with help from the others on the forecast team this can be done.

I shall keep my T+168 checks running with objective marking.

Maybe by the start of next winter we MIGHT be able to suggest which model is best to follow in a particular synoptic set up.

Anyway, thanks everyone who reads this. Your comments, as long as they are constructive are most welcome, even if they are in direct disagreement with my thoughts. rational debate is great.

regards

John

Edited March 14, 2006 by johnholmes

[Evo]

Hm try again, the first post got killed by a crashing Internet Explorer

Hi John,

I am currently working on a program that will download point data from the GFS runs for a number of UK locations automatically at regular intervals at distances from T24 through to T168. It will also download the relevant METAR for the times of the predictions and present the result in a self-updating image like this:

[ignore the actual figures here, it's mostly dummy data]

Once a sufficient data set has built up it should be possible to do some calculations to establish how accurate the GFS is at various ranges and for various locations. Quite exactly what, how, why and when I'm not sure but it should be something else interesting to throw into the pot.

Edit: P.S. The current locations are Bournemouth, Heathrow, Manchester and Glasgow. These will probably change at some point to give a good geographic spread.

Edited March 14, 2006 by Evo

[johnholmes]

hi Evo

that looks very interesting. I'm no computer expert, if OON reads this he will probably die laughing - oh I don't know) but the idea like I say is interesting.

Just one point, in no time at all you will have a huge amount of data, so perhaps best to concentrate on 3 or 4 at the most. How to pick them, not really sure.

Puts in advert for self!

I'm now doing T+168 with Doncaster, or rather my forecast based on T+168, as check.

Any chance of sending me the programme, assuming its simply a matter of click this then that and hey presto its there??!!!

As I say, along with TWS we may actually start getting some proper data to use to get real solid comparisons.

John

just seen your edit re the 3 airports. I would keep it at that for the time being.

John

Edited March 14, 2006 by johnholmes

[Evo]

Hi Mark

It's still very much in the alpha stage at the moment. The daemon is pretty much totally functional, but still a bit shakey and tends to fall over now and again. I'm fixing it up as I go. Within a couple of weeks I should have sufficient data for the output to make sense.

Anyway, I am getting the point data from here:

http://wxweb.meteostar.com/sample/

If you enter an ICAO airfield code, it will give you the point data.

I'm downloading the METARS from lfv.se because the weather.noaa.gov ones are a bit slow to update. The daemon downloads the pages at set times and then parses them to extract the data.

In terms of raw data, we have 16 rows (4 rows per ICAO site) of METAR data a day and up to* 96 rows (6 rows per ICAO site per run) of GFS data a day - it's not that much data to be honest. It's currently going into an access db but I also have MySQL and SQL Servers available to me to use.

* The meteostar site seems to update erratically, sometimes missing runs.

Once I have a proper dataset built up, I'll send you a copy and you can do your statistical magic on it. If you're interested in the code then at some point I can send this over as well. It needs a lot of tidying up first though.

One last thought - ideally the data would come from the GRIB output but this is just too complicated for me to attempt at the moment. Not only that, the GRIB files aren't exactly small!

hi Evo

that looks very interesting. I'm no computer expert, if OON reads this he will probably die laughing - oh I don't know) but the idea like I say is interesting.

Just one point, in no time at all you will have a huge amount of data, so perhaps best to concentrate on 3 or 4 at the most. How to pick them, not really sure.

Puts in advert for self!

I'm now doing T+168 with Doncaster, or rather my forecast based on T+168, as check.

Any chance of sending me the programme, assuming its simply a matter of click this then that and hey presto its there??!!!

As I say, along with TWS we may actually start getting some proper data to use to get real solid comparisons.

John

just seen your edit re the 3 airports. I would keep it at that for the time being.

John

Hi John

To be honest the program wouldn't do a lot for you, it needs to be running on a server 24x7 and have permanent internet access. You are quite welcome to share the data it creates though, in whatever format is best for you. Your input is obviously valued so please feel free to let me know what you think!

[johnholmes]

Hi Mark

It's still very much in the alpha stage at the moment. The daemon is pretty much totally functional, but still a bit shakey and tends to fall over now and again. I'm fixing it up as I go. Within a couple of weeks I should have sufficient data for the output to make sense.

Anyway, I am getting the point data from here:

http://wxweb.meteostar.com/sample/

If you enter an ICAO airfield code, it will give you the point data.

I'm downloading the METARS from lfv.se because the weather.noaa.gov ones are a bit slow to update. The daemon downloads the pages at set times and then parses them to extract the data.

In terms of raw data, we have 16 rows (4 rows per ICAO site) of METAR data a day and up to* 96 rows (6 rows per ICAO site per run) of GFS data a day - it's not that much data to be honest. It's currently going into an access db but I also have MySQL and SQL Servers available to me to use.

* The meteostar site seems to update erratically, sometimes missing runs.

Once I have a proper dataset built up, I'll send you a copy and you can do your statistical magic on it. If you're interested in the code then at some point I can send this over as well. It needs a lot of tidying up first though.

One last thought - ideally the data would come from the GRIB output but this is just too complicated for me to attempt at the moment. Not only that, the GRIB files aren't exactly small!

Hi John

To be honest the program wouldn't do a lot for you, it needs to be running on a server 24x7 and have permanent internet access. You are quite welcome to share the data it creates though, in whatever format is best for you. Your input is obviously valued so please feel free to let me know what you think!

ok Evo, I'll have a think about that and pm you.

John

[Evo]

Evo, what language did you use to parse the meteostar document? Would you be willing to share the source code for it?

VB6.

If you give me a mo, I'll post the cost. Don't expect it to be pretty :blush:

Code PMd.

Sorry for cluttering up your thread John!

Edited March 15, 2006 by Evo

[johnholmes]

re this

If John's prepared to dish out a spec, and help with the more meteorological apsects (that always evade a beginner - ie ME!) I think we can produce some software that might provide some significant benefit to the community at large.

John, as a forecaster what sort of analysis have you always wanted but never had?

I'll get back to you in a bit but thanks for that offer.

John