12z Model Comparisons Results

[Thundery wintry showers]

Model comparisons file now up to date, but for some reason it refuses to upload (just sits there saying "Uploading File" for eternity and it's only 323kb), I don't know if the University network is stopping me or if it's something to do with my computer or the file.

Starting my PhD means I won't have as much time as last year. Over the coming months I won't be keeping up the daily written summaries, but will be keeping a log of the daily scores in that Excel file, and from time to time will post entries illustrating how the models have been doing.

Thus, while I won't be able to keep up what I have done over the past year, I will endeavour to make sure that the 12Z Model Comparisons keeps going in some form.

[The PIT]

Model comparisons file now up to date, but for some reason it refuses to upload (just sits there saying "Uploading File" for eternity and it's only 323kb), I don't know if the University network is stopping me or if it's something to do with my computer or the file.

Starting my PhD means I won't have as much time as last year. Over the coming months I won't be keeping up the daily written summaries, but will be keeping a log of the daily scores in that Excel file, and from time to time will post entries illustrating how the models have been doing.

Thus, while I won't be able to keep up what I have done over the past year, I will endeavour to make sure that the 12Z Model Comparisons keeps going in some form.

Pity that as this thread was a great myth buster for model thread discussion when people posted drivel. Look forward to the excel sheets when you can get them loaded up.

[johnholmes]

Model comparisons file now up to date, but for some reason it refuses to upload (just sits there saying "Uploading File" for eternity and it's only 323kb), I don't know if the University network is stopping me or if it's something to do with my computer or the file.

Starting my PhD means I won't have as much time as last year. Over the coming months I won't be keeping up the daily written summaries, but will be keeping a log of the daily scores in that Excel file, and from time to time will post entries illustrating how the models have been doing.

Thus, while I won't be able to keep up what I have done over the past year, I will endeavour to make sure that the 12Z Model Comparisons keeps going in some form.

I'm sure someone will be able to sort out why it will not download Ian.

The amount of time you have put into this has been very large and we owe you a large debt. As Pit says, fact rather than fiction with the data you have shown.

Hope all goes well with the meteorology, I'm sure it will and you will find it absolutely fascinating although pretty hard work.

Again my thanks Ian and I look forward to seeing what you can output on the models in the future.

Edited October 4, 2007 by johnholmes

[Thundery wintry showers]

Update on last three days: GFS doing okay (three 7/10s) but often positioning HP too far north giving east winds, UKMO and ECM doing a bit better, JMA and GEM also doing well, but NOGAPS coming out with low marks with low pressure frequently positioned close to Britain.

[Thundery wintry showers]

Last few days: all models having some trouble with high pressure positioning, but the GFS, ECM and UKMO getting it broadly right each time. The model comparisons analysis (I may have to email it to someone so that he or she can upload the file, as I can't!) shows that ECM is performing slightly above UKMO and GFS.

[Thundery wintry showers]

For 13-15 October, UKMO performed very well, while GFS and ECM (more especially GFS) were a little too progressive at the T+144 end. NOGAPS and GEM were similar to GFS, while JMA kept bringing in rampant zonality and so scored low marks.

[Thundery wintry showers]

16-18 October: the models rather underdid the northerly, and were too progressive in moving the high pressure eastwards on 17th/18th October. In a reversal of the previous few days, when UKMO and ECM were doing well, the UKMO was particularly dodgy at handling the northerly, while GFS and ECM did okay, ECM being most consistent over T+96-T+144, but GFS being the most accurate at T+96.

[Thundery wintry showers]

Update for 19-22 October: the models performed reasonably well at judging the high-pressure spell but for 22 October they were too progressive at bringing the Atlantic in. ECM was over-progressive at all three timeframes, UKMO and GFS likewise at T+144 but were okay at T+120 and good at T+96.

[Thundery wintry showers]

model_comparison_update.xlsIt transpires that uploading files doesn't work on Firefox (my normal browser of choice) on the UEA server, but it does work using Internet Explorer.

So, without further ado, using IE:

[The PIT]

model_comparison_update.xlsIt transpires that uploading files doesn't work on Firefox (my normal browser of choice) on the UEA server, but it does work using Internet Explorer.

So, without further ado, using IE:

Thanks TWS.

[Thundery wintry showers]

Anticyclonic spell, 22-25 October: all of the models performed reasonably well, as there is a large margin for error with high pressure.

Return of westerlies on 26-28 October: GFS performed very poorly, keeping high pressure in charge over Britain for too long, while ECM got it close to spot on, and UKMO fell midway between GFS and ECM in terms of accuracy. JMA and GEM were also about level with UKMO.

[Stratos Ferric]

Great thread TWS, and top marks from me for effort. Two questions: what's the basis for scoring? What's the model lead time you're testing against?

[Thundery wintry showers]

I'm using the T+96, T+120 and T+144 period for comparisons. I used to do a full written analysis, but with having a fair influx of work for my PhD, I've stopped the detailed written summaries (though still keep up a record of the models' performances).

It's a 1-10 ranking system which is quite subjective but I'd hope that I would give marks that are consistent, generally speaking, relative to each other. Usually, say, if an output is poor at T+144 but good at T+120 and T+96, I will give a 7.

model_comparison_update.xls

Performances have indeed dropped recently, but mainly on the lesser models. GFS in particular seems to have been having issues recently, although there have also been occasions where it came out on top.

[Thundery wintry showers]

Sadly I'm going to have to come clean and admit I can't keep these up any more; it's been good to have a consistent record over the last year and a half, but commitments with forecasting, moderating and PhD work mean that I don't have as much resolve/time to keep them updated.

Thanks for all who have supported the analysis over the period.

[The PIT]

Sorry to hear that I enjoyed reading your reports. It was also a very good myth buster as well over in the Model discussion thread.

[johnholmes]

thanks for all the hard work Ian, as Pit has posted it was a good antidote to some of the wilder hopecasts on the model thread. Its still a great stack of data that maybe someone else could take over??

[pottyprof]

Thanks for all the work you have put into this Ian. It's been a very informative exercise and I'm sure that any of the serious model watchers have enjoyed the comparisons.

Hope the PhD work is going well..