ECMWF, GFS, NMM And UKMO: How Have The Models Been Performing?

[johnholmes]

An interesting post from Thundery wintry showers regarding the performance of the main models:

And kinda stays true to the fact of the ECMWF performing a little better than the GFS when looking at the latest scores on the thread. (ECMWF: 9 points. GFS: 5 points).

Does seem to be the case that the models are having a difficult time predicting this possible cold spell (which still looks very likely for far Northern parts this week) with Shortwaves and the toppling of the high much later on in the outputs making it tricky to predict what is going to happen for later on this week, the weekend and further beyond that.

We need to be rather careful with this sort of comparison that we are in fact comparing like with like.

Both the checks I carried out and those you refer too were done for T+240. The remark by TWS referred to the T+144 time scale not T+240.

UK Met stops, for us anyway, at T+144, so any comparison using UK Met has to stop at T+144.

Other models do go out to T+240 and can obviously be compared at that time scale.

Edited March 7, 2011 by johnholmes

[summer blizzard]

An interesting post from Thundery wintry showers regarding the performance of the main models:

And kinda stays true to the fact of the ECMWF performing a little better than the GFS when looking at the latest scores on the thread. (ECMWF: 9 points. GFS: 5 points).

Does seem to be the case that the models are having a difficult time predicting this possible cold spell (which still looks very likely for far Northern parts this week) with Shortwaves and the toppling of the high much later on in the outputs making it tricky to predict what is going to happen for later on this week, the weekend and further beyond that.

Indeed, when you have true zonality the GFS is brilliant at picking up the shortwaves however during my checks during the Autumn, this only occured once. In the other situations where you have an amplified Jet Stream, the GFS has performed poorly.

The big suprise for me was the GEM model which while fourth best at day 6 (144 hours), was embarrasing even the ECWMF at day ten during early Autumn however the ECWMF did almost claw its way back.

Is the GEM model catching the big three at 144?

In regards to the ten day checks, the GEM only goes out to 240 hours on the 0z run and i am rarely awake early enougth to catch the GFS0z (student). I will try at the weekend however if aybody can get the 240 hour charts in the morning and post them, it would be much obliged.

[summer blizzard]

Ressurection commencing.

Okay, so as we enter the more exciting (and unpredictable) time of year i have decided to begin comparisons again between the ECWMF, GFS and GEM models at 240 hours out.

Rules are that the winner gets 2 points for being closest, second gets one point and loser gets nothing.

Final 2010 results were..

GEM: 10 points

ECWMF: 9 points

GFS: 5 points

Last year was very suprising as the GEM was pretty brilliant at 240 hours despite being behind at day 6 (144 hours).

Todays then are..

Three different solutions although in general it is the ECWMF vs GEM + GFS.

[Gibby]

Well I thought it was about time I put my input into here as I suppose I ought to be well qualified as I report on the big three models twice daily on this site and others though I must first state that I keep no statistical evidence of what I feel on how they have performed recently.

Anyway, my feelings are that they all have been guilty of some wild fluctuations at times through this summer. Most of these have been founded from them all trying to lower pressures over Greenland when in reality for much of the summer it just didn't happen. Consequently Low pressures were repeatedly steered towards the UK instead of Iceland and Greenland a situation that is not common in summer. In fact, because the normal pattern set is for the Jet Stream to be well north in summer long term model outputs default setting is just that and I feel that is why so many times the long term charts frequently showed High pressure returning from the Southwest in FI only to see it removed as time grew closer and the computers latched on to the fact that pressure wasn't going to lower over Greenland.

My gut feeling though is that when it comes to long term predictions for Europe and the UK ECM generally has a better handle on this although this is more apparent in Winter when it deals with cold scenarios quite well. GFS on the other hand can overcook Low pressures again particularly in winter and can power cold blocks over Europe out of the way with total ease when in reality they end up backtracking towards an ECM solution close to the time. GFS I find totally frustrating when it comes to their precipitation forecasts in relation to both intensity and location and generally are about as useful as a chocolate teapot.

With regard to the UKMO we can of course only see data up to 144hrs so we never know what their data sets show beyond that time but they are the best in the time frame of the 6 days we see and rarely undercook or overcook Low and High pressure. In fact if you ever see a powerful storm or severe cold shown on UKMO then take cover 'cause it usually means there is a very good chance its going to happen.

Having highlighted the errors of the models in my comments above I will also praise them in the fact that they have generally improved markedly in recent years in the short term predictions they show and very rarely nowadays to we see a model get things totally wrong within the immediate 24hr time prediction. Of course the further you go out from the present the reliability of such output falls markedly but when used in conjunction of the ensemble sets incorporating the Operational and Control runs for trendsetting only they become a very useful and reasonably reliable tool of a general indication of how the weather will pan out on a national basis in the upcoming 2 weeks.

It will be interesting to see how they all handle the up and coming Autumn/Winter and Spring seasons storms and snowfall and I will let you know my feelings of how they've faired next Spring.

[johnholmes]

thanks for those comments Gibby

Yes the rainfall continues to be a problem on most occasions. In that comment folks lies an issue that will crop up time and time again through the winter. The wails of anguish from snow lovers that it did not snow for them, it rained instead or whatever.

A very rough guide to the difficulty in comparing trying to forecast rain and snow is this. Forecasting rain has no more than about 3 variables, snow has another 5 or six ON TOP of these rainfall variables.

[summer blizzard]

Ressurection commencing.

Okay, so as we enter the more exciting (and unpredictable) time of year i have decided to begin comparisons again between the ECWMF, GFS and GEM models at 240 hours out.

Rules are that the winner gets 2 points for being closest, second gets one point and loser gets nothing.

Final 2010 results were..

GEM: 10 points

ECWMF: 9 points

GFS: 5 points

Last year was very suprising as the GEM was pretty brilliant at 240 hours despite being behind at day 6 (144 hours).

Todays then are..

Three different solutions although in general it is the ECWMF vs GEM + GFS.

Will wait for the final outcome, but it looks to me as if the GEM model may be making the 'big' models look silly again.

[summer blizzard]

G

Ressurection commencing.

Okay, so as we enter the more exciting (and unpredictable) time of year i have decided to begin comparisons again between the ECWMF, GFS and GEM models at 240 hours out.

Rules are that the winner gets 2 points for being closest, second gets one point and loser gets nothing.

Final 2010 results were..

GEM: 10 points

ECWMF: 9 points

GFS: 5 points

Last year was very suprising as the GEM was pretty brilliant at 240 hours despite being behind at day 6 (144 hours).

Todays then are..

Three different solutions although in general it is the ECWMF vs GEM + GFS.

The GEM model won that one with GFS second and ECWMF third.

Table stands at..

GEM - 2 points

GFS - 1 point

ECWMF - 0 points

Next set of charts...

All fairly similar though differences in the strength of the low.

[johnholmes]

G

The GEM model won that one with GFS second and ECWMF third.

.

I'm not sure that is a correct decision SB, none really got the position of ex Katria correctly, ECMWF was the deepest and the low certainly was not over the UK on Sunday?

Maybe I've been looking at the wrong chart-I'm not making a big thing of it but can you show your actual chart to come to the conclusion you did please?

thanks

[bluearmy]

tend to agree john - GEM was closest with its placement of the depression near the uk. however, its upstream pattern was very poor. quite honestly, wouldn't have awarded any points at all !!

[summer blizzard]

I'm not sure that is a correct decision SB, none really got the position of ex Katria correctly, ECMWF was the deepest and the low certainly was not over the UK on Sunday?

Maybe I've been looking at the wrong chart-I'm not making a big thing of it but can you show your actual chart to come to the conclusion you did please?

thanks

While Katia was not over the UK sunday, the other low was which is why the ECWMF came last as it still had ridging over the UK. GFS had it too far south and while GEM did not pick up Katia, it did have the previous low in a very similar position.

Will have to wait for the chart to become available in the archives as i forgot to save it.

tend to agree john - GEM was closest with its placement of the depression near the uk. however, its upstream pattern was very poor. quite honestly, wouldn't have awarded any points at all !!

Perhaps the biggest flaw with this comparison is that it is UK focused although that it because at 10 days out, most models will have a different upstream pattern to the actual and we are only looking at 1 day rather than several days in a row.

Edited September 13, 2011 by summer blizzard

[summer blizzard]

Just taking a look at the odels for the 22nd, it looks like GFS will be upset with another potential win for the GEM model.

[summer blizzard]

Table stands at..

GEM - 4 points

GFS - 1 point

ECWMF - 1 points

It seems that GEM has won again with the ECWMF in a close second but too cyclonic. All three models had the Jet Stream to weak at ten days out.

Edited September 22, 2011 by summer blizzard

[johnholmes]

I am afraid to say SB I disagree again with your marking giving GEM he lead. I do not believe either of them show enough +'s to be given the top mark, I would rate them about equal. GFS is, I agree, 3rd.

The comments I hope are seen as a constructive comment about your markings SB, the upper air pattern is best on ECMWF, the surface features are about even with ECMWF and GEM in my view?

Like the last time I commented the marks are given on the features over the area close by and over the Uk, say out to about 20-25W.

[bluearmy]

i think thats all three models made a decent fist of this one. GEM and ECM slightly ahead of GFS. i wouldnt split the two closest in marks.

[summer blizzard]

i think thats all three models made a decent fist of this one. GEM and ECM slightly ahead of GFS. i wouldnt split the two closest in marks.

Sadly there must be a winner otherwise its not a game (adds an element of fun to the model watching), GFS made too much of the high pressure over western Russia and GEM was the only one to really have high pressure south of the UK.

[johnholmes]

sadly SB as ba and I have indicated the 3 are too close, or perhaps too far away from the actual to mark a winner, ECMWF and GEM are closest but with no clear winner.

To do a scientific approach as to which model is performing best it needs total objectivity and I'm afraid you are not showing that.

I'm afraid you bias towards GEM is colouring your view or that is how it seems to me.

[bluearmy]

Sadly there must be a winner otherwise its not a game (adds an element of fun to the model watching), GFS made too much of the high pressure over western Russia and GEM was the only one to really have high pressure south of the UK.

a game cannot have a draw ???? only in the states SB !! i would have though awarding equal marks to models which are equally close is reasonable. as it happens, i have always paid attention to GEM in deep fi. it certainly lags the other two in accuracy at day 6 but by day 10, it often has a reasonable handle on things.

[summer blizzard]

sadly SB as ba and I have indicated the 3 are too close, or perhaps too far away from the actual to mark a winner, ECMWF and GEM are closest but with no clear winner.

To do a scientific approach as to which model is performing best it needs total objectivity and I'm afraid you are not showing that.

I'm afraid you bias towards GEM is colouring your view or that is how it seems to me.

a game cannot have a draw ???? only in the states SB !! i would have though awarding equal marks to models which are equally close is reasonable. as it happens, i have always paid attention to GEM in deep fi. it certainly lags the other two in accuracy at day 6 but by day 10, it often has a reasonable handle on things.

Very well, i shall instigate a draw although since GFS was a pretty clear loser, i shall award one point each to the GEM and ECWMF.

Don't have a bias towards any model really.

GEM - 3 points

GFS - 1 point

ECWMF - 1 points

Next set of charts as well, pretty different solautions so there should be an easy winner/loser.

[J10]

Many years ago I started to do this and I am pleased that SB has continued to do this. Personally though I disagree with 2 things about the competition. Firstly I would make the scores out of 10 and do this for each of the weeks. As such if there is nothing between say GEM and ECM they could each be awarded 7 out of 10, so although there is an overall competition, there is no need to artificially separate the entries if they are too close.

Edit already changed by SB above

On the marks out of 10, I would award GFS 5/10. GEM 6.5/10 and ECM 7/10. Even though GEM has got the pattern better than the others it has the LP too far west, so IMO ECM pips it.

[summer blizzard]

Many years ago I started to do this and I am pleased that SB has continued to do this. Personally though I disagree with 2 things about the competition. Firstly I would make the scores out of 10 and do this for each of the weeks. As such if there is nothing between say GEM and ECM they could each be awarded 7 out of 10, so although there is an overall competition, there is no need to artificially separate the entries if they are too close.

Edit already changed by SB above

On the marks out of 10, I would award GFS 5/10. GEM 6.5/10 and ECM 7/10. Even though GEM has got the pattern better than the others it has the LP too far west, so IMO ECM pips it.

Interesting idea and i would be quite happy to do it that way if the others agree.

GEM actually had the low the furthest east (centred pretty much north of Scotland), had the correct flow but ECWMF did have the better low placement out west with the wrong flow. Hard to say.

[OldGreggsTundraBoy]

models not performing great this summer

[johnholmes]

Interesting idea and i would be quite happy to do it that way if the others agree.

GEM actually had the low the furthest east (centred pretty much north of Scotland), had the correct flow but ECWMF did have the better low placement out west with the wrong flow. Hard to say.

thanks SB I think that would be the best way forward.

I would rather it not thought of as a contest though but an attempt to honestly find out over a long period which of those 3 models is performing best over the period.

We have 1st class stats for the northern hemisphere for days 5 and 6 but nothing as straightforward to my eyes for day 10.

I hope you do it.

[Neilsouth]

models not performing great ever

Fixed

[summer blizzard]

thanks SB I think that would be the best way forward.

I would rather it not thought of as a contest though but an attempt to honestly find out over a long period which of those 3 models is performing best over the period.

We have 1st class stats for the northern hemisphere for days 5 and 6 but nothing as straightforward to my eyes for day 10.

I hope you do it.

Yes, we do need to decide how we award the points though as i may give a different score to what somebody else thinks.

[johnholmes]

how about overall marks out of 5 for the major synoptic pattern. Is the trough in the right place and depth, is the nearest upper ridge likewise?

Then 5 for the bit zoomed in over the UK area, direction of the isobars over the UK, their value and how close they are, what 500mb temperatures are predicted. That should give as good a guide to trying to use the model for predicting what the surface weather for the UK might be 10 days ahead?

For instance in the latest charts you show GEM has isobars from a diffferent direction to the other two as one difference.

Edited September 25, 2011 by johnholmes