ECMWF, GFS, NMM And UKMO: How Have The Models Been Performing?

[Thundery wintry showers]

I'm not a fan of looking at more than the 3 main models I'm afraid. In my view, and of those ex colleagues I still have contact with, other than the main 3, none of the other models have sufficient routine accuracy to be worth me looking at. Yes UK Met look at all but its the main 3 in terms of synoptic length which they generally take notice of.

Re Julian below

Yes out to T+240 Julian with ECMWF and GFS, at least on the ones I check, admittedly not very often but I will be putting them in more regularly in the forecast discussion thread I posted in a day or so ago. For the NOAA checks for all the main models then T+144, 6 days, is the furthest I've seen them routinely have checks for?

I recall that my 12Z Model Comparisons in 2006-07 bore out your assertions- I looked at GEM, NOGAPS and JMA and all lagged behind the "big three" by a large margin, with NOGAPS performing particularly poorly.

[johnholmes]

Interesting data now showing the Canadian data is regularly up with the big 3, seems to out perform GFS quite often, the current situation with the longer term averages are in the link below

http://www.emc.ncep.noaa.gov/gmb/STATS/STATS.html

I'm not sure if its coincidence but that seems unlikely but GFS has certainly improved since late July when its latest update started. Both at 5 and 6 days.

Another interesting statistic is that all models show lower scores now than they did Jan-March. Click on the archive button in the link above to browse the data available.

Just goes to prove that the weather will, as always, do what it wants regardless of what any weather model may suggest. But its fun trying to pick which one is the more correct at any one time?

[summer blizzard]

I recall that my 12Z Model Comparisons in 2006-07 bore out your assertions- I looked at GEM, NOGAPS and JMA and all lagged behind the "big three" by a large margin, with NOGAPS performing particularly poorly.

I do not think the GEM model existed in 2006, i believe you were thinking of GME.

Which one is the Canadian model?

[johnholmes]

I do not think the GEM model existed in 2006, i believe you were thinking of GME.

Which one is the Canadian model?

on the link I posted Can?

[summer blizzard]

As you can see, at 240 hours it is effectively the GFS and GEM Vs ECWMF, i was unable to get the JMA after 72 hours, there may be a discrepency because the GFS is 6z, but we will have a look at the 0z models on 22nd August.

Looks like the GFS and GEM models won this round, the ECWMF was nowhere near the mark, however all three models underestimated the height rises over Greenland and the strength of the Sub-Tropical Jet Stream.

This is the actual chart..

As you can see, no model got it spot on, however i think the GEM models has this one, GFS was just about 24 hours too progressive.

I will post the next set of charts tommorow.

[Thundery wintry showers]

I do not think the GEM model existed in 2006, i believe you were thinking of GME.

Which one is the Canadian model?

GEM existed as far back as 2004/05.

[johnholmes]

Hopefully I will post an update on my T+240 checks between GFS and ECMWF before too long.

[summer blizzard]

Here are the next set of charts for 240 hours, all very different.

Will be interesting to see which model gets it right.

[Harry]

Here are the next set of charts for 240 hours, all very different.

Will be interesting to see which model gets it right.

Hopefully ECMWF

[bluearmy]

Here are the next set of charts for 240 hours, all very different.

Will be interesting to see which model gets it right.

maybe if GEM is first again, JH might consider making it GFS/ECM/GEM at T240 rather than just ECM/GFS? annecdotally, i feel that GEM has done a reasonable job over the past couple of months, having never really gone for any extended fi settled options which has turned out to be the right call.

[summer blizzard]

Here are the next set of charts for 240 hours, all very different.

Will be interesting to see which model gets it right.

http://www.wetterzentrale.de/pics/Rtavn001.png

Actual chart saw a high to the east and a low to the west. All models got the low to the west modeled correctly however the GFS and GEM were too progressive meaning that the ECWMF won, with GFS second and GEM third.

[johnholmes]

maybe if GEM is first again, JH might consider making it GFS/ECM/GEM at T240 rather than just ECM/GFS? annecdotally, i feel that GEM has done a reasonable job over the past couple of months, having never really gone for any extended fi settled options which has turned out to be the right call.

hi ba

Thanks for the comment but I am doubtful I will find time to start including it in my updates due to restart this week.

http://www.wetterzen...cs/Rtavn001.png

Actual chart saw a high to the east and a low to the west. All models got the low to the west modeled correctly however the GFS and GEM were too progressive meaning that the ECWMF won, with GFS second and GEM third.

yes I would agree ECMWF provided the best guide. One of its few successes, at least on the 7 or 8 I've kept a check on. I was surprised when this happened so maybe it was just a shortish spell, mind you it has been over about 5 months I've done those few checks.

[summer blizzard]

Next set of charts..

[BlueHedgehog074]

I know it's far out in FI, but I couldn't help thinking that ECMWF were over doing that Low far out to the West and now show something different with the 00Z output.

...Though interestingly, they do bring back a mean-looking Low the day after (at 240 hrs), while the GFS has the strong Low sticking to and lasting a little beyond September 21st. Wonder how they and the other models will shape this in future runs and whether they've spotted any particular trends from this.

Edited September 12, 2010 by Rainbow Snow

[peterf]

Hopefully ECMWF

i'll second that lol.

[johnholmes]

the latest check the 11th so far over a period of several months and ECMWF was the clear winner. It forecast well the synoptic chart we see at 12z today. GFS was close to no use whatever, its worst so far.

Currently on the 11 checks I've done GFS has got 5 correct, ECMWF 2 with 4 showing neither offering acceptable advice.

the next check, T+240 is when the ex hurricane may affect this end of the Atlantic. Currently the two models are diametrically opposed, GFS has pressure of 1020mb in about the same place as ECMWF has pressure of about 980mb. So an interesting 10 days to wait and see which is giving the best advice.

Edited September 15, 2010 by johnholmes

[summer blizzard]

Here was the actual chart, and yet again, the GEM model has victory, with GFS second and ECWMF well off.

Have the days of GEM being regarded as a back water model passed, quite possibly?

In order to best find out which model has it best, we will operate a table format, 2 points for victory, 1 point for second and 0 points for last.

Here is the running from the three checks thus far..

GEM: 4 points

GFS: 3 points

ECWMF: 2 points

Close going thus far, but should be interesting over the longer time period (240 hours).

I will post the next set of charts tommorow probably.

[summer blizzard]

Next set of charts..

[johnholmes]

Update on 7-15 day outlooks and performance at 10 ahead by models(T+240 hours) checks I've done

General comment is that charts suggest unsettled into first week of October at least seems the most likely pattern of weather. There are some differences of the 3 500mb anomaly charts but essentially they all show a strongish westerly flow over the Atlantic into a major trough over/near the UK. This mostly fits with some of the rather longer term teleconnections. The 10 day synoptic outputs from GFS and ECMWF are, as is often the case, quite different. GFS shows an upper ridge and surface high developing south or south west of the UK. ECMWF continues with the them mentioned above. I’ll use this as the next check. The general 5 and 6 day NOAA checks are a bit variable with the 3 main models exchanging the lead. The last 10 day check I did, finishing on 25 September showed GFS once again providing by far the best advice for the UK; the upper ridge to the west/north west and upper trough east of the UK being well predicted. Currently the position after 12 checks spanning about 9 months with GFS being the best guidance on 6 occasions, ECMWF on 2 with 4 offering no helpful guidance from either of them.

[summer blizzard]

Next set of charts..

No model got it spot on, however GEM picked up on the primary low being to the north west, thus it is another victory for the GEM model, ECWMF was close and GFS was nowhere near.

Current table after four 240 hour checks..

GEM: 6 points

GFS: 3 points

ECWMF: 3 points

[bluearmy]

given how well ecm picked up on this upcoming scandi high over a week ago with GFS completely wrong, can we not put a tick in ecm's box. seems churlish not to reward it for picking up a pattern change. incidentally, GEM wasnt too rusty on this either.

normally, when the two main models are a long way apart, a middle road solution is the outcome. on this one, GFS was nowhere near.

[johnholmes]

I'll drop my score sheet in here, just with ECMWF and GFS. I'm on to my 13th with 12 results so far, the first one started way back in March so there is a reasonable coverage of differing seasons.

Currently GFS has given good guidance on 6 occasions

ECMWF on 2

with neither giving good guidance on 4.

To me the surprise has been how much better GFS has performed, and remember these checks started well before any upgrade. The fact that neither gave good guidance on 4 occasions is rather disappointing, as that amounts to around 30%.

given how well ecm picked up on this upcoming scandi high over a week ago with GFS completely wrong, can we not put a tick in ecm's box. seems churlish not to reward it for picking up a pattern change. incidentally, GEM wasnt too rusty on this either.

normally, when the two main models are a long way apart, a middle road solution is the outcome. on this one, GFS was nowhere near.

I'm not too sure about how well its done to be honest as the 13th check, see below for dates, and ECMWF had a large upper trough from Greenland, through Iceland across the UK and into Iberia; GFS had a surface high centred just off SW England?

I'll put the whole check in during Thursday for all to see how the 2 models predicted the overall weather pattern for Thur 27 Oct 2010.

No 13 check

13the

T+240 checks us12z Monday 27 September for Thursday 7 October 2010

[Nick F]

given how well ecm picked up on this upcoming scandi high over a week ago with GFS completely wrong, can we not put a tick in ecm's box. seems churlish not to reward it for picking up a pattern change. incidentally, GEM wasnt too rusty on this either.

normally, when the two main models are a long way apart, a middle road solution is the outcome. on this one, GFS was nowhere near.

GFS does still seem to have an in-built default, once the Atlantic lows/westerlies awaken, for Atlantic systems to affect the UK to the end of the its runs, despite signals picked up by ECM and perhaps other models that this may not be the case. I still think that GFS is good at handling more mobile westerly weather patterns or northerlies and picking up on them to develop first, while ECM is rather good at handling blocking situations and picking up on them first. Though this perhaps is a rather personalised and generalised view which is not backed up by stats!

[bluearmy]

I'll drop my score sheet in here, just with ECMWF and GFS. I'm on to my 13th with 12 results so far, the first one started way back in March so there is a reasonable coverage of differing seasons.

Currently GFS has given good guidance on 6 occasions

ECMWF on 2

with neither giving good guidance on 4.

To me the surprise has been how much better GFS has performed, and remember these checks started well before any upgrade. The fact that neither gave good guidance on 4 occasions is rather disappointing, as that amounts to around 30%.

I'm not too sure about how well its done to be honest as the 13th check, see below for dates, and ECMWF had a large upper trough from Greenland, through Iceland across the UK and into Iberia; GFS had a surface high centred just off SW England?

I'll put the whole check in during Thursday for all to see how the 2 models predicted the overall weather pattern for Thur 27 Oct 2010.

No 13 check

13the

T+240 checks us12z Monday 27 September for Thursday 7 October 2010

ecm picked the scandi high on its 00z run 29/09. (with only one subsequent op run not consistent). i believe gfs took until friday 01/10 to catch on with any consistancy.

[johnholmes]

Following on from that post, the pdf below shows how the 13th check turned out.

No 13 check-mond 27 sept for thur 7 oct.pdf