12z Model Comparisons Results

[Thundery wintry showers]

model_comparisons.xlsModel Comparisons Results for Monday 20th March 2006.

Current Situation: Easterlies over Britain with northerlies flooding into Shetland. 1060mb Greenland High, and there is a strong 985mb low over west Russia. 1005mb troughs can be seen over W Scandinavia but not close enough to affect Britain.

GFS: Had us in cold northerlies at T+144, having the Scandinavian Low too far west. Northerlies were still prevailing at T+120, but at T+96 it was spot on with the evolution.

Only 6/10 because while it got the general low/high positions correct, it had us in cold northerlies at T+144 and T+120.

UKMO: Had northerlies over Britain at T+144, similar to the GFS. At T+120 it showed an evolution similar to ECMWF's phantom north-easterly for yesterday, with a low over Denmark propelling NE winds in from Scandinavia. At T+96 it was close to the correct pattern but still had northerly winds across the whole British Isles in association with the Arctic airmass.

Only 4/10 for the UKMO. The general positioning of lows and highs was good, but it had us in north/north-easterly winds at all three timeframes.

ECMWF: Had us in northerlies at T+144. Changed to cold NE winds at T+120, so far almost identical to UKMO. At T+96 however the outcome was spot on.

A 6/10 for ECMWF; it wasn't great, but it did better than UKMO and about the same as GFS.

NOGAPS: Well off the rails at T+144 with a big low over France and strong NE winds over Britain and a strong Arctic influence. It went for straight northerlies at T+120, and light NNE winds as per UKMO at T+96.

Only 3/10, it was slightly further out even than the UKMO.

JMA: Not too bad at T+144, with the coldest air only just reaching Shetland, although winds were shown as N/NE'ly. At T+120 it went for straight northerlies and cold air though, but then T+96 was spot on.

I'd have to give this a 7/10- T+120 was well out but T+96 was good, and T+144 was better than GFS and ECMWF.

GEM: Went for straight Arctic northerlies at T+144. At T+120 it was close to the correct outcome though. At T+96, sadly, it was unavailable.

I don't know how the GEM performed at T+96, but on the evidence of it being good at T+120 I will give it a 7/10.

The models for today showed a classic "downgrade" situation with potent northerlies/north-easterlies being replaced by only moderately cold easterlies at T+96. UKMO and NOGAPS still persisted with a northerly regime at T+96, while JMA and GEM hinted at the downgrade at T+144 and T+120 respectively.

[Thundery wintry showers]

model_comparisons.xls21st March 2006 Comparison Results.

Actual Situation: We have light easterlies over Britain at the moment, with northerlies in the far N of Scotland, 1055mb Greenland High, 1005mb trough over W Scandinavia, deep 985mb low over NW Russia.

GFS: Had northerlies right over Britain at T+144, at T+96 it had high pressure right over Britain, but at T+96 it had light easterlies over Britain as turned out to be the correct outcome.

Only 5/10. The GFS is certainly consistently good out to T+96 at the moment, but seems to have big problems at T+144 and T+120.

UKMO: Kept northerlies over Britain at T+144 and T+120, but went for light easterlies at T+96. Also 5/10.

ECMWF: It went for northerlies at T+144, but at T+120 and T+96 went for the correct evolution with light easterlies, although specific details were diffferent especially at T+120 where high pressure was more dominant.

ECMWF didn't handle it great, but since it was much closer at T+120 I'd have to give it a 7/10.

NOGAPS: Another model to get 5/10; went for northerlies at T+144 and T+120, but correctly went for light easterlies at T+96.

JMA: Another 5/10- it went for northerlies at T+144, a high to the east of Britain at T+120, but light easterlies at T+96.

For GEM only the T+144 is available, not enough data for me to give it a mark I'm afraid.

A tendency for all models to struggle at T+144 and T+120, becoming correct at T+96.

[johnhan61]

I've just been reading your comparison results - looks like the sort of thing I'm looking for. See my topic titled "UKMO STRENGTHS/WEAKNESSES"

Will your end result be compiled data validating specific strengths and weaknesses of the models given the same synoptic parameters?

[Thundery wintry showers]

That kind of thing is envisaged, but this model comparisons idea has only been running a couple of weeks and it will take a lot more time before any conclusions can be made.

[johnhan61]

Well, keep up the great work TWS - impressive stuff that I plan on paying attention to daily. We go through a twice-daily initialization and verification procedure here on the UKMO global model based on both the 00Z and 12Z outputs. This is part of our daily forecast process. We then attempt to apply adjustments to TAF's - i.e. speed/slow movement of trofs/fronts; decrease/increase ceiling forecasts etc. etc.

[johnholmes]

hi TWS

Perhaps this thread should be put in the Extra area if USAF are going to use it?!

Seriously PAUL if you read this, how about some money from them?

John

[The PIT]

TWS very good.

One Question what is the spread sheet showing? Marks out of ten???

Needs better labeling but graphs and spreadsheets generally seem poorly labeled by People interested/ doing the weather.

Guess the teacher didn't drill it into them enough at School.

[Thundery wintry showers]

They are marks out of ten. The row at the top gives the average mark.

[Paul]

TWS very good.

One Question what is the spread sheet showing? Marks out of ten???

Needs better labeling but graphs and spreadsheets generally seem poorly labeled by People interested/ doing the weather.

Guess the teacher didn't drill it into them enough at School.

I think teachers in certain schools in Sheffield also weren't too hot on manners either.

[pottyprof]

paul.. must be a north south split.. :unsure:

Great work TWS.. thanks for taking the time to do this.. only found it tonight.. follows my train of thought.. great stuff!! :unsure:

[johnhan61]

Guys, just to make something clear (before I get myself into trouble) - the USAF aren't looking at this. I'm personally looking at it and I just happen to work for the USAF. This is all my own personal interest in the subject.

[Thundery wintry showers]

model_comparisons.xlsSo, who takes the prize for the prediction for today?

Model Comparisons for Wedensday 22nd March 2006: low at 1005mb over W Scandinavia, at 1010mb over W Scotland, and a weak 1015mb high over England & Wales.

GFS: It wasn't too bad at T+144; had the high a bit too far east giving southerlies, but the general pattern was correct. Trough not picked up, or at best underdone, off W Scotland. At T+120 it again had a similar pattern to the outcome, but the high was a bit too far east again and ESE winds were indicated for southern Britain. Trough picked up off W Scotland. At T+96 it correctly had the 1015mb high over Britain and the low off NW Scotland.

I will give this run a 8/10. The trough off W Scotland was picked up at T+120, and while the high was centred a bit too far east at T+144 and T+120, the general sunny anticyclonic outlook was correct.

UKMO: North-westerlies at T+144: not good. At T+120 the anticyclonic outlook was correct (it had a 1020mb high over Britain), and kept it at T+96. However, the trough affecting W Scotland was underdone, unlike on the GFS.

I give this 7/10- T+144 was off, but T+120 and T+96 were good, though with the high overdone and the trough for W Scotland underdone. Slightly inferior to the GFS this time around.

ECMWF: Spot on at T+144 with 1015mb high over Britain. At T+120 it was also good, with 1020mb high over Britain, but at T+96 it went off a bit. Correctly spotted the trough over W Scotland but overdid it somewhat, with our high to the east of Lincolnshire rather than over Britain.

I will give this an 8/10 as it was excellent at T+144 and T+120, but have to dock it a mark for being less accurate than either UKMO or GFS at T+96.

NOGAPS: North-westerlies at T+144, no T+120, at T+96 had the high over Ireland and cold northerlies elsewhere.

Only 4/10 for NOGAPS- it over-egged those northerlies even at T+96, though at least it got the idea of high pressure correct.

JMA: North-westerlies at T+144. At T+120 it got the high correctly over Britain, and at T+96 it picked up the trough off W Scotland.

A 7/10, same as the UKMO, a mark taken off relative to GFS because of the poorer T+144 output.

GEM: Completely unavailable, so no mark.

Overall quite a good performance by the models, with the GFS and ECMWF having the best all-round outputs, UKMO and JMA waywayd at T+144 but good at T+120 and T+96. NOGAPS again did poorly though.

The overall standings are inconclusive so far, ECMWF has the highest score but not by much, with only the NOGAPS significantly behind.

[Thundery wintry showers]

model_comparisons.xlsWe've already seen how poorly the GFS did at T+168 for today; now for the general model comparisons at T+96, T+120 and T+144. Today's situation on 23 March 2006: 1015mb high just to the east of England, slight 1010mb trough over NE Scotland, 980mb low situated at around 48N, -20W (to the W of southern Britain) There was still a whopping 1050mb Greenland High.

GFS: Not too bad at T+144, it had the lows and highs in the right places but the specifics were slightly different with an ESE flow projected for southern areas that didn't materialise. It went off at T+120, having a 995mb low SW of Iceland (where did that come from?), the high slightly overdone at 1020mb, and south-easterly winds across the country. At T+96 it was well away from the correct outcome, 1000mb trough over southern Britain (?) and easterly winds over Scotland.

I give it 4/10; it was quite good at T+144, but given the very poor output at T+96, it certainly wasn't one of the GFS's more reliable runs.

UKMO: At T+144 it wasn't bad, the high to the east was slightly overdone at 1020mb and the low to the W placed further west than it turned out, but light southerlies and sunshine was still on the cards. The same mistakes were committed at T+120 (this time 1025mb high) but generally good, and at T+96 it was almost spot on.

This run gets 8/10, it was clear at all three timeframes what the correct outcome would be, just a tendency to overdo the high to the east at T+144 and T+120.

ECMWF: Again not bad at T+144, with the high slightly overdone (1020mb), the low placed a little to the west of where it turned out, and light southerlies. At T+120 it wasn't as good, with south-easterlies shown over Britain and high pressure to the N as well as the E. At T+96 the lows and highs were in optically similar positions but Britain was under uniform pressure rather than southerlies.

6/10 for the ECMWF, some inconsistency showing in the near timeframe, but not as bad as GFS.

NOGAPS: 1000mb low over NW Scotland at T+144 with SE England High and cold origin south-westerlies over Britain- miles off. However, it was remarkably accurate at T+120 and again T+96.

This was certainly one of the best NOGAPS runs I have seen, despite being poor at T+144, and scores 8/10.

JMA: Southerlies over Britain at T+144, albeit rather overdone in strength with the high pressure to the E somewhat underdone. At T+120 it was poor, with a 1010mb trough in the North Sea, north-westerlies over NE Britain and south-easterlies in the SW. It was quite good again at T+96 with the high slightly overdone to the E, but the general pattern spot on with southerlies.

I give this 6/10 also- it was rather poor at T+120 with a rather different outcome to what actually happened, but good at T+96.

GEM: Only T+96 is available, which I don't think is enough to give it a rating.

So a very good UKMO for today, and surprisingly, a good NOGAPS too. My findings also back up John's comments in his GFS Checks thread that the GFS for today was a poor run.

[Thundery wintry showers]

Results for Friday 24th March 2006, let's see if the GFS can improve on yesterday, it needs to, as it is falling rather behind the Euro models (totally against my initial expectations)

We had a 985mb low covering southern Ireland, SE winds over Scotland and southerlies pushing up into England and Wales, cold with cloud and rain for most but mild with a spring mix of sunshine and showers in the south. There was still a 1045mb high over Greenland.

GFS: At T+144 it had the low too far to the SW, and Britain under ESE winds and still in the cold air. At T+120 it had a channel low, and strong easterly winds over Britain, and us still very much in the cold air. It was closer at T+96 with the 985mb low over SW England, but still didn't have the mild air making much progress, us still very much in the cold easterlies.

Surprising, given the GFS's reputation for being too progressive, it simply wasn't progressive enough this time. Only 3/10, it repeated the same mistake at all three timeframes.

UKMO: At T+144, if anything, it over-egged the mild air, with the southerlies sweeping in over the whole country and the low positioned too far north-west, nonetheless it was closer than the GFS to the right outcome. At T+120 the lows were too far south, and it had us in cold easterlies. T+96 was okay, with the low situated a bit far north, but the pattern of SE winds was correct, and the mild air moving into the south.

I give this a 6/10, it had its difficulties with the low never in the right sort of position, but at least it was closer to the right outcome than the GFS and had the mild air pushing north at T+144 and T+96.

ECMWF: Good at T+144- the low in approximately the right place, the low in the wrong shape resulting in strong easterlies for Scotland and N England and southerlies for the south, but the general pattern of mild with sun/showers in the south, cold with rain elsewhere was accurate. However at T+120 it committed the same error as the GFS with a channel low and easterlies, and at T+96 also committed the same error as the GFS- low too far south, mild air too far south, easterlies for all of us.

This run gets 4/10 because of the poor T+96 and T+120 evolution, but as T+144 was accurate, it deserves a higher score than the GFS which was poor at all three timeframes.

NOGAPS: Not too bad at T+144, the low too far west and south-easterlies over Britain, but at least it had the milder air moving in. At T+120, same error as many others, low too far south, easterlies for Britain, and again at T+96.

This also gets 4/10- it was closer at T+120 than ECMWF but further away at T+144.

JMA: Quite good at T+144, the low close to the right place and SE winds over Britain. At T+120 it had the mild southerlies pushing north, if a little over-egged, and the low in the right sort of position. However, at T+96, once again, there was the problem of low too far south, cold air too far south.

This gets 6/10, let down by being wrong at T+96, but decent at T+144 and T+120.

GEM: Unavailable at T+144. T+120 was pretty much perfect in terms of the Britain details, with SE winds in the north and southerlies in the south, though detail was lacking (as is always the case with GEM) and the low was too far west. Detail again let it down at T+96 with the main low too far west, it had easterlies in Scotland and SSW winds in the south, but again it had the battleground in exactly the right place.

I'd have to dock a mark or two for lack of detail which can make all the difference in these setups, but a good 8/10 I think.

So today's winner, surprisingly, is the GEM which had the mild air coming in to the correct extent. It was remarkable how all of the models underestimated the arrival of the mild air at T+96, apart from the GEM. In the overall standings the two Euro models- ECMWF and UKMO- are in the lead with an average close to 7/10, while GFS slips below 6/10 average.

model_comparisons.xls

[Thundersquall]

So today's winner, surprisingly, is the GEM which had the mild air coming in to the correct extent. It was remarkable how all of the models underestimated the arrival of the mild air at T+96, apart from the GEM. In the overall standings the two Euro models- ECMWF and UKMO- are in the lead with an average close to 7/10, while GFS slips below 6/10 average.

It seems that about half the models had the cold air too far south,

which begs the question was it too far south or is this part of an emerging trend?

[Thundery wintry showers]

Results for Saturday 25th March 2006- did the models correctly predict the onset of milder weather?

We still had a 1045mb Greenland High, and a 980mb low out in the Atlantic, with a 995mb trough over western Scotland, and south-westerly winds over the British Isles.

GFS: A long way off at T+144, with northerlies in occupation. The northerlies were replaced with easterlies at T+120, but at T+96 it went for a SSW flow, which was close to the actual outcome.

I give this run 4/10, it was very good at T+96, but very poor at T+144 and T+120.

UKMO: At T+144 it had low pressure over East Anglia, the mild air getting into the far S and easterlies elsewhere. Closer at T+120 with easterlies for Scotland but SW winds over England/Wales, but went for south-westerlies at T+96, though the low to the NW was overdone somewhat.

This run gets 6/10, it was close to the right outcome at T+120 and very good at T+96, but T+144 lets the UKMO down again.

ECMWF: Very poor at T+144- Alps low, and north-easterlies over Britain. The low was over the Baltic at T+120 with light easterlies with a northerly source over Britain. However, it was accurate at T+96.

The ECMWF scores 4/10 because it was about as accurate (or inaccurate?) as the GFS.

NOGAPS: Had Britain in northerlies at T+144. +120 was unavailable, but T+96 was still off, with Britain in the mild air but covered by light westerlies rather than moderate to strong south-westerlies.

This run gets 3/10 because of the less accurate T+96.

JMA: Similar to UKMO at T+144 with the low straddling southern Britain, westerlies in the S and easterlies in the N. Worse at T+120 with cold easterlies and a low over the Baltic. It wasn't bad at T+144 though, with the trough to the NW overdone/put over the British Isles, but SSW winds projected.

I'd give this one a 5/10, it was better than the GFS and ECM, but not quite as good as the UKMO.

GEM: Strong easterlies for all at T+144, and had northerlies over most areas at T+120. However I don't think I can give this run a mark as T+96 was missing, and T+96 was the period that bolstered the mark for most of the other models.

Overall a rather poor set of outputs for today, with UKMO just edging it to the winning post. I still don't know what on earth has happened to the GFS, I'm sure it was far more accurate than this before, and it currently languishes in second last place. ECMWF and UKMO are approximately equal on an average of 6.6/10.

model_comparisons.xls

[johnholmes]

I have to agree with you Ian. It is nothing like as good at T+168 as it was and even down to T+120 is nothing like as good as it was. It really is a puzzle. I've never had much luck e mailing NOAA for any response so no idea how to discover what the problem is.

John

[Thundery wintry showers]

The results are coming in for the 26th March 2006. With the UKMO and ECMWF currently neck and neck, which model will do best today?

Synoptic Situation: 995mb low over N Scotland, SW winds over S Scotland, England and Wales. 980mb low in the Atlantic to the west of Britain.

GFS: Everything was too far south at T+144, with the trough over central England (as opposed to Scotland) and the main low at 975mb. At T+120 it correctly had southerlies over Britain, but there was a secondary low over N Ireland, rather than over N Scotland, and intensified. At T+96 it made the same mistake as at T+144, with the mild air too far south.

This run gets a 5/10, as it was never correct, but at least it had some of the specifics correct at T+144 and T+120.

UKMO: Lacking some detail at T+144 but more or less spot on with south-westerlies sweeping NE over Britain. Not bad at T+120 also, the low over N Scotland a bit overdone and positioned a little too far west, but south-westerlies sweeping over Britain was correct. At T+96, had the low too far south, much like GFS, though other details were accurate, consequently winds were shown as light over Britain.

This run gets a 7/10, with T+144 and T+120 being very good, but T+96 being disappointing.

ECMWF: Well out at T+144 with the mild air having not reached us yet, and south-easterlies over Britain, and a 1030mb high to the north. At T+120 similar mistake to the GFS with the main battleground low too far south, and south-westerlies restricted to southern areas, easterlies in Scotland. At T+96, it was spot on.

It's hard to mark this relative to the UKMO: worse T+144 and T+120 but much better T+96. On balance I'll give it a 6/10 as the T+120 and more particularly T+144 were well out.

NOGAPS: Had us under north and east winds at T+144. However, it had SW winds over Britain at T+120- lacking detail, but the pattern was there. It was pretty close also at T+96, with a 990mb low to NW of Britain, and south/south-westerly winds, though the specific details were different.

This run gets a 7/10, poor T+144, but goot at T+120 and T+96.

JMA: Same error as the ECMWF at T+144, with SE winds over Britain. However it was good at T+120, then at T+96 was also pretty accurate, the only niggle being the low over W Scotland a bit further south than turned out to be the case, but still SW winds accurately predicted over England & Wales.

This gets 7/10.

GEM: Way out at T+144, showing easterlies over Britain. No T+120. At T+96, it had ESE winds over Britain, so was still inaccurate even at T+96.

Although there is no T+120, I'll award this run a 2/10, and even that's probably a bit generous. GEM was the model that consistently kept the cold air locked in, and it showed.

model_comparisons.xls

[reef]

Its amazing how poorly the GFS has done over the last 4 days or so when zonal conditions have often been noted as its strong point. It seems the achilles heel of the model is when any sort of blocking is around, especially so when the Greenland high is involved.

The UKMO seems to really be excelling itself in the current synoptics, interesting as this was often called as "inaccurate" at T+144.

As ever, great work TWS and thank you for all the time and effort put into these comparisons.

[Thundery wintry showers]

Model Comparisons Results for Monday 27th March 2006.

We have a strong 980mb low centred to the W of Scotland, light winds over Scotland, and strong westerlies over England and Wales.

GFS: At T+144 it had an incredibly complex picture with lows to the W, SW, N and NE! Despite this, the pattern was for westerlies over Britain, turning southerly in the south- overall I'd have to say a poor output though. Poor again at T+120 with lows to the W and E, light winds over Britain, and north-easterlies in Scotland. However, it was spot on at T+96.

This gets a 5/10, it was very poor at T+144 and T+120, but saved from disaster by T+96, which was excellent.

UKMO: Had S or SW winds over Britain at T+144, and lows to the west. Poor at T+120 with a low off NE England and easterlies in Scotland, low too far out in the Atlantic, but SW winds over England and Wales correctly predicted. It was relatively close at T+96 but had our low too far to the SW, so although westerlies were indicated over England and Wales, there were easterlies for the far north of Scotland.

Only a 4/10 for this run- it was sub-par at all timeframes.

ECMWF: Cold air still over the north at T+144, ESE winds in Scotland. Quite good at T+120 with light winds over Scotland and SW over England and Wales, details close to the actual outcome. However, at T+96 veered off-course again with the low too far south, easterlies in N Scotland, westerlies in S England.

Only 5/10 for the ECMWF.

NOGAPS: Quite good at T+144, with strong westerlies over Britain. Sadly T+120 and T+96 were unavailable so I can't give this one a mark; it has the best T+144 of any model.

JMA: Also pretty good at T+144, with WSW winds. Had the lows too far west at T+120 though, with SW winds in the south rather than WSW. The JMA let itself down at T+96 with a low to the SW of Britain, easterlies in N Scotland. though south-westerlies were predicted for England and Wales.

Another 5/10, nothing spectacular but at least the general pattern was there.

GEM: This run gets another 2/10 as it had the cold air over all but the extreme south of Britain at both T+120 and T+96 (T+144 unavailable).

A rather poor set of outputs from all of the models today.

model_comparisons.xls

[shuggee]

Amazing that the JMA is perceived as more accurate than the GFS using your methodology Ian. Obvisouly we need lots more runs and data comparisons - but an interesting start!

[Thundery wintry showers]

Model Comparison Results for Tuesday 28th March 2006.

Actual: 985mb low to N and NE of Britain, strong westerly winds veering NW'ly in the west. Secondary low, 1000mb, to the W of Ireland.

GFS: Had northerlies over Britain at T+144. It was close to being spot on at T+120 though, with the only error being the low to the N being slightly further west than it turned out. It wasn't bad either at T+96, again the main low a little too far west and slightly underdone in intensity this time, but overall a good punt.

I'd give this a 7/10, it's still not great, but certainly one of the best GFS runs for a while.

UKMO: A very poor T+144 with low pressure over the S and easterlies for Scotland. It looked vaguely recognisable at T+120 but had northerlies over Scotland and N England, the main problem being that the low to the N was positioned over Scandinavia. Closer at T+96 but still had the low positioned too far east, with northerlies for Scotland and N England again.

Only a 4/10 for this one.

ECMWF: Still the mild air not getting to northern areas at T+144 with easterlies over Scotland. Closer at T+120, but there was a low to the NW that didn't come off. The westerly pattern over England and Wales was correctly predicted though, and overall similar to what actually happened. Pretty accurate at T+96.

The ECMWF deserves a 7/10 also, about the same as GFS.

NOGAPS: Same as ECMWF at T+144. No T+120, but decent at T+96, showing north-westerlies over Britain, the low to the north positioned slightly too far east. I give this 6/10.

JMA: Wrong at T+144, with south-westerlies over Britain except in N Scotland (easterlies) and lowest pressure to the NW. Wrong at T+120 too, with northerlies. It was good at T+96 though, the low a little further south than actually happened but everything else spot on.

Because of the good T+96 I will give this a 4/10, but with T+144 and T+120 being so poor, it would be a stretch to give it anything upwards of half marks.

GEM: Similar to ECMWF at T+144. It had southerlies at T+120 with low pressure to the west. Low over southern England (as opposed to N Scotland) at T+96.

One of the worst runs of any model I have seen. I feel justified in giving this a 1/10, it didn't even resemble the correct outcome at any timeframe and the standards of the GEM are really slipping.

So, a victory for the GFS and ECMWF today, which were both poor at T+144, but good out to T+120.

model_comparisons.xls

[Thundery wintry showers]

Model Comparisons Results for Wednesday 29th March 2006.

We have: 990mb low off SW Ireland, 985mb off NW Scotland, westerlies over Britain, starting to change to SW in the far west.

GFS: At both T+144 and T+120 it had us in a west to north-westerly pattern, with low pressure to the north and northeast, but had the secondary lows to the W a bit far west, so convective potential for the day would have been higher and the rain wouldn't have reached western areas. T+120 the more errant of the two, with north-westerlies. Quite good at T+96, but underdid the strength of the westerlies and had the intensities of the lows wrong.

This gets a 7/10, it had the general pattern right, but the approaching front to the W was consistently underdone. Again not a great run but decent, and an improvement on many recent GFS runs.

UKMO: Had high pressure over Britain at T+144. Had southerlies at T+120, with the low to our SW instead positioned just to the W of Ireland. Still overdid the low at T+96, with southerlies pushing eastwards across the country.

It had the general pattern vaguely correct at T+120 and T+96 but consistently overdid the low to the west. I'd say a 4/10 for this one. T+144 lived up to its name this time.

ECMWF: Similar to GFS at T+144- correctly got us in westerlies, but the low to the SW was underdone. Same at T+120. Very good at T+96 with the lows and winds of correct intensity.

ECMWF gets 8/10- a very solid run, which could easily have been used to make a decent forecast for the day.

NOGAPS: Had north-easterlies at T+144, T+120 unavailable, T+96 had a low over East Anglia (?) but the westerly pattern was there.

Hard to mark this one because of the lack of a T+120. I say 5/10.

JMA: Had little wind in the south, easterlies in the north at T+144. Similar at T+120 but with northerlies instead of easterlies in the north. At T+96 had southerlies over most of Britain with the UKMO's mistake of having the low west of Ireland rather than to the SW.

It hit on a vaguely recognisable pattern at T+96, but this run is only a 2/10. T+120 and T+144 were nothing like what actually happened.

GEM: Easterlies at T+144. However, it had westerlies at T+120- lacking detail, but a good start- and at T+96 was also pretty good. Low to the SW slightly underdone.

This gets 6/10, reasonable at T+120 and T+96 but poor at T+144.

Overall a good run with the ECMWF, with GFS quite good too. JMA and UKMO were letdowns.

After a horrendous patch, GFS returns to joint third with JMA, while ECMWF with a mean of 6.6/10 heads the UKMO.

model_comparisons.xls

[Thundery wintry showers]

Model Comparisons Results for Thursday 30th March 2006.

985mb low covering the northern half of Britain, strong W'ly winds for the south. 985mb low out to the west and 985mb low south of Iceland.

GFS: T+144 poor- low over southern Britain with northerlies for Scotland. At T+120 had the low that is currently over us in the North Sea, and a low index pattern, but the other two lows were correctly positioned. Also problems at T+96- it had a generally WSW flow over Britain, but this included Scotland where a tongue of Arctic air was expected to penetrate (but didn't), and our low stuck over southern Ireland.

Just 3/10 for this one, it was inconsistent, and never really correct, though all of the runs identified a zonal pattern.

UKMO: The low pressure to the west was missing at T+144 with northerlies flooding into Scotland. Same errors as GFS at T+120. Same errors as GFS also at T+96!

A 3/10 also, it was as bad as the GFS, but no worse.

ECMWF: Had rain moving east and southerlies at T+144. Southerlies again at T+120. Southerlies again at T+96! The error was consistently having low pressure situated too far west.

It was consistent- but consistently wrong. I give this 3/10 as well.

NOGAPS: Had north-easterlies at T+144, cyclonic ESE winds at T+120, and no T+96.

Since there was no T+96 I can't give it a mark- depending on the T+96 output it could have been anything between 1/10 and 4/10.

JMA: Easterlies over England & Wales at T+144 and northerlies for Scotland. Had strong westerlies and a low off NW Scotland at T+120- not really correct, but at least the right pattern for England and Wales, wrong for Scotland. At T+96 same error as ECMWF- lows situated too far west, and resulting southerlies.

This too only gets 3/10.

GEM: Not bad at T+144- south-westerlies, and lows in similar positions to where they actually turned out, though a little too far west. Similar problems at T+120, low further west this time. Too little detail at T+96, had a cyclonic WSW flow over the whole of Britain.

This run gets a 4/10 as it at least had a W/SW flow correct over England and Wales, but everything else was generally out.

One of the worst days I have ever seen for model performances. Not a single run at a single timeframe got the arrangement of lows and winds even close to the correct positions.

model_comparisons.xls

[Evo]

I think a pattern is emerging from this - we have probably all thought through empirical evidence that each model would cope with certain set-ups well, whilst struggling with others. I was slightly sceptical of this view but your analyses seem to be showing that this is the case.

The JMA has handled certain recent setups well but it hasn't got a clue with others. This even seems to apply to the GFS and the UKMO, though obviously to a lesser degree. The leader does seem to be the ECM, which given that statistically is the most accurate global model, makes sense. More often than the others, it has something which, whilst obviously not dead on, is good enough to make a pretty decent mid range forecast.