12z Model Comparisons Results

[Thundery wintry showers]

10 August

1012mb low off W Scotland with Atlantic westerlies starting to come into the west. Most eastern areas had another dry sunny day.

GFS did better this time, scoring 7/10. T+144 was poor again with a low in the North Sea giving strong NNE winds. T+120 was reasonable but overdid the extent of the low pressure slightly, then T+96 was very good.

UKMO also scores 7/10. T+144 was dodgy with low pressure (slack) over southern Britain, but T+120 and T+96 were good; T+96 had the low pressure a bit too far west though, with pressure at 1020mb over Britain rather than 1015mb.

ECMWF scores only 5/10- it overdid the Atlantic at T+144 and T+120 giving westerlies for all, then T+96 was quite accurate.

NOGAPS scores 5/10- T+144 unavailable, T+120 had low pressure to the east giving strong northerlies, T+96 had high pressure over Britain and the Atlantic lows way too far west.

JMA scores 6/10- reasonable at T+144 and T+96 (though with low pressure too far east at T+96), T+120 was a dodgy mishmash of weak lows.

GEM scores 5/10- high pressure over Britain at T+120 and T+96, a low in the North Sea giving northerlies for the east.

Overall, GFS and UKMO did okay, while the ECM was over-progressive.

[Thundery wintry showers]

11 August

1012mb low over western Scotland with light south-westerly winds.

GFS scores only 6/10. T+144 was good, but then T+120 and T+96 had a strong ridge from the Azores High with 1020mb pressure over the south.

UKMO also scores 6/10- T+144 was very poor with low pressure over the SE, and NE winds for most, but T+120 was reasonable (ridge from Azores overdone) and T+96 very accurate.

ECMWF scores 5/10- T+144 had a large low over the east, T+120 had northerlies but T+96 was accurate. Overall another disappointing ECM run.

NOGAPS scores 5/10- unavailable at T+144, at T+120 it had high pressure over Britain but at T+96 it was quite accurate, low slightly too far east.

JMA scores only 3/10- at T+144 and T+120 northerlies were shown with lowest pressure to the East, while T+96 was recognisable, but still had too much of a pronounced 'northerly' setup, low accurately positioned over W Scotland.

GEM scores 5/10- northerlies at T+120, T+96 had low over Scotland and Azores ridge over the south.

The models still appear to be struggling.

[Thundery wintry showers]

12 August

1004mb low over Scotland and cyclonic WNW flow.

GFS scores 7/10- T+144 and T+96 were quite accurate, but T+120 had Britain in a very 'flat' westerly flow with no pronounced low over Scotland.

UKMO scores only 3/10, in contrast to earlier outputs. T+144 had a strong Azores ridge over the British Isles, T+120 had a low in the North Sea giving northerlies while T+96 had a low over SE England giving NE winds.

ECMWF scores 4/10- T+144 had a strong Azores ridge, T+120 had a low over SE England giving NE winds, T+96 was okay.

NOGAPS scores 5/10- T+144 had an Azores ridge, T+120 had the low too far east giving NW winds, T+96 was quite accurate.

JMA scores 5/10 also- T+144 and T+120 had low pressure too far east giving N or NW winds but T+96 was accurate.

GEM scores 4/10- low too far east at both T+120 and T+96 giving NW winds.

Again the models continue to struggle. In contrast to the past few days, GFS had the best stab at the outcome, while UKMO was particularly poor.

[Thundery wintry showers]

13 August

996mb low over N Scotland with strong westerly winds, but most places had a bright day with a few showers.

GFS scores 6/10- T+144 was reasonable but with too 'flat' a westerly flow, T+120 was accurate, then T+96 was fairly poor with a secondary low over SW England turning the winds to a south-westerly direction.

UKMO scores only 3/10- T+144 had Britain under weak high pressure, T+120 had low pressure too far south and east giving a generally north-easterly flow, while T+96 was similar to GFS, with southerly winds and a secondary low over the SW.

ECMWF available only at T+96 when it had Britain under a weak area of high pressure.

NOGAPS scores 5/10- low too far east at T+144 giving northerlies, underdone and too far south at T+120 and T+96 giving little in the way of wind over Britain.

JMA scores 4/10- low too far east at T+144 giving NW winds, too far west at T+120 and T+96 giving S or SW winds.

GEM scores 5/10- low too far north at T+120, and too far west at T+96 giving SSW winds.

The Net-Weather Model Comparison charts seem to be missing ECMWF all of a sudden, ever since the runs of 8 August.

[johnholmes]

tks again Ian.

One begins to despair of the current standard at T+96 and beyond. When I think back 2-3 years ago and I would have bet at least the cost of a good dinner with anyone for GFS to be 80-90% correct out to T+144 and on occasion even further.

Now all 3 seem to be having trouble. ECMWF is perhaps the worst in my view as it was specifically set up to take forecasting accuracy (80%+) to 10 days and possibly beyond. That was way back in the 70's.

Various upgrades by each of the main 3, your and my money for the UK Met O and still no real 'step' forward in my view. We are close to spending £20m of UK taxpayers money for the UK Met O, so they say, to be able to give really accurate warnings etc.

I wish I could believe it.

Edited August 14, 2007 by johnholmes

[johnholmes]

to add to my comments above

Have a look at the 00z T+240 for ECMWF and GFS? Okay maybe the 06z will be out for GFS by the time you egt to do it. Unless it shows a big change tell me which one you believe.

Without showing the two charts this is a summary

GFS=ridge >1020mb from Eire out to North ermany through the Channel and then back into nw Speain.

ECMWF has a low <1005mb just off our south west approaches !!

And people wonder why the Met O so called 16 day outlook changes from day to day!

I shall look with interest at the 00z chart for that day, 24 August

decision made!

Once I come back from hols, 24 August in fact, I'll start a T+240 hours check with GFS and ECMWF. Just that one run, well if I have time and enough patience, I'll do the 00z and 12z runs for that time. Then 10 days later compare the charts to see which is nearest.

don't panic Paul, I'll use the Net Wx versions!

Edited August 14, 2007 by johnholmes

[The PIT]

Got to admit I'm disappointed how the Models are performing of late. The upgrades don't seem to worth the money as the previous lot seemed to be more accurate. The bad part of it people think T180 is accurate. I know last Winter GFS managed to track a storm impressively from T160 ish. That now seems a distant memory now.

[Thundery wintry showers]

14 August

988mb low off SW England with warm, wet south/south-westerly winds across Britain.

GFS scores 6/10- low pressure to the north at T+144 and T+120 giving a slack westerly flow over Britain and low underdone to the SW, but very accurate at T+96.

UKMO scores 5/10- T+144 had a low over NE Scotland giving northerlies, T+120 was okay but had the low to the SW vastly underdone, giving light southerlies, T+96 was reasonable but had low pressure about 200 miles too far east.

ECMWF unavailable for the time being.

NOGAPS scores 4/10- T+144 had low pressure to the NE giving NW winds, T+120 had a slack low over Scotland giving westerlies, T+96 unavailable.

JMA scores 7/10- T+144 had a low over the British Isles, but T+120 and more especially T+96 were very good.

GEM was also pretty good, scoring 7/10- T+120 had the low to the SW underdone, T+96 was very good.

Overall another dubious set of outputs. Surprisingly JMA and GEM were the most accurate.

[Thundery wintry showers]

15 August

984mb low off NE Scotland and a generally westerly flow, changing to NNW in the north. Sunshine and showers in the north, some big thunderstorms in some southern areas, esp. S Midlands, N Home Counties and later Norfolk.

GFS scores only 5/10- T+144 was its closest output with low pressure out in the North Sea giving northerlies, while T+120 and T+96 had low pressure too far west, situated over central and western Britain giving a strong cyclonic regime. It had the outlook of low pressure right, but positioning was badly out meaning the weather details couldn't be pinned down.

UKMO scores 7/10. T+144 was reasonable with low pressure over NE Scotland, just slightly too far south, with a cyclonic/WNW flow. T+120 and T+96 had the low fairly accurately positioned but without the secondary feature over southern areas, giving a straight WNW flow.

ECMWF unavailable (for the time being) again.

NOGAPS scores 6/10- T+144 had the low underdone but accurately placed, T+120 unavailable, T+96 had the low too far south, out in the North Sea giving northerlies for Scotland.

JMA scores 6/10- T+144 and T+120 were quite good but low slightly too far west giving westerlies, T+96 worse with a low complex on top of Britain.

GEM scores 6/10 also- low pressure over W Scotland, too far west, at T+120 and T+96.

Not too bad, but not great either. UKMO was the most accurate overall.

[Thundery wintry showers]

16 August

980mb low off Shetland and a cyclonic/north-westerly flow. It was a bright showery day for most parts.

GFS scores 7/10- a reasonable output, T+144 had the low too far south, situated right over northern Scotland, and was poor as a result, but both T+120 and T+96 were very good.

UKMO scores only 5/10- all three timeframes, especially T+120 and T+96, had the low pressure overdeepened and too far south, situated over northern Scotland.

ECMWF still unavailable at the moment.

NOGAPS unavailable at T+144. Scores 6/10, as T+120 and T+96 were both reasonable but had low pressure too far south, situated just off northern Scotland (so less far out than the UKMO)

JMA also unavailable at T+144. It scores just 5/10 as the low was over northern Scotland at T+120 and T+96.

GEM unavailable.

Overall the GFS did pretty well, with T+120 and T+96 the first impressively accurate outputs I've seen for some time, though T+144 was poor and brought the mark down to 7/10. Other models struggled. Hopefully the ECMWF will be up soon.

[Thundery wintry showers]

17 August

1000mb low off western Ireland but not yet impacting the British Isles.

GFS scores a 6/10- T+144 and T+120 had the Atlantic system at least 500 miles too far west, giving Britain a moderate WNW flow, but T+96 was quite good with the low knocking on our doorstep.

UKMO also scores 6/10- T+144 and T+120 were worse than GFS with an overdeepened low over NE Scotland, but T+96 was extremely accurate, so I think overall it was about as good/bad as GFS.

Still unable to access ECMWF, but hopefully it should be on the archive shortly.

NOGAPS scores an 8/10- T+144 unavailable, but it was very good at T+120 and T+96 with the low off W Ireland and a moderate W flow.

JMA scores 7/10- poor at T+144 with low pressure over NE Scotland, but T+120 and T+96 were reasonable, Atlantic low only slightly too far west.

GEM available only at T+96 when it wasn't very good; Atlantic low way too far west.

Overall the models struggled a bit again. I was surprised by how good the NOGAPS outputs were.

[Thundery wintry showers]

18 August

1004mb low centred over eastern Ireland. Many fronts embedded in the system, thus it was widely a dull and wet day. Southerly winds in the east.

GFS scores 6/10- low nowhere to be seen at T+144 with a slack westerly flow over Britain, T+120 had the low quite accurately placed but underdone, then T+96 was very good.

UKMO scores 5/10- cyclonic NW flow at T+144, T+120 was quite good but low slightly too far west, then T+96 had the low vastly underdone.

ECMWF again unavailable.

NOGAPS scores 7/10- T+144 was very poor with northerly winds shown, but then T+120 and T+96 were both very accurate- every bit as much so as GFS T+96.

JMA scores 5/10- it had the low at all three timeframes, but too far SW and vastly underdone. Westerlies over Britain.

GEM unavailable.

I hope the ECM outputs will start coming back through soon on the archives. In the meantime, NOGAPS did surprisingly well again, while GFS and UKMO struggled, although GFS was slightly the more convincing.

[Thundery wintry showers]

19 August

1008mb low over eastern England with a cyclonic/northerly flow. Due to embedded fronts in the system most places had a dull cold wet day rather than the bright showery weather associated with straight Arctic flows.

GFS scores 6/10- low off SW England at T+144, over southern England at T+120 but accurately placed at T+96.

UKMO scores 4/10. It was excellent at T+144, but then T+120 and T+96 were poor, with low pressure over SE England and high pressure coming into the north- well out.

ECMWF unavailable. I'll have to get back to Paul about the ECM charts as they still aren't appearing at all.

NOGAPS scores 7/10- another decent set of NOGAPS outputs, with low pressure slightly too far west (by about 100 miles) at T+144 and T+120 but accurately placed at T+96.

JMA scores 6/10- low pressure generally too far south with NE winds, T+120 was poor while T+144 and T+96 were quite close.

GEM only available at T+96 when it showed HP over the north-west of Britain.

[Thundery wintry showers]

20 August

1004mb lows over the south-eastern quarter of Britain with north-easterly winds for most.

GFS scores only 5/10- T+144 had high pressure too close by, giving light northerlies. T+120 was accurate, but T+96 had high pressure to the SW giving NW winds for Britain.

UKMO scores only 3/10 because all three timeframes had high pressure to the SW giving an anticyclonic NW flow.

I still can't get ECM charts; it seems that the 12Z ECM runs don't appear on the Model Comparisons page at the moment.

NOGAPS scores 5/10- good at T+144, unavailable at T+120, poor at T+96 with an anticylonic/NW regime and high to the SW.

JMA scores 5/10 also- low pressure to the south at T+144 giving easterlies, high pressure almost on top of us at T+120, then northerlies at T+96 with low pressure in the North Sea. A dodgy set of outputs.

GEM scores 4/10- high pressure to SW at T+120 giving W winds for all, then T+96 was better but still overdid high pressure to the W, giving NNW winds.

Overall a poor set of outputs- all overdid the Azores High.

[Thundery wintry showers]

21 August

1028mb high to the west, 1004mb low over Holland, and a stiff north/north-easterly flow.

GFS scores 6/10- reasonable at T+144 and T+96 though with high pressure too far east, but poor at T+120 with high pressure centred to the SW giving a north-westerly flow.

UKMO scores 5/10- poor at T+144 and T+120 with high pressure to the S and SW giving westerlies, but T+96 was very good.

The ECM outputs are showing up back to 21 August so in a few days' time I should be getting ECM outputs back, but unavailable for the 15-17 Aug period.

NOGAPS also unavailable except T+120 which had Britain in a straight northerly flow.

JMA scores 5/10- identical to UKMO with a high-pressure outlook and westerlies at T+144 and T+120 but good at T+96.

GEM scores 6/10- T+120 had high pressure to the SW with northerlies over eastern Britain but T+96 was very good.

Overall a tendency to position high pressure too far south and east.

[Thundery wintry showers]

22 August

1024mb high just covering NW Scotland with a NE flow across the country. It was a NW-SE split, but not in the traditional way- south-eastern areas were widely dull, cold & damp while the north-west was sunny.

GFS scores 5/10- T+144 had anticyclonic NW flow around strong Azores High and T+120 wasn't much better, but it was accurate at T+96.

UKMO scores 7/10- poor T+144 with anticyclonic NW flow around strong Azores High, but T+120 and T+96 were both very good, particularly T+96 with almost everything accurately placed.

ECMWF will still be unavailable for a few days.

NOGAPS scores 5/10- T+144 had high pressure to the SW giving a WNW flow, T+120 unavailable, T+96 had a high in the Atlantic giving N winds.

JMA scores 6/10- T+144 had a high to the W/SW giving northerly winds swinging to the west, but T+120 and T+96 had the high only a little too far south, giving NNE winds.

GEM scores 6/10 also- NNE winds at T+120 and T+96 with high slightly too far east/south.

Apart from the usual dodgy T+144, the UKMO did pretty well. Otherwise a dodgy set of outputs with tendency to position high pressure too far south and east, giving a more settled outlook with far more sunshine in the south-east.

[Thundery wintry showers]

No 12Z Model Comparisons over the Bank Holiday weekend, but they're back today- and the ECMWF makes a welcome return.

27 August

1028mb high to the west of Britain with slack northerlies in the east. Some northern and eastern areas had a fair amount of cloud but some other areas were sunny.

GFS scores 7/10- too flat a pattern at T+144 with high centred to the SW and an anticyclonic-north-westerly flow in the north, but T+120 and T+96 were both very good.

UKMO scores a 9/10 as it was almost spot on at all three timeframes- high perhaps a little too far west at T+120.

ECMWF scores 8/10- T+144 more dubious with a cold northerly flow and high too far west, but T+120 and T+96 both very good.

NOGAPS scores 5/10- T+144 unavailable, T+120 had high pressure right over Britain while T+96 had a high to the SW, giving WNW winds.

JMA scores 6/10- high too far west at T+144, too far south at T+120 giving westerlies, but T+96 was good.

GEM scores 6/10 as it had Britain in northerlies, but high pressure was quite a bit too far west.

Overall the models returned to 'situation normal' with the top three models performing well, especially the Euro models, and the three lesser models having difficulties in placement of the high.

[The PIT]

Interesting as the Atlantic calms down the Model scores go up. Doesn't bode well for Winter but it may mean we get old style surprises.

[Thundery wintry showers]

28 August

1028mb high to the west of Britain, a very slack northerly flow over Britain with pressure between 1020 and 1024mb.

GFS was okay, scoring 7/10. T+144 and T+120 had the high too far north, giving easterly winds in the south, though accurate for the north, then T+96 was good.

UKMO also scores 7/10- T+144 had the high too far south giving westerlies in the north, but T+120 was reasonable (high still marginally too far south) and T+96 very good.

ECMWF scores 8/10- high marginally too far east at T+144, covering the UK, with T+120 and T+96 very accurate.

NOGAPS scores 5/10- T+144 unavailable, T+120 had low pressure in the North Sea and northerlies, but T+96 was quite accurate.

JMA scores 6/10- poor at T+144 with low to east and SE and NE winds, but T+120 was okay (high slightly too far west) and T+96 accurate.

GEM scores 6/10- high slightly too far south at T+120 and T+96 with westerly winds in the north.

Not too bad; the ECM was slightly more accurate than UKMO and GFS.

[Thundery wintry showers]

29 August

A ridge of high pressure from 1028mb mid-Atlantic high, bringing settled weather, cloudy for some, sunny for others.

GFS scores 8/10- high slightly too far north at T+144 with easterlies in the far south, but T+120 and T+96 were excellent.

UKMO scores only 5/10- T+144 had a pronounced northerly flow and T+120 also to a lesser extent, due to high pressure being too far west. However, T+96 was good.

ECMWF scores 7/10- T+144 and T+96 both had high pressure too far north, much like GFS at T+144, while T+120 was accurate.

NOGAPS scores 5/10- T+120 had high pressure to the SW giving westerlies, and the same at T+96.

JMA scores 7/10- T+144 had northerlies over Britain and the high too far west, T+120 was very accurate, but T+96 underdid the high a bit.

GEM also scores 7/10- decent at T+120 and T+96 but with the high a little too north and west.

Overall decent outputs (except UKMO and NOGAPS, which were disappointining, especially in the case of UKMO which has been doing well recently). GFS had a slight edge over the ECM, JMA and GEM.

[johnholmes]

UKMO scores only 5/10- T+144 had a pronounced northerly flow and T+120 also to a lesser extent, due to high pressure being too far west. However, T+96 was good.

to see that is really very disappointing, one would have hoped that our own Met O would be able to deal with this much better. There is a huge amount of research been done on northerlies, will they or won't they.

[The PIT]

More worrying is how badly the models handled the southerly tracking Jet stream and the northerb blocking.

Now that things are slightly more normal look at how the scores have gone up. Poor programming ??? Dunno.

[Thundery wintry showers]

30 August

1032mb high to the SW and an anticyclonic north-westerly flow. Many of the model runs went for this scenario for earlier days; will they have gone for it this time?

GFS scores 6/10- it didn't go for the high to the SW scenario at T+144 and T+120, instead placing the high to the west and giving a slack northerly flow, but T+96 was reasonable.

UKMO scores 8/10. In contrast to yesterday's performance, UKMO had the high placed correctly at all three timeframes, but low pressure to the North was underdone at T+144 and to a lesser extent T+120, giving a slacker north-westerly flow over Britain. T+96 was the most accurate of any output.

ECMWF scores 7/10- it was very good at T+144 and T+120, but lost marks due to a poor T+96, with high pressure to the west and a northerly flow originating within the Arctic Circle.

NOGAPS scores 8/10- T+120 and T+96 both had the high a bit too far north, but the pattern of NW winds over Britain around the high was correct, and lows to the north were well judged.

JMA scores 5/10- high to the west at T+144 and T+120 giving NNW winds; T+96 had the high correctly placed, but a low off NE Scotland giving a stronger NNW flow in the NE.

GEM scores 6/10- had the high to the west at T+120 and T+96 giving NNW winds.

Overall a reasonable set of outputs again, with the UKMO coming out on top, in contrast to yesterday.

[johnholmes]

with the UKMO coming out on top, in contrast to yesterday.

the question to ask is why the considerable difference in two days for it?

None of what you are showing Ian leaves me with any great confidence in trying to give meaningful forecasts, at even short time intervals, during this winter.

Once again I pose the question what on earth is the matter with the 'Big 3' ?

[johnholmes]

just to add to my post above.

Ian has now been doing these checks, off and on, and mostly on, since 09/03/06, so getting on for 18 months of data.

Quite a decent amount of data to start to draw some conclusions from.

Okay I know its his interpretation for marking but he is the only one so we should be able to accept his marking fairly through the 18 months.

Some of his tables of how the models are doing in differing weather patterns is pretty interesting, by the way Ian can we have your latest table when you have time please?

I've run my own data on Ian's' basic set with marks over varying periods. Just done one now back to 12/08, although unfortunately ECMWF was not available to Ian for much of that time.

The scores over that period, ignoring ECMWF, were GFS 6.1(6.154) and Met 5.9(5.846).

Neither of these scores really gives me a great deal of confidence at the time scales involved, as I've just posted above. None of the 3 were really as accurate as they should have been in the short term lead up to the two major flood events of this summer.

It leaves me, on here, and my own web site, feeling increasingly reluctant to pretend I can give reliable forecasts beyond about T+72 probably much of the time T+96.

So what some may say. The point I want to make is that I was able to do this in 1990 with an average 'score' of 80% with a very strict client, namely the Royal Air Force.

Having begun to think that GFS was the best thing to a forecaster since sliced bread, see my praise for it out to T+168 on my checks which ran for about 18 months

http://www.netweather.tv/forum/index.php?s...mp;#entry658995

only at the end of that spell did I begin to lose faith in it.

I have to say the past 17 months have not returned any of that faith, not for the other two main models either.

Edited August 31, 2007 by johnholmes