Better Than The Models ?

[jethro]

Alas, natural cycles (ie those that can discerned distinctly, and extrapolated forward - which, I presume, is what MB is doing here) fail dismally in the LI energy exchange. I didn't think you'd take me for a cynic, J?

OK - but I've no idea what the universal margin of error is ....

I think sceptical cynicism is a natural state of mind for most folks these days, myself included.

I've no idea what the universal margin of error is, seems quite large if the model runs are anything to go by.

[VillagePlank]

I think sceptical cynicism is a natural state of mind for most folks these days, myself included.

I've no idea what the universal margin of error is, seems quite large if the model runs are anything to go by.

Unfortunately, the alternate methods used for forecasting weather (RJS/BFTP/MB) against climate (LI) seem incompatible. Which is a shame since climate really is the average of weather so one presumes that if you can get the weather accurately enough for long enough in the future, climate must follow.

We shall see, I guess, within four weeks what constitutes a hit or miss.

Edited January 7, 2011 by VillagePlank

[Timmytour]

I would say the first test of accuracy for his forecast can be judged on the whether the conditions become a newsworthy event or not.

I'm not familar with how often Hamburg experiences extreme weather, but I do know it's not something that crops up on the news much.

If it were to do so, then I'd be inclined to overlook slight differences in pressure and position that might exist and be ready to pay closer attention to what he says in the future. On the other hand if it's not newsworthy I may not even end up checking how accurate he was!

I'm sure there was a guy a couple of years back who used to post up stuff about dead people telling him it was going to snow in a week or so's time. Well one lucky strike gets me faintly curious but it needs to be followed up!

As far as I'm concerned with MB....he's specific in the detail of his forecast and honest in his appraisal of them. He's done enough to get me interested. If someone was telling me lottery numbers that came up regularly, I certainly wouldn't be ignoring him because he hadn't sufficiently explained the science behind his predictions!!!!

[La Bise]

A couple of points:

- None of the models do come with a 100% guaranteed hit rate, they are speculative scenarios. Very few if any forecasters issue a 100% guaranteed forcecast. MB has and has also pointed out the margin of error he allows himself is tiny. Very bold of him and on that aspect, he has clarified things a lot. The verification of his February forecast will need to be extremely close to his forecast to verify, by HIS own conditions. If that canard that nasty sceptics are giving him a hard time could be dropped now, ta very much...

- MB has no commercial interest in his method, as he has stated many times already so any notion he is keeping it for himself as he has to earn a living is not really relevant. Which begs the question why so much secrecy? A broad outline would suffice really, one that makes a bit more sense than what has been said previously.

I do suspect MB has actually sterner test conditions for his experiment than a number of people who would be quite happy and have indeed been quite happy to give the thumbs up to his method despite notable failings. It is a well known effect of the human brain, our capacity to blank out what does not tie up with our convictions in order to highlight what does. You only have to look at the money been made by astrologists and their haphazard, vague and banal predictions...

[callmedick]

if I correctly understand you use a "DNA" signature for the global weather pattern on a day you can find a historic match and if the match is strong enough you have confidence they will follow each other closely for about 5 weeks. Does matching dna values for days following your prediction (where possible) of this great storm still point to the same outcome? If you tried to find a DNA match for conditions right now does it point to a similar forecast, or are strong DNA matches like that which you based your prediction on so rare you dont expect to have another between prediction and verification date. you say you are 100% condient this will happen -yes?

[Pie Man]

I'm finding this thread increasingly like an argument in the pub at closing time.

Its getting boring.

Time to move on.

Pieman

[dinger]

http://

Could this be the first signs of MB's storm. Albeit 2 days early

[AtlanticFlamethrower]

If he is consistently on the right lines then that would be credibility enough; how it is done is irrelevant, even if he read it in tea-leaves.

This is my view.

On a forum like this fun is the first priority.

PhDs matter if MB gets to the stage he wants to publish in an academic journal. Until then these forecasts are for everyone, for whom it's just a bit of fun... and maybe more if he gets another direct hit.

[Timmytour]

This is my view.

On a forum like this fun is the first priority.

PhDs matter if MB gets to the stage he wants to publish in an academic journal. Until then these forecasts are for everyone, for whom it's just a bit of fun... and maybe more if he gets another direct hit.

Totally agree AF

In fact this prediction is keeping my interest up at a time when the lack of prospects for intense cold and snow in the near future would otherwise have it on the wane

Far reaching FI for GFS is now just two days away from MB's first chart posted for 28th Jan..... and it seems a long way from shaping up to how MB would have things looking...

[virtualsphere]

Interesting thread and forecast. I notice the BBC Monthly Outlook for 24th Jan - 6th Feb by Rob McElwee says (with low confidence at this stage):

"The northwest quarter wind prevails but surprisingly rainfall is above average for England and Wales and well above in eastern Scotland. Temperatures are well below average in Scotland and Northern Ireland whilst sunshine is well above average for northern Scotland but below in the Midlands and northeast England.As I like to try and fit pressure patterns to these indications, I suggest a slow-moving depression in the North Sea."

http://news.bbc.co.u.../forecast/10209

[weatherguy]

Indeed...Certainly signs of a storm (coming with -15 uppers in places if I remember correctly from when I looked earlier) seems to be appearing in deep FI on the charts, and in a similar location to where MB suggests; few hundred miles out at the moment but it's not expected to be bang on yet. Long time to go of course, but this could be very good to watch

(Sorry, I don't know how to add a thumbnail...)

Edited January 11, 2011 by weatherguy

[k19]

Indeed...Certainly signs of a storm (coming with -15 uppers in places if I remember correctly from when I looked earlier) seems to be appearing in deep FI on the charts, and in a similar location to where MB suggests; few hundred miles out at the moment but it's not expected to be bang on yet. Long time to go of course, but this could be very good to watch

(Sorry, I don't know how to add a thumbnail...)

Nice to hear someone mention the uppers. Since the airflow around low pressure is anti clockwise, could this potentially give a notherly, and therefore, a very snowy storm?

[weatherguy]

Nice to hear someone mention the uppers. Since the airflow around low pressure is anti clockwise, could this potentially give a notherly, and therefore, a very snowy storm?

It certainly would have the potential if things turn out like MB predicts, but as shown in the last cold spells, predicting snow can be near impossible even T+24 at times

(In that GFS chart however, the high pressure is probably hugging too close to the UK to give the low a chance to penetrate, MB's charts would be more promising though I reckon...)

Anyway, @MB - How is your prediction looking at the moment? Is there a way in your method to "update" your forecast to see if the accuracy is holding up, or do those "DNA matches" restrict you to how often you can forecast for a set day?

Edited January 11, 2011 by weatherguy

[Roger J Smith]

As this forecast period 1-5 Feb begins to come into distant view soon, I thought it might be a good idea to establish how an independent validation of this forecast might proceed, so there are no questions of fitting the validation to circumstances.

First of all, there are maps provided which could give rise to a correlation study. I will undertake that for the map on Feb 3 and extend it to other days if there is any indication of high significance (if Feb 3 has a low or negative correlation then I think the scientific merit of the forecast would be difficult to establish). However, most of us would consider this a good forecast if it verified with a time error of 2 days or less, so I will be open to correlating actuals from Feb 1, 2, 4 or 5 with predicted 3 Feb.

Correlation would tend to overcome the problem that is almost certain to arise, that the pressure pattern is somewhat overcooked on the maps. Granted, you will not have winds of 140-160 mph without lows well down below 960 mbs but looking at the 3 Feb map, I feel that perhaps if you redrew it in 3 mb intervals keeping 1015 constant, you might have a more realistic map (and wind forecasts would drop by 40% as the interval on MB's map is 5 mbs). Correlation would ignore that feature, a perfect correlation would occur if the map verified at 60% strength. By the way, anyone looking at the 3 Feb map, the 1010 isobar is labelled 1100 mbs (typo).

If correlation proved significant, then the question of pattern fitting would arise, because you could get this map on 3 Feb after tracks and evolutions at a sharp angle to those implied on maps for 1 and 2 Feb. So any reasonable validation would probably need to meet these tests of significance, given the forecasts, and after granting that even a half-strength result would be acceptable for scientific significance (in other words, winds gusting "only" to 70 or 80 mph around the North Sea).

The validation, then, should demonstrate these key elements, in a proposed 100-point validation exercise:

(1) Flow through the period is generally between WNW and NNW. 30% of score from this, with 30 points for NW flow and one point deducted for each 6 deg departure from 325 deg in mean 500-mb wind (using Shetland as an index). If there is chaotic flow then we will take segments of the five-day period and score them by weighted time, if the flow co-operates and remains fixed or varying slightly against an average, then we will take the average. (edit ... I originally said 12 deg but the max error would be 180 deg not 360 deg, doh).

(2) Intensity of pattern, 20 per cent. I will base this on the strongest wind gust in the UK or Ireland in the five day period, with 40 knots worth zero, and each 2 kt improvement on 40 knots worth a point, so anything over 80 knots here will earn a maximum (and that's far below the predicted 150 knots).

(3) Timing of primary low, 20 per cent. This will relate to the 945 mb low shown over Sweden on the 3rd. I will take the closest actual low to verifying this, and calculate the position error in kms for each of the five days, comparing in each case with the track of the low on the forecast maps. This will involve five different calculations, each worth 4 points. To get 4 points, the position error must be less than 300 kms (quite generous). Each increment of 100 kms over that will reduce the score by 1. A position error of 700 kms will score zero. In terms of the low over central Sweden, that would place the verifying low somewhere like southern England or the Kara Sea or Minsk.

(4) Jet stream validation. We could all agree that looking at the maps presented, there would need to be a very strong jet stream over the UK and/or North Sea around the 3rd to 5th in particular. I could not imagine the winds at surface reaching even half the predicted values unless we have 100 kt winds at 500 mbs and 200 kt winds at 250 mbs. Thus, using charts for Feb 1 to 5, I will validate this component by giving up to 20 points if these criteria are met, and taking off one point for every 5 kts by which they fall short. This will be a snapshot of the map grid at any point in the period and should yield points even if the rest of the forecast falls flat. The max must occur somewhere in the zone north of 50 deg, east of 20W, south of 70N and west of 20E, some raging jet over Portugal does not count.

(5) Correlation study ... a further ten points will be added or subtracted from the total, based on the correlation coefficient times 10. A totally negative correlation on 3 Feb will assess a 10-pt penalty on total score, a random result will not change the accumulated score, a good correlation could add 10 points.

For comparative purposes, I will also score my own forecast for the same period based on these parameters:

(1) Mean flow in the period SE, S, WSW, WNW, NNW for the five days, at 500mbs for Shetland. Nothing like a raging northwest flow throughout.

(2) strongest wind gust 46 knots (same scoring technique).

(3) tracking low Feb 1 12z 62N 25W, 2nd 12z 57N 14W, 3rd 12z 55N 5W, 4th 12z 54N 5E, 5th 12z 53N 18E

(note, I also foresee a weak low somewhere near Sweden but the above is my version of the active weather event in this time frame ... I also foresee low pressure moving through Iberia into the Med).

(4) I foresee a weaker jet stream with peaks around 70 knots at 500 mbs and 140 knots at 250 mbs in the grid as defined above.

(5) I don't have a map to correlate but will correlate on these 12z Feb 3 pressures at 5 deg intervals between 70N and 45N and between 20W and 20E, the same grid that would be used to correlate the forecast map (the bold entry is close to the MB 945-mb Sweden low) :

70N .. 1025 . 1015 . 1008 . 1002 . 0995 . 0990 . 0992 . 0994 . 0998

65N .. 1022 . 1012 . 1005 . 1004 . 1006 . 1008 . 1012 . 1015 . 1014

60N .. 1020 . 1011 . 1015 . 1017 . 1020 . 1022 . 1024 . 1026 . 1028

55N .. 1012 . 1002 . 1014 . 1020 . 1027 . 1030 . 1034 . 1035 . 1036

50N .. 1015 . 1014 . 1014 . 1018 . 1022 . 1025 . 1027 . 1028 . 1029

45N .. 1014 . 1010 . 1008 . 1010 . 1016 . 1020 . 1021 . 1022 . 1022

A rather cold pattern, would be predicting sleet or snow in parts of Ireland from this 12z map, clear and cold in most of central Europe with an implied 1038 mb high over the southern Baltic, 20-40 knot westerly flow over northern Scandinavia. UK forecast for the 3rd, compare and contrast, would be increasing cloud, cold, E-SE winds 15-30 mph, possible sleet or snow by overnight into 4th. The implied central pressure of the low northwest of Ireland would be 1000 mbs at map time and I would predict a track across the north central UK with central pressures in the 995-1000 mb range on the 4th. This is a much weaker low than MB is forecasting, further southwest at comparable times, and likely to be more of a wintry mix than a windstorm. A stronger low might be expected near Seville Spain into the western Med at this time (990 mbs). That would be off the validation grid.

Okay then, would appreciate any feedback about this proposed validation scheme. I am really cutting MB a break here by not exposing his surface pressures to a direct comparison but instead correlation. That 945 mb low could bust by 100 mbs in a worst case scenario.

Edited January 12, 2011 by Roger J Smith

[lorenzo]

Excellent work, Roger looks like you've got it mapped. Am sure this verification will assist MB with future forecasts also.

Good to see you have put some predictions out there too - gives us something to look in to for the start of Feb and in extreme FI charts.

[Hammer]

Are MB's predicted wind speeds more likely kph rather than mph?

Just seem way to high if mph.

[Roger J Smith]

I don't think we want to have any sort of "hostile" validation, which is why I am proposing correlation and pattern matching as the basis. If we went on some of the numbers for pressure and wind speed, there could be a very negative result that might obscure any reasonable match between the maps and the outcome.

The maps presented, while quite extreme for lowest pressures, still don't seem to indicate winds as strong as 140 or 160 mph, but would certainly validate for those speeds in km/hr. I think we should ask MB to clarify whether he meant sustained winds or peak gusts, and if he has derived those forecast speeds from his maps or tried to draw maps to represent the speeds that he got directly from his research. Could tell you that the return period of 140 mph winds at Hamburg must be longer than the place has been there. Sustained winds over 70 mph are rather rare in all regions around the North Sea, gusts to 120 mph are about the highest I've ever seen or read about -- even the Daniel Defoe storm did not have gusts as strong as 140 mph. Gusts of 90 mph would do a lot of damage in places like Hamburg or Amsterdam.

Lows that reach 945 mbs do occur from time to time in the North Sea but rarely in Sweden or the Baltic where a pressure of 960 mbs would be considered exceptionally low. Another unusual feature of the forecast is the depicted gradient over the UK on 5 Feb, that looks very counter-intuitive to me (strong gradient on land, slack over the North Sea, in a northwest flow).

[MurcieBoy]

My word! I go away for a few days......

Many thanks Roger for the effort you have obviously put into this; I do welcome such an appraisal from a third party. As invited, I thought I would make just a few short feedback points:

The forecast is “out there†so folk can do as they wish in its review, but please remember this is only an “experimental forecast†that actually only took less than half an hour “thinking time†(ie: less time than to write some of these posts!).

I could have spent more time trying to factor in the effects of the DNA variances (and there are variances; and a material one in the case of one of the DNA constituents). I should say that one of the reasons I did not alter the Feb 2011 forecast (to accommodate these DNA variances) is that I am trying to let Nature tell me what the impact of the variances actually is. The same with the Xmas 2010 forecast, I did not adjust for the DNA variances at all (which were much more material and in more than one DNA constituent); but as you can see there were some significant successes in that forecast (see #2 appraisal video).

So folks, as I stated in #3 Storm video, (i) I do not expect the “actual†to be 100% the same as the “forecastâ€; and (ii) it would be the “degree of intensity†of the forecast that should be the subject of my follow up appraisal video; that is, the “degree of intensity†of the lows, the wind speeds and the sea surges. I do however expect the forecast to be close to the actual (...how close? We shall see!).

In #169, I stated: “It’s far too early at this stage to see the upside of the “methodâ€; as it is being “tested†and hopefully refined. You can however see the worst sides to it (at its current level of evolution).†The refinement that I am referring to here is in factoring DNA variances into the forecast.

If possible, the same validation procedure should be applied to the earliest GFS output – after all, the heading of this thread is “Better than the Models?â€

Degree of tolerance: as I have discussed already in #169, I would like the validation procedure (applied to all forecasts put forward) to be tightened. That is, the event has to happen on the date (and within an hour, say, of the given time) of the forecast.

Wind speed: as requested, I am able to clarify the wind speeds that I refer to in the forecast are in mph and refer to the Gradient Winds. The strongest gusts (not gradient winds) over the North Sea during the 1-5 Feb 2011 are forecasted to be around 110-140 mph. I hope this clears up any confusion that may have arisen, for which I apologise (I wanted to get the forecast out before the Bells!)

Just one final thought, hope this does not turn out to be a version of the "Spanish Inquisition!" Its only an "Experimental Forecast" based upon less than a year's weather research. Please reflect on AF's valid point in #183

Edited January 12, 2011 by MurcieBoy

[Buzzit]

Getting there

[Interitus]

Seems to me that there are two separate skills required by this forecast - the ability to produce the charts and then the ability to interpret them accurately. Certainly the charts don't look as extreme as the text forecast and I think this is something to bear in mind if the rating of an otherwise accurate set of charts was dragged down by poor interpretation.

Roger's comprehensive scoring system is commendable but I'm not sure it is fair to include values not included in the forecast and either inferred or implied, such as the jet stream.

Also how do you fairly determine the proportion of the 100 point total score to each aspect?

For the pressure charts alone it would be simpler to use a standard gridded error score (mean absolute, rmse etc) and a correlation versus climatology for the skill score - ie. the standard approaches for model verification.

Edited January 12, 2011 by Interitus

[Roger J Smith]

I'm just proprosing to do this validation, nobody has to pay any attention to it. There are no unfair inferences, anyone can see there's a prediction of a northwest flow and a strong jet stream, where else would you get 140 mph winds from in February?

[Coast]

I could have spent more time trying to factor in the effects of the DNA variances (and there are variances; and a material one in the case of one of the DNA constituents).

Hello MuruceBoy, can you just clear up the following post for me from UKWW that seems to contradict the statement above please? Cheers!

The trouble with looking at the "effect" (eg: the weather or stock market or lottery results), is that it has so many different patterns. It is impossible to work it out. Might as well ignore the possibility of working it out. If it could be solved by just looking at the effect, someone would have done it by now (given our super computers at our disposal)

The trick is to look at the factors that the determine or drive the weather (or other effects). These follow precise and easy to see mathematical patterns. As there are less than five variables, it is much easier to match "cause" to "effect".

http://www.ukweatherworld.co.uk/forum/forums/thread-view.asp?tid=39682&start=81

[MurcieBoy]

Hi Coast, have read both quotes quite a few times, but cannot see where there is a contradiction. Where is the contradiction in your eyes? If you can advise, I shall try and answer.

[k19]

Hello MuruceBoy, can you just clear up the following post for me from UKWW that seems to contradict the statement above please? Cheers!

http://www.ukweather...=39682&start=81

I think he is confusing the idea of DNA with the idea of effects?

[Interitus]

I'm just proprosing to do this validation, nobody has to pay any attention to it. There are no unfair inferences, anyone can see there's a prediction of a northwest flow and a strong jet stream, where else would you get 140 mph winds from in February?

Sorry, you said you would appreciate any feedback.