Using CET Statistics

[VillagePlank]

This weekend I was musing over the use (and abuse) of statistics, especially when talking of CET’s and records being broken when something occurred to me; one should think of me sitting enjoying a cup of Lady Grey and watching the clouds go by when a light-bulb appeared above my head . . .

We talk of CET averages in a pseudo-Bayesian fashion; especially when we try to forecast future CET values. It is common to look at the CET record and cross-correlate with existing (and known) data to provide a hypothesis of why the CET will be average, or above or below the mean (there is a case for median, but I’ll leave that to another day) We look at what’s happened last Winter, and we look at trends for highly differentiate periods such as seasons, and try to anticipate what the future holds for us.

I contend that this approach could be wrong, and that the CET, with today’s analysis skills, can only ever be considered as a historic record.

Consider one million words of real English text. We, perhaps, all know that the letter ‘e’ is the most common letter in English. Studies have shown that the frequency of the letter ‘e’ is about 25% of all letters used; sometimes this is more, sometimes this is less.

If we choose to play a game of hangman with the following (hidden word) _ _ _ _ _ _ _ _ our first guess would be the letter ‘e’ giving e _ e _ _ _ _ _;

This gives us, at the moment, a guess/success hit-rate of 100% We’ve identified the scope of the problem domain, analysed it, and currently are enjoying success.

The next most common letter is ‘t’: this gives us e _ e _ _ _ _ t. The more observant amongst us will probably be able to successfully guess from this what the word (the outcome) actually is. Not only that, considering the next most common letter is ‘a’ which gives us e _ e _ _ a _ t which only goes to confirm our theory; there are enough letters to successfully predict what the outcome is without all of the information. We have 100% success.

However if we were to consider an approach that is, dare I say it, more rigorous, and less reliant on our unique human pattern matching skills and hunches, the success rate goes significantly down. What happens if we start with the first letter and then iterate successively across the word?

In the above example we used statistics to have five attempts where, for the majority of people, the sixth attempt is the whole word ‘elephant’

The success rate for the first letter is, once again, 100%, but it, obviously, quickly goes downhill. The letter ‘l’ for instance is the 11th most used letter so out of twelve attempts we now only have 1 hit – and appalling success rate. Of course, it goes up to 2/13 with the next letter, but subsequent letters pull the success rate down and down to make the method useless.

So what’s the point of all this rambling? Well, it was analogous to using the CET and trends of CET in what can only be considered a logical statistical manner in order to predict future values of the same.

If you accept that this analogy is good, then you’ll agree that the method is a significant factor of success. In the hangman case it is so significant that it denotes success or failure. The data’s the same, the analysis of the domain is just the same and is excellent.

It’s how you use the data, and the conclusions of the analysis that count. It may well be, and I’m convinced that it is, that numerical computation and analogous forecasting of future trends is doomed to fail. Always.

There is something more special about us, as human beings. We are expert pattern matchers, we can compute vast amounts of information in sub-second precision; a human weather forecaster, given the data for the domain in question, is a far more effective tool, than anything technology can currently create.

And this probably makes no sense at all. Oxymoronic, huh?

Edited October 2, 2006 by Wilson

[Dawlish]

Hi Wilson,

Nah. The bergamot has gone to your head!

You are using the analogy of tossing a coin. Toss it 10 times, get one head, then 9 tails.....what's the chances of the next one being a head?

In this case, I'd look at it from the point of view of a Frequentist, the sample space is completely defined. I know what the probability is and I'd be prepared to bet if someone offered me 5/4 heads and continued to offer me those odds, for more tosses. I'd have an edge (see below). Casinos make their money playing odds like that all the time. In the end, at roulette, given a non-based table and ball, the casino will rake in approximately 1/37th of all the money staked - they will always win on the "0" ball. I say approximately, bacause they'll have undergone the meteorite impact before having spun the wheel an infinite number of times, for an infinite number of punters, but they'll be happy enough. They get enough punters to ensure success. 1/37th of £100,000,000 staked pays the gas bill. Same principle with slot machines, though the rake-off is much higher.

Same with bookies. Given equal amounts to be paid out (not stakes) on either side of a book, the bookie will win. Bookies fall down, in the short-term, to either high, winning, stakes that they can't lay off, or a small book (explains my losses in Sept ). Over time, even these extreme events will even out. The other way they can be beaten is to have an edge.

Using numerical prediction to predict future events really does pay off.

Lets turn to another game. Blackjack. The basic house advantage, at Blackjack, is 5.9%. Some sytems claim to be able to reduce that, by obeying certain rules, but the house will still win, in the long-term. So what is the house afraid of? What causes them to lose, blacklist the people concerned, hound them by video surveillance at all major casinos and pass detailed information about these people between themselves? Card counters. Card counters have an edge. They can calculate the probability of a particular card coming up and bet accordingly. Professional card counters have won millions and are not allowed into casinos - hence the security. Long-term winners, at Blackjack, may employ a marketable system (Don't go there anyone, you'll lose, honest - same with Technical Analysis of shares (TA), or horse tipsters. If their system was so good, do you think they'd be sharing it with you??) but they also count cards. It is the only way to win, long-term, when the odds are against you.

Horse racing and especially greyhound racing have their own special species of people with an edge. It involves cheating and I'm not going into all the possible ways it can happen. There isn't space! look up "Flockton Grey" for a really good one!

I've never been to a casino and when I make a visit to a racecourse, it is an unmitigated disaster (Newton Abbot, this summer. Me, Missus, Eldest went. Great entertainment, but: 6 races, 18 bets, all on different horses, but how many winners..........Yup; 0 Gawd, you'd have expected one of us to have a payout).

Betting on the weather.

Over the course of the previous 2 years, I placed 2 bets. Both on the lack of Cristmas Day snow, in a particular area. Last year, I won £50 at 1/4 on no snow in Birmingham and the year before I won the same amount, betting 1/5 against no snow in London. This summer, I discovered that betfair had a book for people to bet on monthly summer temps - not, sadly for Autumn months. Over the course of the summer, I won every bet I placed, except for 100F being achieved (I did get 19/1 and saw the odds fall to 6/1 at one stage, but I just missed out). I did pick up >97F, at 12/1, though on a small covering bet. I thought one, or the other would be achieved.

Why did I achieve that level of success, when I achieved no success whatsoever at backing horses? Because I had an edge. I backed myself to understand the possibilities of the gfs charts at T+240+ better than the people laying the odds...and I did. I confidently expected to pick up a result from every one of those bets. Many were at very short odds, as short as 1/12, but you pay no tax online and as I limited my initial stake to £200, I didn't make a fortune, but the results are in my signature, below. I claim the 100% as I regard the 100F/97F as simply a split stake, on the same bet.

What has this got to do with CET?? Well, if you come back to your hangman example, my monthly CET prediction, against a long-term average, is like being able to guess an "e" every time, only better. I suppose I'm a Bayesian on this. I've assigned probabilities (mainly in my head, you are correct about us being amazing supercomputers) to the uncertain factors and as a result, I've formulated a degree of belief in the outcome - namely 70:10:20 warmer:average:colder and I forecast warm for every future month. It has achieved >70% success rate over a period of, aproaching, 200 months since 1990. In terms of years, I'm well above that 70% figure.

I'm not a mathematician, but I'm very willing and able to learn. From your interesting and really enjoyable post, this short paragraph, below, stirred me to return to probability theory, via the web (well, it's a wet morning and I'm still recuperating; there's not much else to do!!) and I'm always grateful when someone makes me learn. There is no way I'm getting into the formulae, unless I'm really, really, forced(!), but, on the basis of the evidence I've offered, I don't think this, overly certain, statement can be true:

"It’s how you use the data, and the conclusions of the analysis that count. It may well be, and I’m convinced that it is, that numerical computation and analogous forecasting of future trends is doomed to fail. Always.".....

........you just need an edge.

Paul

Edited October 2, 2006 by Dawlish

[Snowly does it]

Hmmmm, thats fine and you can sing while you're winning, but eventually 70:20:10 will let you down, whether it be by reversal of trend, acceleration of trend or simple statistical quirk.

Statistical analysis of weather may give you some success, but forecasting is about more than probability, it is taking the current, accepting the historic and prognosing (a word?) the future based on the most likely evolution.

An imperfect science at the moment, but LRF do not have the luxury of giving 'probably warmer because its been getting warmer' as a forecast, they have to justify it in the particular, not the abstract.

Ultimately LRF will (one hopes) exceed statistical analysis for reliability and will at very least not fall down a well of oblivion when things (eventually) change.

The warming trend noted and frequency of warmer months is clearly a part of the forecasting process, but it is not enough on its own.

Edit - for example, you said in another thread that 10-20 years is needed to confirm a change. Not good enough in the 10-20 years the change is occuring will probability be (Yoda lessons)

Edited October 2, 2006 by snowmaiden

[Dawlish]

Hmmmm, thats fine and you can sing while you're winning, but eventually 70:20:10 will let you down, whether it be by reversal of trend, acceleration of trend or simple statistical quirk.

Statistical analysis of weather may give you some success, but forecasting is about more than probability, it is taking the current, accepting the historic and prognosing (a word?) the future based on the most likely evolution.

An imperfect science at the moment, but LRF do not have the luxury of giving 'probably warmer because its been getting warmer' as a forecast, they have to justify it in the particular, not the abstract.

Ultimately LRF will (one hopes) exceed statistical analysis for reliability and will at very least not fall down a well of oblivion when things (eventually) change.

The warming trend noted and frequency of warmer months is clearly a part of the forecasting process, but it is not enough on its own.

Edit - for example, you said in another thread that 10-20 years is needed to confirm a change. Not good enough in the 10-20 years the change is occuring will probability be (Yoda lessons)

True, the trend will eventually reverse, but the odds are in favour (rather unfortunately) of me not being here to see it when it does - or anyone else reading this - though I expect 70:10:20 to have changed to at least 80:5:15, maybe higher, in my lifetime, when compared to the long term mean.

LRF is much more than using probability, you are right and the Met Office does not have the luxury of saying; "well, I'd give you 7/2 against a colder winter", each year that passes, but one point I've made, often, is that LRF is basically not accurate enough at present. I hope it will exceed astatistical analysis, sooner, rather than later, but at present, over 70% accuracy is a fair bit higher than the Met Office, or anyone else, can claim for the next season. You are right about the 10-20 years bit being needed to confirm a change; but the last 10-20 years, in them, probabilty good has been! No reason, I see for future change to be. [tilts ears, knowingly]

Paul

[Snowly does it]

True, the trend will eventually reverse, but the odds are in favour (rather unfortunately) of me not being here to see it when it does - or anyone else reading this - though I expect 70:10:20 to have changed to at least 80:5:15, maybe higher, in my lifetime, when compared to the long term mean.

LRF is much more than using probability, you are right and the Met Office does not have the luxury of saying; "well, I'd give you 7/2 against a colder winter", each year that passes, but one point I've made, often, is that LRF is basically not accurate enough at present. I hope it will exceed astatistical analysis, sooner, rather than later, but at present, over 70% accuracy is a fair bit higher than the Met Office, or anyone else, can claim for the next season. You are right about the 10-20 years bit being needed to confirm a change; but the last 10-20 years, in them, probabilty good has been! No reason, I see for future change to be. [tilts ears, knowingly]

Paul

No reason for future change would you see until in the red you be.

[VillagePlank]

You are using the analogy of tossing a coin. Toss it 10 times, get one head, then 9 tails.....what's the chances of the next one being a head?

Evens. Unless you have an awful lot of time in order to approach an infinite series, or you get lucky, then you have just as much chance of winning as you have as losing based in historical data

In this case, I'd look at it from the point of view of a Frequentist, the sample space is completely defined. I know what the probability is and I'd be prepared to bet if someone offered me 5/4 heads and continued to offer me those odds, for more tosses. I'd have an edge (see below). Casinos make their money playing odds like that all the time. In the end, at roulette, given a non-based table and ball, the casino will rake in approximately 1/37th of all the money staked - they will always win on the "0" ball. I say approximately, bacause they'll have undergone the meteorite impact before having spun the wheel an infinite number of times, for an infinite number of punters, but they'll be happy enough. They get enough punters to ensure success. 1/37th of £100,000,000 staked pays the gas bill. Same principle with slot machines, though the rake-off is much higher.

Yup

Using numerical prediction to predict future events really does pay off.

Urmmm . . . .

Lets turn to another game. Blackjack. The basic house advantage, at Blackjack, is 5.9%. Some sytems claim to be able to reduce that, by obeying certain rules, but the house will still win, in the long-term. So what is the house afraid of? What causes them to lose, blacklist the people concerned, hound them by video surveillance at all major casinos and pass detailed information about these people between themselves? Card counters. Card counters have an edge. They can calculate the probability of a particular card coming up and bet accordingly. Professional card counters have won millions and are not allowed into casinos - hence the security. Long-term winners, at Blackjack, may employ a marketable system (Don't go there anyone, you'll lose, honest - same with Technical Analysis of shares (TA), or horse tipsters. If their system was so good, do you think they'd be sharing it with you??) but they also count cards. It is the only way to win, long-term, when the odds are against you.

Yup

Horse racing and especially greyhound racing have their own special species of people with an edge. It involves cheating and I'm not going into all the possible ways it can happen. There isn't space! look up "Flockton Grey" for a really good one!

Yup, but 'cheating' cannot be considered, in mathematical terms, as part of the problem domain.

I've never been to a casino and when I make a visit to a racecourse, it is an unmitigated disaster (Newton Abbot, this summer. Me, Missus, Eldest went. Great entertainment, but: 6 races, 18 bets, all on different horses, but how many winners..........Yup; 0 Gawd, you'd have expected one of us to have a payout).

Oh dear . . .never bet on horses so I can't say I've shared the experience

Betting on the weather.

Over the course of the previous 2 years, I placed 2 bets. Both on the lack of Cristmas Day snow, in a particular area. Last year, I won £50 at 1/4 on no snow in Birmingham and the year before I won the same amount, betting 1/5 against no snow in London. This summer, I discovered that betfair had a book for people to bet on monthly summer temps - not, sadly for Autumn months. Over the course of the summer, I won every bet I placed, except for 100F being achieved (I did get 19/1 and saw the odds fall to 6/1 at one stage, but I just missed out). I did pick up >97F, at 12/1, though on a small covering bet. I thought one, or the other would be achieved.

Why did I achieve that level of success, when I achieved no success whatsoever at backing horses? Because I had an edge. I backed myself to understand the possibilities of the gfs charts at T+240+ better than the people laying the odds...and I did. I confidently expected to pick up a result from every one of those bets. Many were at very short odds, as short as 1/12, but you pay no tax online and as I limited my initial stake to £200, I didn't make a fortune, but the results are in my signature, below. I claim the 100% as I regard the 100F/97F as simply a split stake, on the same bet.

Regardless of the probabilities, by using your understanding of the problem domain, you have effectively cheated. I use the word 'cheated' because I can't think of another less acute word. Should you have solely used the historical record then your chances would have been, at best, evens. This is exactly what I'm saying above: success comes from stepping outside the problem domain. In the hangman example, the first bit relies almost entirely on the readers ability to pattern match the word 'elephant' and has, in essence nothing to do with rigorous statistics - I just dressed it up that way.

What has this got to do with CET?? Well, if you come back to your hangman example, my monthly CET prediction, against a long-term average, is like being able to guess an "e" every time, only better. I suppose I'm a Bayesian on this.

Indeed.

I've assigned probabilities (mainly in my head, you are correct about us being amazing supercomputers) to the uncertain factors and as a result, I've formulated a degree of belief in the outcome - namely 70:10:20 warmer:average:colder and I forecast warm for every future month. It has achieved >70% success rate over a period of, aproaching, 200 months since 1990. In terms of years, I'm well above that 70% figure.

You've used your understanding of current climatic conditions to skew the bets. I agree with your spread, here, but I'd go even less for average and up the cold one a bit. Even so, the basic underlying philosophy is that you expect the weather to change, and you expect it to change to warmer.

I'm not a mathematician, but I'm very willing and able to learn. From your interesting and really enjoyable post, this short paragraph, below, stirred me to return to probability theory, via the web (well, it's a wet morning and I'm still recuperating; there's not much else to do!!) and I'm always grateful when someone makes me learn. There is no way I'm getting into the formulae, unless I'm really, really, forced(!), but, on the basis of the evidence I've offered, I don't think this, overly certain, statement can be true:

"It’s how you use the data, and the conclusions of the analysis that count. It may well be, and I’m convinced that it is, that numerical computation and analogous forecasting of future trends is doomed to fail. Always.".....

........you just need an edge.

I am overly certain, and in that respect, I've cheated. The entire possibilities of weather combinations is so vast that it would take all of the computing power available now, and expected in the next millenia to search the vast pattern space we call climate and even then you'd need about a million years to wait for a result. (Using the top ten super-computers in the world, today, it would longer than the expected lifetime of the universe)

You need to, as you have, step outside the problem domain and find shortcuts. There are very effective methods of reducing the pattern space such as neural networks, genetic algorithms, simulated annealing etc etc; but the problem of weather, in my opinion, cannot simply be deterministically reduced until the underlying 'truth' is revealed, and then rebuilt into a model.

The curve of current model building is coming to the point of diminishing returns (in terms of pure numerical forecasting) and they are, now, implementing more subjective measures such as probability of ENSO occurences, volcanic eruptions (I say subjective because the probability of these events occuring is so poorly understand, and that it's a matter of forming an opinion of which study is right and which is wrong)

If you need an example telephone the MetO and ask them for 'near certain' predictions of where convective rainfall will occur

More examples of my irrational ranting can be found:

http://www.netweather.tv/forum/index.php?s...mp;#entry766861

http://www.netweather.tv/forum/index.php?s...mp;#entry753905

Ultimately LRF will (one hopes) exceed statistical analysis for reliability and will at very least not fall down a well of oblivion when things (eventually) change.

If we continue to use reductionist mathematics (breaking the problem domain down into smaller and smaller chunks until they're easy to solve) we'll end up with quantum mechanics which is, I'm afraid, all about statistics.

I hope it will exceed astatistical analysis, sooner, rather than later, but at present, over 70% accuracy is a fair bit higher than the Met Office, or anyone else, can claim for the next season. You are right about the 10-20 years bit being needed to confirm a change; but the last 10-20 years, in them, probabilty good has been! No reason, I see for future change to be. [tilts ears, knowingly]

I wonder what the results would be if we analysed a few weather parameters say Wp={Precipitation,Sun,MaxT,MinT,Wind} and predicted that each subsequent day will be with a 5% error of the previous day? I've done something similar to this and my results were >80% - I'll dig out my work and post it. Does this mean I understand the problem domain better than the MetO - no! Does it mean I've discovered some hitherto method of weather forecasting - no! It means I understand that weather systems normally take more than 24 hours to clear the UK.

In the same manner, although I accept that your spread is fairly consistent with the short-term outcomes - and I expect you to continue to win at least for the next 2/3 years (win = you're right more times than others, not that the frequency matches your spread) it cannot, by definition, hold for the future of our climate.

Edited October 2, 2006 by Wilson

[Dawlish]

Hi Wilson,

Yes; I'd happily concede that purely numerical forecasting is doomed to faliure. I don't think you cheated there, I just think you are right. The advances in accuracy in numerical forecasting have been painfully small over the last 10-20 years. I would have expected it to be greater, but it isn't. Despite hugely improved conmputing power, it seems to have hit tha law of diminishing returns. Something else is needed.

However, to say that analagous forecasting, is doomed to fail, in the way that I'm doing it - that's really the "nah" from my last post and why I said that I feel the whole of the italicised quote is wrong (if a part is wrong, the whole is wrong - pedantic, I know and it doesn't add to my point!). Analagous forecasting of a dicrete event (say a month, or a season, is not accurate, but analagous forecasting in terms of probability is.

Once you start to try to forecast actual weather events like convectional rainfall, with actual values, or in specific areas, I wholeheartedly agree, you're really in trouble! Still people want to know...."will it snow in_____ at such and such a time. For that one; see below!

Paul

PS. You are right, that I "cheat", but it is only cheating because I'm betting against people whose knowledge and skills, in this field, I determine, are less than my own. Exactly the same belief that allows people to have the confidence to win $1,000,000 in poker tournaments. It's taken me about 18 months to develop my technique in LRF in gfs to cut the corners (right again), I just wonder if I could do the same, in 18 months at poker. It is, honestly, tempting and I've never played before!

Anyone fancy giving me about £100k as an initial stake, with which I could learn the game and finance the mortgage and the family for a year? If you then give me another £100k, I'll give you 25% of all my future winnings ad infinitum. The stake in me would be non-refundable, but the entertainment would be in watching my progress ........if you'd backed the right person! (there's no avatar for tongue in cheek!)

No reason for future change would you see until in the red you be.

I'm in the red now, but I'll take the long-term view!

Edited October 2, 2006 by Dawlish

[VillagePlank]

However, to say that analagous forecasting, is doomed to fail, in the way that I'm doing it - that's really the "nah" from my last post and why I said that I feel the whole of the italicised quote is wrong (if a part is wrong, the whole is wrong - pedantic, I know and it doesn't add to my point!). Analagous forecasting of a dicrete event (say a month, or a season, is not accurate, but analagous forecasting in terms of probability is

Analagous forecasting, in my opinion, will always have a significant error rate in it.

To perform an analogy, in my opinion, you need to have the complete picture of what it is you are comparing. In order to express the idea that a comparison has merit you must be sure that you have all of the significant information available to your fingertips.

I haven't seen any charts of butterfly's turbulence

[edit] Of course you can use the same ideas to claim that weather behaves like a strange attractor and that given certain initial states the weather has to decay into said analagous patterns, and like Henon's strange attractor the weather is self-similar at all scales (fractal) so knowing temporal initial states is actually irrelevant because you already have the information on the outcomes - but that's an entirely new (and different) ball game [/edit]

Edited October 2, 2006 by Wilson

[Dawlish]

Analagous forecasting, in my opinion, will always have a significant error rate in it. My (very) basic reasoning is simply that the months from year to year are not the same - we correct this by using leap-years, and leap-seconds. If we can have some measure against Julian time, then that would be a different kettle of fish.

I understand that most (if not all) people will certainly say that the difference between civil and sidereal are so small that it's negligable, but my own experiments show that day-length, longtitude, latitude (ie where you are on the planet, and where you are in orbit (affects insolation including the angle of attack of the sun)) show some significance. When I'm done I'll post results. If I never post results you'll know, with absolute certainty, that I was absolutely wrong

Your last sentence made me smile! I'll look forward to the results.

I'm sorry, I've misread your doubts about forecasting using the past, I think. I don't mean analagous forecasting in the sense that you can figure future months from a past month, or months. That is riddled with error and I don't know of any studies that have shown, to a reasonable confidence level, that if we have a cold----(eg summer), we'll have a warm-----(eg winter). I mean forecasting from trends. The analogue here, being that, given the same perameters (the world continues to warm), the same percentage (70:10:20) is likely to continue. With warming, I expect the 70 to increase and the 20 to decrease. The 10 is just totally iffy!

Paul

Edited October 2, 2006 by Dawlish

[VillagePlank]

Your last sentence made me smile! I'll look forward to the results.

I'm sorry, I've misread your doubts about forecasting using the past, I think. I don't mean analagous forecasting in the sense that you can figure future months from a past month, or months. That is fraught with error and I don't know of any studies that have shown, to a reasonable confidence level, that if we have a cold----(eg summer), we'll have a warm-----(eg winter). I mean forecasting from trends. The analogue here, being that, given the same perameters (the world continues to warm), the same percentage (70:10:20) is likely to continue. With warming, I expect the 70 to increase and the 20 to decrease. The 10 is just totally iffy!

Paul

I also completely edited my above post to avoid the obvious clash with my incomplete knowledge and research that would ultmately arise!!

[VillagePlank]

YOu could always make real money on betting on your spread here Bet on the weather futures market!!