Climatology

[VillagePlank]

I have a significant, but passing interest, in meteorology. I have been reading this forum for well over a year, and have contributed (albeit under a different account for most of the time) where I thought that my thoughts perhaps were merited or might indeed of been some interest to others, who like me, peruse this forum regularly.

There is, however, a concurrent theme raging through this forum, and others very similar. That theme is determinism.

Now, the vast majority of serious contenders would never disagree that the climate is chaotic; that, in essence, we cannot solve the difficulties of weather prediction by mathematics and computing power.

The problem is, of course, that everyone says it; but no-one believes it. Chaotic, or to avoid ambiguous colloquialisms, non-linear, mathematics is defined, at it’s very worse, as an outcome being highly sensitive to initial conditions. The so called ‘Butterfly Effect’ The essence (and I’m not trying to ensure the reader sucks eggs, here) is that a butterfly in some far off continent flaps its wings and causes thunderstorms in London by virtue of a complex series of interactions throughout the atmosphere.

This, effectively, is not only a terrible cliché, but it provides some comfort to those who wish to cling to vestiges of determinism; those who wish to carry on believing that the natural universe can be broken down in some laminate floorised fashion of top-down decomposition, until such deterministic laws are understood, measured, and can be further included in our understanding; a step closer to the truth.

Basic combinatorial mathematics shows us pertinently that we will never ever be able to compute with any notional degree of certainty what the weather will do at the very next instant, let alone next week, next month, next season, or next year. Of course, to even contemplate such a terrible monstrosity of computing arrogance relies in totality upon understanding the coupling of each and every particle in the system.

This understanding, this breaking down the system into smaller and smaller fragments, this decomposition, is what the meteorological fraternity is currently striving for. There is belief that if we can find the fundamental deterministic laws that govern particle coupling between different fluids (with their different dynamics) we can then put the system back together as a model inside a computer and provide a deterministic forecast model that is never wrong. This belief is further fed by promises of more computing power, better programmers, and better models.

Any meteorologist with an ounce of sense can immediately sense that this is patently wrong. That to achieve this is not only highly improbably, but it is, in fact, a fool’s errand to even try.

In highly controlled circumstances, such as a pendulum swinging, we can produce non-deterministic results. These results are reproducible every time – that is, that we cannot predict the motion of the pendulum to any exact degree. Modern physics has had a great understanding of many things – we sent man to the moon, but we still do not understand the motion and interaction if a single particle hits two other particles at exactly the same time. What does happen to the target two particles?

Even though we can construct experiments of extremely simple systems (such as pendulums) that produce non-deterministic results and are immune to the scientific dogma of decomposition, the belief that we can eventually ‘discover’ the determinism that underpins all natural systems lives on.

Of all the trillions upon trillions of particle interactions in the atmosphere, and ignoring their thermodynamic nature, we have no method, and no mathematical way of describing the interactions of three particle problem.

This, of course, flies in the face of known and verifiable forecasting. We can tell to some degree what the weather is going to do tomorrow with some small essence of certainty. We can assign probability of risk to thunderstorm activity, and to impact of snow cover for a given region for a given time.

The two viewpoints, it seems, do not match up.

I propose that it is the meteorologist’s methodology that limits his ability to garner continuing success. The ultimately flawed belief in cause and effect, and determinism, only leads to undertaking work that, time and time again, leads to diminishing returns.

We can say accurate things about the pendulum model, for instance. It helped the human race tell the time for many years for a start. In many cases a fluid does not need to broken into its constituent particles to describe its behaviour; you can predict which way a fluid will move under the influence of gravity, for instance.

We can say many things about the weather, and we can conglomerate such things into a forecast. It is known that such forecasts accuracy reduces as the forecast heads out into time.

The reason given, and accepted, as to why forecasts diminish with time, of course, is that the atmosphere is chaotic, and unpredictable.

If we understand that then why do we still try to decompose the system into deterministic parts when we already know, in advance, that it’s a fruitless exercise?

It is clear from anecdotal evidence, both experimentally, and historically, that the best efforts are those who are looking for patterns in long term and short term behaviour.

That science, of course, is the child science of climatology

[John Cox]

Hi Wilson,

Firstly, welcome to NW.

What a wonderfull post.

In a nutshell it you seem to be saying that we are indeed on a fools errand trying to predict what the weather will do at any given time due to the butterly effect.

Am I right in assuming that you think that men / women and their computers are wasting their time or that they are looking in the wrong direction in their programming or even a bit foolish trying to compute advanced weather predictions because it cannot and will never be able to be done with any degree of certainty?

[VillagePlank]

Am I right in assuming that you think that men / women and their computers are wasting their time or that they are looking in the wrong direction in their programming or even a bit foolish trying to compute advanced weather predictions because it cannot and will never be able to be done with any degree of certainty?

Not quite. The non-linear nature of weather has already predefined that we can never produce good weather forecasts using deterministic modelling. This is not just the mammoth property of the task at hand. Even if we could compute each and every variable required, there is the question as to whether (i) we know all the variables, (ii) all the variables at some arbitrary low abstraction are deterministic in nature.

There is a study, the study of emergence, that provides mathematical tools that help us to look at and understand patterns without understanding the details. The question, I suppose, is do we really need to know about particle interaction within, for instance, ocean/atmopshere coupling, when we can look at what emerges from a given set of cirumstances and form forecasts based on that.

I feel that I've been a little hard and the community at large; non-deterministic forecasts are now being published. And these are, quite rightly, about probability.

A good example is look at the vast expertise of meteorlogists (amateur and otherwise) on this forum. Together as a group there is still no conviction, no elaboration of verification, on any convection forecast. It's true that given events give life to given environment causes, which, continue to feed the illusion of a deteministic nature of weather. In reality the closest to a forecast one can give is that, for instance, convective rainfall will occur with n probabality, in Essex.

If one considered a land based grid (transverse mercator?) and looked at trends for each point of the grid as the sum of all the parts (whether known or unknown) and computed trends for each point and compared each point to compute change then, not only would you be able to compute this on a standard PC (for somewhere the size of the UK) you'd be able to assign probability of forecast correctness based on, perhaps, Bayesian inferencing techniques validated statistically (chi-squared analysis?)

This, today, is sadly lacking. The only areas forging forward are those who manage climatological models; but they're still stuck in the trap of determinism - "if only we knew more, had more computing power, understood this and that deterministically, then we could make a good long-term forecast."

It stands today, that we can only predict rain or shine, wind or quiet on a very limited scale for a very limited time span into the future.

Unless there is a sea-change paradigm shift in the nature of meteorological analysis, then I suspect the curse of determinism to be a self-fulfilling prophecy.

A prediction that no prediction can ever be made will, unfortunately, become true.

Edited July 13, 2006 by Wilson

[John Cox]

My colleague at work asked me last Friday what the weather would be like in Wexford, in Ireland this week.

I gave her my detailed weather forecast.

"The second half of the week will be dryer and warmer than the first half of the week."

I think I have a grasp of the situation at hand.

Incidentally, my forecast was correct.

[VillagePlank]

My colleague at work asked me last Friday what the weather would be like in Wexford, in Ireland this week.

I gave her my detailed weather forecast.

"The second half of the week will be dryer and warmer than the first half of the week."

I think I have a grasp of the situation at hand.

Incidentally, my forecast was correct.

I predict the weather tomorrow will be the same as it is today within a given tolerance.

A backward analysis of temperature, pressure, wind direction, and precipitation would, I am sure, show me to be right more than I am wrong.

[Dawlish]

Not quite. The non-linear nature of weather has already predefined that we can never produce good weather forecasts using deterministic modelling. This is not just the mammoth property of the task at hand. Even if we could compute each and every variable required, there is the question as to whether (i) we know all the variables, (ii) all the variables at some arbitrary low abstraction are deterministic in nature.

There is a study, the study of emergence, that provides mathematical tools that help us to look at and understand patterns without understanding the details. The question, I suppose, is do we really need to know about particle interaction within, for instance, ocean/atmopshere coupling, when we can look at what emerges from a given set of cirumstances and form forecasts based on that.

I feel that I've been a little hard and the community at large; non-deterministic forecasts are now being published. And these are, quite rightly, about probability.

A good example is look at the vast expertise of meteorlogists (amateur and otherwise) on this forum. Together as a group there is still no conviction, no elaboration of verification, on any convection forecast. It's true that given events give life to given environment causes, which, continue to feed the illusion of a deteministic nature of weather. In reality the closest to a forecast one can give is that, for instance, convective rainfall will occur with n probabality, in Essex.

If one considered a land based grid (transverse mercator?) and looked at trends for each point of the grid as the sum of all the parts (whether known or unknown) and computed trends for each point and compared each point to compute change then, not only would you be able to compute this on a standard PC (for somewhere the size of the UK) you'd be able to assign probability of forecast correctness based on, perhaps, Bayesian inferencing techniques validated statistically (chi-squared analysis?)

This, today, is sadly lacking. The only areas forging forward are those who manage climatological models; but they're still stuck in the trap of determinism - "if only we knew more, had more computing power, understood this and that deterministically, then we could make a good long-term forecast."

It stands today, that we can only predict rain or shine, wind or quiet on a very limited scale for a very limited time span into the future.

Unless there is a sea-change paradigm shift in the nature of meteorological analysis, then I suspect the curse of determinism to be a self-fulfilling prophecy.

A prediction that no prediction can ever be made will, unfortunately, become true.

Marvellous Wilson. I have enjoyed reading these two posts - and not just because I agree with the vast majority of the content; they were beautifully written.

Great strides were made in the 1960s, 70s and even 80's with forecasting accuracy, but the pace of improvement has slowed to a crawl in the last 5 years. I listened with great interest to the Chief Education Officer for the Met Office in Exeter, give a lecture on forecasting accuracy lasy year. In fact I introduced him and thanked him on behalf of the South Devon Geographical Association.

Tha Met Office have just brought a new computer on line (last year) which has increased their ability to crunch numbers by "x" times, but they do not expect 3 day forecasting accuracy to improve more than 1%, or maybe 2% in the next 10, similar to the improvement that has happened in the last 5 (the increases in computing power, compared to the increases in forecast accuracy is subject to the law of diminishing returns. Why? Because, if I may paraphrase what Wilson rightly says, it is just too bloody difficult!

The atmosphere, in exhibiting the characteristics of a chaotic fluid, requires us to have an almost infinite number of data sampling points to be able to predict its behaviour correctly. Nanotechnology and other unforseen advances in remote monitoring (satellites again) may help in the near future, but the vast volume of mixed gases that we call the troposphere will, hopefully, never be subject to perfect analysis in my lifetime, or, as I've said previously, I'd be better off taking up ballroom dancing (pet hate, a bit like cats) than offering odds on temperatures.

To look at probabilities is the way forward. Forecasts always have an element of mistake in them. It is inbuilt with the complexity of the system. At present, every forecast that you see and hear, as Wilson says, will be wrong. However, for me, a 75% probablity makes much more sense than saying; "there is a good chance of rain". As I like to work in odds, 1/4 on that it will rain makes even more sense, but each to their own here. To use this, implicitly accepts the error which every forecast is subject to.

I accept what you say about the possible future of meteorology being, in effect, climatology and that forecasting could be approached by outcome instead of inputs, however, people are always going to need the short-term forecasts. For the Met Office to express them as probablities would make much more sense.

I believe strongly that the world will continue to warm, but I'd never say it was inevitable. For someone to say that they feel it is 10/1 on (1/10) that the warming trend will continue for the rest of my lifetime says much more about their views and their mindset than someone telling me that they believe in GW. 1/10 means that they are prepared to accept that it may not continue and that their present view may be wrong!

Paul

Edited July 13, 2006 by Dawlish

[johnholmes]

nothing to do with the topic, just my standard request to all new members, after bidding them a big welcome to Net Wx, either in this life or your former one Wilson. May we please be given the benefit of where you are issuing such erudite comments from old chap, ie your town or area in your avatar, or your signature.

Thanks very much

John

[Scribbler]

Hi Wilson and others

I’m a layman, so please excuse my lack of technical terminology.

I don’t believe that we need to know all the ins and outs of quantum mechanics (or other esoteric sciences) to be able to produce a reasonably accurate forecast.

Two atoms colliding may interact in a random way that upsets a nuclear physicist but two molecules of water vapour (for example) will PROBABLY interact in much the same way every time – given normal atmospheric surroundings.

Thus a suitable eddy in the right area of the atmosphere in the tropics will PROBABLY develop into a hurricane – given the normal, average, typical surrounding conditions.

By the use of TYPICAL reactions and interactions a good forecaster or computer can produce a pretty good forecast (for pretty well anything) for some time in advance.

However, quite a few interactions (especially within an atmosphere) will likely become subject to occasional aberrations or abnormalities which will throw any forecaster’s prophecy all to hell. And a blip today becomes a major error in a future scenario. So, much as we’d all like to have perfect forecasting it does seem that, however powerful and intelligent our computers become, local exceptions will always be there to foul things up from time to time.

But even so, given a computer model that recreates much of the earth’s geophysical features (down to a reasonably small scale), there must be a statistically good chance of being reasonably correct – at least quite a lot of the time.

A pretty good forecast, yes – a perfect forecast, no chance!

[Dawlish]

Hi Wilson and others

Two atoms colliding may interact in a random way that upsets a nuclear physicist but two molecules of water vapour (for example) will PROBABLY interact in much the same way every time – given normal atmospheric surroundings.

You are joking Scribbler! Absolutely no chance! (good post though!)

Paul

[Scribbler]

You are joking Scribbler! Absolutely no chance! (good post though!)

Paul

Haha! .......but thanks anyway!

Well, what I mean is something like this.

Atoms collide and the outcome of most atomic interactions has been well dissected and documented. However, much of the outcome is still as good as random. I think???!!!

When two elements of weather meet or interact, the outcome (within broad guidelines) is reasonably foregone. When a warm front meets cold air the warm air rises and it PROBABLY rains/snows, etc. I think (positive!)???!!!

Ergo one can think along broadly accepted terms or probabilities to create a half-way decent forecast.

[Supercell Hunters]

Hello people

I looking forward to my future in meteorologist and climatologist career. I will go to Uni at Lismore in 2 years time to make Supercell Hunters active and we will be storm chasing and forecastings.

[VillagePlank]

I specifically avoided the use of atoms as I wanted, perhaps erroneously, to avoid the detail required to understand even the simplest of atom systems; in particularly I avoided the water molecule which has many strange properties that are not apparent elsewhere in nature; you only need to consider the myocin ratchet reaction that occurs inside all of us . . .

You are indeed correct to presume that the collisions of two atoms are thoroughly studied and documented; it is the interaction of three (or more) atoms that is not at all well understood; in this I mean that there is no known analytic solution using the quantum mechanical motion equation, Schrodinger’s equation. The same, incidentally, is true if you consider three billiard balls interacting at the same time (one ball hits the other two and the impact between the three occurs as precisely the same time)

Given that the atmosphere has trillions upon trillions of atoms within it, I have assumed that the probability of a three (or more) particle system occurring to be so high that insofar as we can measure these things, it is certain. I do not normally subscribe to certainty, or its derivative cousin, determinism, but in the interests of avoiding terseness I shall, for once, concede this point. Indeed, some readers may choose to invoke Heisenberg’s Uncertainty Principle here, but I shall, in this case, conveniently ignore it for brevity.

Although I would argue the reasoning that there is a linear path between understanding the interaction of two atoms to the forming of a visible eddy, I do think that the eddy is a good example of what it is I am trying to say.

The eddy you spoke of is looking at the results of complex, ill-understood, interactions within the atmosphere. My argument is that whilst meteorological science is currently investigating why the eddy has formed, we should, I feel, be asking what the consequences of this eddy are in the system. Although the difference between these two is apparently subtle, and slight, I nevertheless believe that the difference is significant.

I think, actually, that some people are considering the what's over the why's. Advanced predictions of hurricanes and other such systems rely on analysis of current macro atmospheric states such as a minor eddy.

It is the modelling and forecasting of future states, primarily computed by super-computing processes such as GFS, in order to form a forecast, that is so prone to error that most users on this forum find out only to their dismay. How many times has snow been forecast confidently (by one of the most advanced atmospheric coupling systems in the world) only to find out six hours before that it was all, in fact, an error in judgement. I should not personify computing technology, but the word ‘judgement’ I feel is good enough to reflect the opinion of those who watch these models.

In fact it is the lack of understanding of the coupling of micro atmospheric states which has led to the sub-science of tele-connections which attempts to look at patterns on a continental(ish) size basis. Numerical coupling, has never performed to the ability of its promise, and is, as has already been outlined, now subject to the laws of diminishing returns.

Known mathematics already tells us that we cannot understand the physical interactions of the atmosphere in its entirety; this is something, I hope, that we can all agree on.

The question, I suspect, remains at which level of abstraction we should be looking for patterns to further our understanding of the weather, and hence, improve our forecasting ability well out into the future.

I think that considering weather patterns on the basis of risk, correlation of states, and teleconnections is the way forward.

Edited July 14, 2006 by Wilson

[VillagePlank]

Bloody hell, I used to be nice and polite to people once. See what being a member here does to you

[Gray-Wolf]

B*gger the maths let's just use quantum physics to establish a way to warp time and then we can 'fetch back' an accurate forcast from any time (long or short range)........................