Leaky Integrator Discussion

[VillagePlank]

The model you are using is the discrete form of the LI i.e. something like

T(t) = (1 - a * DT) * T(t-1) + { b * S(t-1) + c * V(t-1) [ + d * X(t-1) ...] } * DT

Well, the LI has moved on, so what's contained in the thread is old. Without checking out your formula, critically, I would say that that's on the right track.

(This is the bit where it gets a bit coy, since I don't want to say to much until the testing is done)

Your colours are nailed to the mast as your model is fixed and you appear content with the results you are producing as you tweak it further.

Demonstrably not true. If the LI fails the tests I have already outlined, I will drop it in an instant.

Edited December 14, 2009 by VillagePlank

[Guest diessoli]

(This is the bit where it gets a bit coy, since I don't want to say to much until the testing is done)

Oh come-on what about maximum transparency?

D.

[VillagePlank]

Oh come-on what about maximum transparency?

I can't make me look like the fool I really am, now, can I?

Once I've done the testing, I'll write it up, and everyone will know then ... and, besides, I am being totally transparent about not being transparent on the methods ...

Edited December 14, 2009 by VillagePlank

[Roger J Smith]

I've been testing out some ideas myself here, and found some interesting factoids.

First of all, I extended the annual sunspot number back to 1659 using Schove's estimates and profiles of similar peaks. Then the year-to-year correlation of mean annual CET and sunspot number is 0.20, but if you exclude the Maunder and start at 1714, the correlation drops to 0.14.

Now, I realize that your concept is to correlate a much longer running average with mean temperature, so I tried that for various assumptions and found that the best fit was a 10 year lag of the 10-year average, correlation .33 ... no doubt it's a higher correlation with ten-year temperature averages.

If I extended either the number of years averaged (for sunspot number) or the lag, the correlation very slowly dropped off, but not much, it stays around 0.3 as far as 40 years of running data with no lag (this implies a lag of 20 years), long periods and large lags started to drop off towards random.

So I would have to confer on the math, but I think the leaks are fairly fast out of the atmospheric heat retention system if the main driver is solar variation. From what I'm seeing the best fit is 15-25 years lag. For example, in the coldest part of the Maunder, the lag is almost zero. In the warmup period of the 1720s and 1730s, the lag is at most ten years. In the late 18th century, there is not a very systematic arrangement of CET and any kind of lagged solar activity. In the Dalton, the lag is about 10-15 years to coldest temperatures. From there on it's a bit noisy through the 19th century, the warmth of the 1930s seems to be a lag of about 10-15 years after resumption of strong activity, there is no real explanation for any cooling in the 1940s using any assumption of lag, and then we have the modern period where temperatures continue to plateau, the warmest conditions based on a 20-year lag should have occurred around the 1970s with a very slight decline thereafter, just starting to accelerate now.

So I think this would argue for an internal source of warming such as greenhouse gases since 1970, the amount seems to be increasing slowly into the 1.0 C range.

If I tried to modulate the 20-year lag model with lunar data, I achieved no better correlations despite the added functions having independent correlations of 0.2, but I produced temperature graphs that looked very similar in "bounce" to the actual data. This means I need to look at the error pattern to see if there are lags, or other factors indicated.

The years 1740 and 1879 stand out as being way too cold for the model and of course are anomalous mainly for their very cold winters, but as to the cause there, it may have been random noise more than anything as direct as volcanic dust or whatever, after all, the atmosphere is a fluid and it would be difficult to model it precisely from any perspective.

What would you say the operational lag time would be in the LI concept? It seems longer than what I tested, but the longer you make it, the more trouble you'll have with the cold phase of the Maunder minimum, because there does not seem to be much lag in that postulated cause and effect at all.

The MWP, by the way, looks okay but I am not sure why the 14th century wouldn't have been warmer than the MWP from this hypothesis. As I understand it, the LIA set in around the 14th century and had several "inter-little-glacials."

I would say, before you eliminate other variables from your model, you might want to consider the specificity of the UK climate as your data base for temperature. If global temperature does not necessarily correlate with UK temperature, then what is the significance of the lag time being demonstrated in the model? Is it partly heat stored in the Atlantic circulation? Could this be an indirect way of testing out whether the external heat generated by solar activity is received, stored and dissipated at equal rates in all climate regions, or perhaps hoarded longer in some than others. I suspect the latter, because temperatures have been dropping off to some extent faster in North America than Europe since 2002. In other words, the leak may be faster through our stronger portion of the magnetic field.

[VillagePlank]

What would you say the operational lag time would be in the LI concept?

The main form of the leaky integrator is,

dx/dy=-Ax+C

or, my preferred version,

da/dt = -xa+yb

where x, and y, are constants for the system. This is extended (by me) to mimic a leaking bucket to,

dh/dt = ri-olh

which is described on the main thread (and the PDF) Each of these represent the rate of change at some point in the system. I suppose it is possible to determine what is left over after each iteration to find the 'lag' effect, and then partition that effect to years hence - but I haven't tried (Maybe I should? - perhaps by using some sort of FIFO apportioning system?)

What this means in natural English is that the warmer it becomes the harder it gets to heat up, and the cooler it becomes the slower it gets to cool down. So, the operational lag is therefore entirely dependent on what is in the system.

Being dependent in this manner means that the 'lag' is therefore entirely dependent on the systems history (since it is modelled as a polynomial - and therefore discrete, as dissouli says) -which is why I've used the word 'hysteresis'

Edited December 14, 2009 by VillagePlank

[Methuselah]

Hi VP,

Having spent far too-much time on the 'Night Train to Moscow' of late, I no-longer have the nous to make very many (none? ) useful mathematical contributions to your hypothesis. But, having said that, it's great to see a sceptical application of science!

The LI may well 'fall flat on its face' (as may the MetO's latest predictions for AGW!) I don't know. (As a sceptic myself, I really don't mind either way.) But, whatever the LI's flaws, pitfalls or successes - IMO, the real point is the scientific method...

So, many thanks for all the hard work. (And expense!) Because (IMO) even as an heuristic, it's very-well worthwhile...As I've said before: one doesn't need to deny the recent warming to propose an alternative explanative hypothesis!!

[VillagePlank]

What would you say the operational lag time would be in the LI concept? It seems longer than what I tested, but the longer you make it, the more trouble you'll have with the cold phase of the Maunder minimum, because there does not seem to be much lag in that postulated cause and effect at all.

That's because at low levels of input, the system responds almost instantaneously - so no 'lag effect' should be observed.

The LI may well 'fall flat on its face'

My opinion is that, rather like a game of Texas Hold 'em: never get married to pocket bullets. You'll lose a lot money ... what starts of as the very best hand, quickly can turn into a pair that is easily beaten.

Edited December 14, 2009 by VillagePlank

[Admiral_Bobski]

Just to emphasise VP's point there, the LI's time lag is variable, so the length of lag is greater the warmer the Earth gets.

It occurs to me that I have found several scientific papers which talk about lags in the climate system from solar activity. These lags vary a lot, depending upon whose paper you read: some say the lag is 2-3 years, some say 10-20 years, some say 40-50 years and others talk of lags of over 100 years (sometimes well over).

To my mind this tells us two things: firstly, there may be a variety of lags occurring (a lag for atmospheric processes, a lag for oceanic processes, a lag for concrete-related processes and so on), all superimposed on each other. Secondly, the variety of lags may be a symptom of scientists attempting to impose specific, discrete values to a variable phenomenon - if you're trying to find a repeating pattern in a system that doesn't repeat then you end up with a wide variety of possible correlations.

The answer could be either one of these, or it could be an element of both (or it could be neither, I admit). Interesting, though...

CB

[VillagePlank]

Interesting, though...

It also explains why Fourier analysis doesn't properly yield the solar cycle (and, being the main driving force - quantitatively, speaking - behind the climate, it should do - shouldn't it?)

Although the alternate hypothesis is that the solar effect is (more or less) constant, so the solar cycle is meaningless, but, as far as I can ascertain, that is the case only when looking at insolation. The sunspot cycle looks at solar activity.

Edited December 14, 2009 by VillagePlank

[Admiral_Bobski]

It also explains why Fourier analysis doesn't properly yield the solar cycle (and, being the main driving force - quantitatively, speaking - behind the climate, it should do - shouldn't it?)

Although the alternate hypothesis is that the solar effect is (more or less) constant, so the solar cycle is meaningless, but, as far as I can ascertain, that is the case only when looking at insolation. The sunspot cycle looks at solar activity.

One would certainly expect it to...I think...

Does this then lend further credence to the LI hypothesis...?

</bait>

EDIT - If the sunspot cycle being meaningless is only relevant when looking at insolation then are we, perhaps, not being insolant enough...?

</pun>

CB

Edited December 14, 2009 by Captain_Bobski

[VillagePlank]

Does this then lend further credence to the LI hypothesis...?

Only if the sunspot count (hence solar cycle) is a proxy for something else ...

[Admiral_Bobski]

Only if the sunspot count (hence solar cycle) is a proxy for something else ...

Time for me to go digging for clues, methinks...

[Iceberg]

Sorry VP a rather pointless post from me, just to say that I 've not forgot LI, but I am rather interested in the current weather for the UK and this is taking my netweather time up.

Cheers

[VillagePlank]

Sorry VP a rather pointless post from me, just to say that I 've not forgot LI, but I am rather interested in the current weather for the UK and this is taking my netweather time up.

Aren't we all! You know I once tried to get historical information (slp, humidity, temps) so that I could do a heuristic winter forecast for this very website? Didn't get the data, couldn't do the forecast ...

Here's looking forward to more high temperature records

[Roger J Smith]

First of all, thanks for the explanation of how the lag varies with past circumstances.

I have continued to examine the sunspot - annual CET correlations, by taking lags of one, two, three and four years. The correlations are interesting. You'll remember I reported a correlation of .20 for the same year. The correlations then become .21, .19, .17 and .14 for these four lag cases. In other words, the immediate effect of solar variation on CET appears to have a one-year lag, on average.

Bear in mind that I also found a higher correlation, .33, for the running ten-year average up to the year in question. This correlation fell off for longer averaged periods. So I went back and tested for correlations of temperature with 8, 6, 4 and 2 year averages before (up to and including) the year. Starting with the 10-year average correlation of .33, this gives a table as follows:

10-yr avg ... .33

8 -yr avg ... .29

6 -yr avg ... .26

4 -yr avg ... .23

2 -yr avg ... .21

1 -yr avg ... .20

(prev yr) ... .21

(2nd prev yr) .19

(3rd prev yr) .17

(4th prev yr) .14

This led me to test longer averages (20 years was shown to be a lower correlation). I tested 11, 12, 13 and 15 year averages, finding these correlations: 0.34 for each, but the highest taking the third decimal place was 11 years (.344) after which it slumps to .336.

So, the very best fit is the running eleven-year average, and I suspect if you lagged that by 1-3 years it might edge up to .35 correlation.

A rather imprecise way of visualizing this is that the UK climate responds to solar variation about one cycle behind, and this would suggest that a sharp cooling may lie ahead since the climate is now responding to the ever-falling sunspot number average of the previous 11-12 years. With a quiet sun for 2-4 years longer, this would imply a very cold period to come around 2015-2020.

The LI concept appears to postpone that ... however, decade to decade changes in the past have been fairly steep ... for example, the winters of the 1930s compared to the 1940s.

[VillagePlank]

A rather imprecise way of visualizing this is that the UK climate responds to solar variation about one cycle behind, and this would suggest that a sharp cooling may lie ahead since the climate is now responding to the ever-falling sunspot number average of the previous 11-12 years. With a quiet sun for 2-4 years longer, this would imply a very cold period to come around 2015-2020.

If sunspot count were the only factor, I would expect global temperatures to be plummeting very quickly very soon. However, ENSO, sea-ice, and volcanic activity play an important part in getting to r=0.91. Caveats do apply with two of these parameters such as I am unsure how to derive the degrees of freedom on ENSO and sea-ice.

My, perhaps naieve, analysis of this is that the current very high global temperatures are a consequence of a very active sun for 50 years (ish) that gradually accumulates to the high temperatures, today - for arguments sake, let's call it +0.5. My experiments have shown that even if you negate all other factors, and turn the sun off - it will take less than a decade to reach a median where there are just as many warm anomaly years, and cold anomaly years. That we expect the next solar cycle (so sunspots start to be added into the system) to start in the near future, and in the absence of mitigating ENSO, sea-ice, and volcanic activity, means that that 'resetting' to the norm will take even longer.

Indeed, I expect record hot temperatures since variability might be, say, +/-0.2C, the LI might be losing temps at, say, 0.1C, and since 0.4+0.2=0.6, then that posts a new hot temperature record. Without a -ENSO, or sea-ice extent increase, or a big volcanic eruption - it is difficult to see how the climate, according to the LI, can get to that median point within the next two solar cycles - approximately 25 years from now, and that's assuming that the solar cycles are average, or less than average in magnitude.

I think that that's a point that a lot of people who peruse the LI miss. It predicts further warming for the foreseeable future; it doesn't predict imminent cooling, requiring some natural event to 'reset'

(of course it's bedtime here, so figures are 'off-the-cuff' but close enough)

(EDIT: Quick spurious bedtime thought - that the LI output agrees with, say, IPCC output in the medium term is interesting. Could it be that CO2 and temps do correlate, but only that CO2 is a proxy for something else?)

Edited December 15, 2009 by VillagePlank

[Roger J Smith]

Oddly enough then, if the climate cools significantly in the next five years, this would point more to recent increases being AGW-driven than not, from the perspective of testing which of these two sunspot predictors fared better. The LI model resists cooling and attempts to explain all or at least most of the recent warming, the ad hoc RJS first attempt has no other explanation for warming since 1970 but does predict enough of a cooling to offset the AGW (presumably) warming. Since I haven't committed to the alternative hypothesis yet, I think I'll put it on the table as a what if, and say that a sharp cooling of annual CET values any time now to 2018 would give it more credibility, while a plateau in CET values or continued slight rise would make the LI more credible, and leave us unsure about AGW since the actual answer could be a compromise.

Lunar variables from my research do not combine to produce large century to century temperature swings, this is where I differ from David Dilley to some extent, the main effect of the lunar variables is to modulate a fairly stable climate. These larger swings would require explanation from solar, planetary or other sources. However, I am willing to meet Dilley half-way on his 241-year cycle, I just think it may be a bit overdone and as to its proposed application to very long cycles, implausible given the credibility already given to Milankovitch type orbital forcing.

That's where I stand on these issues, not that it really matters since lunatics are now running the asylum.

[Iceberg]

Morning VP,

I find myself with a bit of time thanks to no snow etc and my mind started wondering back to this.

Firstly I thought it was on this thread, but I cant for the life of me find it, sorry I am talking about the PDF of your draft paper ?

Now, also do you work off SSN, or SI ?

If we have a net balance loss of heat from the earth, would this reduce temps according to your model ? i.e more leaves the bucket than enters it ?

Finally what part of the equation does ENSO impact in your model ? I' ll add to this in a min.

[Iceberg]

Sorry found the pdf now on the other thread.

ENSO should effect the calculation differently to SI/SNN or volcano, indeed volcano should effect differently to SI/SNN and ENSO.

Essentially they all have different ratios of effect on variables in the equation (even if they all have a based effect on global temperature), this effect on global temperature is a by-produce of the their processes, which must be modelled rather than the by product in an equation like this.

Hope the above makes sense.

I'd also like to see and understand how the 0.91 correlation is achieved and how you've factored stuff in before further comment.

Cheers

[VillagePlank]

Now, also do you work off SSN, or SI ?

If we have a net balance loss of heat from the earth, would this reduce temps according to your model ? i.e more leaves the bucket than enters it ?

Sunspot number as published in the references.

I am not sure what you mean by 'net balance loss' Essentially, the argument is this: As a general rule input and output is balanced, such that we see decadel variance but not by that much, or by that little. It comes and goes. However, the system is sensitive enough that if there is sufficient quantity of incoming energy, and sufficient quantity for long enough then the capacity of the system to get rid of that is overwhelmed and some of that energy is stored - somehow. Observational evidence that it is stored is in the temperature record.

This assumes that the GhG effect is constant, or, to be more precise that sufficient GhG are now present that we no longer are on a steep upward slope of the logarithmic curve. Rather, we are at the top flattening off bit. This, of course, has consequences, such that I suspect that the same effect that the LI posits for sun (ie sufficient quantity for sufficient time) is also true of the GhG effect - time being the operative word, here, and whilst on a snapshot the sun appears constant, under the LI theory GhG appears constant, but long periods of time might indeed prove it otherwise.

Of course, this is all conjecture, and theorising.

ENSO should effect the calculation differently to SI/SNN or volcano, indeed volcano should effect differently to SI/SNN and ENSO.

Essentially they all have different ratios of effect on variables in the equation (even if they all have a based effect on global temperature), this effect on global temperature is a by-produce of the their processes, which must be modelled rather than the by product in an equation like this.

yes, they do - that's why they all have multiplication factors associated with them (appears at the top of each graph)

I'd also like to see and understand how the 0.91 correlation is achieved and how you've factored stuff in before further comment.

Stuff is factored in as described in this thread and the LI theead. Clearly, specific details (such as how to do the gradient climb for parameter fitting) are only going to be viewable in the final paper. That's why I did the walk-though, so people could get involved. It should be noted that the parameters have not been plucked out of thin-air.

The correlation is a comparison of the LI output to the Hadley set, and measured using the Pearson method (Excel help files should explain) This simple describes the closeness of how each line goes up an down.

The certainty of this correlation is yet to be done.

[Iceberg]

Sunspot number as published in the references.

I am not sure what you mean by 'net balance loss' Essentially, the argument is this: As a general rule input and output is balanced, such that we see decadel variance but not by that much, or by that little. It comes and goes. However, the system is sensitive enough that if there is sufficient quantity of incoming energy, and sufficient quantity for long enough then the capacity of the system to get rid of that is overwhelmed and some of that energy is stored - somehow. Observational evidence that it is stored is in the temperature record.

Thanks VP, I saw a paper a couple of days ago, (which made me think of this thread), which showed that after 2002 there was a net loss of energy out of the earth, I would have thought that this net loss needs to be factored in.

Also re SSN as an example we have seen very little SSN over the last 2 years, however the actual amount of SI is far above what was experienced when SSN's were at similar levels in the during the Maunder minimum. Therefore SSN's can't be a proxy for energy from the sun. (although there is cyclical agreement).

Can you post your latest PDF here so I can have a look at it, I have a feeling I might have been looking at an old one.

Thanks btw for responding, I do genuinely want to look into this, but need to understand how the correlation is achieved first and that the right things are feeding into the correlation.

[VillagePlank]

Also re SSN as an example we have seen very little SSN over the last 2 years, however the actual amount of SI is far above what was experienced when SSN's were at similar levels in the during the Maunder minimum. Therefore SSN's can't be a proxy for energy from the sun. (although there is cyclical agreement).

Yes - I am very guilty of using the word 'energy' incorrectly. The way I think about it (and very unscientific it is, too) is that each of the factors are there because they have an effect. So, figuratively speaking, SSN's are there to model solar effect of which SI is taken to be a part.

Can you post your latest PDF here so I can have a look at it, I have a feeling I might have been looking at an old one.

Still working on it. Maybe early Feb.

Edited January 19, 2010 by VillagePlank

[sunny starry skies]

Yes - I am very guilty of using the word 'energy' incorrectly. The way I think about it (and very unscientific it is, too) is that each of the factors are there because they have an effect. So, figuratively speaking, SSN's are there to model solar effect of which SI is taken to be a part.

Still working on it. Maybe early Feb.

I too would be interested in seeing a copy of the paper - I can't find your old one (am looking in the wrong place?).

I posted this in the general discussion, but most of the points/questions should lie here:

I like the Leaky Integrator idea, but I think it is merely a way of showing how the climate system works, and not what the various forcong mechanisms are. It's about the most constructive non-GHG discussion I've seen on the Web. It's clear there are various lags and feedbacks in the system, and the LI can demonstrate these to an extent. However, I also don't think it can test the difference between GHG forcing and solar forcing, as by it's definition can't provide the physical mechanism by which one is stronger than the other? Please correct me if I'm wrong!

I should maybe post this in the LI thread, but is there not a way to test LI (solar forcing)? Using palaeoclimate records for the last 500 years or so and sunspot records... if you can show with a record corrected for ENSO and volcanic, that an LI mechanism explains the variations, then you've made a start. [Edit from previous: sounds from the chat here that you've done something like this, so will reserve comment till I see it!]

My biggest caveat is that correlation is not causation. Reproduction of a pattern of wiggles is half the problem, the next thing is finding a mechanism. There's a good physical reason why GHGs raise the climate above what it otherwise would be without GHGs, but so far not the same reason for solar variations. ALso, so far as I can see, you could explain GHG warming with the LI by having more-or-less the same energy input and by steadily reducing the size of your outlet, to mimic the energy retained in the system by GHGs. [Added] Much as AGW proponents have to eliminate the possibility that extra energy is entering the system, do you not have to eliminate the possibility that your outlet is getting smaller?

But one thing I will not say is that the LI idea is silly / dismiss it out of hand. It's a good idea, but I remain cordially unconvinced until you can show me that you can reproduce past and present changes, and have a mechanism why it should be solar and not GHG. I think if I were to be reviewing the hypothesis as a paper, that would be my caveat if you were to be making significant comment about global climate. Are you thinking of publishing it?

[Edit: As it seems there is a paper, will wait on my caveats till I see it!]

All the best,

sss

[VillagePlank]

The convincer for me to carry on with the idea is that after I'd 'fiddled' with parameters until I got the highest Pearson score that I could. This left a big problem in the 1940's (ish) that turned out to be a problem with the Hadley set. Once corrected the line was a very good match. What are the chances of identifying that fault with the Hadley set? It was/is my consideration that the chances of this occuring, by accident or chance, is next to nothing, hence a belief that there's something in it.

As you say, correlation is not causation, but whilst people will put caveats on 'I won't believe a word of it until I see a physical' explanation, that is not the idea. I am no climatologist, I am a computer programmer (by trade - I'm an IT manager, now) so I wouldn't know the first place to look. The idea of it is to see if there is a method whereby the LI beats random data from the same method, such that it is 95% certain (stdev etc) that the correlation is significant, and thus needs further research to look for any physical factors. That is the best conclusion that I can possibly hope for: that there is a significant correlation between sunspots/volcanic/ice-extent/ENSO combined in some fashion and the Hadley temperature set.

I am currently writing the source-code without recourse to hidden libraries so that all of the work can be easily scrutinised.

I will publish whatever the result.

Edited January 19, 2010 by VillagePlank

[Admiral_Bobski]

A while ago I did post up a description of a basic physical process, at an atomic/sub-atomic level, that would explain it. I think it's back in this thread somewhere.

And remember - absence of proof is not proof of absence!

CB