Model Bias?

[Radiating Dendrite]

Are the weather models bias or just statistical number crunchers?

Previously discussed here

Please continue here.

[johnholmes]

ta for this. I think it is not a bad idea as it may enable people to understand how the models work. I'll put something in here during the day.

[recklessabandon]

ta for this. I think it is not a bad idea as it may enable people to understand how the models work. I'll put something in here during the day.

thanks, look forward to it

[johnholmes]

Model bias – true or untrue?

OED definition

noun

• 1 [mass noun] inclination or prejudice for or against one person or group, especially in a way considered to be unfair:there was evidence of bias against black applicantsthe bias towards younger people in recruitment

• a concentration on or interest in one particular area or subject:his work showed a discernible bias towards philosophy

• a systematic distortion of a statistical result due to a factor not allowed for in its derivation.

• 2 a direction diagonal to the weave of a fabric:a turquoise silk dress cut on the bias

• 3 Bowlsthe irregular shape given to one side of a bowl.

• the oblique course taken by a bowl as a result of its irregular shape.

• 4 Electronicsa steady voltage, magnetic field, or other factor applied to a system or device to cause it to operate over a predetermined range.

Not that helpful to meteorologists with only the one highlighted in red perhaps the closest? Indeed one poster has commented about statistical bias.

How about a meteorological definition?

Bias. The degree of correspondence between the mean forecast (<f>) and the mean observation (<x>). This type of bias is also known as overall bias, systematic bias, or unconditional bias. The mean error is a measure of the overall forecast bias for continuous and probabilistic forecasts. A measure of bias for categorical forecasts is equal to the total number of event forecasts (hits + false alarms) divided by the total number of observed events. With respect to the 2x2 verification problem example outlined in the definition of contingency table, bias= (A+ B)/(A+C).

If that means anything to you –good luck!

So best I give what I believe is the definition most often used on Net Weather.

Over the years I’ve been on Net Weather this topic raises its head fairly regularly. A mass of information is available and its often quite complex and misunderstanding it is easy, even for a professional.

The quote by one member from the NOAA source is perhaps fairly typical of this misunderstanding of terms meteorologists use in a different context. With this and any other quote I am NOT getting at anyone.

The NOAA comment talks of, see link below for the full article, they talk about

http://www.hpc.ncep..../biastext.shtml

modification and model performance characteristics. They then go on to use the word bias. This is different to what amateur meteorologists seem to associate the term with. All models show these, biases, model characteristics, call it what you will. That is the reason that the main centres regularly update their information to show what corrections they feel are necessary to arrive at the right conclusion. One example of this is the GFS world model cannot give an accurate idea of the gustiness of certain airflow patterns east of the Pennines but the Fine Mesh model Met UK use does. On a somewhat larger scale both models are able to replicate fairly well wind funnelling up the Forth-Clyde valley. These are model characteristics or biases if you prefer that word.

Another instance of the mis-use (in my view) of the term model bias is that the models revert back to the Atlantic mode/zonal type or they must have a bias because of the earth warming over the past century or so?

No they do not exhibit any bias to either of these. All models, operating to slightly different systems (nothing to do with what the weather is) ‘simply’ take the data available at their cut off time. They then, after quality control to try and remove as many errors, both in time and statistical type mentioned by one poster, and then simply run using the basic laws of physics applied to meteorology and solved by highly complex mathematical equations. The Met O site has a very good explanation of this, see the link below, should keep you occupied for hours!

http://www.metoffice...delling-systems

And yes somewhere in that lot you will find the phrase model bias but PLEASE refer to what I’ve explained above.

I hope this helps explain to some extent the possible confusion about model bias?

Edited November 3, 2011 by johnholmes

[kold weather]

Interesting John...the GFS in the past HAS had a slight cold bias and that has been confirmed by NCEP (There is a link somewhere where it notes some of the small biases that the model has displayed on that side of the Atlantic) and I suspect if it does have a cold bias it is therefore going to often overdo the thermal gradiant and therefore overdoing the jet, especially in the lower resolution part of the run and therefore probably has got a habit of over-deepening low pressure systems, hence why I actually think the comment that the model is 'deafulting' actually may have some sort of merit.

Doesn't always work out like that of course and the clever thing is knowing where and when those biases can cause problems.

There are numerous examples on some american websites where they have noted various baises of the models. For example one poster noted that the NAM model over there constantly was too low in temperature in one location and that has been statistically proven. The models are constantly being updated in order to try and either limit these biases or try and reduce them.

Another real classic bias was what the CMC used to do in the tropics. It nearly constantly tried to produce 'fake' tropical cyclones and would probably produce 50-80 hurricanes a year, which obviously was rubbish. Its been improved lately but it still does have a fair number of false alarms.

Most model biases are small, there are a couple that are quite big though and combined with several other factors/biases they really can throw a forecast off. Its just a case of deciding whether its a REAL signal, or a signal thats been induced by the models bias. Generally if mosst models show the same thing, its the real deal.

[recklessabandon]

Model bias – true or untrue.....?

thanks John, quite interesting. It was I who was raising the question/issue of the difference between statistical bias (especially that of variable omission bias) and cognitive bias. I was certainly not suggesting the any forecast models are influenced by cognitive bias (though reading some of the winter forecasts I do wonder ....)

Anyway, I think part of the confusion came from the definitions, clearly the METO have their interpretation of bias which of course in statistics is perfectly understandable, given how many different 'types' of bias one can encounter.

One question that does remain to me though; if the models are taking a global view over a known square area, how do they compensate for those regions of the globe where data on the ground may be very sparse, lets say the sahara regions of Africa as an extreme example versus western Europe, is provision given in the models for potential 'gaps' in ground-based data sets ( I am assuming that to a degree in-filling will be given by satellite).

[johnholmes]

.

One question that does remain to me though; if the models are taking a global view over a known square area, how do they compensate for those regions of the globe where data on the ground may be very sparse, lets say the sahara regions of Africa as an extreme example versus western Europe, is provision given in the models for potential 'gaps' in ground-based data sets ( I am assuming that to a degree in-filling will be given by satellite).

sorry I don't have the answer to that. Again it might be an idea to e mail Met and ask if they could explain how they deal with that.?

[asjmcguire]

Thanks John,

apologies for causing WW3

Clearly I am never going to get my head around how the models work without being qualified

[johnholmes]

Thanks John,

apologies for causing WW3

Clearly I am never going to get my head around how the models work without being qualified

no need to apologise-thanks - just keep reading and asking questions

[If Wishes made Weather]

On a statistical point that is indicated above by kold weather, there is a phenomenon called 'regression towards the mean'. Effectively this means that if there a potentially variable is extreme on first measurement, it will less extreme (i.e. less variable) on second measurement. Thus, in the FI part of the model runs, where variables are extreme, often we see knock out all of Western Europe sized storms, Arctic cold over Hawaii etc. This regression tendency will almost always moderate these extremes as time gets nearer, so we get, oddly, average weather.

[kold weather]

Here is a link with some of the model biases, note this is certainly not a complete list as I've heard of MANY others from US mets who probably have a career looking for this sort of thing:

http://www.hpc.ncep.noaa.gov/mdlbias/biastext.shtml

"As a result it is not only difficult to isolate consistent model performance characteristics (loosely referred to as “biasâ€) across the model upgrades, but also the source of the bias."

Notice several interesting ones for the GFS, for example overdoing the coverage of very light precip, convective feedback and "Slightly ambitious with magnitude of high amplitude patterns"

[johnholmes]

Here is a link with some of the model biases, note this is certainly not a complete list as I've heard of MANY others from US mets who probably have a career looking for this sort of thing:

http://www.hpc.ncep..../biastext.shtml

"As a result it is not only difficult to isolate consistent model performance characteristics (loosely referred to as “biasâ€) across the model upgrades, but also the source of the bias."

Notice several interesting ones for the GFS, for example overdoing the coverage of very light precip, convective feedback and "Slightly ambitious with magnitude of high amplitude patterns"

I think that is the link I quoted from K but its worth reading it carefully and note also the sentence from K in italics-the word bias is really model characteristics.

There is no bias to one type of weather or another in the way some folk on here use the term. Sorry to harp on about it but the text from NOAA is the one to use, sadly UK Met nor ECMWF do not seem to be as free with their views on their own models unless I'm looking in the wrong place.

Edited November 7, 2011 by johnholmes