Retrospective seasonal model output

[Rain All Night]

The Meteociel site has now been publishing the seasonal model outputs from the contributors to the Copernicus Climate Change Service (C3S) for long enough that we have an opportunity that I suspect (though I've only been here a year) the community has never had before, which is to retrospectively view the seasonal outputs, for each of the previous 12 months, from runs up to 6 months in advance, allowing us to reflect on these models and consider how much credence we might give to their outputs going forward.

The charts are found here: https://www.meteociel.fr/cartes_obs/c3s_panel.php?mois=8&annee=2023&mode=12&ech=1&size=2&map=0

I produced Z500 anomaly monthly analysis charts using the same map projection of Europe as Meteociel using this tool here: https://psl.noaa.gov/cgi-bin/data/composites/comp.pl?var=Geopotential+Height&level=500mb&mon1=7&mon2=7&iy=2023&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&iy=&ipos[1]=&ipos[2]=&ineg[1]=&ineg[2]=&timefile0=&tstype=0&timefile1=&value=&typeval=1&compval=1&lag=0&labelcolor=Color&labelshaded=Shaded+w%2Foverlying+contours&type=2&scale=100&contourlabel=0&switch=0&cint=10&lowr=-160&highr=160&proj=Custom&xlat1=30&xlat2=80&xlon1=-60&xlon2=40&custproj=Cylindrical+Equidistant&level1=1000mb&level2=10mb&Submit=Create+Plot

Models

The models are, in the formation they are displayed on Meteociel's grid:

C3S multi-system — ECMWF SEAS5 — Météo-France System 8

UKMO GloSea6 — CMCC SPS3.5 — ECCC CanSIPSv2.1

JMA CPS3 — NCEP CFSv2 — DWD GCFS2.1

September 2022

——————

October 2022

——————

November 2022

——————

December 2022

——————

January 2023

——————

February 2023

——————

March 2023

——————

April 2023

——————

May 2023

——————

June 2023

——————

July 2023

——————

August 2023

——————

Edited September 3, 2023 by RainAllNight
Added the climate average for each month.

[Rain All Night]

To hopefully spark some conversation in this thread, I have written an amateur review of the seasonal modelling of each of the past 12 months.

Sept 22: The August runs were arguably seeing the potential for mid-latitude blocking to develop (the Atlantic ridge which finally doused the flames of summer), but couldn't pin down where. Before that, not really much good - they all missed the low heights over eastern Europe.

Oct 22: Even the October runs couldn't get this one right. ECCC did seem to pick up the potential for a significant Euro high and/or Atlantic trough on some of its earlier runs. Otherwise, one of the worst-modelled months of the past year.

Nov 22: The September and October runs clearly saw the potential for a significant block to develop, but they underestimated its latitude and (as I recall) overestimated how early it would arrive. By underestimating the latitude of the block, they of course missed the enormous area of low heights in the Atlantic which developed in the middle part of the month.

Dec 22: Perhaps the best-modelled month of the past 12. Models could clearly see the potential for significant blocking from the outset, with ECMWF, UKMO and JMA having done a better job than the other models at picking out on some runs that the blocking would be at high latitude.

Jan 23: A number of runs from a number of models at a number of timescales show the Atlantic blocking that occurred, but there are also too many runs misleading us with suggestions of Scandinavian heights that never came into being.

Feb 23: Too many runs from most models giving the impression that the powered-up Atlantic (which they did correctly foresee in many cases) would be able to reach us in this month, which of course it did not. Many runs did see the block coming, but could not place it correctly.

Mar 23: ECMWF did seem to forecast this reasonably well on later runs, but missed the spoiler Iberian heights. Otherwise poor.

Apr 23: The DWD model did OK here, otherwise generally poor - interestingly some models made their best predictions for April way back in November (Météo-France, JMA, NCEP), and the C3S multi-system mean was one of the better indicators for this month too.

May 23: None of the models had any idea what to do with this month at any point, not even at the beginning of May itself. Very poor, possibly the worst-modelled month out of the last 12.

Jun 23: Even at the beginning of June itself, the models placed the core of the blocking incorrectly. Prior to that, ECMWF and to a lesser extent UKMO generally did a decent job of this one, and you could say the NCEP generally saw the theme of strong northern blocking. CMCC had some OK runs earlier, but dropped the ball later.

Jul 23: ECCC notably seemed to spot that the other models were missing some significant troughing in the middle latitudes. I don't think this month was actually modelled as badly as you might imagine (given the failure rate of human-authored forecasts for the summer) - you can clearly see the models were sniffing out the significant Labrador Sea blocking, though they did not give us the right overall impression about the impact of this.

Aug 23: ECCC again did the best job of giving the right overall impression, and again, the models generally picked up on the theme of heights to the north. The CMCC model was particularly consistent with this.

Generally, it is impressive what some of these models can do some of the time, but there is just not enough consistency. Each month it is often a different model or couple of models that is able to pick up on the right signals. But in a way, it is incredible that any model is ever able to do this at such long range.

Do let me know where you agree or disagree with what I have said, or any other thoughts you might have on the merits of the seasonal modelling output.

Edited September 5, 2023 by RainAllNight

[Dorsetbred]

On 05/09/2023 at 11:39, RainAllNight said:

To hopefully spark some conversation in this thread, I have written an amateur review of the seasonal modelling of each of the past 12 months.

Generally, it is impressive what some of these models can do some of the time, but there is just not enough consistency. Each month it is often a different model or couple of models that is able to pick up on the right signals. But in a way, it is incredible that any model is ever able to do this at such long range.

Do let me know where you agree or disagree with what I have said, or any other thoughts you might have on the merits of the seasonal modelling output.

Nice review, thanks for sharing and taking the time to put this together. It's good idea, I'm just wondering how many folks will catch on and engage, and given the bun fights in the MOD thread, this could easily turn into "I disagree, with what you say...". Let's see how it goes.

[Mike Poole]

Interesting to have a proper look at the performance of the seasonal models, there aren’t (to my knowledge) any publicly available stats we can access on this.  A few thoughts occur to me.

First, I would expect the models to perform much better in winter than at other times of the year, and your analysis highlighting December as the best performance is therefore not a surprise.  In winter, there is a strong driver in terms of the stratospheric/tropospheric polar vortex (PV).  We know that a weak or strong PV can make a massive difference to the presence or absence of blocking highs that persists for some time, so if the models can get a handle on the PV, they will perform well.  To do that needs a decent representation of the stratosphere, and therefore vertical levels and the model top are important.  

In summer, the patterns are flabbier and there is no PV ruling the roost, so the models are picking up smaller signals likely based on SSTS - apart from the stratosphere in winter, the main way the seasonal models differ from the short range ones is that they couple a model of the atmosphere and oceans.  

One thing it would be nice to tease out is what reasonably constitutes a ‘signal’ amongst the noise from these models, I think the mean anomaly results over ensembles members and long times are often rather misinterpreted in discussion.  It is more useful to see the monthly means than the 3 monthly means for sure, as on the 3 month scale, the variability within that timeframe is lost.  All of which means one is often dealing with small shifts in probability away from the climate normal probability which are sometimes perhaps interpreted as being larger than they actually are.  

I would be interested to hear others’ perspectives on the seasonal models.

Edited September 7, 2023 by Mike Poole

[Rain All Night]

3 hours ago, Mike Poole said:

One thing it would be nice to tease out is what reasonably constitutes a ‘signal’ amongst the noise from these models

I am aware of a few ways in which the some of the outputs that we have access to attempt to tackle this. I'll use the August CanSIPS (ECCC) and ECMWF outputs for January 2024 to illustrate.

Here are the some charts from the Copernicus site. Firstly, an ensemble mean anomaly chart, with the added feature of a "1% significance level contour", which as I understand it, means that inside of the grey line, the chance is supposed to be (assuming the model were ideal, which of course it is not) less than 1% that the anomaly shown has resulted from random chance, rather than from an actual signal. This contour is for some reason not present on charts generated from the ECCC (CanSIPS) model.

Alongside that, we have another similar-looking chart, which instead of showing raw anomalies, shows the probability that the Z500/MSLP will be either in the highest 33% (tercile), or lowest 33%, of the historical values observed in the climatology period that is being used. I understand that these probabilities are calculated by using the seasonal model to recreate historical forecasts, then looking at the difference between what the model predicts and what we know actually happened, to determine the model's biases.

Here is another ensemble mean anomaly chart showing the same MSLP output from the same CanSIPS run as shown above, but this time from the Tropical Tidbits site. Alongside it is another similar chart, showing what is referred to as the "normalised anomaly", which I understand means that instead of showing the raw anomaly in millibars/hPa, it is shown in "standard deviations", which means that the strength of the anomaly is adjusted based on the range of anomalies that were observed during the climatology period being used, so for example if you historically only saw anomalies of between -2mb and +2mb in a particular location, but now you've got an anomaly of +4mb, that's going to show up very strongly because it is an "anomalous anomaly", whereas if you've got an anomaly of just +0.5mb, that's going to be only faintly visible on the chart (if at all) because it is well within the range of historical "normal anomalies". I hope I have explained this correctly!

Tropical Tidbits seems to have "normalised" charts for the MSLP parameter only, and it only has the CanSIPS (ECCC) and CFS (NCEP) seasonal models.

Edited September 7, 2023 by RainAllNight