Zenarcher

Partly cloudy with sunshine in the afternoon then around 5.30pm it started raining heavy for a few minutes with the gust of 53mph being recorded. It's now a cold Northerly wind but hasn't brought any wintry weather yet. Temperature 3.3°C Wind speed 25mph gusting to 40mph highest gust today 53mph. Pressure 1007mb Rising

How It Was Done, During the early part of December 2018, I decided to start an experiment to see how the weather models would perform forecasting the weather up to 6 days ahead. To get things started I choose a location near me that being Stornoway on the West coast of Scotland there is a Met Office weather station situated there and I wanted to use a Met Office weather station since it would record accurate weather observations during this experiment and there is also another weather station close by at the airport I would use this as a back up in case the one from the Met Office failed to report any readings during the experiment. Please Note, I plan on doing more of these once the new GFS model is out and the next time I will be using multiple locations across the whole of the UK instead of one location in order to get more accurate results. Please do not take the results from this experiment too seriously as it was just for me to see which weather forecast was the best for my local area and I've decided to share the results here with you all. Weather Models Used, GFS (Global Forecast System) run by the United States' National Weather Service. ECM (European Centre for Medium-Range Weather Forecasts). UKMO The UK's Met Office. ICON (ICOsahedral Nonhydrostatic general circulation model) from the German Weather Service. GEM or as it's sometimes called the CMC is from the Environment and Climate Change Canada weather model. FMI Hirlam Model from the Finnish Meteorological Institute. ARPEGE from Meteo France. HIRLAM or sometimes called KNMI is from the Netherlands Weather Service. Weather Apps and Websites Used, BBC Weather I decided to include it since they use data from the MeteoGroup I'm not sure what weather model they use it could be the ECM or a combination of several weather models. Weather Underground They have the largest weather station community in the world which helps them produce their own forecasts. DarkSky An award-winning weather app. The Weather Channel Which Google use for their weather forecasts. Accuweather. How The Results Were Recorded, Every day over the following weeks I would take notes on what every forecast said from all of the weather models 12z runs and when the time and day came I would check the observation made at the weather station and take a note on how far off their predictions were. Then all of the results were averaged out with a months worth of data to find the mean error from each of the forecasts. Air Temperature Results, 24 Hours (1 day) Average Mean Error, 1st Met Office 0.52 2nd BBC Weather 0.65 3rd ICON 0.66 4th Weather Underground 0.73 5th The Weather Channel 0.74 6th DarkSky 0.75 7th ECM 0.76 8th AccuWeather 0.77 8th GEM 0.88 10th GFS 0.89 11th ARPEGE 0.93 12th FMI 1.61 13th HIRLAM 2.20 Notes: The GFS was a bit of a surprise with it being down in 10th place it was slightly more bias towards having the temperatures warmer than forecast the same goes to FMI and HIRLAM which also both struggled when it came to predicting colder weather and overnight temperatures. 48 Hours (2 days) Average Mean Error, 1st ICON 0.72 2nd Met Office 0.75 3rd ECM 0.77 4th AccuWeather 0.81 5th DarkSky 0.82 6th Weather Underground 0.85 7th GFS 0.90 8th The Weather Channel 0.95 9th GEM 1.06 10th ARPEGE 1.09 11th BBC Weather 1.15 12th HIRLAM 1.29 13th FMI 1.72 72 Hours (3 days) Average Mean Error, 1st ICON 0.79 2nd AccuWeather 0.88 3rd Met Office 0.97 4th BBC Weather 1.04 5th ECM 1.11 6th DarkSky 1.12 7th GFS 1.13 8th Weather Underground 1.19 9th GEM 1.31 10th ARPEGE 1.54 96 Hours (4 days) Average Mean Error, 1st ICON 1.06 2nd Met Office 1.09 3rd BBC Weather 1.22 4th GEM 1.23 5th ECM 1.25 6th Weather Underground 1.30 7th DarkSky 1.31 8th GFS 1.50 Notes: You may notice some of the weather models and have disappeared this is because not all of them go this far out. 120 Hours (5 days) Average Mean Error, 1st ICON 1.29 2nd GEM 1.43 3rd Met Office 1.44 4th Weather Underground 1.47 5th DarkSky 1.50 6th ECM 1.52 7th BBC Weather 1.62 8th GFS 2.00 Notes: This is as far as the ICON 7km data goes and it still remains in first place it's done really well with 3rd place at day 1 but leading all the way from day 2 to day 5 it's the clear winner when it comes to forecasting temperatures at short range. 144 Hours (6 days) Average Mean Error, 1st Weather Underground 1.50 2nd GEM 1.70 3rd DarkSky 1.71 4th BBC Weather 1.72 5th ECM 1.75 6th Met Office 1.76 7th GFS 1.79 Mean Error Between 24 and 144 Hours (1 to 6 days), 1st ICON 0.90 (up to 120 hours 5 days) 2nd Met Office 1.08 3rd Weather Underground 1.17 4th ECM 1.19 5th DarkSky 1.20 6th BBC Weather 1.23 7th GEM 1.26 8th GFS 1.36 Notes: The ICON seem's to be in a different league when it comes to predicting the air temperatures I've looked through the data and it shows just how incredibly consistent it was. Day time vs Night Time Temperatures between 1 to 6 days (24 to 144 hours), I thought it would also be worth showing how each weather model performed at predicting temperatures during the day and at night. Average Mean Error For Daytime Temperature, 1st ICON 0.83 (Up to 120 hours 5 days) 2nd ECM 0.89 3rd Met Office 1.00 4th Weather Underground 1.03 5th BBC Weather 1.10 6th DarkSky 1.18 7th GEM 1.26 8th GFS 1.40 Average Mean Error For Nighttime Temperature, 1st ICON 0.98 2nd Met Office 1.18 3rd DarkSky 1.20 4th GEM 1.28 5th Weather Underground 1.31 6th GFS 1.34 7th BBC Weather 1.37 8th ECM 1.50 Notes: Almost all of the weather models performed more poorly at nighttime compared to the daytime and it's interesting to see how the ECM and BBC Weather done well during the day but are both the worst when it comes to predicting nighttime temperatures. Wind Speed Results, 24 Hours (1 day) Average Mean Error, 1st GFS 2.04 2nd DarkSky 2.29 3rd Met Office 2.56 4th BBC Weather 2.64 5th Weather Underground 2.76 6th The Weather Channel 2.79 7th ICON 3.25 8th GEM 3.47 9th ECM 3.59 10th AccuWeather 3.95 11th FMI 4.09 12th HIRLAM 4.13 13th ARPEGE 4.75 Notes: The ECM underestimated the wind speeds for the whole experiment. In my experience, I've seen it do the same thing for some other locations and when I redo this experiment using multiple locations it will be interesting to see if the ECM still underperforms when it comes to wind speeds. 48 Hours (2 days) Average Mean Error, 1st Met Office 2.15 2nd Weather Underground 2.50 3rd The Weather Channel 2.72 4th BBC Weather 2.83 5th GFS 2.97 6th DarkSky 3.02 7th ICON 3.29 8th ECM 3.43 9th GEM 4.04 10th HIRLAM 4.15 11th ARPEGE 4.38 12th FMI 4.43 13th AccuWeather 5.04 Notes: A few of the weather models seemed to handle the wind speeds poorly those being the HIRLAM, ARPEGE, FMI and the AccuWeather all of them tended to overestimate the wind speeds especially the ARPEGE weather model while the AccuWeather seemed to suffer from the same issue as the likes of the ECM and underestimated the wind speed. 72 Hours (3 days) Average Mean Error, 1st BBC Weather 2.86 2nd Met Office 2.88 3rd Weather Underground 3.00 4th DarkSky 3.25 5th GFS 3.38 6th ECM 3.75 7th ICON 3.77 8th GEM 4.61 9th ARPEGE 4.63 10th AccuWeather 5.44 96 Hours (4 days) Average Mean Error, 1st BBC Weather 2.92 2nd Met Office 3.36 3rd Weather Underground 4.47 4th ICON 4.72 5th ECM 4.76 6th DarkSky 4.86 7th GEM 5.43 8th GFS 5.95 Notes: The GFS has made a big jump back down to last place as many will know the GFS likes to overdo wind speeds while it seems to manage them well within the 1 to 3-day range it seems beyond that it tends to overestimate them more than other weather models. 120 Hours (5 days) Average Mean Error, 1st Weather Underground 4.07 2nd BBC Weather 4.16 3rd Met Office 4.31 4th ECM 4.75 5th DarkSky 4.97 6th GFS 5.40 7th GEM 5.81 8th ICON 6.27 144 Hours (6 days) Average Mean Error, 1st Weather Underground 4.16 2nd BBC Weather 4.50 3rd Met Office 5.26 4th DarkSky 5.42 5th ECM 5.64 6th GFS 5.90 7th GEM 7.18 Notes: GEM seemed to really struggle around this point. While the ECM may have been behind the GFS it has done a better job with the wind speeds at day 5 and 6 compared to the GFS. Mean Error Between 24 and 144 Hours (1 to 6 days) 1st BBC Weather 3.31 2nd Met Office 3.42 3rd Weather Underground 3.49 4th DarkSky 3.96 5th ICON 4.26 (up to 120 hours 5 days) 6th GFS 4.27 7th ECM 4.32 8th GEM 5.09 What about wind gusts? I decided to test that out as well however it was just on one occasion during the early part of January where a storm brought gusts of 70 to 90mph winds to the North of Scotland. 24 hours before I made a note of all the predicted gusts from some of the weather models and I did use up to four different locations all where a Met Office station was located the results are below, Air Pressure Results, 24 Hours (1 day) Average Mean Error, 1st ECM 0.34 2nd DarkSky 0.37 3rd BBC Weather 0.38 4th Met Office 0.50 5th ARPEGE 0.54 6th ICON 0.59 7th GEM 0.65 8th Weather Underground 0.66 9th GFS 0.70 10th HIRLAM 1.04 11th FMI 1.11 Notes: It's surprising to see the GFS so far down the list. As for the HIRLAM and FMI, they both proved to be very inconsistent even at just 24 hours. 48 Hours (2 days) Average Mean Error, 1st BBC Weather 0.70 2nd ICON 0.75 3rd ECM 0.77 4th Met Office 0.81 5th ARPEGE 0.84 6th DarkSky 0.90 7th Weather Underground 1.07 8th GFS 1.20 9th GEM 1.21 10th HIRLAM 1.29 11th FMI 2.52 72 Hours (3 days) Average Mean Error, 1st ICON 1.40 2nd Met Office 1.41 3rd DarkSky 1.50 4th ECM 1.68 5th Weather Underground 1.78 6th BBC Weather 1.90 7th GFS 1.91 8th GEM 1.97 9th ARPEGE 2.11 Notes: ARPEGE which came 5th for both day 1 and 2 now falls to the last place it seems to become more unreliable from day 3 onwards. 96 Hours (4 days) Average Mean Error, 1st ICON 2.22 2nd ECM 2.25 3rd Met Office 2.68 4th GEM 2.77 5th BBC Weather 2.90 6th Weather Underground 3.04 7th DarkSky 3.09 8th GFS 3.47 120 Hours (5 days) Average Mean Error, 1st DarkSky 3.84 2nd ICON 3.95 3rd Met Office 4.18 4th GFS 4.25 5th Weather Underground 4.42 6th ECM 4.75 7th GEM 4.77 8th BBC Weather 5.06 Notes: This may surprise some people but once we got to day 5 DarkSky didn't seem to struggle as much as any other weather model. 144 Hours (6 days) Average Mean Error, 1st DarkSky 4.97 2nd Weather Underground 5.09 3rd Met Office 5.31 4th GEM 5.32 5th GFS 5.50 6th BBC Weather 6.04 7th ECM 6.15 Mean Error Between 24 and 144 Hours (1 to 6 days), 1st ICON 1.78 (up to 120 hours 5 days) 2nd DarkSky 2.44 3rd Met Office 2.48 4th ECM 2.65 5th Weather Underground 2.67 6th GEM 2.78 7th BBC Weather 2.83 8th GFS 2.83 Notes: ICON and DarkSky were the top 2 best are predicting air pressure the Met Office and the ECM were not too far behind. Both the GFS and BBC Weather were the worst. Mean Error From Air Temperature, Wind Speed and Air Pressure 24 to 144 hours (1 to 6 days), Now that we have looked at each individual part of the experiment it's time to add everything together at each timeframe to find the best on average from the temperature, wind and pressure. The chart above shows the results for 24 hours (1 day) and below shows the list of results, 1-day forecast (24 hours), 1st DarkSky 1.13 2nd Met Office 1.19 3rd GFS 1.21 4th BBC Weather 1.22 5th Weather Underground 1.38 6th ICON 1.50 7th ECM 1.56 8th GEM 1.66 9th ARPEGE 2.07 10th FMI 2.27 11th HIRLAM 2.45 2 days forecast (48 hours), 1st Met Office 1.23 2nd DarkSky 1.33 3rd Weather Underground 1.47 4th BBC Weather 1.56 5th ICON 1.58 6th ECM 1.65 7th GFS 1.69 8th GEM 1.90 9th ARPEGE 2.10 10th HIRLAM 2.24 11th FMI 2.89 3-day forecast (72 hours), 1st Met Office 1.64 2nd BBC Weather 1.93 3rd DarkSky 1.95 4th ICON 1.98 5th Weather Underground 1.99 6th GFS 2.14 7th ECM 2.18 8th GEM 2.63 9th ARPEGE 2.70 4-day forecast (96 hours), 1st BBC Weather 2.34 2nd Met Office 2.37 3rd ICON 2.66 4th ECM 2.75 5th Weather Underground 2.93 6th DarkSky 3.08 7th GEM 3.14 8th GFS 3.64 5-day forecast (120 hours), 1st Met Office 3.31 2nd Underground 3.32 3rd DarkSky 3.43 4th BBC 3.61 5th ECM 3.67 6th ICON 3.83 7th GFS 3.88 8th GEM 4.00 6-day forecast (144 hours), 1st Weather Underground 3.58 2nd DarkSky 4.03 3rd BBC Weather 4.08 4th Met Office 4.11 5th GFS 4.39 6th ECM 4.51 7th GEM 4.73 The final overall set of results average mean between 24 and 144 hours (1 to 6 days) for Air temperature, wind speed and air pressure, 1st Met Office 2.30 2nd ICON 2.31 (up to 120 hours 5 days) 3rd Weather Underground 2.44 4th BBC Weather 2.45 5th DarkSky 2.49 6th ECM 2.72 7th GFS 2.82 8th GEM 3.01 Notes: The overall winner from this experiment is the UK Met Office it performed as the most consistent weather model with the least number of mistakes between temperature, wind speed and air pressure between 1 and 6 days over the period of 1 month. As to why the ECM, GFS and GEM are behind? This is because it seems the likes of the BBC Weather, DarkSky and the Weather Underground use more than weather model I explain a bit more in the summaries below and I believe this gives them a slight edge over some of the weather models. My thoughts on each weather model and website/app, GFS: Its one of the worst overall when it came to air temperatures it always had a slight bias towards warmer temperatures and the few cold nights that occurred during December the GFS failed miserably at predicting them even at just 12 hours before the GFS was up to 4°C on a few occasions off the actual recorded temperature. The only strong point that came from it was predicting wind speeds better than any other weather model at 24 hours ahead. The GFS was also one of the worst overall at air pressure this has been known for a number of years now with most low-pressure systems it likes to overdo them which was evident several times during this experiment. In the end it finishes in 7th place just ahead of the GEM model it may come to no real surprise to some people how badly the GFS performed overall and I can only hope that the new upgrade brings it better results. ECM: It was a little bit of a disappointment for me it's still my own personal favourite though. It was one of the best at forecasting temperatures and performed remarkably well with the daytime temperatures but appeared to really struggle with nighttime temperatures. It came 2nd last with the mean wind speeds but actually did a decent job when it came to predicting the wind gusts. The ECM showed it was one of the best at air pressure between day 1 and 4 but fell back a lot for day 5 and 6. Overall it finishes 6th out of 8th that will mainly be because of the wind speeds letting it down it did, however, it did do better than the GFS and GEM. I know the ECM vs GFS happens a lot at the moment the ECM is the overall better one but the gap between how well they perform is actually pretty small which would maybe explain why sometimes one of them does a better job than the other but then this is just all based off this one experiment so it's important not to read too much into it. UK Met Office: It did great with air temperatures both day and night. It came 2nd overall for the wind speeds and 3rd place for the air pressure and it was the overall winner. ICON: It was the best and by some margin for predicting air temperatures and also being the best for day & night time. It wasn't just the air temperatures it was the best at up to 120 hours with the air pressure as well. Coming 2nd overall I think may be a surprise to some people I really didn't expect it to do so well and does seem to be a weather model worth checking. GEM: One of the worst for temperatures although it did an okay job at night time temperatures. For wind speeds, it tended to overestimate them and was actually one of the worst overall for it. It wasn't the worst nor the best for air pressure and by the looks of it, the only thing GEM was a bit decent at. FMI: One of the worst performing weather models of the whole experiment it failed to do well in anything I can't really say anything positive about it. ARPEGE: It didn't do too well with the air temperatures or mean wind speeds but actually was one of the best with the air pressure especially up to 2 days ahead and it was the overall best at predicting the wind gusts. HIRLAM: Very similar to the FMI weather model it was just very inconsistent with everything. BBC Weather: I'm not sure what weather model they use it's very likely to be the ECM but the results from the experiment differ a bit the ECM was better at the temperatures and air pressure while the BBC Weather was the overall best with the mean wind speeds while the ECM was one of the worst which makes me think the BBC Weather use not only the ECM but also perhaps some other weather models as MeteoGroup say on their website "With over 100 meteorologists, a department dedicated to research and development and investment in the five weather models recognised as being the most accurate in the world: ECMWF, UKMO, GFS, HIRLAM and WRF, we are able to provide the most precise forecasts in the market." Might suggest they use more than just the ECM. In the end, I think the BBC Weather may have surprised some people coming 4th out of 8th overall. Weather Underground: With the weather models now out of the way it's time to start with the weather websites and apps something I added into this experiment out of interest to see how good they would be compared to the weather models I really didn't have much hope for them but they certainly did much better than I expected with the Weather Underground coming 3rd overall is pretty amazing. A bit more about how their forecasts work "Our ever-expanding network of 250,000+ personal weather stations is the largest of its kind and provides us with a unique ability to provide the most local forecasts based on actual weather data points. BestForecast™ uses the most innovative forecast models available and cross-verifies their output with all of the localized data points. Only our unrivaled amount of local neighborhood weather data can generate forecasts for your front door." Also uses different weather models "A variety of inputs, including, but not limited to, ECMWF, GFS, and NAM" DarkSky: Just like the Weather Underground it performed much better than I expected coming in 5th place. It done a great job at predicting the nighttime temperatures and was one of the best when it came to air pressure especially at day 5 and 6 where it was the overall best but I couldn't understand how a website/app could so so well so I did a bit of research and found this in an article from a few years ago it seems DarkSky uses a few weather models like the GFS, GEM and ICON, “We first correct for geographically related bias in each model individually (e.g., GFS temperature seems to be annoyingly low in some places, higher in others). And then we monitor the accuracy of each model, geographically, to calculate a standard error which we use to weight each model on-the-fly whenever a request comes in for a forecast." The Weather Channel: It did a decent enough job in its short range 2-day forecasts with temperatures and wind speeds it wasn't the best or the worst and did actually do better than some of the weather models. However, The Weather Channel uses the API data from the Weather Underground so they basically say the same thing it's just that The Weather Channel runs behind and updates later so your best just visiting Weather Underground itself to get the very latest forecast. Accuweather: Seems good enough for temperatures doing a fairly decent job but was bad for underpredicting the wind speeds. How I think the experiment went, I think it went fairly well all of the weather models were presented with a difficult challenge of forecasting the weather for just one very localised location. It threw up a few surprises with some of the weather websites/apps doing a better job than some of the weather models although it looks like we now know why that is. As I said at the start it's important not to take these results too seriously as they are just based on one location over a time period of a month and just looked at temperatures, wind speeds and air pressure. Once the new GFS model is out a new experiment will take place which will include more locations across the UK and I will try to add more weather models and websites/apps to it and I will try to include more things, for example, rainfall and cloud cover. I hope you enjoyed reading this rather long post but found it as enjoyable and fun as I did doing this thank you for reading.

Much colder today compared to yesterday a whole 5°C drop in temperature. Its been partly cloudy and sunny for most of the morning with some showers passing through quickly then back to sunshine. It was windy in the early part of this morning where a gust of 50mph was recorded. Further North in the Western Isles at Stornoway has had snow the picture below is from the Facebook page welovestornoway.com,

Much colder today compared to yesterday. Sunny spells with showers this morning. Temperature 5.8°C Wind speed 14mph gusting to 22mph highest gust so far today 50mph. Pressure 995mb Rising

The afternoon remained cloudy with some rain passing through from time to time gusts stayed strong over 40mph then it quickly dropped in speed in the later part of the evening. Temperature 8.1°C Wind speed 11mph gusting to 17mph highest gust today 47mph. Pressure 995mb Falling

Very mild temperatures today it's currently 10.5°C here this afternoon. It's another cloudy day we had rain all night that stopped this morning. It's a bit windy again with the highest gust being 45mph so far.

A cloudy morning with rain overnight and into the early part of this morning. It's dried up now but looks like more rain will arrive again later. It was a bit windy last night and this morning with gusts over 35mph the highest being 45mph. Temperature 10.4°C Wind speed 27mph gusting to 37mph highest gust so far today 45mph. Pressure 1005mb Falling

Afternoon remained cloudy with the sun trying to make the occasional appearance. This evening the wind has started to increase again. Temperature 9.6°C Wind speed 24mph gusting to 32mph highest today 37mph. Pressure 1010mb Falling

A cloudy morning here and feeling colder than yesterday. It's starting to brighten up at the moment some sunshine out to the West from the Atlantic moving in.

The heavy showers and strong winds eased off overnight. This morning has been cloudy so far and it looks like it's trying to clear up for the afternoon. Temperature 8.2°C Wind speed 12mph gusting to 20mph. Pressure 1017mb Steady

The afternoon saw further sunny spells with heavy showers passing through and gusty strong winds up to 40mph. In the last hour, the wind has started to ease off. Temperature 7.8°C Wind speed 22mph gusting to 36mph highest gust today 50mph. Pressure 1013mb Rising

A mostly cloudy start to the day with sunny spells along with a strong North Westerly wind and frequent gusts just over 40mph through last night and this morning with the highest gust reaching 50mph.

A mostly cloudy start with sunny spells and windy. Temperature 9.2°C Wind speed 25mph gusting to 42mph highest gust so far today 50mph. Pressure 1008mb Steady

A cloudy afternoon that remained mostly dry but still windy. Temperature 10.3°C Wind speed 30mph gusting to 40mph highest today 44mph. Pressure 1012mb Falling

A cloudy day so far here with rain and the usual windy weather is back with gusts just over 50mph yesterday evening and up to 42mph today.

A cloudy morning with rain 1.4mm of rainfall so far. Temperature 9.9°C Wind speed 21mph gusting to 30mph highest so far today 42mph. Pressure 1015mb Falling

The afternoon became more cloudy with some rain as well up to 1.6mm. During the evening the wind started to increase with gusts over 35mph the highest reaching 47mph. Temperature 8.6°C Wind speed 25mph gusting to 35mph highest today 47mph. Pressure 1018mb Falling

Most of the weather forecasts yesterday said cloudy for here today but so far it's been partly cloudy and sunny with a few showers about. Thought I would share some photos I took recently the fog photo is from Wednesday and the rest from yesterday.

Another day so far better than forecast its been partly cloudy and sunny all morning with a few showers about. Temperature 10.1°C Wind speed 7mph gusting to 12mph. Pressure 1028mb Falling

The afternoon was better than forecast partly cloudy and sunny for a while. Temperature 9.1°C Wind speed 8mph gusting to 23mph. Pressure 1032mb Rising

Mostly cloudy with short-lived sunny spells and showers. The fog from yesterday cleared away overnight.

Mostly cloudy with some short-lived sunny spells and showers this morning. Temperature 9.7°C Wind speed 10mph gusting to 21mph. Pressure 1031mb Steady

Foggy in the afternoon with some drizzle about. Temperature 9.7°C Wind speed 4mph gusting to 11mph. Pressure 1030mb Steady

Cloudy and misty with some drizzle here today.

A cloudy and misty start to the day with drizzle. Temperature 7.6°C Wind speed 12mph gusting to 17mph. Pressure 1031mb Falling