Short-term precipitation forecasting using Convolutional LSTM neural networks
--
Introduction and intuition
During the last 4 months, I have moved to Eindhoven, a small city located in The Netherlands, to pursue a Master’s degree in Data Science and Artificial Intelligence. But you know what seems very different in The Netherlands for a group of Mediterranean friends and fellow students ? The weather, and especially the frequency of rain and the low temperatures (below 0 Celsius for many days or even weeks in a row). So when the Winter season kicked-in, all of my friends and fellow students coming from countries of the Mediterranean were awaiting to snow every day, sending screenshots from their weather apps every morning in student group chats that only forecasted snow with a probability of 40% or 50%. Some friends were still arguing, that a snowstorm was close even when sun was apparent and bright the whole day and they were claiming so, just because the weather app was forecasting snow with a low probability for the next hour.
This is when I seriously started thinking about responding to them as a Data Scientist would do. With raw data and the conclusions deriving from data analysis. This is when I started looking into the open datasets and public APIs provided by the Royal Netherlands Meteorological Institute (also known as KNMI).
Data gathering
While browsing through datasets provided by KNMI to see how I could probably utilize them, I came across a dataset containing the composite reflectivity (counted at various angles) captured by 2 weather radars in The Netherlands, the first being located in Herwijnen and the second one in Den Helder. Trying to avoid being too technical regarding how weather radars work, let’s assume that weather radar generates a signal which is being reflected by precipitation (rain, snow, hail etc) when it bounce on it. The intensity of the reflected signal which is then captured by the radar is called reflectivity (counted in dBZ) and we could roughly claim that it is somewhat proportionate to the intensity of precipitation at that point. This reflectivity data, when converted into an image by mapping a color scale according to signal intensity, (by default the color scale provided by KNMI is ‘viridis’ with purple/dark blue for the lower…
