FMI Information Management System
Download as PPTX, PDF0 likes104 views
The document provides information about the Finnish Meteorological Institute (FMI) including: - FMI has roughly 650 full-time employees split between research and operational services related to public safety, commercial, and other sectors. - FMI's software development team consists of 40-50 developers working across several units, with all new development done as open source. - In 2013, FMI began openly providing its data in machine-readable formats through an open data portal.
FMI Information Management System
- 1. FMI Information Management System RoopeTervo LeadArchitect Finnish Meteorological Institute
- 2. If not stated otherwise, omages by FMI. Licence CC4BY Finnish Meteorological Institute 2 • Roughly 650 FTE • Research Institute • Operative services • Safety / Public / B2B • Roughly half of the employees on operative side https://www.youtube.com/watch?v=WzIVAEvLH2E
- 3. If not stated otherwise, omages by FMI. Licence CC4BY Software Development at FMI • ~40 - 50 developers in 5-7 units • + contractors • In principle, all new development as open source • Many projects in collaboration (i.e. MetCoOp, Namcon, WFS3…) 3
- 4. If not stated otherwise, omages by FMI. Licence CC4BY FMI Open Data • Finnish Meteorological Institute opened its data in 2013 • Basically everything that FMI has property rights was opened • Data is provided in freely in machine readable format • https://en.ilmatieteenlaitos.fi/open-data 4
- 5. If not stated otherwise, omages by FMI. Licence CC4BY 5
- 6. If not stated otherwise, omages by FMI. Licence CC4BY Distribution Product Human intervention Post processing Raw data SATELLITES • Polar • Geostationary RADAR NETWORK • National • Baltrad (Norhern Europe) • Opera (Europe) NUMERICAL MODELS • Atmospheric / Ocean / Road • Local 2.5 km / Global 25 km • Ensembles OBSERVATIONS • Global Network • Surface / Balloons / Airplanes LIGHTNNING DETECTION NETWORK • Northern Europe CROWDSOURCING • Social sensor networks • Smartphones • 112 call data KALMAN MOS LAPS KRIGING WWW MOBILE TV NEWSPAPERS B2B OPENDATA SAFETY VISUALIZA TION MODEL SELECTIO N POST PROCESSI NG FIELD MODIFICA TION QUALITY CONTROL DATASERVERS WFS JSON XML MAPSERVERS WMS WCS APPLICATIONS ILMANET B2B MOBILE 6
- 7. If not stated otherwise, omages by FMI. Licence CC4BY 7 Observations FMI operated -- operational Automatic surface stations 184 Rainfall measurement sites 91 Sounding stations 3 Air quality stations 28 Buoys 16 Antennas for lightning detection 8 Mareographs 14 Solar 13 Total 481 Automation 95%
- 8. If not stated otherwise, omages by FMI. Licence CC4BY 8 External air quality 83 Skandinavian lightning detection 29 Baltic mareographs 24 Road weather stations 29 Runway weather stations 63 Foreign surface weather stations 13736 External water measurement stations 63 Road weather stations 629 NetAtmo ~100k Observations External
- 9. If not stated otherwise, omages by FMI. Licence CC4BY 9 Observations Process 1 / 10 minute aggregation LAN / 3G Oracle
- 10. If not stated otherwise, omages by FMI. Licence CC4BY Radars 10 • 10 radars • Range 120-250 km • Typical file formats HDF5, GeoTiff, PNG • Sending power 250 000 W • Receiving power 0,00000000001 – 0,00000001 W
- 11. If not stated otherwise, omages by FMI. Licence CC4BY Radars Products • Single radars • Reflectivity • Cappi (‘rain intensity’) • Doppler speed (‘wind’) • Echotop (‘cloud base’) • Echotop (‘thunder risk’) • Composite • Reflectivity • Rain intensity • Rain accumulation (1h, 3h, 6h…) • Cross sections 11
- 12. If not stated otherwise, omages by FMI. Licence CC4BY Satellites Earth Observation • In weather, used to observe: • cloud, pressure, wind, temperature... • Operated by EUMETSAT and NOAA • Reception, processing and filing of data • Geostationary satellites • Received in Germany • Delivered to Finland via telecommunication satellite and land link • Polar satellites • Received in Helsinki and Sodankylä 12
- 13. If not stated otherwise, omages by FMI. Licence CC4BY • Multicast technology (DVB-S2) • EUMETCast Europe, data rate 46.215 GB/hour, 1109.16 GB/day Satellites Data Transfer 13 Image: FMI
- 14. If not stated otherwise, omages by FMI. Licence CC4BY Lightning Detection • 9 sensor in Finland, 34 in NORDLIS • Practically all ground strikes detected with < 1km accuracy • Also part of cloud strikes • Stored to climate database • Can be delivered to clients in approximately 20 seconds 14
- 15. If not stated otherwise, omages by FMI. Licence CC4BY Numerical Weather Prediction (NWP) • Physical modelling of atmospheric • Models developed in international consortiums • Typical programming languages: Fortran, C++ • Input: observations, boundaries from other model, satellites • Output: grid from several surfaces • Surface, pressure levels, model levels • Typical data formats: • GRIB1, GRIB2 (and NetCDF in marine models) 15
- 16. If not stated otherwise, omages by FMI. Licence CC4BY Weather forecasts are calculated to a regular grid 16 Products (like at https://ilmatieteenlaitos.fi) are often interpolations between grid points
- 17. If not stated otherwise, omages by FMI. Licence CC4BY Numerical Weather Prediction (NWP) ECMWF European Centre for Medium-Range Weather Forecasts (ECMWF) to create products that support weather service ECMWF model gridded data in Meteorological Archival and Retrieval System (MARS). It is the repository of meteorological data at ECMWF (storage size of PBs of data) 17 Model ECMWF ECMWF EPS Spatial Resolution 15km 15 km Coverage Global Global Temporal Resolution 1-3h 1-3h Time range 10 days 10 days Output data Surface, pressure levels Surface 51 members Update interval 6h 6h
- 18. If not stated otherwise, omages by FMI. Licence CC4BY Numerical Weather Prediction (NWP) 18 Model Hirlam Harmonie Spatial Resolution 7,5 km 2,5 km Coverage Europe Skandinavia Temporal Resolution 1h 15 min Time range 48h 48h Update interval 6h 6h Output data Surface, pressure levels, model levels Surface, pressure levels, model levels
- 19. If not stated otherwise, omages by FMI. Licence CC4BY Human Intervention • Meteorologists selects the best source model and refines forecast • The forecast is updated as often as necessary • Output is used as primary data for Scandinavian area • Software ‘SmartMet’ is done at FMI (C++) • Meteorologists draws an analysis maps, fronts, jets, etc… • Software ‘Mirwa’ is done at FMI (Java) • The data is handled as objects 19
- 20. If not stated otherwise, omages by FMI. Licence CC4BY Human Intervention • Meteorologists generates warnings • Software ‘SmartMet Alert’ is done at FMI (Java) • Meteorologists does aviation forecasts • METARS, TAFs, SWC charts… • Software ‘SmartMet Aviation’ is done at FMI (Java) • Several other tasks • Writing texts, generating ice advisories, consulting about weather • Several web based tools 20
- 21. If not stated otherwise, omages by FMI. Licence CC4BY Storing Data • FMI total capacity: 6 PB • Ceph (Sodankylä satellite center): 1.5 PB • Ceph (Main site): 2.5 PB coming 2020 • Fast file system: 300 TB (Sodankylä) + 300 TB (main site) • Archive: 1.5 PB (disk) + 2 PB (tape) • Operational databases: Oracle, PostgreSQL, MySQL and Redis • FMI utilizes ECMWF data storage in daily operations • ECMWF MARS • Archive capacity: 225 PB • Current daily increase: 225 TB • Data size grows exponentially in near future 21 71.7 287 929 0 100 200 300 400 500 600 700 800 900 1000 2017 2020 2025 Global Numerical Weather Model Output Model output per day (TiB)
- 22. AI inWeather Forecasting Examples of Operational Systems and Possibilities 22
- 23. If not stated otherwise, omages by FMI. Licence CC4BY ML can be used in many parts of the process • Making observations (machine vision) • Quality Check (unsupervised learning) • Radar and satellite data (machine vision) • Weather predictions (i.a. neural networks) • Post-prosessing (traditional supervised learning, neural networks) • Impact analysis 23
- 24. If not stated otherwise, omages by FMI. Licence CC4BY MachineVision in Observations 24 https://www.visivis.at/
- 25. If not stated otherwise, omages by FMI. Licence CC4BY Precipitation NowCast (Radar) https://arxiv.org/abs/1912.12132 25 2 years history of radar images 256km U-Net Image: Mehrdad Yazdani, Wikipedia, Licence CC BY-SA 4.0 Prediction (i.e. t +2h)
- 26. If not stated otherwise, omages by FMI. Licence CC4BY Global SyntheticWeather Radar Poster: https://www.star.nesdis.noaa.gov/star/documents/meetings/2019AI/ posters/P2.3_Mattioli.pdf 26 Convolutional Neural Network to create a global synthetic weather radar based on satellites, lightning data and numerical weather prediction data. Image:Mattioli presentation
- 27. If not stated otherwise, omages by FMI. Licence CC4BY Level 1 • Post-processing • Downscaling Level 2 • Emulate existing parametrization scheme • Correct existing schemes Level 3 • Learn new parametrization scheme from simulations or data Level 4 • Model replacement Weather Models and Post-processing 27 Credits: Stephan Rasp, slides
- 28. If not stated otherwise, omages by FMI. Licence CC4BY Post-processing with Blend
- 29. If not stated otherwise, omages by FMI. Licence CC4BY Method used: “Exponentially weighted moving average” / “Moving average” (Cui et al. 2012) BCt = (1-α)BCt-1 + α(FCSTt-1 – OBSt-1) MAEt = (1-α)MAEt-1 + α|FCSTt-1 – OBSt-1| BC = bias correction, α = decaying weight (0.05), OBS = observation, FCST = model forecast, MAE = mean absolute error BC and MAE are calculated for each forecast model, analysis time and lead time separately Bias corrected forecasts are used to calculate MAE values and calculating the latest Blend forecast Bias correction (BC) and MAE
- 30. If not stated otherwise, omages by FMI. Licence CC4BY Biascorrection How large effect will the error have and for how long with different alpha values: equation: BCt = (1-α)BCt-1 (started with BC value 1) Decaying weight, alpha
- 31. If not stated otherwise, omages by FMI. Licence CC4BY Verification results based on MAE are used to produce weights for each model (Woodcock and Engel, 2005): W1 = (1/MAE1) / (1/MAE1+1/MAE2+...+1/MAEn) WhereW = weight, n= total number of models The Blend is calculated using weights and bias corrected model forecasts: Blend =W1*BCFCST1 +W2*BCFCST2+...+Wn*BCFCSTn Where BCFCST = Latest Bias corrected model forecast Calculating weights and Blend
- 32. If not stated otherwise, omages by FMI. Licence CC4BY Neural networks for post-processing ensemble weather forecasts Image: ECMWF Image: Cecbur, source: Wikipedia., License: CC BY-SA 4.0 Loss: where Φ and φ denote CDF and PDF of a standardGaussian distribution Features (X): • Several weather parameters • Embedded station location Label (y): observations Ouput: normal distribution mean μ and standard deviation σ
- 33. If not stated otherwise, omages by FMI. Licence CC4BY PredictingWeather Using Neural Networks is Under Research Jonathan Weyn: Can Machines Learn to Predict Weather? Using Deep Learning to Predict Gridded 500‐hPa Geopotential Height From Historical Weather Data: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019MS001705 https://github.com/jweyn/DLWP Peter Düben: Deep learning for weather and climate (excellent presentation about the topic): https://ossg.bcs.org/wp-content/uploads/04-19-dueben-weather.pdf Challenges and design choices for global weather and climate models based on machine learning: https://www.researchgate.net/publication/326043857_Challenges_and_de sign_choices_for_global_weather_and_climate_models_based_on_machi ne_learning Image: Mehrdad Yazdani, Wikipedia, Licence CC BY-SA 4.0
- 34. TRAINS Predicting Train Delays Inflicted by Weather Photo by Kalevi Lehtonen 1955. Not published until Commons in 2014. https://fi.wikipedia.org/wiki/Tiedosto:Finnish_class_Dm4_locomotive_number_1607_in_the_year_1955.jpg
- 35. If not stated otherwise, omages by FMI. Licence CC4BY Predicting Weather Inflicted Train Delays Data Liikennevirasto (CC4) Delay between stations • Passenger trains • 514 stations Weather observations • 19 parameters
- 36. If not stated otherwise, omages by FMI. Licence CC4BY Image: Venkata Jagannath. License: CC4-BY Image: CC0 Image: BiObserver. License: CC4-BY Results RFR RMSE: 5.37 MAE: 3.21 BSS: 0.11 LSTM RMSE: 4.35 MAE: 2.75 BSS: 0.01 LR RMSE: 5.59 MAE: 3.11 BSS: 0.08 𝐵𝑆𝑆 = 1 − 𝑅𝑀𝑆𝐸 𝑅𝑀𝑆𝐸𝑟𝑒𝑓 , where 𝑅𝑀𝑆𝐸𝑟𝑒𝑓 denotes root mean square error calculated with a mean value over the whole dataset
- 37. If not stated otherwise, omages by FMI. Licence CC4BY LSTM shows no real skill 0 25 50 Delay(minutes) Time 02/2011 Time 06/2016 Time 02/2017 Predicted vs. true delay, case Ahvenus Predicted delay True delay
- 38. If not stated otherwise, omages by FMI. Licence CC4BY RFR works relatively well 0 50 100 Delay(minutes) Time 02/2011 Time 06/2016 Time 02/2017 Predicted vs. true delay, average over all stations Predicted delay True delay
- 39. SASSE Forecasting Electricity Outages Caused by Convective Storms Photo by Bryan Alexander. CC BY 2.0
- 40. If not stated otherwise, omages by FMI. Licence CC4BY • Trained with observed outages (Random Forest Classifier) • Several input features • Few years of old data required • Based on storm cell severity and knowledge about clients electricity grid • For example category 2 storm cell is expected to cause problems in 10-50 % of grid nodes Turn outages to money Power companies has to give compensation for outages Forecasting Electricity Outages Extract storm cells from radar data Classify the storm cells Predict future outages Turn outages to money
- 42. If not stated otherwise, omages by FMI. Licence CC4BY Most Important DataTypes Data Data Type Software Weather forecasts GRIB1, GRIB2, (NetCDF) PanoPly, IDV, ECMWF GRIB tools, WGRIB, WGRIB2, gdal, SmartMet Server, pangeo Marine forecasts, air quality forecasts… NetCDF PanoPly, IDV, gdal, pangeo, GeoServer Observations XML, JSON, csv any BUFR ecCodes, NCEP Aviation products TAF, METAR, SIGMET… any IWXXM fmi-avi-message-converter-iwxxm Radar GeoTiff, HDF5 gdal, GeoServer (see also: https://en.wikipedia.org/wiki/Hiera rchical_Data_Format) Satellites GeoTiff, NetCDF gdal, GeoServer, PyTroll 42