Meteo I/O Introduction
0 likes712 views
MeteoIO is a meteorological data input/output library that aims to remove IO functions from physics engines, make IO easy to use and robust, and simplify data preparation. It supports reading data from various sources into standardized data structures and writing output. Plugins allow adding new data sources. It performs operations like data filtering, unit conversions, and spatial and temporal interpolation to produce complete meteorological datasets.
![MeteoIO: ini files
[Parameters]
DEMSRC = ARC
DEMFILE = ../input/surface-grids/zwischberg.dem
LANDUSESRC = GRASS
LANDUSEFILE = ../input/surface-grids/zwischberg.lus plugins
METEOSRC = A3D
METEOPATH = ../input/meteo
SPECIALPTSSRC = A3D
SPECIALPTSFILE = ../input/surface-grids/zwischberg.pts
OUTPUT = ARC
OUTPATH = ../output/
4](https://image.slidesharecdn.com/meteoiointroduction-091211031540-phpapp02/85/Meteo-I-O-Introduction-4-320.jpg)
![Example: output
[i] meteoio/A3DIO.cc:86: Loading dynamic plugins:
[i] Number of 2D meteo stations: 15
---------- Station: 1 / 16
Name:
Air Temperature: 266.25 This station has no name
---------- Station: 2 / 16
Name: ANET-ZER
Air Temperature: 266.45
---------- Station: 3 / 16
Name: ANET-VIS
Air Temperature: 268.75
(etc)
8](https://image.slidesharecdn.com/meteoiointroduction-091211031540-phpapp02/85/Meteo-I-O-Introduction-8-320.jpg)
Meteo I/O Introduction
- 1. MeteoIO A meteo data I/O library
- 2. Before MeteoIO... • Each application doing its IO – Insufficient robustness – IO functions spread out all over the code – Input data very strictly formatted, several steps in preparing the data – Hard to adapt to new protocols/formats →IO a source of weakness... 2
- 3. MeteoIO: goals • Remove all IO functions out of the physics engine • Make IO easy to use • Make IO robust • Make IO flexible • Greatly simplify data preparation • Allow unmonitored, automatic operation • Allow end user to fine tune his IO... 3
- 4. MeteoIO: ini files [Parameters] DEMSRC = ARC DEMFILE = ../input/surface-grids/zwischberg.dem LANDUSESRC = GRASS LANDUSEFILE = ../input/surface-grids/zwischberg.lus plugins METEOSRC = A3D METEOPATH = ../input/meteo SPECIALPTSSRC = A3D SPECIALPTSFILE = ../input/surface-grids/zwischberg.pts OUTPUT = ARC OUTPATH = ../output/ 4
- 5. MeteoIO: primary data structures • MeteoData: • StationData: – Date object – Longitude – Ta – Latitude – Iswr – Altitude – Vw – Northing (metric grid) – Rh – Easting (metric grid) – Lwr – Name – Nswc – Supports equality comparison – Ts0 – P – ... – Supports equality comparison 5
- 6. MeteoIO: example of use 6
- 7. What is it doing? • Parsing io.ini • Loading plugins supporting requested data sources • Reading data (across files, network, protocols and supporting comments, different EOL, etc) • Converts stations' coordinates if necessary • Filtering data, marking invalid data as “no data” • Conversion of units to SI 7
- 8. Example: output [i] meteoio/A3DIO.cc:86: Loading dynamic plugins: [i] Number of 2D meteo stations: 15 ---------- Station: 1 / 16 Name: Air Temperature: 266.25 This station has no name ---------- Station: 2 / 16 Name: ANET-ZER Air Temperature: 266.45 ---------- Station: 3 / 16 Name: ANET-VIS Air Temperature: 268.75 (etc) 8
- 9. Data sources • Currently supported: – Meteo data: Alpine3D's Ascii, Boschung's XML, Gsn, Imis Oracle database, GeoTop – Grids: ARC, GRASS 9
- 10. Writing a new IO plugin • Provide implementation of some or all of: ReadMeteoData (const Date_IO& dateStart, const Date_IO& dateEnd, std::vector< std::vector<MeteoData> >& vecMeteo, std::vector< std::vector<StationData> >& vecStation, const unsigned int& stationindex=IOUtils::npos) Read2DGrid (Grid2DObject& grid_out, const string& parameter="") ReadDEM (DEMObject& dem_out) ReadLanduse (Grid2DObject& landuse_out) ReadAssimilationData (const Date_IO& date_in, Grid2DObject& da_out) ReadSpecialPoints (CSpecialPTSArray& pts) Write2DGrid (const Grid2DObject& grid_in, const string& name="") 10
- 11. Filters • Available: – Min, max – Max observed changed – No observed change • To be added: – Median Average Deviation • Special processing – Resampling – accumulation • Easy to add new filters 11
- 12. MeteoIO: example of use II 12
- 13. What does it change? • IO are now buffered -> efficient on slow access • Automatic resampling for missing data – Using linear interpolation between two neighbors – To be added: cyclic interpolation, etc 13
- 14. Userland addition In the user's code: add a function checking that the input requirements are met (ie: nb of lwr, nb of ta, etc) 14
- 15. Data structures/methods • Array, Array2D, Array3D – Using vectors for a safe and efficient implementation • Grid2DObject, Grid3DObject – Geolocalized array • DEMObject – Computes slope, aspect, normal's components (Corripio, Hicks, etc) 15
- 16. MeteoIO: example of use III 16
- 17. MeteoIO: example of use IV 17
- 18. TA, 0 to 16° RH, 62% to 90% Gotthard, 20081001 12:00, 19 stations 18
- 19. Spatial interpolations • Available methods: – Std pressure – Constant value – Constant with lapse rate – IDW – IDW with lapse rate – For Rh, Td IDWK with lapse rate and back to Rh – Terrain influenced VW and DW • One method defined for only 1 data input, one for multiple data input (fallback) • Regression: – Linear regression for the moment, removes the worst data point if the regression is too poor 19
- 20. Spatial interpolations 2 Planned additions: – Physical wind interpolation (quick Bernoulli resolution?) – More regression models (dynamically adjusted to the data) – Smarter choice of which method to use (dynamic, including data quality) 20
- 21. Geographic projections • Conversion to/from WGS84 from any coordinate system – Through own implementation (CH1903, ...) – Through proj4 library • Conversion to/from local grid MapProj local_proj(lat_ref, long_ref); local_proj.convert_from_WGS84(latitude, longitude, easting, northing); 21
- 22. MeteoIO: on the side • Date handling (Date_IO class) • Config files handling (ConfigReader class) • Data storage (Arrays & grids classes, sparse matrix (coming)) • Ported to Pop-C++ for parallel execution • Application specific IOs classes 22
- 23. Thank you! Mathias Bavay (SLF) Thomas Egger (Egger Consulting) Laurent Winkler (HES-SO) Florian Hof (SLF) Moustapha Mbengue (SLF) Gael Rosset (HES-SO) Julien Jeanneret (HES-SO) Nora Helbig (SLF) 23