SlideShare a Scribd company logo
Open Data Distributed on Amazon’s Cloud Service
RoopeTervo, Mikko Rauhala, MikkoVisa
Finnish Meteorological Institute
Finnish Meteorological
Institute opened its data
19.11.2018 2
FMI Open Source Software
https://en.ilmatieteenlaitos.fi/open-data
FMI starts to open it’s
software
2013 2016
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
A role of meteorological institutes is changing
Challenges:
Ensure authoritative voice in warnings
More efficiency in development and operations
Ensure the impact of produced information
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
3
Ecosystem is changing
• Meteorological services will not control the whole value chain anymore
• Private sector is responsible increasingly larger share of the value chain
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
4
Charge for data
Charge for Service
Closed ecosystem
Open ecosystem
Streamline collaboration with partners
Collaboration is open and easy
• No long and burden negotiations – just evaluate, use, develop
• Using Open Source Software prevents from vendor locks
• Open data and open source software can boost the development
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
5
ApplicationsObservations
Collaboration and OSS Proprietary software
Ease public-private-partnership
• Open Source Software is a great tool to ease public-private
partnership
• Open data provides seamless access to weather and climate
information from multiple data providers
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
6
Data is valuable only
when it’s used
Maximal coverage requires
several different channels and
services
• One organization can’t handle them
all
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
7
Meteorological services are
successful when weather and
climate don’t cause
unanticipated unwanted impact
to society
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
8
Open data lower usability barrier and thus
increase data usage
Providing prober tools tp handle and analyze the data empowers 3rd
party users to correctly utilize it
• Proper tools ensures consistency between information regardless of
channel and service providers
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
9
Openness makes science repeatable
• Methods are repeatable when anyone can access the
tools
• Easy and open methods to access and analyze the data
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
10
• Finnish Meteorological Institute opened
its data in 2013.
• Basically everything that FMI has
property rights was opened.
• Both (near) real-time and historical and
climatological data.
• Data is provided in freely in machine
readable format.
19.11.2018 12
FMI Open Data
https://en.ilmatieteenlaitos.fi/open-data
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
FMI Open Data Portal follows INSPIRE requirements.
FMI Open Data
Data Portal
Meta data
Services
The very same data portal works as Open Data and INSPIRE
portal.
19.11.2018 13
ISO19115 WFS WMS
CSW
Grid Series
Observations
Time Series
Observations
Data
Models O&M
Simple
Feature
GRIB
NetCDF GeoTiff
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
FMI Open Data
Registration
Registration and API key is required to useView and Download
Services
o Working email address is the only mandatory information
With oneAPI key it’s allowed to
o do at most 20 000 requests per day to Download Service
o do at most 10 000 requests per day toView Service
o do at most 600 requests per 5 minutes to both services
19.11.2018 14Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
Registration and API key is required to useView and Download
Services
o Working email address is the only mandatory information
With oneAPI key it’s allowed to
o do at most 20 000 requests per day to Download Service
o do at most 10 000 requests per day toView Service
o do at most 600 requests per 5 minutes to both services
FMI Open Data
Registration
19.11.2018 15Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
And a little over
830 000 data
downloads
per day
(9,6 req/s)
At the moment
about 11 700 users
19.11.2018 16Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
Data is valuable only when it
reach relevant audience
Maximal coverage requires
several different channels and
services
 FMI joins Amazon’s Public Datasets
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
17
FMI OpenData on AWS
FMI OpenData is also distributed on
AmazonWeb Services (AWS) Cloud platform
• 2-years pilot
• Started in May 2017
• Hirlam surface and pressure levels in the first stage
• The objective is to
• increase the utility and effective use of weather and climate data
• support public-private-partnership
• Specially convenient for users who need the whole model data
• i.e for post-processing or generating map visualizations
• Licence: CC BY 4.0
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
18
FMI OpenData on AWS
• Content
• Hirlam Surface
• Hirlam Pressure Levels
• Coverage: Europe
• Grid resolution: 7,5 km
• Updates: 4 times a day
• Time range: 54 hours (from model run
start)
• Time step: 1 hour
• Archive kept during the pilot
• Parameters:
http://en.ilmatieteenlaitos.fi/open-data-
on-aws-s3
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
19
FMI OpenData on AWS
• Access through buckets:
• Surface data: fmi-opendata-rcrhirlam-surface-grib
• Pressure level data: fmi-opendata-rcrhirlam-pressure-grib
• Browse bucket content:
• http://fmi-opendata-rcrhirlam-surface-grib.s3-website-eu-west-1.amazonaws.com/
• http://fmi-opendata-rcrhirlam-pressure-grib.s3-website-eu-west-1.amazonaws.com/
• Public Amazon SNS topics are available for every new object added to the
Amazon S3:
• arn:aws:sns:eu-west-1:916174725480:new-fmi-opendata-rcrhirlam-surface-grib
• arn:aws:sns:eu-west-1:916174725480:new-fmi-opendata-rcrhirlam-pressure-grib
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
20
FMI OpenData on AWS
See documentation:
http://en.ilmatieteenlaitos.fi/open-data-on-aws-s3
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
21
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
22
41,554
8,516,465
10,893,353
5,317,307
POINT REQUESTS GRID REQUESTS
Simple feature Grid series observations
Point Time Series observations Grid
24 727 125
Most requests in FMI portal are point requests
All data 09/2018
32,118
3,048,071
2,329,841
1,037,145
POINT REQUESTS GRID REQUESTS
Simple feature Grid series observations
Point Time Series observations Grid
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
23
6 415 057
Most requests in FMI portal are point requests
Hirlam model 09/2018
AWS gets more popular
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
24
0
100000
200000
300000
400000
500000
600000
700000
800000
900000
HIRLAM DOWNLOADS FROM DIFFERENT CHANNELS
FMI AWS
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
25
0
10000
20000
30000
40000
50000
60000
70000
HIRLAM DOWNLOADS FROM DIFFERENT CHANNELS
FMI AWS Normalised
AWS gets more popular
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
26
0
10000
20000
30000
40000
50000
60000
70000
HIRLAM DOWNLOADS FROM DIFFERENT CHANNELS
FMI AWS Normalised Trendline of FMI Trendline of AWS
AWS gets more popular
Means roughly 300
users who fetch whole
data operatively
At the moment
54 000 (whole model)
downloads per month
from AWS
19.11.2018 27Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
Point requests are still
by far the most
popular type
For binary data
downloaders S3 is the
most popular channel
19.11.2018 28Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
Case Leanheat
Leanheat is a software solution for optimized heating in centrally heated
buildings. Leanheat generates a building specific heat demand forecast a few
days into the future, and optimally adjusts heating power at all times based on
the generated forecast model.
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
29
Case Leanheat
• Leanheat needs detailed weather forecasts for several areas in Europe
• FMI Hirlam weather models from S3 are used
• Processing done in AWS
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
30
You need to be open to have an
impact
Maximal coverage requires
several different channels and
services
19.11.2018
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
32
www.fmi.fi
https://github.com/fmidev
https://en.ilmatieteenlaitos.fi/open-data
http://en.ilmatieteenlaitos.fi/open-data-on-aws-s3
http://roopetervo.com
http://www.slideshare.net/tervo

More Related Content

PDF
ARIN 47: APNIC Update
PDF
Internet Measurement Networks - SANOG 24
PDF
APNIC Update for SANOG 24
PDF
The New HPC
PPTX
The IANA Stewardship Transition Overview & Background
PPTX
APNIC Update: Infotel 2014
PPTX
IOT, It's the Internet...only bigger
PDF
IPv6 Observatory outomes
ARIN 47: APNIC Update
Internet Measurement Networks - SANOG 24
APNIC Update for SANOG 24
The New HPC
The IANA Stewardship Transition Overview & Background
APNIC Update: Infotel 2014
IOT, It's the Internet...only bigger
IPv6 Observatory outomes

Similar to Fmi Open Data on S3 (20)

PPTX
Why we need open data? FMI Open Data on AWS
PPTX
FMI Open Data on S3
PPTX
Possibilities of Open Source Code
PPTX
20161028 strahlendorff fmi experience in openness
PPTX
FMI Open Data Interface and Usage
PPTX
Open Weather Data as Part of Big Data
PPTX
FMI Open Data Interface and Data Models
PPTX
Producing INSPIRE compliant datasets
PPTX
Producing INSPIRE Compliant Data Sets
PPTX
FMI Open Data on AWS Public dataset program
PDF
Available data sources & Real-time data collection
PDF
Kokemuksia tiedon avaamisesta, Tarja Riihisaari
PPTX
Meteorological and Aviation Weather Open Data implementation utilising OGC st...
PPTX
Open Data and and INSPIRE
PPTX
Open data & crowdsourcing of environmental observations in MMEA
PPTX
FMI Information Management System
PPTX
Inspire Compliant Weather Data
PPTX
Strahlendorff - EO and insitu for weather, water and climate
PDF
Open geodata in Finland
PDF
Environmental data at your fingertips - mmea.fi
Why we need open data? FMI Open Data on AWS
FMI Open Data on S3
Possibilities of Open Source Code
20161028 strahlendorff fmi experience in openness
FMI Open Data Interface and Usage
Open Weather Data as Part of Big Data
FMI Open Data Interface and Data Models
Producing INSPIRE compliant datasets
Producing INSPIRE Compliant Data Sets
FMI Open Data on AWS Public dataset program
Available data sources & Real-time data collection
Kokemuksia tiedon avaamisesta, Tarja Riihisaari
Meteorological and Aviation Weather Open Data implementation utilising OGC st...
Open Data and and INSPIRE
Open data & crowdsourcing of environmental observations in MMEA
FMI Information Management System
Inspire Compliant Weather Data
Strahlendorff - EO and insitu for weather, water and climate
Open geodata in Finland
Environmental data at your fingertips - mmea.fi
Ad

More from Roope Tervo (11)

PDF
FMI Open Data Impact Survey 2019
PDF
Predicting weather inflicted train delays
PPTX
Forecasting Electricity Outages Caused by Convective Storms
PPTX
SmartMet Server in INSPIRE
PPTX
SmartMet Server OSGeo
PPTX
WMTS Performance Tests
PDF
AvoinData-workshop käyttöesimerkki
PDF
AvoinData aineistot
PDF
AvoinData-workshop aikasarjat
PDF
Avoindata workshop tekninen_yleiskuvaus
PDF
Aaltoes opendata 20130206
FMI Open Data Impact Survey 2019
Predicting weather inflicted train delays
Forecasting Electricity Outages Caused by Convective Storms
SmartMet Server in INSPIRE
SmartMet Server OSGeo
WMTS Performance Tests
AvoinData-workshop käyttöesimerkki
AvoinData aineistot
AvoinData-workshop aikasarjat
Avoindata workshop tekninen_yleiskuvaus
Aaltoes opendata 20130206
Ad

Recently uploaded (20)

PDF
The European Green Deal (EU Green Deal)
DOCX
Double Membrane Roofs for Bio CNG Plants Stores biogas.docx
PDF
PAKAM TECHNOLOGY LIMTED PITCH DECK pptx.pdf
PPTX
EME Aerospace.pptx basics of mechanical engineering
PPTX
Soil chemistry lecture 20 presentations agriculture
PPT
MATERI - LABORATORY - SAFETY.ppt
PPTX
Afro-Asia Literature.pptxmmmmmmmmmmmmmmm
PPTX
he document discusses solid waste management. It defines different types of s...
DOCX
Double Membrane Roofs for Biogas Tanks Securely store produced biogas.docx
PPTX
Environmental pollutants for natural res
PPTX
Importance of good air quality and different pollutants.
PPTX
Microbial-Pathogens-and-Parasites-Their-Impact-on-Plant-Health.pptx
DOCX
Biogas Balloon for Bio CNG Plants An efficient solution for biogas storage..docx
PPTX
Biodiversity PPT by Gaithanlung Gonmei.pptx
PPTX
Plant Production 7.pptx in grade 7 students
PPTX
Lecture-05-Audio-lingual. Method & Appro
PPTX
Definition, Causes And Effects Of Greenhouse.pptx
PPTX
102602734019608717246081273460745534.pptx
DOCX
Double Membrane Roofs for Biogas Digesters A sealed cover for biogas producti...
PDF
Cave Diggers Simplified cave survey methods and mapping
The European Green Deal (EU Green Deal)
Double Membrane Roofs for Bio CNG Plants Stores biogas.docx
PAKAM TECHNOLOGY LIMTED PITCH DECK pptx.pdf
EME Aerospace.pptx basics of mechanical engineering
Soil chemistry lecture 20 presentations agriculture
MATERI - LABORATORY - SAFETY.ppt
Afro-Asia Literature.pptxmmmmmmmmmmmmmmm
he document discusses solid waste management. It defines different types of s...
Double Membrane Roofs for Biogas Tanks Securely store produced biogas.docx
Environmental pollutants for natural res
Importance of good air quality and different pollutants.
Microbial-Pathogens-and-Parasites-Their-Impact-on-Plant-Health.pptx
Biogas Balloon for Bio CNG Plants An efficient solution for biogas storage..docx
Biodiversity PPT by Gaithanlung Gonmei.pptx
Plant Production 7.pptx in grade 7 students
Lecture-05-Audio-lingual. Method & Appro
Definition, Causes And Effects Of Greenhouse.pptx
102602734019608717246081273460745534.pptx
Double Membrane Roofs for Biogas Digesters A sealed cover for biogas producti...
Cave Diggers Simplified cave survey methods and mapping

Fmi Open Data on S3

  • 1. Open Data Distributed on Amazon’s Cloud Service RoopeTervo, Mikko Rauhala, MikkoVisa Finnish Meteorological Institute
  • 2. Finnish Meteorological Institute opened its data 19.11.2018 2 FMI Open Source Software https://en.ilmatieteenlaitos.fi/open-data FMI starts to open it’s software 2013 2016 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
  • 3. A role of meteorological institutes is changing Challenges: Ensure authoritative voice in warnings More efficiency in development and operations Ensure the impact of produced information 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 3
  • 4. Ecosystem is changing • Meteorological services will not control the whole value chain anymore • Private sector is responsible increasingly larger share of the value chain 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 4 Charge for data Charge for Service Closed ecosystem Open ecosystem
  • 5. Streamline collaboration with partners Collaboration is open and easy • No long and burden negotiations – just evaluate, use, develop • Using Open Source Software prevents from vendor locks • Open data and open source software can boost the development 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 5 ApplicationsObservations Collaboration and OSS Proprietary software
  • 6. Ease public-private-partnership • Open Source Software is a great tool to ease public-private partnership • Open data provides seamless access to weather and climate information from multiple data providers 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 6
  • 7. Data is valuable only when it’s used Maximal coverage requires several different channels and services • One organization can’t handle them all 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 7
  • 8. Meteorological services are successful when weather and climate don’t cause unanticipated unwanted impact to society 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 8
  • 9. Open data lower usability barrier and thus increase data usage Providing prober tools tp handle and analyze the data empowers 3rd party users to correctly utilize it • Proper tools ensures consistency between information regardless of channel and service providers 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 9
  • 10. Openness makes science repeatable • Methods are repeatable when anyone can access the tools • Easy and open methods to access and analyze the data 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 10
  • 11. • Finnish Meteorological Institute opened its data in 2013. • Basically everything that FMI has property rights was opened. • Both (near) real-time and historical and climatological data. • Data is provided in freely in machine readable format. 19.11.2018 12 FMI Open Data https://en.ilmatieteenlaitos.fi/open-data Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
  • 12. FMI Open Data Portal follows INSPIRE requirements. FMI Open Data Data Portal Meta data Services The very same data portal works as Open Data and INSPIRE portal. 19.11.2018 13 ISO19115 WFS WMS CSW Grid Series Observations Time Series Observations Data Models O&M Simple Feature GRIB NetCDF GeoTiff Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
  • 13. FMI Open Data Registration Registration and API key is required to useView and Download Services o Working email address is the only mandatory information With oneAPI key it’s allowed to o do at most 20 000 requests per day to Download Service o do at most 10 000 requests per day toView Service o do at most 600 requests per 5 minutes to both services 19.11.2018 14Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
  • 14. Registration and API key is required to useView and Download Services o Working email address is the only mandatory information With oneAPI key it’s allowed to o do at most 20 000 requests per day to Download Service o do at most 10 000 requests per day toView Service o do at most 600 requests per 5 minutes to both services FMI Open Data Registration 19.11.2018 15Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
  • 15. And a little over 830 000 data downloads per day (9,6 req/s) At the moment about 11 700 users 19.11.2018 16Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
  • 16. Data is valuable only when it reach relevant audience Maximal coverage requires several different channels and services  FMI joins Amazon’s Public Datasets 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 17
  • 17. FMI OpenData on AWS FMI OpenData is also distributed on AmazonWeb Services (AWS) Cloud platform • 2-years pilot • Started in May 2017 • Hirlam surface and pressure levels in the first stage • The objective is to • increase the utility and effective use of weather and climate data • support public-private-partnership • Specially convenient for users who need the whole model data • i.e for post-processing or generating map visualizations • Licence: CC BY 4.0 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 18
  • 18. FMI OpenData on AWS • Content • Hirlam Surface • Hirlam Pressure Levels • Coverage: Europe • Grid resolution: 7,5 km • Updates: 4 times a day • Time range: 54 hours (from model run start) • Time step: 1 hour • Archive kept during the pilot • Parameters: http://en.ilmatieteenlaitos.fi/open-data- on-aws-s3 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 19
  • 19. FMI OpenData on AWS • Access through buckets: • Surface data: fmi-opendata-rcrhirlam-surface-grib • Pressure level data: fmi-opendata-rcrhirlam-pressure-grib • Browse bucket content: • http://fmi-opendata-rcrhirlam-surface-grib.s3-website-eu-west-1.amazonaws.com/ • http://fmi-opendata-rcrhirlam-pressure-grib.s3-website-eu-west-1.amazonaws.com/ • Public Amazon SNS topics are available for every new object added to the Amazon S3: • arn:aws:sns:eu-west-1:916174725480:new-fmi-opendata-rcrhirlam-surface-grib • arn:aws:sns:eu-west-1:916174725480:new-fmi-opendata-rcrhirlam-pressure-grib 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 20
  • 20. FMI OpenData on AWS See documentation: http://en.ilmatieteenlaitos.fi/open-data-on-aws-s3 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 21
  • 21. 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 22 41,554 8,516,465 10,893,353 5,317,307 POINT REQUESTS GRID REQUESTS Simple feature Grid series observations Point Time Series observations Grid 24 727 125 Most requests in FMI portal are point requests All data 09/2018
  • 22. 32,118 3,048,071 2,329,841 1,037,145 POINT REQUESTS GRID REQUESTS Simple feature Grid series observations Point Time Series observations Grid 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 23 6 415 057 Most requests in FMI portal are point requests Hirlam model 09/2018
  • 23. AWS gets more popular 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 24 0 100000 200000 300000 400000 500000 600000 700000 800000 900000 HIRLAM DOWNLOADS FROM DIFFERENT CHANNELS FMI AWS
  • 24. 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 25 0 10000 20000 30000 40000 50000 60000 70000 HIRLAM DOWNLOADS FROM DIFFERENT CHANNELS FMI AWS Normalised AWS gets more popular
  • 25. 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 26 0 10000 20000 30000 40000 50000 60000 70000 HIRLAM DOWNLOADS FROM DIFFERENT CHANNELS FMI AWS Normalised Trendline of FMI Trendline of AWS AWS gets more popular
  • 26. Means roughly 300 users who fetch whole data operatively At the moment 54 000 (whole model) downloads per month from AWS 19.11.2018 27Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
  • 27. Point requests are still by far the most popular type For binary data downloaders S3 is the most popular channel 19.11.2018 28Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa
  • 28. Case Leanheat Leanheat is a software solution for optimized heating in centrally heated buildings. Leanheat generates a building specific heat demand forecast a few days into the future, and optimally adjusts heating power at all times based on the generated forecast model. 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 29
  • 29. Case Leanheat • Leanheat needs detailed weather forecasts for several areas in Europe • FMI Hirlam weather models from S3 are used • Processing done in AWS 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 30
  • 30. You need to be open to have an impact Maximal coverage requires several different channels and services 19.11.2018 Open Source Software @ Finnish Meteorological Institute | Roope Tervo, Mikko Rauhala, Mikko Visa 32

Editor's Notes

  • #5: - Large players are large enough to operate without NMSs
  • #6: Using Open Source Software prevents from vendor locks Collaboration is open and easy No long and burden negotiations – just evaluate, use, develop If you need any changes or modifications, you can do them yourself or order them from 3rd party Even just finding users to the software boost development
  • #7: - Even just finding users to the software boost development
  • #9: Maximal coverage requires several different channels and services One organization can’t handle them all
  • #10: Open data helps, but.. Weather and climate data is complicated to handle Huge volumes, complex formats, complicated domain Providing tools to handle and analyze the data empowers 3rd party users to correctly utilize it Proper tools ensures consistency between information regardless of channel and service providers
  • #15: - Registration is not very convenient for users and is against open data principles but it provides us a useful information about the usage and makes it easier for us to prevent misusage of the portal
  • #16: - Registration is not very convenient for users and is against open data principles but it provides us a useful information about the usage and makes it easier for us to prevent misusage of the portal
  • #18: We have done well but this is not enough
  • #20: weather forecast model Covers europe Updates 4 times a day Two days ahead Archive is provided
  • #23: The pilot have been going on for 1 and half months now How well has it succeed? Before showing actual numbers I want to show some context This chart show how requests are divided between different types FMI provides the same data in different formats and data models Left column stands for different types of point requests and right (invisible) column stands for grid requests (in binary format) So: most of the requests are point data requests But there are still 40 thousand grid data requests from hundreds of different ip addresses during September
  • #24: The first plot was for all open data but in AWS we have only Hirlam available So, here’s the same data for Hirlam The pattern is the same
  • #25: This chart shows how Hirlam weather forecast data is divided between FMI data portal and S3 for roughly last year The blue line shows number of requests at FMI and yellow line is for requests at AWS Not that scale (and thus axis) is different But data in S3 is divided in a way that every parameter (like temperature or precipitation) is in separate files and from FMI data portal one can download everything with one request (17 for surface data and 8 for pressure level data) If we assume that every S3 user downloads everything, we can compare a popularity of these channels /shown in red column)
  • #26: This chart shows how Hirlam weather forecast data is divided between FMI data portal and S3 for roughly last year The blue line shows number of requests at FMI and yellow line is for requests at AWS Not that scale (and thus axis) is different But data in S3 is divided in a way that every parameter (like temperature or precipitation) is in separate files and from FMI data portal one can download everything with one request (17 for surface data and 8 for pressure level data) If we assume that every S3 user downloads everything, we can compare a popularity of these channels / shown in with green line
  • #27: And if we look trend (shown with dashed lines), we can see that AWS gets more popular and our users are actually moving from FMI portal to AWS
  • #28: Pressure levels 9900 req/day Would mean 309 operative users. Surface data 6000-46000 req/day Mean 15 000 req/day Would mean 220 operative users
  • #30: Leanheat is a software solution for optimized heating in centrally heated buildings. Traditionally, central heating is based purely on outdoor temperature. Leanheat replaces the traditional heating control method by generating a building specific heat demand forecast a few days into the future, and optimally adjusts heating power at all times based on the generated forecast model. As a part of a Leanheat installation, the apartments are equipped with wireless temperature sensors. The heat demand forecast is based on indoor temperature, and it automatically takes into account the thermodynamic properties of the building, temperature- and radiation forecasts as well as resident behavior. The goal of the heating power optimization is to maintain stable indoor conditions with the least amount of heating energy possible. Meanwhile, also peak power demand can be reduced, or the heating can be controlled based on e.g. a price signal from an energy provider.
  • #31: “The Amazon AWS S3 service is a convenient way to share large data sets. It allows Leanheat to get weather forecasts to customized areas and thus enables Leanheat to provide it's world class heating optimization solution anywhere in Europe. Leanheat also uses Amazon AWS servers in its optimization solution which further benefits the overall solution.”
  • #32: “The Amazon AWS S3 service is a convenient way to share large data sets. It allows Leanheat to get weather forecasts to customized areas and thus enables Leanheat to provide it's world class heating optimization solution anywhere in Europe. Leanheat also uses Amazon AWS servers in its optimization solution which further benefits the overall solution.”