Combining Various Data Sources
0 likes495 views
The document discusses the integration of open and closed data sources to analyze the correlation between weather conditions and the prevalence of net blotch, a plant disease affecting barley. It highlights data collection methods from the Finnish Meteorological Institute and the Natural Resources Institute Finland, as well as challenges in data usability. The analysis aims to improve predictions regarding net blotch occurrences to optimize chemical treatments, benefiting agricultural practices.
![DATA ANALYSIS
• Example, Ruukki
• Rainfall per day
• No net blotch
A little amount of
net blotch
22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä
11
0 20 40 60 80 100 120 140
0
5
10
15
20
25
30
35
Ruukki, no net blotch, 2011
Time [day]
Rainfallperday
0 20 40 60 80 100 120 140
0
5
10
15
20
25
30
35
Ruukki, a little amount of net blotch, 2014
Time [day]
Rainfallperday](https://image.slidesharecdn.com/mayracombiningdatasources20150923-151015072016-lva1-app6891/85/Combining-Various-Data-Sources-11-320.jpg)
![DATA ANALYSIS
22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä
12
• Feature; cumulative sum of the rainfall per day
• Green = years with no net blotch, red = years with net blotch
• Feature generation
• Combining variables with different mathematical operations
• Ranking the generated features and selecting the suitable one
1 2 3 4 5 6 7 8 9
0
50
100
150
200
250
300
Ruukki
Time [day]
Rainfallperday,cumulativesum](https://image.slidesharecdn.com/mayracombiningdatasources20150923-151015072016-lva1-app6891/85/Combining-Various-Data-Sources-12-320.jpg)
![1 2 3 4 5 6 7 8 9
0
50
100
150
200
Ruukki
Time [day]
Cumulativesumofgeneratedfeature
DATA ANALYSIS
• Feature; (ln(z)+ln(x))*ln(z);
• z = minimum daily temperature
• x = rainfall per day
• Used algorithm from Ruusunen M.: Signal Correlations in Biomass Combustion - an
Information Theoretic Analysis. Acta Univ Oulu C 459, 2013. PhD Thesis
22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä
13](https://image.slidesharecdn.com/mayracombiningdatasources20150923-151015072016-lva1-app6891/85/Combining-Various-Data-Sources-13-320.jpg)
Combining Various Data Sources
- 1. COMBINING VARIOUS DATA SOURCES Case: The weather and net blotch 1 FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 22.9.2015
- 2. INTRODUCTION • Combining the open and closed data • Challenges • Case from agriculture • Correlation between the weather conditions and existing of net blotch, one plant disease (barley) • Data collection and analysis • Results • Potential usage of the results 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 2
- 3. DATA, WHERE? 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 3 • A lot of data collected every day • Weather conditions • Trafic • Consumer habits • Accidents • … • Measurements from industry, agriculture, health care and so on • Availability of data differs • Open and close data
- 4. CASE; NET BLOTCH AND WEATHER • Aim to find out weather conditions which predicts the appearance of the net blotch • Finnish Meteorological Institute, open data • Data collection • Natural Resources Institute Finland, measurements, close data • Data ready to use in xls-format 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 4 Data from Natural Resources Institute Finland Data analysis Data from Finnish Meteorological Institute New information
- 5. NATURAL RESOURCES INSTITUTE FINLAND, NET BLOTCH DATA • Measurements from different places in different years • Information: • No net blotch = 0 • A lot of net blotch = 5 • The rest = something between 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 5 1980 2000 2020 0 1 2 Jokioinen 1990 2000 2010 0 2 4 Mikkeli 1980 2000 2020 0 2 4 Ruukki 1980 2000 2020 0 2 4 Ylistaro
- 6. FMI, OPEN DATA • Data available for everyone • Requires registration • User can select the measurement station and listed variables • Some challenges with usability • Queries in the internet, data in xml-format • Data includes mixed up letters and numbers • Time limit: Data can be loaded for only one week maximum • Vs. the earliest net blotch data is from 1991 • Maybe easier to use with R-code? 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 6
- 7. EXAMPLE, FMI DATA, ONE QUERY 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 7
- 8. EXAMPLE, FMI DATA OPENED IN EXCEL 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 8
- 9. FMI DOWNLOADER • Simple user interface for downloading FMI data • Available from internet: • http://tumetsu.github.io/FMI-weather-downloader • Developed by Tuomas Salmi • Salmi.tuomas@gmail.com • Fast and easy to use • Data in csv-format => usable • Can’t load every variables • A place and date, rainfall per day, temperature; minimum, maximum and average per day 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 9
- 10. DATA ANALYSIS 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 10 • The weather data were groupped according the place and net blotch • Ruukki: • First group – no net blotch • Second group – a small amount of net blotch • Third group – a big amount of net blotch • Data timing according to growing season => The first point of the weather data is the beginning of the growing season – in every data set • Analysis of the variables from the weather data: What is different between the years groupped according to net blotch
- 11. DATA ANALYSIS • Example, Ruukki • Rainfall per day • No net blotch A little amount of net blotch 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 11 0 20 40 60 80 100 120 140 0 5 10 15 20 25 30 35 Ruukki, no net blotch, 2011 Time [day] Rainfallperday 0 20 40 60 80 100 120 140 0 5 10 15 20 25 30 35 Ruukki, a little amount of net blotch, 2014 Time [day] Rainfallperday
- 12. DATA ANALYSIS 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 12 • Feature; cumulative sum of the rainfall per day • Green = years with no net blotch, red = years with net blotch • Feature generation • Combining variables with different mathematical operations • Ranking the generated features and selecting the suitable one 1 2 3 4 5 6 7 8 9 0 50 100 150 200 250 300 Ruukki Time [day] Rainfallperday,cumulativesum
- 13. 1 2 3 4 5 6 7 8 9 0 50 100 150 200 Ruukki Time [day] Cumulativesumofgeneratedfeature DATA ANALYSIS • Feature; (ln(z)+ln(x))*ln(z); • z = minimum daily temperature • x = rainfall per day • Used algorithm from Ruusunen M.: Signal Correlations in Biomass Combustion - an Information Theoretic Analysis. Acta Univ Oulu C 459, 2013. PhD Thesis 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 13
- 14. THE USE OF RESULTS • Predict the appearance of the net blotch • Optimize the use of chemical plant protectants • Avoid the unnecessary spraying of the fields • Positive effects to crop • Less chemical => save money and nature 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 14
- 15. THANK YOU! 22.9.2015FACULTY OF TECHNOLOGY/ Control Engineering/ Outi Mäyrä 15 • Picture from Bachelor’s thesis of Ville Ruohonen (2015) Disease resistance of the new spring cereal varieties