Application of Distributed processing and Big data in agricultural DSS
- 1. ANUSHA M B 1ST SEM, CNE RVCE
- 2. AGENDA INTRODUCTION TOPIC OVERVIEW LITERATURE SURVEY WORKING CONCLUSION
- 3. INTRODUCTION One of the main challenges in agriculture is to sustainably meet the demand for food while preserving natural resources for future productions. Information Technology can assist producers to make better decisions by providing them with data and tools that enhance decision-making process, consequently allowing better management of the natural resources.
- 4. Topic Overview • Decision Support System has been applied to solve variety of agricultural problems. • It provides the framework that allows both the decision-makers and farmers to make good decisions. • Decision support systems (DSS) use databases, human–machine to combine a large number of models to realize scientific decision- making. • The environmental data plays a vital role in agriculture decision, which changes at a rapid rate. • Keeping these data updated can be done by using a concept of Distributed processing.
- 5. LITERATURE SURVEY Eagle is a bioinformatics solution for management and analysis of genomic data. A web interface provided by Eagle was also similar attempt as Big Weather Solution. Galaxy : An open platform for data intensive biomedical research offering specialized tools through Web interfaces is similar to Big Weather Solution. Climate Corp. helps growers to manage weather risk.
- 6. • From academia there is a clear claim for decision support tools for crop production. • Farmers rely on non-linear decision models with continuous updating of plans, quick analysis, and incremental implementation. • Therefore, IT applications available to farmers do not suit their needs. • In this regard the Big Weather solution is focused on offering simple, intuitive interfaces, and accessible from a multitude of devices using a simple Web browser.
- 7. WORKING • What is Decision making support system(DSS) ? A Decision Support System (DSS) is a computer- based information system that supports business or organizational decision-making activities. Decision support systems can be either fully computerized, human or a combination of both. • What is Big Data ? Big Data is simply means that huge amount of structured, semi-structured and unstructured data that has the potential to process this large data which is not possible in traditional tools.
- 8. • What is Hadoop ? Hadoop is an open source framework for distributed processing and storage of large data sets across a cluster. It has been widely adopted as the appropriate solution to batch processing of big data. Well-known enterprises such as Yahoo, eBay, Facebook, and Twitter are users of this solution. Hadoop supplies a transparent management of a distributed application.
- 9. •What is Big Weather Solution? Big Weather, an agricultural decision-making support system that utilizes Hadoop framework to process large climate related data sets. Big Weather simplifies the usage of big data processing through a friendly and easy to use Web interface. The interface is accessible through any computing device able to connect to the Internet using a Web browser.
- 10. Hadoop : Two main components are 1. HDFS(Hadoop Distributed File System) 2. MapReduce 1.Map Phase 2.Reduce Phase Overview of a MapReduce execution
- 11. BIG WEATHER: DISTRIBUTED PROCESSING SOLUTION Big Weather Architecture: Three main modules are 1. Web Portal 2. Data Server 3. Hadoop Cluster (storage and processing)
- 13. Cloud Computing: Multi-layer System Features • Sharing common computing resources using the same physical machines for different users/jobs can be accomplished by two different approaches: • The first approach is to schedule the job execution requisitions in a queue and execute each one of them sequentially and the second one is to execute the job requisitions in parallel, running them on isolated virtual clusters using the physical machines as hosts to virtual machines guests. • Big Weather allows users to choose between these two approaches.
- 14. • Illustration of two different users running isolated Hadoop jobs on the same Hadoop Cluster.
- 15. A sequence of the main steps in the Web portal interface: (A)User registration (B) Log-in (C) Job request (D) Results.
- 17. The five sequential steps necessary to complete a job processing in Big Weather Solution are: (A) Configuring Slave nodes (B) Loading data sets to HDFS (C) Executing MapReduce job (D) Reporting job progress (E) Transferring results to the Web Portal Web Portal Interface: • Big Weather Web portal is implemented in Java programming language utilizing tools and functionalities from Java Server Faces(JSF) • Big Weather Web portal runs its Java code on Apache Tomcat and for storing user information, job history and job details, Big Weather Web portal uses MySQL.
- 18. TESTS AND RESULTS: •In order to evaluate the integration of Hadoop and cloud computing it has been configured a cloud infrastructure on a test environment. •The size of the HDFS blocks was configured to 128 MB. • It has been validated the performance of Hadoop on top of virtual machines by running a MapReduce job over a climate related data set.
- 19. The first test consists of comparing the processing time to process the job with 4 virtual machines running on the same physical machine, versus the scenario with the same number of virtual machines, but having each one of them in different physical machines.
- 20. Reports of Results: 1. Processing time to execute the MapReduce job with 4 virtual machines. 2. Processing time to execute the MapReduce job by increasing the number of virtual machines in each physical machine
- 21. CONCLUSION •In this presentation we have described the Big Weather solution: an agricultural decision-making support system that can help farmers to improve their productivity based on climate related data. • The results collected indicate relevant performance improvement by increasing the number of virtual machines running in a cluster.
- 22. REFERENCES • Harris, D.: How Climate Corp. is pitting big data against Mother Nature (May 2012), http://gigaom.com/2012/05/02/how-climate-corp-is-pitting-big-data-against-mother- nature/ • J. Shim, M.Warkentin, J. Courtney, D. Power, R. Sharda, and C.Carlsson, "Past,present, and future of decision support technology. Decision Support Systems, vol. 33, no.2, pp. 111-126, Elsevier, 2002. • Ayman Nada, Mona Nasr, Marwa Salah, “Service Oriented Approach for Decision Support Systems”. • Yifan Bo, Haiyan Wang, “Cloud Computing and Internet of Things in Agriculture and Forestry”,2011 International Joint Conference on Service Sciences. • Walter Akio Goya, Marcelo Risse de Andrade, Artur Carvalho Zucchi, Nelson Mimura Gonzalez, Rosangela de F´atima Pereira, Karen Langona, Tereza Cristina Melo de Brito Carvalho, “The use of Distributed Systems and cloud computing in Agricultural Desicion support systems”, 2014 IEEE International conference on Cloud Computing.
- 23. THANK YOU!