Scalable Deep Learning in ExtremeEarth-phiweek19
Download as PPTX, PDF0 likes61 views
This document summarizes a presentation about scalable deep learning techniques for analyzing Copernicus Earth observation data using the Hopsworks platform. The presentation discusses Hopsworks' end-to-end machine learning pipelines, feature engineering capabilities, distributed deep learning techniques like data parallel training, and applications of these techniques to challenges in classifying satellite imagery like sea ice mapping. Deep learning architectures, preprocessing steps, and distributed training methods are highlighted as areas of ongoing work and improvement for analyzing large volumes of remote sensing data on Hopsworks.
Scalable Deep Learning in ExtremeEarth-phiweek19
- 1. ExtremeEarth From Copernicus Big Data to Extreme Earth Analytics This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 825258.
- 2. Frascati - September 12th 2019 Sina Sheikholeslami, KTH Theofilos Kakantousis, Logical Clocks Scalable Deep Learning Techniques for Copernicus Data Phi-Week 2019
- 3. 3 Agenda • What is Hopsworks • End-to-end Machine Learning Pipelines • Deep Learning Pipelines in Hopsworks • Scalable Distributed Deep Learning Techniques • Deep Learning Techniques for Earth Observation data • Summary
- 7. 8 Hopsworks REST API • Manage resources via the REST API o Projects o Datasets o Jobs o FeatureStore o Experiments o ModelServing o Kafka o ...
- 8. End-to-end Machine Learning & Deep Learning Pipelines
- 9. 10 End-to-End ML & DL Pipelines can be factored into stages Data Prep Data Ingest Train Serve Online Monitor Distributed Storage Raw Data Data Lake Resource Manager
- 10. 11 End-to-End ML & DL Pipelines in Hopsworks
- 11. 12 Typical Feature Store pipeline
- 12. Deep Learning Pipelines in Hopsworks
- 13. 14 Data Access - Copernicus EO data
- 14. 15 Data Access - Copernicus EO data
- 17. 18 Experiments Overview - Hopsworks UI
- 18. 19 Experiments Details - Hopsworks UI
- 19. 20 Experiments - Hopsworks API
- 21. 22 TensorFlow Extended with Beam Portability Framework and Flink runner
- 24. 25 Data Scientist Dev View - Jupyter Dashboard
- 25. 26 Dev view - Python first No need to write Dockerfiles Hopsworks Cluster
- 26. 27 Dev View - Orchestration
- 29. 30 Distributed Deep Learning in Hopsworks Executor 1 Executor N Driver HopsFS (HDFS)TensorBoard Model Serving
- 30. 31 Data Parallel Distributed Training Training Time Generalization Error (Synchronous Stochastic Gradient Descent (SGD))
- 31. 32 Distributed Deep Learning in Hopsworks HopsYARN 10 GPUs on 1 host 100 GPUs on 10 hosts with ‘Infiniband’ Hopsworks supports a Hetrogenous Mix of GPUs 4 GPUs on any host
- 32. 33 Ring-AllReduce vs Parameter Server GPU 0 GPU 1 GPU 2 GPU 3 send send send send recv recv recv recv GPU 1 GPU 2 GPU 3 GPU 4 Param Server(s) Network Bandwidth is the Bottleneck for Distributed Training
- 33. 34 Distributed Deep Learning in Hopsworks • Uses Apache Spark/YARN to add distribution to TensorFlow’s CollectiveAllReduceStrategy – Automatically builds the ring (Spark/YARN) – Allocates GPUs to Spark Executors
- 34. Scalable Deep Learning Techniques for Satellite Data
- 35. 36 Polar Use Case EE Ambitions - I • Deep Learning Architecture o Now: Most of the deep architectures developed for classifying remote sensing data focus on VHR optical images. The existing pretrained networks are not optimized to the specific properties of Copernicus data (SAR Images) o KPI: The new deep architecture will improve the classification accuracy of Copernicus data with respect to the use state-of-the- art classifiers • Sea Ice Charts o Now: Sea ice charts are based on trained ice analysts manually segmenting a combination of SAR and other images into smaller polygons to record parameters including, but not limited to, sea ice concentration and stage of development. o KPI: Automatic production of more accurate and reliable sea ice products on the Hops data platform • Distributed Deep Learning: o Now: No work currently exists in the international literature for scaleout deep learning for Remote Sensing data. o KPI: Classification algorithms running on Hops and scaling to PBs of Copernicus data.
- 36. 37 Polar Use Case EE Ambitions - II • Ice Charts o Now: Ice charts are currently manual interpretations with inherent human quality control but also potential for bias. o KPI: Methods must be robust and reliable throughout the sea ice seasonal cycle • Computing power o Now: New ice mapping algorithms currently run only for limited areas and seasons, using limited local processing resources. o KPI: The Hops data platform is able to cope with input data volume and processing load to enable NRT product availability. • Operational Delivery o Now: New automatic ice information products not available to end users o KPI: Demonstrate availability of new automatic ice mapping products with demo users
- 37. 38 Polar Use Case EE Challenges • Training data o Need of enough and accurate training data o Gathering high-quality ground truth is expensive and sometimes not feasible • Resolution o Each pixel represents a large area of land (100m x 100m) o Accurate characterization can be jeopardized by misclassification of even single pixels • Size of SAR Images o The size of each image usually is 1+ GB o Many pixels or patches should be considered in training and inference • Porting and storing data to Hadoop o Data must be prepared in efficient data structures o Appropriate data format for storage of training data is required
- 38. 39 Preprocessing Calibration Speckle Nosie Terrain Correction Feature Extraction Classification Segmentation Non-Linear Function Feature Learning by Deep Learning Sea ice Edges Sea Ice Types - concentration Iceberg Detection Application Road Map
- 39. 40 Deep Learning Transfer Learning Ad hoc Architecture Distributed Training of Existing Architecture Designing Specific Distributed Architecture for Remote Sensing Pixel-Wised VGG16 from Scratch 103,754,000 Patch has been analyzed to Label each pixel Patch-Based VGG16 Fine-Tune 32x32 Patches Sea ice Edges Patch-Based Ad hoc network 32X32 Patches Semi-Supervised Distributed Training (GANs) • All tests have been performed on Hopsworks • High Resolution and Pixel-wise processing requires analyzing 100+M patches - approx. 9hour/image ! • New techniques for distributed training is needed in the remote sensing domain S. Khaleghian, T. Kræmer, T. Eltoft, A. Marinoni, “Distributed Deep learning for sea ice edges detection”, in prep.
- 40. 45 Summary • End-to-end Machine Learning pipelines with Hopsworks • Scalable Deep Learning techniques • Deep Learning techniques for Earth Observation data
- 41. Hopsworks Contributors Jim Dowling, Seif Haridi, Gautier Berthou, Salman Niazi, Mahmoud Ismail, Theofilos Kakantousis, Ermias Gebremeskel, Fabio Buso, Antonios Kouzoupis, Kim Hammar, Steffen Grohsschmiedt, Alex Ormenisan, Robin Andersson, Moritz Meister, Kajetan Maliszewski, Netsanet Gebretsadkan Kidane, Sina Sheikholeslami, Joel Stenkvist, August Bonds, Vasileios Giannokostas, Johan Svedlund Nordström, Rizvi Hasan, Paul Mälzer, Bram Leenders, Juan Roca, Misganu Dessalegn, K “Sri” Srijeyanthan, Jude D’Souza, Alberto Lorente, Andre Moré, Ali Gholami, Davis Jaunzems, Stig Viaene, Hooman Peiro, Evangelos Savvidis, Qi Qi, ... @hopsworks