understanding the planet using satellites and deep learning
Download as PPTX, PDF0 likes81 views
The document discusses the use of satellites and deep learning for understanding and monitoring various problems related to the planet, highlighting the importance of data sources, processing infrastructure, and machine learning applications. It outlines challenges faced in data collection and ground truth annotation and emphasizes the need for robust algorithms and a deep understanding of client requirements. Additionally, the document presents lines of research, lessons learned, and the significance of scalable infrastructure for effective solution development.
understanding the planet using satellites and deep learning
- 1. Understanding the planet using satellites and deep learning bcn.AI June 7th 2019 Albert Pujol Torras @AlbertPT71 Lead Machine Learning Platform
- 2. Agenda ● Satellogic ● Satellogic Data Science and Solutions ● What we can do with satellites, examples of problems we face ● What type of data do we work with ? ● Processing infrastructure, hardware and software ● Some lines of research we are interested on. ● Lessons learned ● Questions
- 4. Data Science & Solutions BCN Delivery platform TLV Headquarters & Design BSAS Manufacturing Plant MVD Comprehensive services PEK
- 5. Object detection/Counting What kinds of problems do we face ?
- 6. Object amount/density estimation / regression with lower image resolution
- 7. Estimation of other image modalities HR RGB LR TIR LR SWIR1 LR SWIR2 HR THERMAL
- 8. Regression: time series image prediction -Estimation of the yield at the end of the season -Monitoring of changes in the estimation to know when and where to act.
- 9. Image semantic segmentation: Land use detection
- 10. “anomaly change” and “semantic change” detection Time T0 Time T1 Diff (T1,T0)
- 11. Satellogic Data 3rd Party Satellite Data Primary Data Sources Derived Layers Temporal Evolution Land Use Maps Advanced Indices Distance to Water Terrain Orientation Superresolution Images ... These sources can be available globally or locally, dynamic or static, high or low res... nKappa: Data science platform with focus on geographic data and satellite imagery. Main goal: To scale solution development by automating/accelerating data science work. nKappa enables solution development using aligned sets of image tiles (Kappas) World Climate Maps Geologic Data Elevation Models Georef: Man-Made Structure Political Boundaries Census Data Maps Data - Data Sources
- 12. Sizes: -Typical project: 20Gb/day. -Daily world remap: continental surface processing 5300 hours of video per day. Sources of image variation: -Clouds….70% of the world is cloud covered. -Perspective changes (off nadir satellite images, drone images). -Shadows orientation, intensity, and longitude variations depending on day hour, clouds, and season. -Chromatic changes due to aerosol and hour of day. -Variations between sensors (different satellites, drone images,..) -Variations/errors in image orthorectification, geolocalization. -Growth and color of seasonal vegetation changes,... Data - Data Sources clouds perspective shadows Chromatic and vegetation changes
- 13. Data - Data Sources Extremely unbalanced datasets
- 14. Rare and expensive: indispensable to train and to assess quality of ML and computer vision approaches. Sources of ground truth: - Land ground truth provided by client. - GT generated using highest resolution imagery. - Human annotation - Our team always annotate ... to understand the problem. - Internal and external annotation (mechanical turk, supahands, ...) - sample what to annotate to preserve variability and input domain coverage. - Measure biases and variances of annotators (discard annotators, images,reconstruct annotation instructions...). - Other GT sources: first world surveyed data annotated from visual imagery or using land ground truth (Corina project, LUCAS, Creaf, Siose in spain, USA USGS land cover dataset,...) Useful one we have to deal with: - Out of data (most of it are correct but small parts are erroneous) - differing resolution (uncertain labels at class borders), - domain/covariate shift: how to transfer it to places that differ in land management culture, climate or relief. Data - Ground truth expensivecheaper
- 15. GT Data: Covariate shift & Domain adaptation Existent “good quality” Ground Truth Rice fields in Europe Target areas without ground truth Urban areas in Europe Urban areas in Lagos Rice fields in China
- 16. ● huge amount of data --> cloud infrastructure. ● nKappa platform for distributed processing (actually using Microsoft Azure) and in-house gpu servers (equipped with 1080ti’s) ● nKappa uses cloud for experiment management to keep track, team share, and audit datasets, algorithms, models ,deploying pipelines and models in to production, and handle all the GIS-ETL related stuff. ● GPU-servers mostly used in the stage of EDA and DS algorithms and models development. Infrastructure - Hardware
- 17. Some lines of research: Domain adaptation ”Deep Visual Domain Adaptation: A Survey”, Mei Wang, Weihong Deng, “Domain Adaptation for Visual Applications: A Comprehensive Survey”, Gabriela Csurka Sampling and sample-weighting based on classifier domain differentiation Adversarial networks to make embeddings invariant to domain change GT Barcelona Target Lasa
- 18. Some lines of research: Usage of generative models Image-to-Image Translation with Conditional Adversarial Networks, Isola, Phillip; Zhu, Jun-Yan; Zhou, Tinghui; Efros, Alexei A. Satellite Image Spoofing: Creating Remote Sensing Dataset with Generative Adversarial Networks, Chunxue Xu,Bo Zhao GeoGAN: A Conditional GAN with Reconstruction and Style Loss to Generate Standard Layer of Maps from Satellite Images Invisible cities. https://opendot.github.io/ml4a-invisible-cities/implementation/ Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros Evaluation of the effects on semantic segmentation of using samples from Conditional Generative Adversarial Networks: - Data augmentation: Generation of satellite images (textures) from land use random labels. - Hiper resolution and image enhancement.
- 19. Some lines of research: Uncertainty measurement and GT cleaning “Dropout as a Bayesian Approximation:Representing Model Uncertainty in Deep Learning”, Yarin Gal,Zoubin Ghahramani “Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels”, Bo Han,, Quanming Yao, Xingrui Yu,, Gang Niu,Miao Xu, Weihua Hu, Ivor W. Tsang, Masashi Sugiyama Measure errors on GT labeling: - Error and entropy on classification distribution when using an ensemble of classifiers. - Entropy of DNN outputs when applying dropout on fully connected layers on inference stage.
- 20. Some lines of research: Distance metric -Invariant embeddings How we can use the huge amount of unlabeled data to train models. -learning deep NN invariant embeddings and transferable models for encoding land use content. “Tile2Vec: Unsupervised representation learning for spatially distributed data” ,Neal Jean, Sherrie Wang, Anshul Samar, George Azzari, David Lobell, Stefano Ermon ANCHOR TILES POSITIVE TILES NEGATIVE TILES
- 21. Lessons learned - Project success : - 5% ML algorithm and algorithm parameters selection, - 95% really understanding what the client needs, how to generate value, and anticipate how your output is going to be consumed, defining good features, good ground truth, good sampling data policy, pre and post processing. - Dedicate the time first to ensure success, … after that improve: - Using fast ML algorithms. - Starting with small datasets with the input and output variability of the original one. - Predictive models: accuracy is not always the most important: explainability, consistency. - Worth invest on automatically measure dataset quality before start training on big datasets. - Missing values, constant variables, unaligned bands, duplicated variables, unbalancing… - Most of our in production costs are ETL (extract, transform, load) - Deep Learning is amazing (sometimes too much for the problems to solve) ….and it is expensive: - In production: computational cost. - In development: Fine tuning and network cooking. (does not scale quite well) - Context knowledge + common sense heuristics + ML vs end-to-end (is all tarjet domain variability in your train set?)
- 22. Questions ?