20211118 AI+ Remote Sensing

AI+ Remote Sensing:

Applying Deep Learning to Data Enhancement,
Analytics, and its Business

Dr. Jui-Hsin (Larry) Lai 賴瑞欣

US Research Lab, Ping An Technology

Nov 2021

About Dr. Jui-Hsin (Larry) Lai
2
• Staff Research Scientist

• US Research Lab of Ping An Technology in Silicon Valley

• Lead the Remote Sensing team working on
 
- Image enhancement for remote sensing imagery
 
- few-shot learning and unsupervised learning
 
- Model generalization and multi-modality modeling
• Research Publications

• Granted 52+ worldwide patents

• Published 27+ IEEE/ACM papers

• More research demos http://www.larry-lai.com
Dr. Jui-Hsin(Larry) Lai @Ping An Technology

About Ping An Group
3
• The largest insurance company worldwide
Fortune Global 500 Rank
2020 No.21
2018 No.29

2016 No.41

2014 No,128

2012 No.242

2010 No.383

2008 No.462

1988 Established
• Primary business

• Insurance

• Banking

• Financial services

• Healthcare

• Now, empowering each sector with technology
• Ping An Finance Center(PAFC)
 
- 115-story, skyscraper in Shenzhen, China

AI+ Remote Sensing in Ping An Group
4
Agriculture Insurance Smart City
Environment, Society, Government (ESG)
Investment — Option Trading
• A close loop: Research Innovation <=> Business Applications

Challenges and Opportunities
5
• Business driven v.s. Tech driven
Data Management & Computation
Data Enhancement & Fusion
Analytics & Applications
• Decloud, dehaze, harmonization, and
 
spatial/temporal/spectral resolution
• Data cost, accessibility, and computation
speed

Outline
6
• Data Fusion

• 4X Super-Resolution Image Enhancement

• The Proposed PAII-SR Model
• Data Enhancement

• Haze Occlusion Removal

• The Proposed PAII-Haze Model
• Analytics

• Parcel Detection Model

• Crop Recognition Model
• More Applications in Ping An Group

• Agriculture and Carbon Neutrality

Image Resolution is a Key Limitation in Many Remote Sensing Analysis
7
Some application like road or car detection needs
 
high-resolution images
Definition of 10m resolution：
 
Each pixel size means 10-meter in ground distance
Public(Free) Commercial
Data Source
NASA(USA),
 
ESA(European)
Planet Lab,
 
NavInfo Co.
Resolution
10m, 30m, and
above
0.5m, 1m, and 2m
Application
Greenfield
detection,
 
forest analysis
Road
segmentation,
 
crop recognition

Data Cost is the Entry Barrier in Remote Sensing Applications
8

How Does Super-Resolution(SR) Model Enhance Image Details?
9
SR Model
How does the model reconstruct the missing information?

The Input for SR Model Training: (1) Spatial Pattern Correlations
10
The model learns the pattern correlations between low-resolution and high-resolution images.
Low-Resolution Image
V.S.
High-Resolution Image

The Input for SR Model Training: (2) Details under Temporal Changes
11
The model learns the details from lighting changes, ground changes, or occlusions.
April 30, 2021
May 3, 2021
May 5, 2021

The Input for SR Model Training: (3) Attention from Other Sensors
12
The model learns the attention from other sensors.
Low-Resolution RGB Image Synthetic Aperture Radar(SAR) High-resolution RGB Image
+ =

The Proposed PAII-SR Dataset
13
● 2 bands, VV/VH

● 10m resolution

● 4 temporal captures

● Geolocation aligned
with Sentinel-2
● 4 bands, BGRN

● 10m resolution

● 4 temporal captures

● Various terrain coverage
Geolocation
aligned

Sentinel-2
 
Multi-spectrum Imagery
Unmanned Aerial Vehicle (UAV)

BGRN Channels, 0.6m and 2.4m Resolutions
Sentinel-1

SAR Imagery
Input for Model Training The Target of Model Output

The 4x Enhancement by PAII-SR Model
14
PAII-SR Model adapting the deep learning & GAN training

- Proposed the PAII-SR deep learning model with 4X Enhancement
Ground truth: UAV 0.6m Bilinear 4X: 2.4m => 0.6m

PSNR: 34.5dB
PAII-SR 4X Model: 2.4m => 0.6m

Sharp details & PSNR: 34.2dB

15
Test the PAII-SR Model Generalization

- Trained on Sentinel-2 10m => NAIP 2.5m

- Tested on Sentinel-2 40m => Sentinel-2 10m
Ground truth: Sentinel-2 10m Bilinear 4X: 40m => 10m PAII-SR 4X Model: 40m => 10m

Model generalization seems working

16
Test the PAII-SR Model Generalization

- Trained on Sentinel-2 10m => NAIP 2.5m

- Tested on Gaufen-2 4m => Gaufen-2 1m
Ground truth: Gaufen-2 1m Bilinear 4X: 4m => 1m PAII-SR 4X Model: 4m => 1m

Model generalization seems working

Quick Summary in SR Image Enhancement
17
• Image resolution is a key limitation in many remote sensing analytics

• SR technique plays a critical role in translating free image source (low-resolution) into valuable
imagery (high-resolution)
• Our PAII-SR Dataset

• The largest remote sensing dataset for SR model training

• Including 1.6M+ image pairs

• Two 4x scaled imageries: (1) 10m => 2.5m, (2) 2.4m=>0.6m
• Our PAII-SR 4x Enhancement Model

• Achieve the average PSNR 34.19 dB

• Effectively enhance the image details & preserve the consistent color tone

• Model generalization to more resolutions (2.4m, 10m, 40m)

• Model generalization to other satellites (Sentinel-2, Gaufen-2 satellites)

Outline
18
• Data Fusion




• Analytics




Cloud and Haze Occlusion in Remote Sensing (RS) Imagery
19
Cloud occlusion is common on RS images.
Haze occlusion is more common than cloud

but not easy to be noticed.
Problems: Cannot see the ground activity. Problems: Cannot see the true ground
reflectance. E.g., NDVI for growth monitoring.

Cloud Detection & Haze Removal in Sen2Cor
20
• Sentinel-2 L2A Product

• The product after atmospheric correction

• Should have accounted for the hazy/aerosol/
cirrus

• The cloud mask in L2A is highly accurate

• But, the haze is still an unsolved problem
From Sen2Cor User Guide By ESA

Le
ft
: L1C RGB, Right: L2A RGB

Middle: Aerosol Op
ti
cal Thickness (AOT) derived from B10
=> Need to correct the data that have already
been corrected but not done in a good way.

How to Design a Model for Removing Haze?
21
Haze Removal
Model
Challenge 1: Design a model outperforming the Sen2Cor?
Challenge 2: Define an image which is haze-free?

We Need to Index the Haze Density First!
22
• There is no universal standard for haze-level in industry or academy

• We adapt the Dark Channel Prior* to measure haze density

• Observation: at least one color channel has very low intensity at some pixels
* He et al, Single Image Haze Removal Using Dark Channel Prior, CVPR 2009

Haze Index Applying to Remote Sensing Imagery
23
DCP: 43 DCP: 58 DCP: 71 DCP: 88
Below images are Sentinel-2 L2A product.

They should be “atmospheric correction”, but you can easily see the haze occlusion.
• Dark Channel Prior(DCP)

• The higher DCP value, the heavier haze level

• The DCP level seems consistent to visual perception

The Proposed PAII-Haze Dataset
24
- Image pair on the same geo-location but having different haze-level

- Image pair captured within 5 days, to avoid ground change

- Various coverage of terrain types and seasons
DCP: 25 DCP: 26 DCP: 21 DCP: 25
DCP: 43 DCP: 58 DCP: 71 DCP: 88

The Proposed Deep Learning PAII-Haze Model
25
PAII-Haze Model

- BGRN 4-Channel Deep Learning model

- We try to highlight the NDVI accuracy

- Model loss: L1-loss of RGB & N & NDVI

- Single model trained on various haze levels

Evaluation of PAII-Haze Model
26
Hazy image GT image Haze removal image
Heavy
Haze
Light
Haze
The PAII-Haze Model can cope with various haze density
- Average PSNR: 30.5 dB

- Subjective evaluation shows dramatic improvement

Quick Summary in Haze Removal
27
• Over 90% remote sensing images are covered with haze-level over 30+ DCP

• We can NOT see the true ground reflectance, e.g., the NDVI for crop growth monitoring
• Our PAII-Haze Dataset

• The largest remote sensing dataset for Haze-Removal model

• Including 800K+ image pairs

• Each image with BGRN 4-channel and 512x512 pixel-size
• Our PAII-Haze Model

• Effectively remove haze under various haze-level

• Achieve the average PSNR 30.5 dB

• Subjective evaluation shows dramatic improvement

Outline
28
• Data Fusion




• Analytics




Crop Recognition is Not Easy to Machine, Even to Human
29
Rice
Rice
Rice
Rice
Rape
Rape
Rape
Rape
Rape Rape
Rape
Rice
Rice
Rice
Rice
Rice
Rape
Maize
Maize
Rape
Rice
• The tasks under crop recognition

• Cropland segmentation

• Parcel detection

• Crop recognition
• Each task is challenging

• Cropland and forest look similar

• Parcel shapes are irregular

• Crop recognition cannot be done
based on a single image

• And …

In 2019, We Proposed a Framework — Cropland & Parcel & Crop Models
30
Graph-based Topology
Connecting Model
PAII-Cropland Model
Cropland Segmentation Result
Probabilities
Segmentation
Topology Connecting Result
Post-processing
(morphological
operation)
Segmentation
Result
Cropland
segments
Mul
ti
-spectral &
Mul
ti
-temporal

satellite images
Topology connected
graph
*The topology model
connects the broken ridges.
Input Data Ridge/Cropland/Non-Cropland Segmentation Topology Post-Processing

In 2019, We Did a Good Job on Prove of Concept (PoC) Counties
31
Crop Type Precision/Recall
Rice 0.85/0.86
Wheat 0.88/0.86
Corn 0.87/0.83
Co
tt
on 0.92/0.82
Greenhouse 0.93/0.90
- Train 5 di
ff
erent models for 5
di
ff
erent crop types

- Tes
ti
ng on 9 PoC coun
ti
es

- Average recall/precision are high
Corn in
Caoxian
County
Greenhouse
in Shouguang
County
Satellite Image Annota
ti
on Predic
ti
on

In 2020, We Marched Quickly and Broadly!
32
From PoC to massive area produc
ti
on, the wheat detec
ti
on on the mainland China in March 2020
However, the challenges between the PoC and massive production are totally different scales.

In 2020, Challenges in Data labeling & Model Generalization
33
Labeling cost for crop recognition: $12 USD/KM2
Huge training data, high labeling cost Model generalization is difficult.
How the model to cope with various geo-
locations, crop types, and satellite imageries?
Various lighting conditions
 
under log/lat coordinates
Different crop seeds Different satellite imaging

In 2021, Thinking Outside the Box — Semi-Supervised Parcel Detection
34
The PAII-Parcel Model
• Core ideas

• Reduce the dependency of supervised training data

• Adapt machine learning and reduce model parameters

• Adapt semi-supervised learning trained from general tasks

• Leverage other sensors, attention map from SAR
Multi-spectrum & SAR images
Results of parcel detection
Semi-supervised learning
for boundary refinement
Pixel clustering
model

Results of PAII-Parcel Model
35
The PAII-Parcel Model is trained from a universal dataset to output mul
ti
ple levels of parcel sizes.
Input Image Output: fine-grain parcel detection Output: coarse-grain parcel detection

In 2021, Thinking Outside the Box — Few-Shot Learning Crop Recognition System
36
Recognition results
Expert
Knowledge
Dataset
UI for collecting
 
training labels
+
Time series of crop
growth analysis
Multi-stage & light-
weight recognition model
+
PAII-Crop Model: time-series crop recognition model with few-shot learning
Multi-inputs
• Core ideas

• Adapt the Expert Knowledge Dataset, e.g., local crop history, growth period, terrain conditions.

• Introduce the interactive UI system to collect training labels for few-shot learning

• Leverage SAR imagery to get additional ground reflectance e.g., water detection for rice

• Leverage DEM imagery to get terrain slop and hill shade

System of PAII-Crop Labeling, Model Training, and Model Deployment
37
- A real-time and interactive labeling system with few-shot learning model

Recognition Results of PAII-Crop Model
38
Category Province Precision Recall
Almond CA, US 72.420 76.036
Corn Henan, China 89.445 88.431
Cropland Henan, China 91.128 92.554
Cropland Hubei, China 91.974 93.946
Wheat Hubei, China 80.927 92.318
Cropland Jilin, China 93.327 91.179
Massive Deployment for PAII-Crop Model:

Test the system running on massive area
Few-Shot Learning for PAII-Crop Model:

40 labels/10,000 KM2 collecting from the interactive UI
Ground Truth:

Cropland in Henan Province
Model Output:

Gray-scale [0-1] probability

Quick Summary in Crop Recognition
39
• Crop recognition is one of the core agriculture applications

• A combination of cropland segmentation, parcel detection, and recognition
• Our PAII-Parcel Model

• Abandon the requirement of huge label data

• Proposed the semi-supervised learning for pixel clustering and boundary refinement

• Trained from a universal dataset and output multiple levels of parcel sizes
• Our PAII-Crop Model

• Abandon the requirement of huge training data

• Introduce the interactive UI system to collect training labels for few-shot learning

• Synergy of the Expert Knowledge Dataset, multi-spectrum, SAR, and DEM imageries

• Test the system running on province-level and its performance precision over 80%, recall 80%

Outline
40
• Data Fusion




• Analytics




41
农业⽣产：⼤宗农产品监测指数——辅助期货交易决策
案例：北美⼤⾖ – 种植面积、长势⽔平、产量预测
• 建模快、覆盖全：4周完成模型构建，8周完成全美泛化覆盖

• 指数化输出：3⼤类，14小类：种植面积指数/长势⽔平指数/产量预测指数

• 模型精度⾼：种植面积准确率>90%，长势/产量准确率>85%（对比USDA）

• 模型时效优：与USDA同频可比，单次输出相比USDA同期数据提前48小时以上
指数分类指数细类
北美⼤⾖产量
预测指数
北美全境-总产预测指数
北美全境-总产预测同比指数（vs 去年同期）
北美全境-平均亩产预测数据
北美全境-平均亩产预测数据同比（vs 去年同期）
州级别亩产预测指数
州级别-亩产预测同比指数（vs 去年同期）
州级别总产预测指数
州级别-总产预测同比指数（vs 去年同期）
北美⼤⾖长势
⽔平指数
州级别-⼤⾖较好长势区域的占比指数
州级别-⼤⾖综合长势同比指数（vs 去年同期）
北美⼤⾖种植
面积指数
北美全境-总种植面积同比指数
北美全境-总种植面积预测指数
州级别-种植面积同比指数（vs 去年同期）
州级别-种植面积预测指数

Flooding and Natural Damage Analysis for Smart City
42
July 2020 Hubei Province, Flooding detection
Mul
ti
-spectrum Image SAR Image
Eleva
ti
on (DEM) Image

Carbon Emission Analysis for Environment, Society, and Government (ESG)
43
2019/06/01 - 2019/09/01
PAII-Carbon Analysis for global county-level CO2 emission.

We are working on data enhancement for high-temporal
and high-spatial resolution emission analysis.
CO2 Data processing for OCO-2 Satellite.
2019/06/01 - 2019/09/01

45
碳中和投资管理 — 光伏电厂识别绿⾊项目投资的监测管理
全国、省級⼤範圍光伏电厂识别

- 识别电厂建造日期、开发进度、电厂面积

- 结合⽓象数据分析电厂发电效能

46
碳中和投资管理 — 風能电厂识别绿⾊项目投资的监测管理
全国、省級⼤範圍風能电厂识别

- 识别电厂建造日期、开发进度、电厂面积

- 结合⽓象数据分析电厂发电效能

Summary
47
• AI + Remote Sensing is Booming

• High computation capability — Large scale analytics with spatial and temporal coverage

• Semi-supervised learning, few-shot learning — Lower the barrier of model training & generalization
• Carbon neutrality is the TREND

• Remote sensing is important, but its business market was limited

• Now, remote sensing is one of the third-party certification in Carbon Neutrality metric

• And more applications in green economy
• Welcome to join us in Silicon Valley or 深圳

• Intern: PhD student

• Full-time employee: PhD degree or 3+ working-year with master degree

Thank You!

Dr. Jui-Hsin(Larry) Lai

20211118 AI+ Remote Sensing

More Related Content

What's hot (20)

Similar to 20211118 AI+ Remote Sensing (20)

Recently uploaded (20)

20211118 AI+ Remote Sensing