Exploring Fused Convolutional Neural Networks for Aerial Imagery Segmentation
- 1. Exploring RGB+DSM Convolutional Neural Networks Fusion Strategies for Satellite Imagery Segmentation Mendrika Ramarlina www.haizaha.com
- 2. Dataset ● 38 tiles of 6000x6000px Orthorectified RGB Imagery ● DSM: ground height removed for each pixel => height above the ground ● Ground Sampling Distance of 5cm for both DSM and TOP ● Two classes: ○ Background ○ Building ● Pre-processing: ○ Each tile is split into patches of 2048x2048 pixels ○ Resized down to 256x256
- 3. Baseline Architecture: Unet-ResNet34 ● Backbone: Unet - https://arxiv.org/abs/1505.04597 ● Encoder: 34 Layers Residual Network - https://arxiv.org/abs/1512.03385 ● Layer fusion strategy: Conv2d Fusion ○ First stack feature maps ○ Then run a 2d convolution with 1x1 kernel filter
- 4. Model: RGB Unet ResNet-34 ● Input: RGB images ● Weights initialization: ImageNet ● IOU: 0.836
- 5. Model: 1-Channel DSM Only ● Input: DSM ● Weight Initialization: average of RGB channels from ImageNet pre-trained weights ● IOU: 0.832
- 6. Model: Elevation Detection Pre-training ● Pre-training phase: train an elevation detection model ● Re-training the model on ground truth ● IOU: 0.829
- 7. Model: Fused Segmentation-Heads ● Practically a model ensemble ● Fusion mechanism: Conv2d Fusion ● IOU: 0.865
- 8. Model: Fused Decoders ● Fusion mechanism: Conv2d Fusion ● IOU: 0.863
- 9. Model: Fused Encoders ● Fusion mechanism: Conv2d fusion and summation fusion ● IOU: 0.87
- 10. Model: 4-Channels RGB-D ● Fusion mechanism: Conv2d Fusion ● IOU: 0.888
- 11. Results: Final IOU vs. Fusion Location
- 12. Results Input Data Initialization Fusion Location Epochs 0 - 200 Epochs 200 - 400 RGB ImageNet 0.833 0.836 RGB Elevation Detection 0.819 0.829 DSM ImageNet 0.813 0.832 RGB-DSM ImageNet Input 0.877 0.888 RGB-DSM ImageNet Encoder 0.876 0.87 RGB-DSM ImageNet Decoder 0.207 0.863 RGB-DSM ImageNet Segmentation Heads 0.164 0.865
- 13. Ways to Get in Touch Mendrika Ramarlina mendrika@haizaha.com linkedin.com/in/mendrikaramarlina github: @ramarlina