Skip to content

h1063135843/EDFT

 
 

Repository files navigation

Efficient Depth Fusion Transformer for Aerial Image Semantic Segmentation

Abstract

Taking depth into consideration has been proven to improve the performance of semantic segmentation through providing additional geometry information. Most existing works adopt a two-stream network, extracting features from color images and depth images separately using two branches of the same structure, which suffer from high memory and computation costs. We find that depth features acquired by simple downsampling can also play a complementary part in the semantic segmentation task, sometimes even better than the two-stream scheme with the same two branches. In this paper, a novel and efficient depth fusion transformer network for aerial image segmentation is proposed. The presented network utilizes patch merging to downsample depth input and a depth-aware self-attention (DSA) module is designed to mitigate the gap caused by difference between two branches and two modalities. Concretely, the DSA fuses depth features and color features by computing depth similarity and impact on self-attention map calculated by color feature. Extensive experiments on the ISPRS 2D semantic segmentation dataset validate the efficiency and effectiveness of our method. With nearly half the parameters of traditional two-stream scheme, our method acquires 83.82% mIoU on Vaihingen dataset outperforming other state-of-the-art methods and 87.43% mIoU on Potsdam dataset comparable to the state-of-the-art.

Paper can be download here.

Installation

Please refer to get_started.md for installation

Data

Two ISPRS Contest Datasets have been preprocessed to form RGB-D images and organized as a custom of mmsegmentation. Please download from aistudio: Vaihingen, Potsdam

Results

DataSet Backbone Crop Size Lr schd mIoU mIoU(ms+flip) config download
Vaihingen Segformer-B0 256x256 80000 80.49 81.63 config model
Vaihingen Segformer-B1 256x256 80000 81.28 82.13 config model
Vaihingen Segformer-B2 256x256 80000 82.17 82.88 config model
Vaihingen Segformer-B3 256x256 80000 82.27 83.04 config model
Vaihingen Segformer-B4 256x256 80000 83.02 83.82 config model
Vaihingen Segformer-B5 256x256 80000 82.48 83.23 config model
Potsdam Segformer-B4 512x512 80000 87.22 87.40 config model

password for BaiduNetdisk: dshs

# Single-gpu testing
python tools\test.py configs\edft\segformer_mit_fuse-b0_256x256_80k_vai.py mit_fuse_b0.pth --eval mIoU

Training

# Single-gpu training
python tools\train.py configs\edft\segformer_mit_fuse-b0_256x256_80k_vai.py

Citation

@Article{rs14051294,
	AUTHOR = {Yan, Li and Huang, Jianming and Xie, Hong and Wei, Pengcheng and Gao, Zhao},
	TITLE = {Efficient Depth Fusion Transformer for Aerial Image Semantic Segmentation},
	JOURNAL = {Remote Sensing},
	VOLUME = {14},
	YEAR = {2022},
	NUMBER = {5},
	ARTICLE-NUMBER = {1294},
	URL = {https://www.mdpi.com/2072-4292/14/5/1294},
	ISSN = {2072-4292},
	DOI = {10.3390/rs14051294}
}

About

OpenMMLab Semantic Segmentation Toolbox and Benchmark.

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 98.8%
  • Shell 1.1%
  • Other 0.1%