|
|
论文:2022,Vol:40,Issue(6):1404-1413 |
|
|
引用本文： |
|
|
钱坤, 李湉雨, 李喆, 陈美杉. 基于拆分注意力残差网络的红外和可见光图像融合算法[J]. 西北工业大学学报 |
|
|
QIAN Kun, LI Tianyu, LI Zhe, CHEN Meishan. Infrared and visible image fusion algorithm based on split-attention residual networks[J]. Journal of Northwestern Polytechnical University |
|
|
|
|
|
|
|
| 基于拆分注意力残差网络的红外和可见光图像融合算法 |
|
| 钱坤1,2, 李湉雨1, 李喆2, 陈美杉1 |
|
1. 海军航空大学 岸防兵学院, 山东 烟台 264000;
2. 中国人民解放军 32127部队, 辽宁 大连 116100 |
| 摘要: |
| 在红外和可见光图像融合算法中,图像信息的丢失始终是制约融合图像质量提升的关键问题,为此,提出了一种基于拆分注意力残差网络的红外和可见光图像融合算法,使用带有拆分注意力模块的深层残差网络拓展感受野和提高跨通道信息融合能力,运用平滑最大值单元函数作为激活函数进一步提升网络性能;特征提取后运用零相位分量分析和归一化算法得到融合权重后完成图像融合。实验结果表明,融合后的图像细节丰富,边缘锐利;在峰值信噪比、结构相似性指数度量和基于梯度的融合性能等指标上与经典的6种算法相比均有不同程度提升。 |
| 关键词:
图像融合
拆分注意力
残差网络
零相位分量分析
平滑最大值单元函数
|
|
| Infrared and visible image fusion algorithm based on split-attention residual networks |
|
| QIAN Kun1,2, LI Tianyu1, LI Zhe2, CHEN Meishan1 |
|
1. Coastal Defense Academy, Naval Aeronautical University, Yantai 264000, China;
2. No. 32127 Unit of PLA, Dalian 116100, China |
| Abstract: |
| In the infrared and visible image fusion algorithm, the loss of image information is always the key problem to restrict the improvement of fusion image quality. Therefore, an infrared and visible image fusion algorithm based on split-attention residual network is proposed. The residual network with split-attention block is used to expand the receptive field and improve the ability of cross-channel information fusion, and the smooth maximum unit function is used as the activation function to further improve the network performance. Then, the extracted feature map is whitened by using the zero-phase component analysis method to project it in the same subspace, and the initial weight map is obtained by L1-norm. Then the bicubic interpolation algorithm is used for upsampling and the softmax function is used for weight normalization to obtain the weight matrix consistent with the size of the original image. Finally, the weighted average strategy is used to weighted average the original infrared and visible images to obtain the final fused image. In order to verify the performance of the algorithm, the subjective and objective evaluation is compared with the six classical fusion algorithms in experiment. In the subjective evaluation, the fusion image of the present algorithm is in detail, which not only reflects the thermal information in the infrared image, but also retains the texture details of the visible image, with sharp edges, high color restoration and natural appearance. Among the three evaluation indexes of peak signal-to-noise ratio, structural similarity index measure and gradient-based fusion performance, the present algorithm improves at least 1.78％, 2.00％ and 3.10％ by comparing with the other six algorithms. |
| Key words:
image fusion
split-attention
residual network
zero-phase component analysis
smooth maximum unit
|
|
| 收稿日期: 2022-03-14
修回日期:
|
| DOI: 10.1051/jnwpu/20224061404 |
| 通讯作者: 李湉雨(1993—),海军航空大学讲师,主要从事机器学习及图像处理研究。e-mail:2823426785@qq.com
Email：2823426785@qq.com |
| 作者简介: 钱坤(1986—),海军航空大学博士研究生,主要从事计算机视觉及机器学习研究
|
|
| 相关功能 |
|
|
|
|
| 作者相关文章 |
|
| 钱坤 在本刊中的所有文章 |
| 李湉雨 在本刊中的所有文章 |
| 李喆 在本刊中的所有文章 |
| 陈美杉 在本刊中的所有文章 |
|
|
|
|
|
|
|
|
参考文献: |
|
|
[1] KUMAR S B K. Image fusion based on pixel significance using cross bilateral filter[J]. Signal Image & Video Processing, 2015, 9(5):1193-1204
[2] BAVIRISETTI D P, DHULI R. Fusion of Infrared and visible sensor images based on anisotropic diffusion and Karhunen-Loeve transform[J]. IEEE Sensors Journal, 2015, 16(1):203-209
[3] MA J, CHEN C, LI C, et al. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31:100-109
[4] FU Z Z, ZHAO Y F, XU Y W, et al. Gradient structural similarity based gradient filtering for multi-modal image fusion[J]. Information Fusion, 2020, 53:251-268
[5] LI H, WU X J. Infrared and visible image fusion using latent low-rank representation[EB/OL].(2018-4-24)[2022-11-11]. https://arxiv.org/abs/1804.08992v4.
[6] 裴佩佩, 杨艳春, 党建武, 等. 基于滚动引导滤波器和卷积稀疏表示的红外与可见光图像融合方法[J]. 激光与光电子学进展, 2022, 59(12):56-63 PEI Peipei, YANG Yanchun, DANG Jianwu, et al. Infrared and visible image fusion method based on rolling guidance filter and convolution sparse representation[J]. Laser & Optoelectronics Progress, 2022, 59(12):56-63 (in Chinese)
[7] 杨勇, 李露奕, 黄淑英, 等.自适应字典学习的卷积稀疏表示遥感图像融合[J]. 信号处理,2020, 36(1):125-138 YANG Yong, LI Luyi, HUANG Shuying, et al. Remote sensing image fusion with convolutional sparse presentation based on adaptive dictionary learning[J]. Journal of Signal Processing, 2020, 36(1):125-138 (in Chinese)
[8] ZHOU Z, WANG B, LI S, et al. Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with gaussian and bilateral filters[J]. Information Fusion, 2016, 30:15-26
[9] PRABHAKAR K R, SRIKAR V S, BABU R V. DeepFuse:a deep unsupervised approach for exposure fusion with extreme exposure image pairs[C]//Proceeding of the 2017 IEEE International Conference on Computer Vision, 2017
[10] ZHANG Y, LIU Y, SUN P, et al. IFCNN:a general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54:99-118
[11] LI H, WU X J, KITTLER J. Infrared and visible image fusion using a deep learning framework[C]//Proceeding of the 24th International Conference on Pattern Recognition, 2018
[12] LI H, WU X J, DURRANI T S. Infrared and visible image fusion with resnet and zero-phase component analysis[J]. Infrared Physics & Technology, 2019, 102:103039
[13] 陈广秋, 王帅, 黄丹丹, 等. 基于多尺度局部极值分解与ResNet152的红外与可见光图像融合[J]. 光电子·激光, 2022, 33(3):283-295 CHEN Guangqiu, WANG Shuai, HUANG Dandan, et al. Infrared and visible image fusion based on multiscale local extrema decomposition and resnet152[J]. Journal of Optoelectronics·Laser, 2022, 33(3):283-295 (in Chinese)
[14] ZHANG H, WU C, ZHANG Z, et al. ResNeSt:split-attention networks[C]//Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022
[15] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognitio[C]//Proceeding of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016
[16] JIE H, LI S, GANG S, et al. Squeeze-and-excitation networks[J]. IEEE Trans on Pattern Analysis and Machine Intelligence, 2020, 42(8):2011-2023
[17] LI X, WANG W, HU X, et al. Selective kernel networks[C]//Proceeding of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020
[18] DENG J, DONG W, SOCHER R, et al. ImageNet:a large-scale hierarchical image database[C]//Proceeding of the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2009:248-255
[19] MAAS A, HANNUN A, NG A. Rectifier nonlinearities improve neural network acoustic models[C]//Proceeding of the 30th International Conference on Machine Learning, 2013
[20] CLEVERT D, UNTERTHINER T, HOCHREITER S. Fast and accurate deep network learning by exponential linear units(ELUs)[EB/OL]. (2015-11-23)[2022-11-11]. https://arxiv.org/abs/1511.07289.
[21] BISWAS K, KUMAR S, BANERJEE S, et al. SMU:smooth activation function for deep networks using smoothing maximum technique[EB/OL]. (2021-11-23)[2022-11-11]. https://arxiv.org/abs/2111.0468223
[22] 马旗, 朱斌, 张宏伟. 基于VGG网络的双波段图像融合方法[J]. 激光与红外, 2019, 49(11):7 MA Qi, ZHU Bin, ZHANG Hongwei. Dual-band image fusion method based on VGGNet[J]. Laser & Infrared, 2019, 49(11):7
[23] TOET A. The TNO multiband image data collection[J]. Data in Brief, 2017, 15:249-251
[24] ZHANG X, YE P, XIAO G. VIFB:a visible and infrared image fusion benchmark[C]//Proceeding of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020
[25] 徐东东. 基于无监督深度学习的红外与可见光图像融合方法研究[D]. 长春:中国科学院大学, 2020:72-74 XU Dongdong. Research on infrared and visible image fusion based on unsupervised deep learning[D]. Changchun:Chinese Academy of Sciences, 2020:72-74 (in Chinese)
[26] JAGALINGAM P, HEGDE A V. A review of quality metrics for fused image[J]. Aquatic Procedia, 2015, 4:133-142
[27] WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment:from error visibility to structural similarity[J]. IEEE Trans on Image Processing, 2004, 13(4):600-612
[28] ESKICIOGLU A M, FISHER P S. Image quality measures and their performance[J]. IEEE Trans on Communications, 1995, 43(12):2959-2965
[29] XYDEAS C S, PV V. Objective image fusion performance measure[J]. Military Technical Courier, 2000, 56(4):181-193 |
|
|
|
|
|
|
|
