|
|
论文:2021,Vol:39,Issue(2):423-429 |
|
|
引用本文： |
|
|
李海燕, 马艳, 郭磊, 李海江, 陈建华, 李红松. 基于双判别生成对抗网络的不规则孔洞图像修复[J]. 西北工业大学学报 |
|
|
LI Haiyan, MA Yan, GUO Lei, LI Haijiang, CHEN Jianhua, LI Hongsong. Image restoration for irregular holes based on dual discrimination generation countermeasure network[J]. Northwestern polytechnical university |
|
|
|
|
|
|
|
| 基于双判别生成对抗网络的不规则孔洞图像修复 |
|
| 李海燕1, 马艳1, 郭磊1, 李海江2, 陈建华1, 李红松1 |
|
1. 云南大学 信息学院, 云南 昆明 650000;
2. 云南交通投资建设集团有限公司, 云南 昆明 650000 |
| 摘要: |
| 为解决现有算法在修复随机、不规则大面积孔洞时出现色差和细节模糊的缺陷，提出了基于双判别生成对抗网络的不规则大孔洞图像修复网络架构。图像生成器是部分卷积定义的U-Net架构，归一化的部分卷积仅对有效像素完成端到端的掩码更新，U-Net中的跳过链接将图像的上下文信息向更高层分辨率传播，用重建损失、感知损失和风格损失的加权损失函数优化模型的训练结果；使用对抗性损失函数，单独训练包含合成判别器和全局判别器的双判别网络，判断生成图像与真实图像的一致性；加权所有损失函数，结合生成网络和双判别网络一起训练，进一步增强待修复区域的细节和整体一致性，使修复结果更自然。在Place365标准数据库上进行仿真实验，实验结果表明：提出方法在处理随机、不规则、大面积孔洞修复时，其结果的整体和细节语义一致性优于现有方法的结果，有效克服了细节模糊、颜色失真和出现伪影等缺陷。 |
| 关键词:
图像修复
随机不规则形状孔洞
部分卷积
合成判别器
全局判别器
|
|
| Image restoration for irregular holes based on dual discrimination generation countermeasure network |
|
| LI Haiyan1, MA Yan1, GUO Lei1, LI Haijiang2, CHEN Jianhua1, LI Hongsong1 |
|
1. School of Information Science and Engineering, Yunnan University, Kunming 650000, China;
2. Yunnan Communications Investment and Construction Group Co., Ltd, Kunming 650000, China |
| Abstract: |
| In order to solve the problem that the global and local generated countermeasure network cannot inpaint the random irregular large holes, and to improve the standard convolution generator, which demonstrates the defects of color difference and blur, a network architecture of inpainting irregular large holes in an image based on double discrimination generation countermeasure network is proposed. Firstly, the image generator is a U-net architecture defined by partial convolution. The normalized partial convolution only completes the end-to-end mask update for the effective pixels. The skip link in U-net propagates the context information of the image to the higher resolution, and optimizes the training results of the model with the weighted loss function of reconstruction loss, perception loss and wind grid loss. Subsequently, the adversary loss function, the dual discrimination network including the synthetic discriminator and the global discriminator are trained separately to judge the consistency between the generated image and the real image. Finally, the weighted loss functions are trained together with generating network and double discrimination network to further enhance the detail and overall consistency of the inpainted area and make the inpainted results more natural. The simulation experiment is carried out on the Place 365 standard database. The subjective and objective experimental results show that the results of the proposed method has reasonable overall and detail semantic consistency than those of the existing methods when they are used to repair random, irregular and large-area holes. The proposed method effectively overcomes the defects of blurry details, color distortion and artifacts. |
| Key words:
image restoration
random irregular shape hole
partial convolution
synthetic discriminator
global discriminator
|
|
| 收稿日期: 2020-07-01
修回日期:
|
| DOI: 10.1051/jnwpu/20213920423 |
| 基金项目: 云南省万人计划"教学名师"、云南省重大科技专项（2018ZF017）与国家自然科学基金（61861045）资助 |
| 通讯作者: 郭磊(1974-),云南大学教授,主要从事智能控制研究。e-mail:lei_guo@ynu.edu.cn
Email：lei_guo@ynu.edu.cn |
| 作者简介: 李海燕(1976-),女,云南大学教授、博士生导师,主要从事人工智能、图像处理研究。
|
|
| 相关功能 |
|
|
|
|
| 作者相关文章 |
|
| 李海燕 在本刊中的所有文章 |
| 马艳 在本刊中的所有文章 |
| 郭磊 在本刊中的所有文章 |
| 李海江 在本刊中的所有文章 |
| 陈建华 在本刊中的所有文章 |
| 李红松 在本刊中的所有文章 |
|
|
|
|
|
|
|
|
参考文献: |
|
|
[1] WEXLER Yonatan, SHECHTMAN Eli, IRANI Michal. Space-time completion of video[J]. IEEE Trans on Pattern Analysis and Machine Intelligence, 2007, 29(3):463-476
[2] BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. Patchmatch:a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics-TOG, 2009, 28(3):24:1-24:11
[3] HUANG Jiabin, KANG Singbing, NARENDRA Ahuja, et al. Image completion using planar structure guidance[J]. ACM Transactions on Graphics, 2014, 33(4):1-10
[4] SATOSHI Iizuka, EDGAR Simo-Serra, HIROSHI Ishikawa. Globally and Locally Consistent Image Completion[J]. ACM Transactions on Graphics, 2017, 36(4):1-14
[5] PATHAK Deepak, KRAHENBUHL Philipp, DONAHUE Jeff, et al. Context encoders:feature learning by inpainting[C]//Conference on Computer Vision and Pattern Recognition, 2016
[6] GOODFELLOW Ian, POUGET-ABADIE Jean, MIRZA Mehdi, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems, 2014
[7] CAI W, WEI Z. PiiGAN:generative adversarial networks for pluralistic image inpainting[J]. IEEE Access, 2020, 8(1):48451-48463
[8] LI Z, ZHU H, CAO L, et al. Face inpainting via nested generative adversarial networks[J]. IEEE Access, 2019, 7:1-1
[9] YU Jiahui, LIN Zhe, YANG Jimel, et al. Generative image inpainting with contextual attention[C]//Conference on Computer Vision and Pattern Recognition, 2018
[10] YAN Zhaoyi, LI Xiaoming, LI Mu, et al. Shift-net:image inpainting via deep feature rearrangement[C]//The European Conference on Computer Vision, 2018
[11] 刘波宁,翟东海. 基于双鉴别网络的生成对抗网络图像修复方法[J]. 计算机应用,2018,38(12):3557-3562, 3595 LIU Boning, ZHAI Donghai. Image completion method of generative adversarial networks[J]. Journal of Computer Application, 2018, 38(12):3557-3562, 3595(in Chinese)
[12] LIU Guilin, FITSUM A REDA, KEVIN Shih, et al. Image Inpainting for irregular holes using partial convolutions[C]//The European Conference on Computer Vision, 2018
[13] RONNEBERGER O, FISCHER P, BROX T. U-net:convolutional networks for biomedical image segmentation[C]//Medical Image Computing and Computer Assisted Intervention, 2015
[14] JOHNSON Justin, ALAHI Alexandre, LI Feifei. Perceptual losses for real-time style transfer and super-resolution[C]//The European Conference on Computer Vision, 2016:694-711
[15] GATYS L A, ECKER A S, BETHGE M, et al. Image style transfer using convolutional neural networks[C]//IEEE Conference on Computer Vision and Pattern Recognition, 2016
[16] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]//International Conference on Learning Representations, 2015
[17] RUDIN L I, OSHER S, FATEMI E. Nonlinear total variation based noise removal algorithms[J]. Physica D:Nonlinear Phenomena, 1992, 60(1):259-268 |
|
|
|
|
|
|
|
