|
|
论文:2023,Vol:41,Issue(4):784-793 |
|
|
引用本文： |
|
|
杨瀚森, 樊养余, 吕国云, 刘诗雅, 郭哲. 基于语义概念的图像情感分析[J]. 西北工业大学学报 |
|
|
YANG Hansen, FAN Yangyu, LYU Guoyun, LIU Shiya, GUO Zhe. Image emotion analysis based on semantic concepts[J]. Journal of Northwestern Polytechnical University |
|
|
|
|
|
|
|
| 基于语义概念的图像情感分析 |
|
| 杨瀚森1, 樊养余1, 吕国云1, 刘诗雅2, 郭哲1 |
|
1. 西北工业大学 电子信息学院, 陕西 西安 710072;
2. 虚拟现实内容制作中心, 北京 101318 |
| 摘要: |
| 随着越来越多的用户通过社交媒体表达自己的情感,图像情感分析技术受到了研究人员的密切关注。但是由于情感的模糊性和主观性,相比较于其他计算机视觉任务,图像情感分析更具挑战性。该领域既有的工作仅研究了图像到情感之间的直接映射关系。然而,心理学中有关情感感知的理论揭示了人们感知情感的过程是分步式的。因此,提出了一种新的图像情感分析框架,利用情感概念作为中级语义来辅助建立图像和情感之间的关系。将情感和概念的关系用知识图谱来描述并嵌入到语义空间中,再将图像的视觉特征投影至该语义空间与情感进行对齐,从而学习图像和情感之间的关系。另一方面,提出了一种多层次深度度量学习方法,从标记层面以及示例层面同时对模型进行优化。在2个情感图像数据集上进行实验,结果表明提出的方法在情感图像检索以及分类任务上,相对于现有方法表现良好。 |
| 关键词:
图像情感分析
知识图谱
视觉-语义嵌入
深度度量学习
|
|
| Image emotion analysis based on semantic concepts |
|
| YANG Hansen1, FAN Yangyu1, LYU Guoyun1, LIU Shiya2, GUO Zhe1 |
|
1. School of Electronics and Information, Northwestern Polytechnical University, Xi'an 710072, China;
2. Content Production Center of Virtual Reality, Beijing 101318, China |
| Abstract: |
| With the increasing number of users express their emotions via images on social media, image emotion analysis attracts much attention of researchers. For the ambiguity and subjectivity of emotion, image emotion analysis is more challenging than other computer vision tasks. Previous methods merely learn a direct mapping between image feature and emotion. However, in emotion perception theory of psychology, it is demonstrated that human beings perceive emotion in a stepwise way. Therefore, we propose a novel image emotion analysis framework that makes use of emotional concepts as middle-level feature to bridge image and emotion. Firstly, the relationship between the concept and the emotion is organized in the form of knowledge graph. The relation between the image and the emotion in the semantic embedding space is explored where the knowledge is encoded into. On the other hand, a multi-level deep metric learning method to optimize the model from both label level and instance level is proposed. Extensive experimental results on two image emotion datasets, demonstrate that the present approach performs favorably against the state-of-the-art methods on both affective image retrieval and classification tasks. |
| Key words:
image emotion analysis
knowledge graph
visual-semantic embedding
deep metric learning
|
|
| 收稿日期: 2022-09-20
修回日期:
|
| DOI: 10.1051/jnwpu/20234140784 |
| 基金项目: 国家自然科学基金面上项目(62071384)与陕西省重点研发计划(2023-YBGY-239)资助 |
| 通讯作者: 吕国云(1975—),西北工业大学副教授,主要从事音频、视频、图像处理及虚拟现实研究。e-mail:lvguoyun102@163.com
Email：lvguoyun102@163.com |
| 作者简介: 杨瀚森(1986—),西北工业大学博士研究生,主要从事机器学习、图像识别研究。
|
|
|
|
|
|
|
|
参考文献: |
|
|
[1] YOU Q, LUO J, JIN H, et al. Cross-modality consistent regression for joint visual-textual sentiment analysis of social multimedia[C]//Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, 2016: 13-22
[2] ZHANG Y, LAI G, ZHANG M, et al. Explicit factor models for explainable recommendation based on phrase-level sentiment analysis[C]//Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, 2014: 83-92
[3] ZHAO S, YAO X, YANG J, et al. Affective image content analysis: two decades review and new perspectives[J]. IEEE Trans on Pattern Analysis and Machine Intelligence, 2021, 44(10): 6729-6751
[4] OLKIEWICZ K A, MARKOWSKA-KACZMAR U. Emotion-based image retrieval-an artificial neural network approach[C]//Proceedings of the International Multi Conference on Computer Science and Information Technology, 2010: 89-96
[5] ZHANG H, YANG Z, GÖNEN M, et al. Affective abstract image classification and retrieval using multiple kernel learning[C]//International Conference on Neural Information Processing, 2013: 166-175
[6] ZHAO S, YAO H, YANG Y, et al. Affective image retrieval via multi-graph learning[C]//Proceedings of the 22nd ACM International Conference on Multimedia, 2014: 1025-1028
[7] YOU Q, LUO J, JIN H, et al. Robust image sentiment analysis using progressively trained and domain transferred deep networks[C]//Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015
[8] YANG J, SHE D, SUN M. Joint image emotion classification and distribution learning via deep convolutional neural network[C]//International Joint Conference on Artiticatl Intetttgcnce, 2017: 3266-3272
[9] YANG J, SHE D, LAI Y K, et al. Retrieving and classifying affective images via deep metric learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2018
[10] YAO X, SHE D, ZHANG H, et al. Adaptive deep metric learning for affective image retrieval and classification[J]. IEEE Trans on Multimedia, 2020, 23: 1640-1653
[11] YAO X, SHE D, ZHAO S, et al. Attention-aware polarity sensitive embedding for affective image retrieval[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 1140-1150
[12] YAO X, ZHAO S, LAI Y K, et al. APSE: Attention-aware polarity-sensitive embedding for emotion-based image retrieval[J]. IEEE Trans on Multimedia, 2020, 23: 4469-4482
[13] GOI M T, KALIDAS V, YUNUS N. Mediating roles of emotion and experience in the stimulus-organism-response framework in higher education institutions[J]. Journal of Marketing for Higher Education, 2018, 28(1): 90-112
[14] CHEN T, BORTH D, DARRELL T, et al. Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks[J/OL].(2014-10-30)[2022-08-15].https://arxiv.org/abs/1410.8586
[15] BORTH D, JI R, CHEN T, et al. Large-scale visual sentiment ontology and detectors using adjective noun pairs[C]//Proceedings of the 21st ACM International Conference on Multimedia, 2013: 223-232
[16] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778
[17] GROVER A, LESKOVEC J. Node2vec: scalable feature learning for networks[C]//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016: 855-864
[18] XU H, QI G, LI J, et al. Fine-grained image classification by visual-semantic embedding[C]//International Joint Conference on Artiticatl Intetttgcnce, 2018: 1043-1049
[19] YOU Q, LUO J, JIN H, et al. Building a large scale dataset for image emotion recognition: the fine print and the benchmark[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2016
[20] MIKELS J A, FREDRICKSON B L, LARKIN G R, et al. Emotional category data on images from the International Affective Picture System[J]. Behavior research methods, 2005, 37(4): 626-630
[21] PENG K C, CHEN T, SADOVNIK A, et al. A mixed bag of emotions: model, predict, and transfer emotion distributions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 860-868
[22] EKMAN P. What emotion categories or dimensions can observers judge from facial behavior?[J/OL].(2013-11-17)[2022-08-15]. https://www.sciencedirect.com/science/article/abs/pii/B9780080166438500252
[23] GAO Y, WANG M, TAO D, et al. 3-D object retrieval and recognition with hypergraph analysis[J]. IEEE Trans on Image Processing, 2012, 21(9): 4290-4303
[24] MANJUNATH B S, OHM J R, VASUDEVAN V V, et al. Color and texture descriptors[J]. IEEE Trans on Circuits and Systems for Video Technology, 2001, 11(6): 703-715
[25] DENG J, DONG W, SOCHER R, et al. Imagenet: a large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009: 248-255
[26] SCHROFF F, KALENICHENKO D, PHILBIN J. Facenet: A unified embedding for face recognition and clustering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 815-823
[27] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Imagenet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 83-90
[28] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]//3rd International Conference on Learning Representations, San Diego, 2015
[29] VAN DER MAATEN L, HINTON G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9(11): 2579-2605 |
|
|
|
|
|
|
|
