Explainable Multimodal Knowledge Processing for Digital Media Integrity Risk Assessment under Uncertainty

MATHEW, ROSE MARY; Joy, Aqueena

doi:10.22034/kps.2026.599041.1304

Explainable Multimodal Knowledge Processing for Digital Media Integrity Risk Assessment under Uncertainty

Document Type : Original Research Manuscripts

Authors

ROSE MARY MATHEW ¹

Aqueena Joy ²

¹ Associate Professor, Department of Computer Applications, Federal Institute of Science and Technology, Angamaly, Kerala.

² PG Student, Department of Computer Applications, Federal Institute of Science and Technology, Kerala, India.

10.22034/kps.2026.599041.1304

Abstract

The rapid growth of social media and generative artificial intelligence has increased the prevalence of misinformation and manipulated digital media, creating challenges for reliable content verification. This study presents a soft computing-based explainable multimodal framework for digital media integrity risk assessment under uncertainty. The framework integrates transformer-based textual credibility and emotion analysis, OCR-assisted text extraction, CLIP-based image–text semantic consistency, and ResNet50-based visual integrity assessment. The resulting evidence is integrated through a weight-adjusted Mamdani fuzzy inference mechanism to produce continuous integrity-risk scores and graded risk levels. SHAP and Grad-CAM provide supporting textual and visual explanations. On the benchmark test set, the framework achieved 95.6% accuracy and 95.1% F1-score, with a ROC-AUC of 0.97, indicating strong threshold-independent discrimination under the evaluated test conditions. Ablation and sensitivity analyses demonstrated performance changes associated with the integrated components and fusion configuration. Five repeated runs showed limited performance variation under the evaluated protocol. External validation on approximately 1,500 previously unseen multimodal samples achieved a 92.9% F1-score, indicating performance variation under a different data distribution. The framework provides an interpretable, uncertainty-aware approach to multimodal digital media integrity assessment and supports graded risk interpretation rather than relying solely on binary classification.

Keywords

Digital media integrity

Fuzzy inference

Misinformation detection

Multimodal learning

Risk assessment

Uncertainty.

Subjects

Intelligent agents

Comito, C., Caroprese, L., & Zumpano, E. (2023). Multimodal fake news detection on social media: A survey of deep learning techniques. Social Network Analysis and Mining, 13, 101. https://doi.org/10.1007/s13278-023-01104-w.

Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., & Ravi, S. (2020). GoEmotions: A dataset of fine-grained emotions. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4040–4054. https://doi.org/10.18653/v1/2020.acl-main.372

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4171–4186. https://doi.org/10.18653/v1/N19-1423

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778. https://doi.org/10.1109/CVPR.2016.90

Jin, R., Fu, R., Wen, Z., Zhang, S., Liu, Y., & Tao, J. (2024). Fake news detection and manipulation reasoning via large vision-language models. arXiv preprint arXiv:2407.02042.

Khattar, D., Goud, J. S., Gupta, M., & Varma, V. (2019). MVAE: Multimodal variational autoencoder for fake news detection. Proceedings of the 2019 World Wide Web Conference, 2915–2921. https://doi.org/10.1145/3308558.3313552

Li, Q., Gao, M., Zhang, G., Zhai, W., Chen, J., & Jeon, G. (2024). Towards multimodal disinformation detection by vision-language knowledge interaction. Information Fusion, 102, 102037. https://doi.org/10.1016/j.inffus.2023.102037

Liu, X., Li, Z., Li, P., Huang, H., Xia, S., Cui, X., Huang, L., Deng, W., & He, Z. (2025). MMFakeBench: A mixed-source multimodal misinformation detection benchmark for LVLMs. International Conference on Learning Representations (ICLR 2025).

Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 4765–4774.

Mamdani, E. H., & Assilian, S. (1975). An experiment in linguistic synthesis with a fuzzy logic controller. International Journal of Man-Machine Studies, 7(1), 1–13.

Nakamura, K., Levy, S., & Wang, W. Y. (2020). Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. Proceedings of the Twelfth Language Resources and Evaluation Conference, 6149–6157.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning, 139, 8748–8763.

Sagar, A. S. M., Bennamoun, M., Boussaïd, F., Sharif, N., Xu, L., Sahmoud, S., & Kishk, A. (2026). Fact or fake? Assessing the role of deepfake detectors in multimodal misinformation detection. IEEE Access, 14, 86992–87011. https://doi.org/10.1109/ACCESS.2026.3686437

Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, 618–626. https://doi.org/10.1109/ICCV.2017.74

Shu, K., Mahudeswaran, D., Wang, S., Lee, D., & Liu, H. (2020). FakeNewsNet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big Data, 8(3), 171–188. https://doi.org/10.1089/big.2020.0062

Singhal, S., Kabra, A., Sharma, M., Shah, R. R., Chakraborty, T., & Kumaraguru, P. (2020). SpotFake+: A multimodal framework for fake news detection via transfer learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(10), 13915–13916. https://doi.org/10.1609/aaai.v34i10.7230

Singhal, S., Shah, R. R., Chakraborty, T., Kumaraguru, P., & Satoh, S. (2019). SpotFake: A multimodal framework for fake news detection. 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), 39–47. https://doi.org/10.1109/BigMM.2019.00-44

Tahmasebi, S., Müller-Budack, E., & Ewerth, R. (2024). Multimodal misinformation detection using large vision-language models. Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM 2024), 2189–2199. https://doi.org/10.1145/3627673.3679826

Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., & Ortega-Garcia, J. (2020). DeepFakes and beyond: A survey of face manipulation and fake detection. Information Fusion, 64, 131–148.

Vosoughi, S., Roy, D., & Aral, S. (2018). The spread of true and false news online. Science, 359(6380), 1146–1151. https://doi.org/10.1126/science.aap9559

Wang, W. Y. (2017). “Liar, liar pants on fire”: A new benchmark dataset for fake news detection. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 422–426. https://doi.org/10.18653/v1/P17-2067

Yan, Z., Qi, P., Hsu, W., & Lee, M.-L. (2025). TRUST-VL: An explainable news assistant for general multimodal misinformation detection. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 5588–5604. https://doi.org/10.18653/v1/2025.emnlp-main.284

Yang, P., Ma, J., Liu, Y., & Liu, M. (2023). Multi-modal transformer for fake news detection. Mathematical Biosciences and Engineering, 20(9), 16115–16134.

Yang, Y., Yang, Y., Ying, Q., Qian, Z., & Zhang, X. (2023). Multi-modal fake news detection on social media via multi-grained information fusion. arXiv preprint arXiv:2304.00827.

Zadeh, L. A. (1965). Fuzzy sets. Information and Control, 8(3), 338–353. https://doi.org/10.1016/S0019-9958(65)90241-X