A Weighted Multi-k Extension of ReliefF for Robust Feature Selection

Doğan, Alican; Toçoğlu, Mansur

doi:10.22034/kps.2026.588386.1283

A Weighted Multi-k Extension of ReliefF for Robust Feature Selection

Document Type : Original Research Manuscripts

Authors

Alican Doğan ¹

Mansur Toçoğlu ²

¹ Assistant Professor, Department of Management Information Systems, Bandırma Onyedi Eylul University, Balıkesir, Turkey.

² Assistant Professor D Department of Computer Engineering, Katip Çelebi University, İzmir, Turkey.

10.22034/kps.2026.588386.1283

Abstract

This research proposes an improved version of ReliefF method for feature selection tasks in data mining. ReliefF-based algorithms are versatile and effective feature evaluators that identify conditional dependencies among features across instances, and they are commonly used in the preprocessing stage for tasks such as classification and regression. Since its introduction, numerous extensions of the ReliefF approach have been proposed to address redundancy, irrelevance, and noisy features, as well as the inherent limitations of the model in handling both binary and multi-class datasets. In this paper, we examine the effect of the usage of multiple number of neighbors in ReliefF algorithm concurrently with the help of standard majority voting and weighted majority voting strategies. The proposed method, is called Multi-k Weighted ReliefF (MW-ReliefF). It requires a set of different number of neighbors to calculate near hit and near miss values for each. Then it updates those values with respect to the weights of the predetermined number of nearest neighbors. The method is implemented as a preprocessing step for feature reduction before classifying 20 benchmark binary and multi-class datasets and classification performance results are obtained and compared to the counterparts of ReliefF method and its traditional variants in order to prove the validity of MW-ReliefF. In fact, the experimental findings demonstrated that the proposed algorithm outperformed the original ReliefF methods across almost all datasets.

Keywords

Feature selection

ReliefF

Classification

Data mining

Machine learning

Subjects

Data mining

Chen, J., Song, A., & Zhang, W. (2011). A novel hybrid gene selection approach based on ReliefF and FCBF. International Journal of Digital Content Technology and Its Applications, 5, 404-411.

Draper, B., Kaito, C., & Bins, J. (2003). Iterative Relief. In Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshop (pp. 62-62). Madison, WI, USA. https://doi.org/10.1109/CVPRW.2003.10040

Eppstein, M. J., & Haake, P. (2008). Very large scale ReliefF for genome-wide association analysis. In 2008 IEEE Symposium on Computational Intelligence in Bioinformatics and Computational Biology (pp. 112-119). Sun Valley, ID, USA. https://doi.org/10.1109/CIBCB.2008.4681570

Granizo-Mackenzie, D., & Moore, J. H. (2013). Multiple threshold spatially uniform ReliefF for the genetic analysis of complex human diseases. In L. Vanneschi, W. S. Bush, & M. Giacobini (Eds.), Evolutionary Computation, Machine Learning and Data Mining in Bioinformatics (EvoBIO 2013) (Lecture Notes in Computer Science, Vol. 7833, pp. 1-10). Berlin, Heidelberg: Springer. https://doi.org/10.1007/978-3-642-37195-0_1

Greene, C. S., Penrod, N. M., Kiralis, J., & Moore, J. H. (2009). Spatially uniform ReliefF (SURF) for computationally-efficient filtering of gene-gene interactions. BioData Mining, 2, 5. https://doi.org/10.1186/1756-0381-2-5

Karizaki, A. A., & Tavassoli, M. (2019). A novel hybrid feature selection based on ReliefF and binary dragonfly for high dimensional datasets. In 2019 9th International Conference on Computer and Knowledge Engineering (ICCKE) (pp. 300-304). Mashhad, Iran. https://doi.org/10.1109/ICCKE48569.2019.8964852

Kılıç, İ., & Yalçın, N. (2025). A novel hybrid methodology based on transfer learning, machine learning, and ReliefF for chickpea seed variety classification. Applied Sciences, 15(3), 1334. https://doi.org/10.3390/app15031334

Kira, K., & Rendell, L. A. (1992). A practical approach to feature selection. In D. Sleeman & P. Edwards (Eds.), Machine Learning Proceedings 1992 (pp. 249-256). San Francisco, CA: Morgan Kaufmann. https://doi.org/10.1016/B978-1-55860-247-2.50037-1

Kira, K., & Rendell, L. A. (1992). The feature selection problem: Traditional methods and a new algorithm. In Proceedings of the Tenth National Conference on Artificial Intelligence (AAAI-92) (pp. 129-134). Menlo Park, CA: AAAI Press.

Kohavi, R., & John, G. H. (1997). Wrappers for feature subset selection. Artificial Intelligence, 97(1-2), 273-324. https://doi.org/10.1016/S0004-3702(97)00043-X

Kononenko, I. (1994). Estimating attributes: Analysis and extensions of RELIEF. In F. Bergadano & L. De Raedt (Eds.), Machine Learning: ECML-94 (Lecture Notes in Computer Science, Vol. 784, pp. 171-182). Berlin, Heidelberg: Springer. https://doi.org/10.1007/3-540-57868-4_57

Kononenko, I., Šimec, E., & Robnik-Šikonja, M. (1997). Overcoming the myopia of inductive learning algorithms with RELIEFF. Applied Intelligence, 7, 39-55. https://doi.org/10.1023/A:1008280620621

Le, T. T., Urbanowicz, R. J., Moore, J. H., & McKinney, B. A. (2019). STatistical inference Relief (STIR) feature selection. Bioinformatics, 35(8), 1358-1365. https://doi.org/10.1093/bioinformatics/bty788

Liu, X., Zhang, Y., Zhang, L., & Yang, Y. (2020). A novel process monitoring method based on dynamic related ReliefF-SFA method. IEEE Access, 8, 41673-41683. https://doi.org/10.1109/ACCESS.2020.2977225

McKinney, B. A., Reif, D. M., White, B. C., Crowe, J. E. Jr., & Moore, J. H. (2007). Evaporative cooling feature selection for genotypic data involving interactions. Bioinformatics, 23(16), 2113-2120. https://doi.org/10.1093/bioinformatics/btm253

McKinney, B. A., White, B. C., Grill, D. E., Li, P. W., Kennedy, R. B., Poland, G. A., & Oberg, A. L. (2013). ReliefSeq: A gene-wise adaptive-k nearest-neighbor feature selection tool for finding gene-gene interactions and main effects in mRNA-Seq gene expression data. PLoS ONE, 8(12), e81527. https://doi.org/10.1371/journal.pone.0081527

Moore, J. H., & White, B. C. (2007). Tuning ReliefF for genome-wide genetic analysis. In E. Marchiori, J. H. Moore, & J. C. Rajapakse (Eds.), Evolutionary Computation, Machine Learning and Data Mining in Bioinformatics (EvoBIO 2007) (Lecture Notes in Computer Science, Vol. 4447, pp. 166-175). Berlin, Heidelberg: Springer. https://doi.org/10.1007/978-3-540-71783-6_15

Park, H., & Kwon, H. C. (2007). Extended Relief algorithms in instance-based feature filtering. In Proceedings of the Sixth International Conference on Advanced Language Processing and Web Information Technology (ALPIT 2007) (pp. 123-128). Luoyang, China. https://doi.org/10.1109/ALPIT.2007.18

Peker, M., Arslan, A., Şen, B., Çelebi, F. V., & But, A. (2015). A novel hybrid method for determining the depth of anesthesia level: Combining ReliefF feature selection and random forest algorithm (ReliefF+RF). In 2015 International Symposium on Innovations in Intelligent Systems and Applications (INISTA) (pp. 1-8). Madrid, Spain. https://doi.org/10.1109/INISTA.2015.7276748

Stokes, M. E., & Visweswaran, S. (2012). Application of a spatially-weighted Relief algorithm for ranking genetic predictors of disease. BioData Mining, 5, 20. https://doi.org/10.1186/1756-0381-5-20

Sun, L., Kong, X., Xu, J., Xue, Z., Zhai, R., & Zhang, S. (2019). A hybrid gene selection method based on ReliefF and ant colony optimization algorithm for tumor classification. Scientific Reports, 9(1), 8978. https://doi.org/10.1038/s41598-019-45442-9

Sun, Y. (2007). Iterative RELIEF for feature weighting: Algorithms, theories, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6), 1035-1051. https://doi.org/10.1109/TPAMI.2007.1093

Sun, Y., Todorovic, S., & Goodison, S. (2010). Local-learning-based feature selection for high-dimensional data analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9), 1610-1626. https://doi.org/10.1109/TPAMI.2009.187

Urbanowicz, R. J., Meeker, M., La Cava, W., Olson, R. S., & Moore, J. H. (2018). Relief-based feature selection: Introduction and review. Journal of Biomedical Informatics, 85, 189-203. https://doi.org/10.1016/j.jbi.2018.07.014

Uzun, B., & Ballı, S. (2022). A novel method for intrusion detection in computer networks by identifying multivariate outliers and ReliefF feature selection. Neural Computing and Applications, 34, 17647-17662. https://doi.org/10.1007/s00521-022-07490-3

Yang, Y., Chen, X., Zhang, Y., & Liu, X. (2019). A novel decentralized weighted ReliefF-PCA method for fault detection. IEEE Access, 7, 140478-140487. https://doi.org/10.1109/ACCESS.2019.2943772

Zhang, B., Li, Y., & Chai, Z. (2022). A novel random multi-subspace based ReliefF for feature selection. Knowledge-Based Systems, 252, 109400. https://doi.org/10.1016/j.knosys.2022.109400

Zhang, L. X., Wang, J. X., Zhao, Y. N., & Yang, Z. H. (2003). A novel hybrid feature selection algorithm: Using ReliefF estimation for GA-Wrapper search. In Proceedings of the 2003 International Conference on Machine Learning and Cybernetics (pp. 380-384). Xi'an, China. https://doi.org/10.1109/ICMLC.2003.1259764

Zhang, M., Wang, J. S., Hou, J. N., et al. (2023). RG-NBEO: A ReliefF guided novel binary equilibrium optimizer with opposition-based S-shaped and V-shaped transfer functions for feature selection. Artificial Intelligence Review, 56, 6509-6556. https://doi.org/10.1007/s10462-022-10317-0

Zhang, Y., Wang, S., Leng, Y., Shan, Z., Jin, A., Guo, Z., Lin, C., Zhang, J., & Zhang, S. (2023). A novel process monitoring method of heat treatment furnace based on ReliefF-SFA. Proceedings of SPIE, 12636, 126360H. https://doi.org/10.1117/12.2674302