From Signal to Noise: How Class Imbalance Erodes Trust in Credit-Scoring Explanations
DOI:
https://doi.org/10.62270/jirms.v7i3.165Keywords:
Explainable AI, Credit Scoring, Class Imbalance, SHAP, LIME, Cross-Country Comparison, Machine LearningAbstract
Purpose—This study examines whether SHAP and LIME (SHapley Additive exPlanations and Local Interpretable Model-agnostic Explanations) produce stable explanations as class imbalance increases in credit scoring, with imbalance as the independent condition and stability as the outcome, tested across two credit markets.
Study Design/methodology/approach—This paper replicates and extends Chen, Calabrese, and Martin-Barragan (2024), adding a third model and a second country. XGBoost, Random Forest, and LightGBM are trained on Taiwan and LendingClub USA (2007-2018) data across eleven imbalance levels, 1% to 50% default rate. SHAP and LIME explanations are generated for a fixed, seeded subsample of 50 held-out observations. Stability is measured via Sequential Rank Agreement (SRA) and Coefficient of Variation (CV) across B = 50 bootstrap iterations, with 95% confidence intervals and Friedman/Wilcoxon tests reported.
Findings— Explanation stability declines sharply as imbalance increases on the Taiwan dataset, confirming Chen et al. (2024). LIME deteriorates most severely: at a 1% default rate, its rank agreement collapses to statistical noise for all three models (SRA as low as 0.038 for XGBoost, 95% CI −0.26 to 0.29), a severity not previously reported. SHAP's lowest stability is SRA = 0.360 (LightGBM, 1%). CV rises rather than falls as balance improves, reflecting bootstrap variation, not instability. USA results follow the same direction but remain more stable.
Practical Implications— Financial institutions using SHAP or LIME for GDPR Article 22 or Equal Credit Opportunity Act compliance should treat explanations below a 5% default rate as tentative and adopt stability testing as a standard part of model validation.
Originality/Value— This study makes three contributions: the first open-data, cross-country replication of Chen et al. (2024); the first report that LIME's feature-ranking agreement collapses to statistical noise under extreme class imbalance, rather than degrading gradually; and the first test of LightGBM's explanation stability alongside XGBoost and Random Forest.
References
Ariza-Garzón, M. J., Sanz-Guerrero, M., & Arroyo Gallardo, J. (2024). Lending club loan dataset for granting models. Lending Club loan dataset for granting models. https://doi.org/https://doi.org/10.5281/zenodo.11295916
Ballegeer, M., Bogaert, M., & Benoit, D. F. (2025). Evaluating the stability of model explanations in instance-dependent cost-sensitive credit scoring. European Journal of Operational Research, 326(3), 630-640. https://doi.org/10.1016/j.ejor.2025.05.039 DOI: https://doi.org/10.1016/j.ejor.2025.05.039
Breiman, L. (2001). Random forests. Machine learning, 45(1), 5-32. https://doi.org/https://doi.org/10.1023/A:1010933404324 DOI: https://doi.org/10.1023/A:1010933404324
Brown, I., & Mues, C. (2012). An experimental comparison of classification algorithms for imbalanced credit scoring data sets. Expert systems with applications, 39(3), 3446-3453. https://doi.org/10.1016/j.eswa.2011.09.033 DOI: https://doi.org/10.1016/j.eswa.2011.09.033
Bueff, A. C., Cytryński, M., Calabrese, R., Jones, M., Roberts, J., Moore, J., & Brown, I. (2022). Machine learning interpretability for a stress scenario generation in credit scoring based on counterfactuals. Expert systems with applications, 202, 117271. https://doi.org/10.1016/j.eswa.2022.117271 DOI: https://doi.org/10.1016/j.eswa.2022.117271
Bussmann, N., Giudici, P., Marinelli, D., & Papenbrock, J. (2021). Explainable Machine Learning in Credit Risk Management: N. Bussmann et al. Computational Economics, 57(1), 203-216. https://doi.org/https://doi.org/10.1007/s10614-020-10042-0 DOI: https://doi.org/10.1007/s10614-020-10042-0
Chan, C.-P., Tsai, C.-H., Tang, F.-K., & Yang, J.-H. (2025). A SHAP-based comparative analysis of machine learning model interpretability in financial classification tasks. Journal of Applied Economic Sciences, 20(3), 385-400. https://doi.org/https://doi.org/10.57017/jaes.v20.3(89).03 DOI: https://doi.org/10.57017/jaes.v20.3(89).03
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16, 321-357. https://doi.org/https://doi.org/10.1613/jair.953 DOI: https://doi.org/10.1613/jair.953
Chen, T., & Guestrin, C. (2016). Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, DOI: https://doi.org/10.1145/2939672.2939785
Chen, Y., Calabrese, R., & Martin-Barragan, B. (2024). Interpretable machine learning for imbalanced credit scoring datasets. European Journal of Operational Research, 312(1), 357-372. https://doi.org/10.1016/j.ejor.2023.06.036 DOI: https://doi.org/10.1016/j.ejor.2023.06.036
Dastile, X., Celik, T., & Potsane, M. (2020). Statistical and machine learning models in credit scoring: A systematic literature survey. Applied Soft Computing, 91, 106263. https://doi.org/10.1016/j.asoc.2020.106263 DOI: https://doi.org/10.1016/j.asoc.2020.106263
Diakopoulos, N. (2016). Accountability in algorithmic decision making. Communications of the ACM, 59(2), 56-62. https://doi.org/10.1145/2844110 DOI: https://doi.org/10.1145/2844110
Dong, B., Zhang, X., Hong, Z., Yan, C., Zhu, W., Hou, L., & Feng, Y. (2026). Explainable Deep Tabular Learning for Credit Risk Assessment: An Information-Theoretic Cross-Attentional Transformer Approach. Entropy, 28(8), 837. https://doi.org/https://doi.org/10.3390/e28080837 DOI: https://doi.org/10.3390/e28080837
Han, H., Wang, W.-Y., & Mao, B.-H. (2005). Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning. International conference on intelligent computing, DOI: https://doi.org/10.1007/11538059_91
Hand, D. J. (2009). Measuring classifier performance: a coherent alternative to the area under the ROC curve. Machine learning, 77(1), 103-123. https://doi.org/https://doi.org/10.1007/s10994-009-5119-5 DOI: https://doi.org/10.1007/s10994-009-5119-5
He, H., Bai, Y., Garcia, E. A., & Li, S. (2008). ADASYN: Adaptive synthetic sampling approach for imbalanced learning. 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence),
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30.
Kozodoi, N., Lessmann, S., Alamgir, M., Moreira-Matias, L., & Papakonstantinou, K. (2025). Fighting sampling bias: A framework for training and evaluating credit scoring models. European Journal of Operational Research, 324(2), 616-628. https://doi.org/10.1016/j.ejor.2025.01.040 DOI: https://doi.org/10.1016/j.ejor.2025.01.040
Krawczyk, B. (2016). Learning from imbalanced data: open challenges and future directions. Progress in artificial intelligence, 5(4), 221-232. https://doi.org/https://doi.org/10.1007/s13748-016-0094-0 DOI: https://doi.org/10.1007/s13748-016-0094-0
Lazar, A., Pokhrel, P., & Das, S. (2026). Beyond accuracy: A comprehensive comparative study of gradient boosting versus tabular deep learning and explainability techniques for mixed-type tabular data models using SHAP and LIME. International Journal on Artificial Intelligence Tools, 35(02), 2640003. https://doi.org/https://doi.org/10.1142/S0218213026400038 DOI: https://doi.org/10.1142/S0218213026400038
Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human factors, 46(1), 50-80. https://doi.org/https://doi.org/10.1518/hfes.46.1.50_30392 DOI: https://doi.org/10.1518/hfes.46.1.50.30392
LendingClub Dataset. (2007-2018). LendingClub Dataset Full 2007 to 2018 (Kaggle. https://www.kaggle.com/datasets/panchammahto/lendingclub-dataset-full-2007-to-2018
Liu, W., Fan, H., & Xia, M. (2022). Credit scoring based on tree-enhanced gradient boosting decision trees. Expert systems with applications, 189, 116034. https://doi.org/10.1016/j.eswa.2021.116034 DOI: https://doi.org/10.1016/j.eswa.2021.116034
Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions Advances in neural information processing systems,
Machado, M. R., & Karray, S. (2022). Assessing credit risk of commercial customers using hybrid machine learning algorithms. Expert systems with applications, 200, 116889. https://doi.org/10.1016/j.eswa.2022.116889 DOI: https://doi.org/10.1016/j.eswa.2022.116889
Misheva, B. H., Osterrieder, J., Hirsa, A., Kulkarni, O., & Lin, S. F. (2021). Explainable AI in credit risk management. arXiv preprint arXiv:2103.00949. https://doi.org/https://doi.org/10.48550/arXiv.2103.00949
Moscato, V., Picariello, A., & Sperlí, G. (2021). A benchmark of machine learning approaches for credit score prediction. Expert systems with applications, 165, 113986. https://doi.org/10.1016/j.eswa.2020.113986 DOI: https://doi.org/10.1016/j.eswa.2020.113986
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). " Why should i trust you?" Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, San Francisco, California, USA. DOI: https://doi.org/10.1145/2939672.2939778
Stiglitz, J. E., & Weiss, A. (1981). Credit rationing in markets with imperfect information. The American economic review, 71(3), 393-410. https://doi.org/https://www.jstor.org/stable/1802787
Vapnik, V. N. (1998). Statistical learning theory {Adaptive and learning systems for signal processing, communications, and control}. Wiley and Sons.
Visani, G., Bagli, E., Chesani, F., Poluzzi, A., & Capuzzo, D. (2022). Statistical stability indices for LIME: Obtaining reliable explanations for machine learning models. Journal of the Operational Research Society, 73(1), 91-101. https://doi.org/https://doi.org/10.1080/01605682.2020.1865846 DOI: https://doi.org/10.1080/01605682.2020.1865846
Wang, Z., Tian, Y., Li, S., & Xiao, J. (2025). A secure cross-silo collaborative method for imbalanced credit scoring. European Journal of Operational Research, 326(2), 357-373. https://doi.org/10.1016/j.ejor.2025.04.020 DOI: https://doi.org/10.1016/j.ejor.2025.04.020
Yeh, I.-C., & Lien, C.-h. (2009). The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert systems with applications, 36(2), 2473-2480. https://doi.org/10.1016/j.eswa.2007.12.020 DOI: https://doi.org/10.1016/j.eswa.2007.12.020
Zhu, J., Zhang, M., Zhang, X., & Yu, L. (2026). Triplet guided dual-discriminator filtering network for extremely imbalanced credit classification. International Review of Financial Analysis, 105474. https://doi.org/https://doi.org/10.1016/j.irfa.2026.105474 DOI: https://doi.org/10.1016/j.irfa.2026.105474
Downloads
Published
Data Availability Statement
The data that support the findings of this study are openly available. The Taiwan Credit Card Dataset is available from the UCI Machine Learning Repository at https://archive.ics.uci.edu/dataset/350. The LendingClub USA Dataset is available via Zenodo at https://zenodo.org/records/11295916.
Issue
Section
License
Copyright (c) 2026 Abdullah Tufail, Muhammad Khalid Sohail
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution (CC-BY) 4.0 License that allows others to share the work with an acknowledgment of the work’s authorship and initial publication in this journal.
How to Cite
