Identification and Measurement of Self-Technical Debt in Deep Learning Frameworks: A Systematic Review
Abstract
Technical Debt in software development refers to the consequences of decisions prioritizing quick solutions over optimal ones. This concept, introduced by Ward Cunningham in 1992, has been widely studied to improve software quality. In the context of deep learning, Technical Debt is also present due to the use of tools that, while facilitating model creation, may generate debt and negatively impact performance.
Through a three-phase process, this study presents a systematic literature review to identify the types of Technical Debt found in deep learning tools and the techniques used for its identification and measurement. The reviewed studies show that Technical Debt can arise in various development phases, such as design, requirements definition, testing, documentation, source code, algorithms, and compatibility. Other affected aspects include data, models, knowledge, and infrastructure. Several approaches have been used to identify technical debt, such as analyzing comments in static code, pull requests, and commits, applying manual techniques, text mining, neural networks, and natural language processing algorithms. In terms of measurement, statistical methods are predominantly used.
The findings of this review provide a better understanding of how Technical Debt impacts deep learning tools and offer a foundation for guiding future research on its management and mitigation in the development of systems within intelligent environments.
Downloads
References
R. Y. Choi, A. S. Coyner, J. Kalpathy-Cramer, M. F. Chiang, and J. Peter Campbell, “Introduction to machine learning, neural networks, and deep learning,” Transl Vis Sci Technol, vol. 9, no. 2, 2020, doi: 10.1167/tvst.9.2.14.
R. Elshawi, A. Wahab, A. Barnawi, and S. Sakr, “DLBench: a comprehensive experimental evaluation of deep learning frameworks,” Cluster Comput, vol. 24, no. 3, pp. 2017–2038, Sep. 2021, doi: 10.1007/s10586-021-03240-4.
C. Janiesch, P. Zschech, and K. Heinrich, “Machine learning and deep learning,” Electronic Markets, vol. 31, no. 3, pp. 685–695, Sep. 2021, doi: 10.1007/s12525-021-00475-2.
M. H. M. Noor and A. O. Ige, “A Survey on State-of-the-art Deep Learning Applications and Challenges,” Mar. 2024.
N. L. Rane, S. K. Mallick, Ö. Kaya, and J. Rane, “Tools and frameworks for machine learning and deep learning: A review,” in Applied Machine Learning and Deep Learning: Architectures and Techniques, Deep Science Publishing, 2024. doi: 10.70593/978-81-981271-4-3_4.
C. Yang, P. Liang, L. Fu, and Z. Li, “Self-Claimed Assumptions in Deep Learning Frameworks: An Exploratory Study,” in Evaluation and Assessment in Software Engineering, New York, NY, USA: ACM, Jun. 2021, pp. 139–148. doi: 10.1145/3463274.3463333.
D. Sculley, D. Holt, E. Davydov, and T. Phillips, “Hidden technical debt in machine learning systems,” Adv Neural Inf Process Syst, Jan. 2015.
D. OBrien, S. Biswas, S. Imtiaz, R. Abdalkareem, E. Shihab, and H. Rajan, “23 shades of self-admitted technical debt: an empirical study on machine learning software,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, New York, NY, USA: ACM, Nov. 2022, pp. 734–746. doi: 10.1145/3540250.3549088.
P. Brereton, B. A. Kitchenham, D. Budgen, M. Turner, and M. Khalil, “Lessons from applying the systematic literature review process within the software engineering domain,” Journal of Systems and Software, vol. 80, no. 4, pp. 571–583, Apr. 2007, doi: 10.1016/j.jss.2006.07.009.
B. Kitchenham and P. Brereton, “A systematic review of systematic review process research in software engineering,” Inf Softw Technol, vol. 55, no. 12, pp. 2049–2075, Dec. 2013, doi: 10.1016/j.infsof.2013.07.010.
S. Dong, P. Wang, and K. Abbas, “A survey on deep learning and its applications,” Comput Sci Rev, vol. 40, p. 100379, May 2021, doi: 10.1016/j.cosrev.2021.100379.
S. Shamshirband, M. Fathi, A. Dehzangi, A. T. Chronopoulos, and H. Alinejad-Rokny, “A review on deep learning approaches in healthcare systems: Taxonomies, challenges, and open issues,” J Biomed Inform, vol. 113, p. 103627, Jan. 2021, doi: 10.1016/j.jbi.2020.103627.
C. Aracena, F. Villena, F. Arias, and J. Dunstan, “Applications of machine learning in healthcare,” Revista Medica Clinica Las Condes, vol. 33, no. 6, pp. 568–575, Nov. 2022, doi: 10.1016/j.rmclc.2022.10.001.
C. Yang, P. Liang, and Z. Ma, “An exploratory study on automatic identification of assumptions in the development of deep learning frameworks,” Sci Comput Program, vol. 240, p. 103218, Feb. 2024, doi: 10.1016/j.scico.2024.103218.
J. Liu, Q. Huang, X. Xia, E. Shihab, D. Lo, and S. Li, “Is using deep learning frameworks free?,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Society, New York, NY, USA: ACM, Jun. 2020, pp. 1–10. doi: 10.1145/3377815.3381377.
A. Bhatia, F. Khomh, B. Adams, and A. E. Hassan, “An Empirical Study of Self-Admitted Technical Debt in Machine Learning Software,” Nov. 2024.
E. Nascimento, A. Nguyen-Duc, I. Sundbø, and T. Conte, “Software engineering for artificial intelligence and machine learning software: A systematic literature review,” Nov. 2020.
A. Serban, K. van der Blom, H. Hoos, and J. Visser, “Software engineering practices for machine learning — Adoption, effects, and team assessment,” Journal of Systems and Software, vol. 209, p. 111907, Mar. 2024, doi: 10.1016/j.jss.2023.111907.
A. Potdar and E. Shihab, “An Exploratory Study on Self-Admitted Technical Debt,” in 2014 IEEE International Conference on Software Maintenance and Evolution, IEEE, Sep. 2014, pp. 91–100. doi: 10.1109/ICSME.2014.31.
E. da S. Maldonado and E. Shihab, “Detecting and quantifying different types of self-admitted technical Debt,” in 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), IEEE, Oct. 2015, pp. 9–15. doi: 10.1109/MTD.2015.7332619.
J. Liu, Q. Huang, X. Xia, E. Shihab, D. Lo, and S. Li, “An exploratory study on the introduction and removal of different types of technical debt in deep learning frameworks,” Empir Softw Eng, vol. 26, no. 2, p. 16, Mar. 2021, doi: 10.1007/s10664-020-09917-5.
Y. Tang, R. Khatchadourian, M. Bagherzadeh, R. Singh, A. Stewart, and A. Raja, “An Empirical Study of Refactorings and Technical Debt in Machine Learning Systems,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), IEEE, May 2021, pp. 238–250. doi: 10.1109/ICSE43902.2021.00033.
Y. Tang, R. Khatchadourian, M. Bagherzadeh, R. Singh, A. Stewart, and A. Raja, “An Empirical Study of Refactorings and Technical Debt in Machine Learning Systems,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), IEEE, May 2021, pp. 238–250. doi: 10.1109/ICSE43902.2021.00033.
F. Pepe, F. Zampetti, A. Mastropaolo, G. Bavota, and M. Di Penta, “A Taxonomy of Self-Admitted Technical Debt in Deep Learning Systems,” Sep. 2024.
C. Liu, R. Cai, Y. Zhou, X. Chen, H. Hu, and M. Yan, “Understanding the implementation issues when using deep learning frameworks,” Inf Softw Technol, vol. 166, p. 107367, Feb. 2024, doi: 10.1016/j.infsof.2023.107367.
Z. Liu, Q. Huang, X. Xia, E. Shihab, D. Lo, and S. Li, “SATD detector,” in Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, New York, NY, USA: ACM, May 2018, pp. 9–12. doi: 10.1145/3183440.3183478.
R. Dyer, H. A. Nguyen, H. Rajan, and T. N. Nguyen, “Boa:Ultra-Large-Scale Software Repository and Source-Code Mining,” ACM Transactions on Software Engineering and Methodology, vol. 25, no. 1, pp. 1–34, Dec. 2015, doi: 10.1145/2803171.
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations,” Sep. 2019.
O. S. Ekundayo and A. E.-S. Ezugwu, “Deep Learning: Historical Overview from Inception to Actualization, Models, Applications and Future Trends,” May 07, 2024. doi: 10.36227/techrxiv.171504655.52310452/v1.
G. Giray, “A software engineering perspective on engineering machine learning systems: State of the art and challenges,” Journal of Systems and Software, vol. 180, p. 111031, Oct. 2021, doi: 10.1016/j.jss.2021.111031.
- Conceptualization
- Data curation
- Formal Analysis
- Investigation
- Methodology
- Project administration
- Software
- Validation
- Visualization
- Writing - original draft
- Writing - review & editing
- Conceptualization
- Data curation
- Formal Analysis
- Investigation
- Methodology
- Project administration
- Resources
- Supervision
- Validation
- Visualization
- Writing - review & editing
- Conceptualization
- Formal Analysis
- Methodology
- Supervision
- Validation
- Visualization
- Writing - review & editing
- Conceptualization
- Formal Analysis
- Methodology
- Supervision
- Validation
- Visualization
- Writing - review & editing
Copyright (c) 2026 Innovation and Software
This work is licensed under a Creative Commons Attribution 4.0 International License.
The authors exclusively grant the right to publish their article to the Innovation and Software Journal, which may formally edit or modify the approved text to comply with their own editorial standards and with universal grammatical standards, prior to publication; Likewise, our journal may translate the approved manuscripts into as many languages as it deems necessary and disseminates them in several countries, always giving public recognition to the author or authors of the research.
