Home »
Archives
Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.
Enhancing Reliability and Efficiency of Imperative Deep Learning Programs
Introduction
Efficiency is essential to support responsiveness w.r.t. ever-growing datasets, especially for Deep Learning (DL) systems. DL frameworks have traditionally embraced deferred execution-style DL code that supports symbolic, graph-based Deep Neural Network (DNN) computation. While scalable, such development tends to produce DL code that is error-prone, non-intuitive, and difficult to debug. Consequently, more natural, less error-prone imperative DL frameworks encouraging eager execution have emerged at the expense of run-time performance. In this project, we examine the challenges, tools, theories, and techniques involved in writing reliable and efficient imperative DL programs.
Researchers
Exploring Challenges Associated with Migrating Imperative Deep Learning Programs to Graph Execution
Introduction
While hybrid approaches aim for the “best of both worlds,” the challenges in applying them in the real world are largely unknown. We conduct a data-driven analysis of challenges—and resultant bugs—involved in writing reliable yet performant imperative DL code by studying 250 open-source projects, consisting of 19.7 MLOC, along with 470 and 446 manually examined code patches and bug reports, respectively. The results indicate that hybridization: (i) is prone to API misuse, (ii) can result in performance degradation—the opposite of its intention, and (iii) has limited application due to execution mode incompatibility. We put forth several recommendations, best practices, and anti-patterns for effectively hybridizing imperative DL code, potentially benefiting DL practitioners, API designers, tool developers, and educators.
Publications
My and my research students‘ names are boldfaced, undergraduate students are italicized, and female students are underlined:
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, and Anita Raja. Challenges in migrating imperative Deep Learning programs to graph execution: An empirical study. In International Conference on Mining Software Repositories, MSR ’22, pages 469–481, New York, NY, USA, May 2022. IEEE/ACM, ACM. (45/138; 32.6% acceptance rate). [ bib | DOI | arXiv | video | data | slides | poster | http ]
Study Data
Our dataset is hosted on Zenodo.
Blog Posts
Presentations
Slides
Posters
Videos
Automatically Migrating Imperative Deep Learning Programs to Graph Execution
Introduction
Though hybrid approaches aim for the “best of both worlds,” using them effectively requires subtle considerations. Our key insight is that, while DL programs typically execute sequentially, hybridizing imperative DL code resembles parallelizing sequential code in traditional systems. Inspired by this, we present an automated refactoring approach that assists developers in determining which otherwise eagerly-executed imperative DL functions could be effectively and efficiently executed as graphs. The approach features novel static imperative tensor and side-effect analyses for Python. Due to its inherent dynamism, analyzing Python may be unsound; however, the conservative approach leverages a speculative (keyword-based) analysis for resolving difficult cases that informs developers of any assumptions made. The approach is: (i) implemented as a plug-in to the PyDev Eclipse IDE that integrates the WALA Ariadne analysis framework and (ii) evaluated on nineteen DL projects consisting of 132 KLOC. The results show that 326 of 766 candidate functions (42.56%) were refactorable, and an average relative speedup of 2.16 on performance tests was observed with negligible differences in model accuracy. The results indicate that the approach is useful in optimizing imperative DL code to its full potential.
Publications
My and my research students‘ names are boldfaced, undergraduate students are italicized, and female students are underlined:
Raffi Khatchadourian, Tatiana Castro Vélez, Mehdi Bagherzadeh, Nan Jia, and Anita Raja. Speculative automated refactoring of imperative Deep Learning programs to graph execution. In International Conference on Automated Software Engineering, ASE ’25. IEEE/ACM, IEEE, November 2025. (245/1190; 20.6% acceptance rate). To appear. [ bib | DOI | arXiv | video | data | slides | poster | http ]
Raffi Khatchadourian, Tatiana Castro Vélez, Mehdi Bagherzadeh, Nan Jia, and Anita Raja. Hybridize Functions: A tool for automatically refactoring imperative Deep Learning programs to graph execution. In Artur Boronat and Gordon Fraser, editors, Fundamental Approaches to Software Engineering, FASE ’25, pages 89–100, Cham, May 2025. ETAPS, Springer Nature Switzerland. (11/31; 35% acceptance rate). EAPLS Distinguished Paper Award 🏆. [ bib | DOI | tool | slides | poster | http ]
Nan Jia, Anita Raja, and Raffi Khatchadourian. ReLESS: A framework for assessing safety in Deep Learning systems. In Workshop on Artificial Intelligence Safety at the International Joint Conference on Artificial Intelligence, AISafety ’24 at IJCAI ’24. IJCAI, August 2024. Best Paper Award 🏆 nominee. [ bib | http ]
Raffi Khatchadourian, Tatiana Castro Vélez, Mehdi Bagherzadeh, Nan Jia, and Anita Raja. Towards safe automated refactoring of imperative Deep Learning programs to graph execution. In International Conference on Automated Software Engineering, ASE ’23, pages 1800–1802. IEEE, September 2023. NIER track. (25/70; 35.7% acceptance rate). [ bib | DOI | slides | http ]
Research Prototype
Study Data
Our dataset is hosted on Zenodo.
Presentations
Slides
Posters
Video
Audio
Awards & Nominations
Funding
Software Evolution for Machine Learning
Introduction
In this project, we examine the evolution of Machine Learning systems, focusing on the refactorings (source-to-source semantics preserving program transformations) involved and the technical debt such systems incur.
Refactorings and Technical Debt in Machine Learning Systems
Introduction
Machine Learning (ML), including Deep Learning (DL), systems, i.e., those with ML capabilities, are pervasive in today’s data-driven society. Such systems are complex; they are comprised of ML models and many subsystems that support learning processes. As with other complex systems, ML systems are prone to classic technical debt issues, especially when such systems are long-lived, but they also exhibit debt specific to these systems. Unfortunately, there is a gap in knowledge about how ML systems actually evolve and are maintained. In this project, we fill this gap by studying refactorings, i.e., source-to-source semantics-preserving program transformations, performed in real-world, open-source software, and the technical debt issues they alleviate. We analyzed 26 projects, consisting of 4.2 MLOC, along with 327 manually examined code patches. The results indicate that developers refactor these systems for a variety of reasons, both specific and tangential to ML, some refactorings correspond to established technical debt categories, while others do not, and code duplication is a major cross-cutting theme that particularly involved ML configuration and model code, which was also the most refactored. We also introduce 14 and 7 new ML-specific refactorings and technical debt categories, respectively, and propose several recommendations, best practices, and anti-patterns. The results can potentially assist practitioners, tool developers, and educators in facilitating long-term ML system usefulness.
Researchers
Publications
My and my research students‘ names are boldfaced, undergraduate students are italicized, and female students are underlined:
Yiming Tang, Raffi Khatchadourian, Mehdi Bagherzadeh, Rhia Singh, Ajani Stewart, and Anita Raja. An empirical study of refactorings and technical debt in Machine Learning systems. In International Conference on Software Engineering, ICSE ’21, pages 238–250. IEEE/ACM, IEEE, May 2021. (138/615; 22% acceptance rate). [ bib | DOI | data | slides | http ]
