How Do Model Export Formats Impact the Development of ML-Enabled Systems? A Case Study on Model Integration
This program is tentative and subject to change.
Machine learning (ML) models have greatly improved their predictive and generative capabilities in recent years. They are therefore often integrated into ML-enabled systems to provide software functionality that would otherwise be impossible. This integration requires the selection of an appropriate ML model export format, for which many options are available. These formats are crucial for ensuring a seamless integration, and choosing a suboptimal one can negatively impact system development, e.g., via increased dependencies and higher maintenance costs. However, little evidence is available to guide practitioners during the export format selection.
We therefore aim to comprehensively evaluate various model export formats regarding their impact on the development of ML-enabled systems from an integration perspective. Based on the results of a preliminary questionnaire survey (n=17), we designed an extensive embedded case study with two ML-enabled systems in three versions with different technologies. We then analyzed the effect of five popular export formats, namely ONNX, Pickle, TensorFlow’s SavedModel, PyTorch’s TorchScript, and Joblib. In total, we studied 30 units of analysis (2 systems * 3 tech stacks * 5 formats) and collected data via structured field notes.
The holistic qualitative analysis of the results indicated that ONNX provided the most efficient integration across most cases, which shows its great flexibility and portability. SavedModel and TorchScript were very convenient to use in Python-based systems, but otherwise required workarounds (TorchScript more than SavedModel). The TensorFlow format also allowed the easy incorporation of preprocessing logic into a single file, which made it scalable for complex deep learning use cases. Pickle and Joblib were the most challenging to integrate, even in Python-based systems. Regarding technical support, all model export formats demonstrated excellent documentation quality and strong community support across platforms such as Stack Overflow and Reddit. Practitioners can use our findings to inform the selection of ML export formats suited to their context.
This program is tentative and subject to change.
Sun 27 AprDisplayed time zone: Eastern Time (US & Canada) change
11:00 - 12:30 | |||
11:00 15mTalk | How Do Model Export Formats Impact the Development of ML-Enabled Systems? A Case Study on Model Integration Research and Experience Papers Shreyas Kumar Parida ETH Zurich, Ilias Gerostathopoulos Vrije Universiteit Amsterdam, Justus Bogner Vrije Universiteit Amsterdam Pre-print | ||
11:15 15mTalk | MLScent: A tool for Anti-pattern detection in ML projects Research and Experience Papers | ||
11:30 15mTalk | RAGProbe: Breaking RAG Pipelines with Evaluation Scenarios Research and Experience Papers Shangeetha Sivasothy Applied Artificial Intelligence Institute, Deakin University, Scott Barnett Deakin University, Australia, Stefanus Kurniawan Deakin University, Zafaryab Rasool Applied Artificial Intelligence Institute, Deakin University, Rajesh Vasa Deakin University, Australia | ||
11:45 15mTalk | On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Content Research and Experience Papers Vince Nguyen Vrije Universiteit Amsterdam, Hieu Huynh Vrije Universiteit Amsterdam, Vidya Dhopate Vrije Universiteit Amsterdam, Anusha Annengala Vrije Universiteit Amsterdam, Hiba Bouhlal Vrije Universiteit Amsterdam, Gian Luca Scoccia Gran Sasso Science Institute, Matias Martinez Universitat Politècnica de Catalunya (UPC), Vincenzo Stoico Vrije Universiteit Amsterdam, Ivano Malavolta Vrije Universiteit Amsterdam Pre-print | ||
12:00 15mTalk | Rule-Based Assessment of Reinforcement Learning Practices Using Large Language Models Research and Experience Papers Evangelos Ntentos University of Vienna, Stephen John Warnett University of Vienna, Uwe Zdun University of Vienna | ||
12:15 15mTalk | Investigating Issues that Lead to Code Technical Debt in Machine Learning Systems Research and Experience Papers Rodrigo Ximenes Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Antonio Pedro Santos Alves Pontifical Catholic University of Rio de Janeiro, Tatiana Escovedo Pontifical Catholic University of Rio de Janeiro, Rodrigo Spinola Virginia Commonwealth University, Marcos Kalinowski Pontifical Catholic University of Rio de Janeiro (PUC-Rio) | ||