How to Sustainably Monitor ML-Enabled Systems? Accuracy and Energy Efficiency Tradeoffs in Concept Drift Detection
Tue 25 Jun 2024 17:37 - 18:00 at Stefan Arnborg - Remote Presentation #1 Chair(s): Gauthier Rousillhe
ML-enabled systems that are deployed in a production environment typically suffer from decaying model prediction quality through concept drift, i.e., a gradual change in the statistical characteristics of a certain real-world domain. To combat this, a simple solution is to periodically retrain ML models, which unfortunately can consume a lot of energy. One recommended tactic to improve energy efficiency is therefore to systematically monitor the level of concept drift and only retrain when it becomes unavoidable. Different methods are available to do this, but we know very little about their concrete impact on the tradeoff between accuracy and energy efficiency, as these methods also consume energy themselves. To address this, we therefore conducted a controlled experiment to study the accuracy vs. energy efficiency tradeoff of seven common methods for concept drift detection. We used five synthetic datasets, each in a version with abrupt and one with gradual drift, and trained six different ML models as base classifiers. Based on a full factorial design, we tested 420 combinations (7 drift detectors × 5 datasets × 2 types of drift × 6 base classifiers) and compared energy consumption and drift detection accuracy. Our results indicate that there are three types of detectors: a) detectors that sacrifice energy efficiency for detection accuracy (KSWIN), b) balanced detectors that consume low to medium energy with good accuracy (HDDM W, ADWIN), and c) detectors that consume very little energy but are unusable in practice due to very poor accuracy (HDDM A, PageHinkley, DDM, EDDM). By providing rich evidence for this energy efficiency tactic, our findings support ML practitioners in choosing the best suited method of concept drift detection for their ML-enabled systems.
Tue 25 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:00 - 13:00 | ConverStation #1Research Papers / Journal First at ConverStations Room (A108) This session will follow the ConverStation format where all papers will be discussed simultaneoiusly, in three rounds. | ||
11:00 30mTalk | ConverStation - Introduction Research Papers Elina Eriksson KTH Royal Institute of Technology, Sweden | ||
11:30 90mResearch paper | “That is what we can influence”: Exploring energy practices negotiability in households with solar panels using an always-on display Research Papers Jorge Luis Zapico , Arjun Rajendran Menon KTH Royal Institute of Technology, Sweden, Björn Hedin KTH Royal Institute of Technology | ||
11:30 90mResearch paper | Empowering Organizations for Sustainable Digitalization: a Corporate Digital Responsibility Maturity Model Approach Research Papers | ||
11:30 90mPaper | ICT sector electricity consumption and greenhouse gas emissions – 2020 outcome Journal First Jens Malmodin Ericsson Research, Nina Lovehagen Ericsson, Pernilla Bergmark Ericsson Research, Ericsson AB, Dag Lunden Telia DOI | ||
11:30 90mResearch paper | PluriCards: Engaging with the Pluriverse to Find New Sustainability Research Directions Research Papers Maurizio Teli Aalborg University, Denmark, Markus Löchtefeld Aalborg University, Denmark, Petko Karadechev Aalborg University, Denmark, Rikke Hagensby Jensen Aarhus University, Denmark, Victor Vadmand Jensen Aalborg University, Denmark, Helena Amalie Haxvig University of Trento, Italy | ||
11:30 90mResearch paper | Exploring the Impact of K-Anonymisation on the Energy Efficiency of Machine Learning Algorithms Research Papers Yixin Hu Sun Yat-sen University, Pepijn de Reus University of Amsterdam, Ana Oprescu University of Amsterdam, Ivano Malavolta Vrije Universiteit Amsterdam, Vit Zemanek University of Amsterdam | ||
11:30 90mResearch paper | How to Sustainably Monitor ML-Enabled Systems? Accuracy and Energy Efficiency Tradeoffs in Concept Drift Detection Research Papers Rafiullah Omar , Justus Bogner Vrije Universiteit Amsterdam, Vincenzo Stoico Vrije Universiteit Amsterdam, Patricia Lago Vrije Universiteit Amsterdam, Henry Muccini University of L'Aquila, Italy, Joran Leest Vrije Universiteit Amsterdam Pre-print | ||
11:30 90mPaper | Overcoming challenges in life cycle assessment of smart energy systems – A map of solution approaches Journal First Daniela Wohlschlager Research Center for Energy Economics (FfE e.V.) & TU Munich, Hannes Blum Institute for Ecological Economy Research (IÖW), Severin Beucker Borderstep Institute for Innovation and Sustainability, Johanna Pohl Technische Universität Berlin, Magnus Fröhling Technical University of Munich DOI | ||
11:30 90mResearch paper | Life cycle assessment of digitalization in buildings: The case of a building monitoring system Research Papers Shoaib Azizi , Anna Furberg , Marco Molinari KTH Royal Institute of Technology, Sweden, Goran Finnveden KTH Royal Institute of Technology, Sweden | ||
11:30 90mResearch paper | The Effect of Analytical Tools on Energy Consumption in Websites Research Papers Panu Puhtila University of Turku, Lauri Kivimaki University of Turku, Timi Heino University of Turku, Jari-Matti Mäkelä University of Turku, Sampsa Rauti University of Turku, Tuomas Mäkilä University of Turku | ||
11:30 90mResearch paper | Evidence synthesis of indirect impacts of digitalisation on energy and emissions Research Papers Charlie Wilson Environmental Change Institute, University of Oxford, Maureen Agnew Environmental Change Institute, University of Oxford, Felippa Amanta Environmental Change Institute, University of Oxford, Yee Van Fan Environmental Change Institute, University of Oxford, Poornima Kumar Environmental Change Institute, University of Oxford, Marcel Seger Environmental Change Institute, University of Oxford | ||
11:30 90mResearch paper | From Crop to Click – Organic and Digital Transformation of Out-of-Home Catering Value Chains in Germany Research Papers Tamara Scheerer Reutlingen University, Dieter Hertweck Reutlingen University, Tim Hakenberg Rottenberg University | ||
11:30 90mResearch paper | The Potential and Limits of Digital Energy Advisors Research Papers Nelson Sommerfeldt KTH Royal Institute of Technology, Mattias Höjer KTH Royal Institute of Technology | ||
11:30 90mResearch paper | Decision-Making under Environmental Complexity: Shifting from Avoided Impacts of ICT Solutions to Systems Thinking Approaches Research Papers David Ekchajzer , Jacques Combaz Verimag/CNRS, Catherine Letondal ENAC, Laetitia Bornes , Rob Vingerhoeds |
16:30 - 18:00 | Remote Presentation #1Research Papers at Stefan Arnborg Chair(s): Gauthier Rousillhe Group 3Zoom - https://kth-se.zoom.us/j/68775095116(Join Breakout room - Stefan Arnborg) | ||
16:30 22mResearch paper | Exploring the Impact of K-Anonymisation on the Energy Efficiency of Machine Learning Algorithms Research Papers Yixin Hu Sun Yat-sen University, Pepijn de Reus University of Amsterdam, Ana Oprescu University of Amsterdam, Ivano Malavolta Vrije Universiteit Amsterdam, Vit Zemanek University of Amsterdam | ||
16:52 22mResearch paper | MLCA: a tool for Machine Learning Life Cycle Assessment Research Papers Clément Morand Université Paris-Saclay, CNRS, Inria, LISN, Anne-Laure Ligozat Université Paris-Saclay, CNRS, Inria, LISN, Aurélie Névéol Université Paris-Saclay, CNRS, Inria, LISN | ||
17:15 22mResearch paper | Energy Efficiency of AI-powered Components: a Comparative Study of Feature Selection Methods Research Papers | ||
17:37 22mResearch paper | How to Sustainably Monitor ML-Enabled Systems? Accuracy and Energy Efficiency Tradeoffs in Concept Drift Detection Research Papers Rafiullah Omar , Justus Bogner Vrije Universiteit Amsterdam, Vincenzo Stoico Vrije Universiteit Amsterdam, Patricia Lago Vrije Universiteit Amsterdam, Henry Muccini University of L'Aquila, Italy, Joran Leest Vrije Universiteit Amsterdam Pre-print |
Please ensure that you take a ticket to a particular paper in each round
Virtual Only session - https://kth-se.zoom.us/j/68775095116
(Join Breakout room - Stefan Arnborg)
- Check the ICT4S email you received on Monday for the password or the link with the password included
- On-site attendees, use the QR Code on the ICT4S 2024 guide you received at registration