Context. Our research tackles a crucial challenge in Natural Language Processing (NLP). While models like BERT and GPT are powerful, they require substantial resources. Knowledge distillation can be employed as a technique to enhance their efficiency. Yet, we lack a clear understanding on their performance and energy consumption. This uncertainty is a major concern, especially in practical applications, where these models could strain resources and limit accessibility for developers with limited means. Our drive also comes from the pressing need for environmentally-friendly and sustainable applications in light of growing environmental worries. To address this, it is crucial to accurately measure their energy consumption.

Goal. This study aims to determine how Knowledge Distillation affects the energy consumption and performance of NLP models.

Method. To explore the impact of distillation techniques on NLP models, we benchmark BERT, Distilled-BERT, GPT-2, and Distilled-GPT-2 using three different tasks from three different categories selected from a third-party dataset. During the experiment, the energy consumption, CPU utilization, memory utilization, and inference time of the considered NLP models are measured and statistically analyzed.

Results. The study reveals notable differences between the original and the distilled version of the measured NLP models. Distilled versions generally consume less energy, with distilled GPT-2 having reduced CPU utilization. These results provide evidence and insights about the possible trade-offs in using distilled models for NLP.

Conclusion. The results of this study highlight the critical impact of model choice on performance and energy consumption metrics. Future research should consider a wider range of distilled models, diverse benchmarks, and deployment environments, as well as explore the ecological footprint of these models, particularly in the context of environmental sustainability.