Code Executing Reasoning is becoming a new non-functional metric that assesses the ability of large language models (LLMs) in programming tasks. State-of-the-art frameworks (CodeMind or REval) and benchmarks (CruxEval) usually focus on LLM’s prediction of a given code’s input/output or intermediate variable states/values on limited programs. However, there is no tool for more in-depth analysis of the results. Without such a tool, the observations about LLM’s code execution reasoning cannot be generalized to more datasets, preventing the research community and practitioners from devising the next generation of LLMs with better code execution reasoning abilities. This paper introduces ExeRScope, a series of tools and heuristics to analyze the result of code execution reasoning frameworks to understand better the impact of code properties in the studied benchmarks on code execution reasoning. With such tooling, analysis can be generalized to code with similar properties without the urgent need to design more benchmarks, which is a cumbersome effort.

The implementation of ExeRScope is publicly available, and it currently assesses the impact of different (1) program constructs, (2) program complexities, (3) dynamic programming properties such as recursion length, and (4) variable types on code execution reasoning abilities of LLMs. Evaluation of ExeRScope on four programming benchmarks (1450 Python programs) and six LLMs from three existing code execution reasoning techniques shows its effectiveness and practicality in providing important insights, highlighting the strengths and limitations of LLMs concerning code execution reasoning.