Background: Eye-tracking studies in software engineering have become popular providing valuable insights into thought processes of developers and their interaction with visual information. Such insights can support developers in their daily work, for example, by providing evidence-based coding guidelines. However, the analysis of eye-tracking data in the research community does not yet follow established standards. Particularly concerning for the comparability of study findings is the large variety of tools for evaluating eye-tracking data, all of which use different algorithms and preset parameters, sometimes even without disclosing them.

Aims: Our study has three objectives: (1) to characterize the analysis tool landscape for eye-tracking data within software-engineering research; (2) to analyze justifications of eye-tracking study authors for the selection of a particular analysis tool; and (3) to investigate whether the choice of the analysis tool affects the results and conclusions of an eye-tracking study.

Method: First, we conducted a lightweight systematic mapping study to find reports of eye-tracking studies in software engineering, from which we extracted used analysis tools and the authors’ justifications for their use. Second, we reproduced the statistical analyses on three publicly available eye-tracking data sets of the corresponding original studies utilizing different eye-tracking analysis tools.

Results: We found that the most used analysis tools are Tobii Software, iTrace, and Ogama. The majority of studies do not disclose the used analysis tool, however, with an upward trend for more recent publications. Most publications did not provide any justification or consequence for their choice of analysis tool. Our case study revealed that the choice of analysis tool significantly affects the experimental results of a study.

Conclusions: We conclude that the lack of standardization in the analysis of eye-tracking data poses a significant problem for the comparability and reproducibility of eye-tracking studies in SE. We suggest extended reporting guidelines for eye-tracking studies.