GitHub is a collaborative platform for global software development, where Pull Requests (PRs) are essential to bridge code changes with version control. However, software developers often trade software quality for faster implementation, incurring Technical Debt (TD).

When PRs are evaluated by reviewers (e.g., other developers), the latter can often detect TD instances, and this can lead to either rejection of the PR or can spark discussion about it.

We investigated whether PRs’ comments indicate TD by assessing three large-scale repositories: Spark, Kafka, and React. We combined a manual classification with automated detection using machine learning and deep learning models. We classified two datasets and found that 37.7% and 38.7% of the comments indicate TD, respectively. Our best model achieved an 85% F1 score in classifying TD during the validation phase. However, we also faced several challenges during this process, which may hint that TD in PR comments is discussed differently from other software artifacts (e.g., code comments, commits, issues or discussion forums). Thus, we present challenges and lessons learned meant to assist researchers in pursuing this area of research