Will You Trust Me More Than ChatGPT? Evaluating LLM-Generated Code Feedback for Mock Technical Interviews
This program is tentative and subject to change.
Technical interviews are an opportunity for candidates to showcase their technical proficiency to employers. Feedback on code correctness, optimality and complexity can reveal deficiencies and be invaluable in candidates’ preparation for future interviews. Unfortunately, current technical interview practices lack feedback for candidates. To resolve this, we designed a website to simulate technical interview programming experiences and provide users with LLM-generated feedback on code using ChatGPT. We devised between-subject study and conducted mock technical interviews with 46 participants across two settings: human-administered (a live human sharing generated feedback) and automated. We focus on: 1) evaluating the quality of ChatGPT generated code feedback; and 2) dissecting factors influencing trust and usefulness with regard to feedback delivery. Our results show that candidates perceive coding feedback as useful. However, they regard the automatic feedback as less trustworthy compared to feedback in the human-administered setting. In light of our findings, we discuss implications for increasing trust in AI systems and guidelines for designing technical interview feedback systems.