A Tool for Mutation Analysis in Racket
Racket is a functional programming language designed for programming language design and implementation, and is the basis for programming education at many colleges and universities. Recent research results have shown that mutation analysis can be an effective substitute for manual grading of student test cases. In order to evaluate its efficacy in our college’s introductory programming courses, we created a prototype mutation analysis tool for Racket. We describe the design and implementation of the tool and perform a feasibility study using two assignments from an intro CS course where student test suite thoroughness was evaluated by hand by human graders. In our results, we find a moderate correlation between mutation score and hand-grading test suite quality score and conduct a qualitative analysis to identify situations where mutation score and hand-grading score do not correlate. We find that, compared to hand-grading, mutation analysis may require more stringent adherence to the interface specified in an assignment as well as more precisely specified assignments. On the other hand, inter-reviewer reliability is a known challenge of hand-grading, and we observe several instances where hand-graders may have assigned the wrong score. Given the relatively cheap cost to providing mutation analysis feedback to students (compared to hand-grading feedback), mutation analysis still provides the opportunity to provide faster, more frequent, feedback to learners, enabling them to improve their testing practices further. Future work will study the effectiveness of various mutation operators in racket and perform larger-scale evaluations.
Sun 16 AprDisplayed time zone: Dublin change
11:00 - 12:30 | |||
11:00 30mTalk | Analysis of mutation operators for FSM testing Mutation | ||
11:30 30mTalk | A Tool for Mutation Analysis in Racket Mutation Bambi Zhuang Northeastern University, James Perretta Northeastern University, Arjun Guha Northeastern University and Roblox Research, Jonathan Bell Northeastern University | ||
12:00 30mTalk | The Inversive Relationship Between Bugs and Patches: An Empirical Study Mutation Pre-print |