Ensuring safety and providing rigorous behavioral guarantees are critical for robotic systems operating in high-stakes environments such as autonomous driving. Field testing is common, but costly and risky. Simulation-based testing offers a safer and lower-cost alternative for automatically generating traces for analysis and performance assessment. An \textit{oracle}, a mechanism to determine whether a robot’s behavior is correct, is essential for evaluating the performance of each trace based on how well the robot completes its intended tasks while satisfying key criteria such as safety, efficiency, and reliability. Many supervised and unsupervised algorithms have been proposed for oracle learning. Supervised learning is powerful but costly and time-consuming due to manual labeling, whereas unsupervised learning does not require labels but often struggles with accuracy due to the lack of ground truth. To mitigate this drawback, we argue that introducing a human into the learning process, namely human-in-the-loop oracle learning, can help the oracle learn more precise distinctions between good and bad behaviors with a lower worst-case expected cost during testing. With the help of simulations, we can discover and prevent major mistakes that could cause injuries and other harm. We illustrate this approach through a conceptual framework for integrating human-in-the-loop learning into robotic system evaluation.