Autonomous driving systems (ADSs) rely on real-time data from various sensors, including cameras and LiDAR, to make time-critical decisions using deep neural networks. The accuracy of these decisions is crucial for the widespread adoption of ADSs, as errors can have serious consequences. LiDAR-based detection, in particular, is sensitive to point cloud data (PCD) noises from various sources. However, the robustness of the current 3D detection against specification-based perturbations remains unevaluated. These perturbations are derived from the specification of LiDAR sensors and previous research on LiDAR’s ability to capture objects of different colors and materials. They can manifest as very subtle sensor-based noises or obstacle-specific perturbations. Hence, we propose SORBET, a framework that tests the robustness of 3D obstacle detection in ADS against such perturbations to the PCD to evaluate the robustness of LiDAR-based 3D obstacle detection. We applied SORBET to evaluate the robustness of five classic 3D obstacle detection models, including one from an industry-grade Level 4 ADS (Baidu’s Apollo). Furthermore, we studied how the deviated obstacle detection results would propagate and negatively impact trajectory prediction. Our evaluation emphasizes the importance of testing 3D obstacle detection against specification-based perturbations. We find that even very subtle changes in the PCD (i.e., removing two points) may introduce a non-trivial decrease in the detection performance. Furthermore, such a negative impact will further propagate to other modules, and endanger the safety of the ADS.