Rapid iteration of large language models (LLMs) improved their performance on software engineering tasks. While prevailing benchmarks effectively measure LLMs’ functional correctness, they often overlook non-functional aspects, such as coding styles, which are crucial in real-world development and can introduce evaluation bias. To investigate such implicit bias, we propose BenchPrism that automatically disperses diverse coding styles and generates style-fulfilled benchmark variants for LLM evaluation, and thus explore how the bias behaves and is influenced across different benchmarks, LLMs, and code tasks. Experiments confirm the existence of bias, which indeed leads to unstable performance deviations in LLM evaluation (from a 14.92% drop to a 7.51% increase), even distorting model rankings. We also raise valuable takeaways and hope to inspire future researchers to be mindful of the robustness of LLM evaluation when considering the possibility of such hidden bias.