Log parsing is a critical step in analyzing software logs by transforming unstructured raw logs into structured formats, which are essential for system monitoring, failure debugging, and anomaly detection purposes. Creating and maintaining log parsing rules for specific systems requires both domain expertise and significant time and manual effort. To address this challenge, various automated log parsers have been proposed to alleviate the parsing efforts and enhance parsing efficiency. These approaches commonly evaluate their parsing performance using off-the-shelf benchmarks (i.e., Loghub-2k and Loghub-large) that comprise manually labeled logs (i.e., oracle templates) collected from a limited number of datasets. However, such benchmarks are not designed to be extended or adapted to evaluate logs from different sources, making them less applicable to real-world scenarios. On the other hand, crafting new benchmarks requires substantial domain expertise and manual effort, rendering generating log-parsing benchmark datasets an extremely challenging task. Consequently, there is a growing need for a standardized process to assist practitioners in efficiently generating high-quality log parsing benchmarks. In this paper, we propose LPB-Gen, a semi-automated process designed to facilitate the log parsing benchmark datasets generation for practitioners. We present a comprehensive evaluation of LPB-Gen through a user-based assessment to evaluate the correctness of the log benchmark data generated by LPB-Gen as well as the efficiency and reproducibility of LPB-Gen. The user study results not only show that LPB-Gen can generate a benchmark that outperforms the state-of-the-art benchmark in both coverage and accuracy, but also prove that LPB-Gen can provide actionable solutions that apply to generalized datasets, where practitioners may leverage our semi-automated process to produce oracle templates based on their specific datasets. Finally, we summarize seven factors that contribute to inconsistencies during developers’ manual refining of logs. These factors shall provide valuable insights for future research aimed at advancing log parsing and analysis.