Large language models (LLMs) have demonstrated their potential in performing complex software engineering (SE) tasks. Rigorous evaluation of LLMs and LLM-based tools requires massive, up-to-date data derived from real-world SE processes. Existing continuous integration & delivery (CI/CD) datasets, such as BugSwarm, provide evolving data mined from software build processes, including complete context of build failures. To harness the CI/CD data, we propose CI-Bench, a unified benchmarking framework designed to evaluate LLM-based program repair tools on software failures from CI/CD processes. CI-Bench retrieves data from the BugSwarm dataset, parses the build logs, and constructs appropriate prompts before invoking LLM-based program repair tools. Additionally, CI-Bench includes an executor that facilitates dynamic evaluation in the identical environment as the original build process. With CI-Bench, we evaluate three state-of-the-art LLM-based program repair tools, Agentless, SWE-Agent, and AutoCodeRover, on a code repair task involving 100 real-world CI/CD failures using GPT-4o, Claude-3.5-Sonnet, and Deepseek-V3 as foundation models. The evaluation shows that Agentless, SWE-Agent, and AutoCodeRover achieve success rates up to 32%, 36%, and 13% in generating correct patches, respectively. CI-Bench is available at https://github.com/bugswarm/ci-bench and a demo of the tool can be found at https://youtu.be/BM0K-P38MOg.