Recently Large Language Models (LLMs) have increasingly been used in security-related tasks, including gen- erating proof-of-concept (PoC) exploits. Several LLM-assisted approaches have been proposed; they typically generate PoCs from vulnerability descriptions and use additional guidance. However, such approaches are often ineffective because the signals—such as printed markers, generated files, or runtime side effects—that they use for validation may not imply that the vulnerability is triggered. Research for more reliable PoC generation is required, yet remains challenging. We propose PoC-Gym, a pipeline for LLM-based PoC generation for Java security vulnerabilities. PoC-Gym uses both static and dynamic information, e.g., CVE-tailored prompts, static traces, and coverage-based feedback, and iteratively generates PoC candidates. Each candidate goes through a series of validation steps: whether the execution is complete, manifests a success signal, and reaches the sink of the target trace. We evaluate PoC-Gym using 20 Java CVEs. PoC-Gym generates many PoCs that appear valid at runtime but fail to actually reach the vulnerability. To assess the effectiveness of the generated PoCs, we validate these runtime-valid candidates against ground-truth vulnerable locations using dynamic trace analysis. Across 338 runs, we find that 65 candidates successfully reach the vulnerable locations, covering 12 of the 20 CVEs. To better understand how to achieve more reliable PoC generation, we present an in-depth analysis of such PoCs and identify common sources of failures. We believe that our work provides insights for future research.