Test-Driven Development (TDD) is a widely recognized practice for building reliable software, yet its adoption is often constrained by the effort required to write tests before implementing functionality. Recent advances in Large Language Models (LLMs) offer new opportunities to automate test generation directly from natural language specifications. We present TestForge, a benchmarking framework for systematically evaluating LLM-based test case generation, which defines the core evaluation components in line with well-established benchmarking guidelines. As a first instantiation, we apply TestForge to JUnit test generation using a curated subset of the IBM CodeNet dataset, which includes problem statements, limited reference tests, and both correct and faulty solutions. Our study encompasses problems ranging from basic algorithms to more complex scenarios, and it explores three prompting strategies: zero-shot, few-shot, and chain-of-thought. Results show that while LLMs can generate functionally useful tests for clearly specified problems, they continue to struggle with producing comprehensive, semantically rich suites for more complex cases.