Automating test suite generation requires knowledge about the system and the underlying test case generation approach. Querying a smart Chatbot like ChatGPT using the program under tests for obtaining test cases is an appealing alternative for reducing costs and effort. However, the quality of the obtained test suite in terms of code coverage or mutation score might be questionable. Hence, an experimental evaluation focusing on the quality metrics of resulting test suites is important. In this paper, we provide such an experimental evaluation considering 12 Python programs and five different Large Language Models including ChatGPT4o. We measure the statement and line coverage and the mutation score of the resulting test suite. Furthermore, we vary the temperature used by a Large Language Model and the prompting strategy. Besides reporting on the corresponding research questions, we also provide a comparison with already published similar studies.