Smart contracts, self-executing contracts with the terms of the agreement directly written into code, have become integral to blockchain technology, particularly in decentralized finance and Web3. The prevalence of Ponzi schemes in smart contracts poses a significant scam, causing substantial financial losses and undermining trust in blockchain-based systems. Existing detection methods heavily rely on large amounts of labeled data and static information, resulting in poor reliability and the inability to detect unseen Ponzi schemes. In this paper, we propose PonziSleuth, the first LLM-driven method for detecting Ponzi smart contracts. PonziSleuth leverages the advanced language understanding capabilities of LLMs to analyze smart contract source code directly, using a novel two-step zero-shot chain-of-thought prompting technique. We conducted a comprehensive performance evaluation using widely adopted benchmark datasets and real-world smart contracts. Our results demonstrate that PonziSleuth significantly outperforms state-of-the-art detection methods, achieving high accuracy and reliability. Specifically, PonziSleuth achieved a balanced detection accuracy of 96.06% with GPT-3.5-turbo, 93.91% with LLAMA3, and 94.27% with Mistral, showcasing its superior performance over existing models. In real-world detection, PonziSleuth effectively identified 15 new Ponzi schemes from 4,597 contracts verified by Etherscan in Mar 14-24, 2024, achieving a false negative rate of 0% and a false positive rate of 0.29%. We believe PonziSleuth represents a significant step in leveraging LLMs for mitigating scams and enhancing blockchain security.