Sequential decision-making processes (SDPs) are essential in addressing complex real-world challenges, such as autonomous driving, robotic control, and traffic management. While recent advances in Deep Learning (DL) have led to mature solutions for solving these complex problems, SDMs remain vulnerable to learning unsafe behaviors, posing significant risks in safety-critical applications. The state-of-the-art testing framework for SDMs primarily concentrates on identifying crash-triggering scenarios but neglects the diversity aspect of the crashes. As a result, redundant crash-triggering scenarios may be generated. This can reduce the overall testing performance and introduce additional analysis costs. To address this, we propose CureFuzz, a novel curiosity-driven black-box fuzz testing approach for SDMs. CureFuzz proposes a curiosity mechanism that allows a fuzzer to effectively explore novel and diverse scenarios, leading to improved detection of crash-triggering scenarios. Additionally, we introduce a multi-objective seed selection technique to balance the exploration of novel scenarios and the generation of crash-triggering scenarios, thereby optimizing the fuzzing process. We evaluate CureFuzz on various SDMs and sequential decision-making problems, such as autonomous driving and video game playing. Experimental results demonstrate that CureFuzz outperforms the state-of-the-art method by a substantial margin in the total number of faults and distinct types of crash-triggering scenarios. We also demonstrate that the crash-triggering scenarios found by CureFuzz can repair SDMs, which highlights CureFuzz as a valuable tool for testing SDMs and optimizing their performance.