Generative language models have made tremendous progress in software engineering tasks such as code generation, code refinement and debugging. Many of these tasks require strong reasoning skills over code and code executions. This raises the question whether we can utilize the reasoning skills of ChatGPT in the formal setting of software verification and complement existing automatic verifiers such as CPAchecker. To answer this question, we study the abilities of ChatGPT in the context of automatic software verification. More specifically, we focus on the task of loop invariant generation. Loop invariant generation requires reasoning over the code semantics to infer invariants that hold for all loop executions and good loop invariants can significantly simplify the verification process. Our initial investigation on the SV-COMP Loops category shows that ChatGPT can effectively annotate C programs with valid loop invariants, many of which can be useful for automatic software verification. Interestingly, we find that many generated loop invariants are far from trivial and have a similar quality as invariants produced by a human annotator.