One very attractive property of Large Language Models (LLMs) is their emergent in-context learning capability, which enables us to simply describe our requirements in natural languages and get the corresponding souce code generated in programming languages. While LLMs as a generative model are known to hallucinate, i.e., generate factually incorrect contents, the fact that code can be executed can be used to fight this phenomenon. We briefly look at existing techniques designed to improve the correctness of code generated by LLMs, and will try to imagine the future of Genetic Improvement that is supported, enhanced, and driven by LLMs.