Machine learning operations (MLOps) automate common tasks throughout the entire life cycle of a machine learning model. In Industry 4.0 environments, cyber-physical production systems increasingly utilise reinforcement learning (RL) models to optimise and automate production processes, giving rise to reinforcement learning operations (RLOps). However, manually configuring RLOps pipelines requires specialised development operations (DevOps) knowledge and can limit the potential for automation. We propose a template-based approach for rapidly creating and deploying RLOps pipelines in Industry 4.0 environments that minimises the required programming effort and expertise. Based on the Pipes and Filters pattern, our modular solution leverages large language models (LLMs) for automated pipeline creation. It enables fully automated execution, including model training, testing, and deployment, with built-in quality control ensuring correct configurations. We demonstrate our approach with a case study derived from an Industry 4.0 platform for smart factories, in which we empirically evaluate seven LLMs based on four representative RLOps use cases. This evaluation provides the first systematic comparison of LLM capabilities for RLOps pipeline generation in Industry 4.0. Our results demonstrate that our solution, used with a suitable LLM, can reliably generate and execute RLOps pipelines with low error rates, thereby reducing development time and democratising RLOps deployment by eliminating the need for specialised DevOps knowledge.