In the domain of Industry 4.0 Cyber-Physical Production Systems (CPPSs), Reinforcement Learning (RL) has gained momentum as an effective strategy for training intelligent agents in digital twins. Whilst the practice of Machine Learning Operations (MLOps) has become established as a holistic approach to automating workflows in supervised and unsupervised Machine Learning (ML), the extent to which MLOps practices are applicable to RL, particularly due to major differences between ML and RL concerning model deployment and model training, are not currently well-understood. The literature on RLOps as a paradigm is scarce. We tackle this open question by conducting an exploratory, qualitative, deductive-inductive industry case study on a CPPS, performing content analysis of CPPS artefacts, such as architectural schematics and source code, and understanding their relation to 22 known Architectural Design Decisions and 86 associated decision options through classification into four distinct emergent categories. Our findings help bridge the gap between MLOps and RLOps architectures, contributing novel insights into understanding the application of MLOps to RL and providing practical insights and inspiration for further research.