Reinforcement learning (RL), and in particular its combination with deep neural networks, has seen major breakthroughs in the recent years, most notably outperforming humans in games like Atari, Go, and StarCraft. RL use is also extensively investigated in a range of practical self-adaptive applications and cyber physical systems, however, existing algorithms often fall short of being suitable for use in such complex environments. My research focuses on developing techniques that enable the use of RL for optimization in large-scale adaptive systems, for example, urban traffic control, smart grid, and communication networks. These applications share properties with many other large-scale systems, i.e., are characterized by distributed control, heterogeneity, presence of multiple and often conflicting goals, reliance on diverse sources of information, and above all, the need for continuous adaptation. In this talk I will present a range of techniques for enabling the use of RL for optimization in such environments, in particular multi-agent multi-objective optimization, adaptation in non-stationary environments, online transfer learning, and state space adaptation. I will discuss further challenges in enabling RL deployment in self-adaptive systems, including development of new algorithms that can ensure seamless lifelong adaptivity and highlight the need for explainability and software testing techniques for RL-based applications.