Adaptive User Interfaces (AUIs) have the potential to enhance the usability and personalization of modern software applications by dynamically adjusting layout, content, and interaction strategies. However, fully automated adaptation often compromises user agency, transparency, and trust. To ensure that interface adaptations remain understandable and user-centered, human judgment must be systematically integrated into the UI adaptation process. In this paper, we propose a human-in-the-loop reinforcement learning (RL) framework for adaptive UIs, where humans participate in two complementary stages. First, during RL agent training, human feedback informs reward shaping and guides policy learning to better reflect subjective user preferences, usability criteria, and context-specific needs. Second, during runtime interaction, users interact with the adaptive system and can accept, reject, or fine-tune adaptation decisions suggested by the trained RL agent thus, maintaining transparency and control as the interface evolves.
Specifically, we present the modular software environment designed and implemented to support this framework and report findings from initial empirical studies applying it in specific usage scenarios. The results show promising potential for generating personalized adaptations that improve the overall user experience. Finally, we discuss directions for advancing the second stage of our framework (enhancing runtime interaction) to foster adaptive user interfaces that evolve collaboratively with humans. Our vision is to combine human insight and machine intelligence to achieve adaptive user interfaces that are not only efficient and effective but also trustworthy, transparent, and aligned with human values.