Autonomous systems increasingly depend on black-box predictive models to drive adaptation decisions. Yet, the opacity of these models typically reduces the transparency of decisions and efficient runtime planning. In systems based on the standard MAPE-K control loop architecture, planners must often rely on black-box optimization without prior knowledge of how model predictions relate to system goals. In such circumstances, planning becomes costly and difficult to interpret. We present a layered control loop for transparent decision-making in autonomous systems. The approach is enabled by a novel multi-layer architecture that extends the MAPE-K loop by incorporating model-agnostic explanations into the adaptation process. The layers on top of the MAPE-K operate asynchronously and include: (i) an explanation layer, generating local rules describing the behavior of the embedded predictive models under specific operating conditions, thereby curating them as part of the system knowledge base; and (ii) a knowledge augmentation layer, in charge of exploring new operating conditions to enhance the quality of the predictive models as well as the corresponding explanations. These explanations become instrumental for the layer below to plan effective adaptations by reducing the exploration space and improving the interpretability of planning outcomes. Our empirical evaluation using two study subjects in the domain of robotic systems shows that our multi-layer approach reduces runtime planning cost while maintaining or improving the satisfaction of dependability requirements compared to existing baseline methods.