Reinforcement learning is a powerful methodology that enables self-adaptive systems to relearn and update their adaptation policy when dealing with unforeseen changes. To update the policy more efficiently, several knowledge reuse approaches have been proposed to speed up relearning. However, the current studies treat and reuse the knowledge integrally, which may result in increased relearning costs if the reused knowledge is inappropriate in the changed situation. Generally, some localized pieces of the knowledge are still appropriate for reuse if they are not related to the changes, while some pieces may become inappropriate for reuse if they are affected by the changes. This paper proposes a goal-oriented curriculum evolution method to realize finer-grained knowledge reuse, combining goal-oriented modeling and curriculum learning. The method is twofold: (1) at design time, we apply goal-oriented modeling to design a curriculum in which an RL problem is decomposed into sub-problems, so that knowledge can be decomposed into several pieces of localized knowledge for sub-problems, and (2) at runtime, we evolve the curriculum to reflect changes (i.e., update the sub-problems related to the changes), so that the affected pieces of knowledge can be locally updated to make them appropriate for reuse in the changed situation. The evaluation based on a cleaning robot shows that the relearning time was shortened, demonstrating the effectiveness of our method.