Multiple machine learning (ML) models are often incorporated into real-world ML systems. However, updating an individual model in these ML systems frequently results in regression errors, where the new model performs worse than the old model for some inputs. While model-level regression errors have been widely studied, little is known about how regression errors propagate at system level. To address this gap, we propose RegTrieve, a novel retrieval-enhanced ensemble approach to reduce regression errors at both model and system level. Extensive experiments across various model update scenarios show that RegTrieve significantly reduces system-level regression errors with almost no impact on system accuracy, outperforming all baselines by 20.43% on average.