Pre-trained models (PTMs) have revolutionized machine learning by significantly enhancing reusability and reducing resources required for model training. Despite their advantages, selecting appropriate PTMs for specific system requirements remains difficult due to application heterogeneity and variable task performance. This has led to the proposal of PTM compositions to enhance capabilities beyond individual models, with an on-the-fly approach being applied further to meet dynamic requirements. However, composing PTMs on-the-fly can result in objective mismatch, leading to inefficiencies and errors in constituent PTMs. This necessitates the timely and efficient replacement of underperforming PTMs. The process is a major challenge due to the vast number of candidates and the extensive time required for evaluation. Therefore, we propose the Sample-Infer-Predict framework for efficient on-the-fly PTM replacement which comprises three phases: sampling, inference, and prediction. First, our novel Density and Diversity sampling algorithm efficiently selects representative user inputs. Second, the inference phase evaluates candidate PTMs from model hubs for the prediction dataset. Third, the prediction phase utilizes the dataset to predict the optimal PTM replacements. We evaluate our methodology using a vehicle detection PTM composition and a dataset of 5849 vehicle images, focusing on the efficiency of the replacement process, the quality of the meta-predictions, and the effectiveness of our sampling technique. Our approach reduces replacement time (155s vs. 28293s), achieves high precision@k values, and lowers mean absolute error values compared to other sampling techniques.