Auto-tuning DL compilers are gaining ground as an optimizing back-end for DL frameworks. While existing work can generate deep learning models that exceed the performance of hand-tuned libraries, they still suffer from prohibitively long auto-tuning time due to repeated hardware measurements in large search spaces. In this paper, we take a neural-predictor inspired approach to reduce the auto-tuning overhead and show that a performance model can be trained prior to compilation to accurately identify high-performing tensor operation codes and eliminate repeated search and hardware measurements. To generate a sample-efficient training dataset, we extend input representation to include task-specific information and guide data sampling methods to focus on learning high-performing codes. We evaluated the resulting predictor model, One-Shot Tuner, against AutoTVM and other prior work, and the results show that One-Shot Tuner speeds up compilation by 2.81 to 67.7x compared to prior work while providing comparable or improved inference time for CNN and Transformer models.