Test flakiness, characterized by non-deterministic test outcomes despite unchanged code, poses a significant challenge to continuous integration system efficiency and developer productivity. We comprehensively study a year-long initiative to mitigate test flakiness within a large-scale engineering organization. We adjust test failure sensitivity thresholds to account for alert fatigue and resolution quality, refocus data visualization for prioritization and pattern analysis, and introduce incentives to improve the rate of solutions contributing to test stabilization. Key findings reveal the importance of balancing technical efficiency gains with organizational dynamics.