The integration of Large Language Models (LLMs) into software applications necessitates informed design choices across various configurations, including LLM selection, prompting techniques, and their parameters, and prompt templates. Many of these choices are arbitrary, and developers often lack guidance on optimizing configurations. In this work, we define the Experiment Configuration Optimization Problem and illustrate it with a real-world Text-to-SQL application we developed. Our results show that most configurations are sub-optimal, with only a few offering a favorable trade-off between accuracy and cost. Highlighting the critical need for systematic exploration, we show that extensive experimentation is expensive, underscoring the importance for cost-effective methods to navigate the configuration space. Our findings motivate further research into methodologies that effectively optimize LLM application configurations.