This session will provide an in-depth exploration of Foundation Model (FM) routing for FMware, focusing on balancing quality and inference cost. Covered topics include:

• An introduction to FM routing, where requests are routed to FMs of varying sizes and capabilities
• A survey of existing routing methods that rely on data-driven learning to make optimal routing decisions
• The challenges posed by existing approaches, such as reliance on curated data, complex computations, and the evolution of weaker FMs
• The introduction of Real-time Adaptive Routing (RAR), a novel approach that continuously adapts FM routing decisions using guided in-context learning
• How RAR reduces dependence on stronger, more expensive FMs while maintaining high response quality
• The intra-domain generalization benefits of RAR’s guided learning approach in enhancing weaker FMs