This session will detail the challenges and techniques that can be used to evaluate an AIware. Covered topics include:

• Overview of evaluating an AIware - Importance
• Evaluation primitives - Evaluation with ad hoc vibe checks, benchmarks, manually curated datasets, trace data, data splits, and repetitions
• Evaluation metrics overview
• Evaluating individual components of AIware - Agents, RAG, etc.
• Testing AIware - Unit tests, summary evaluations, response evaluations, regression testing, backtesting
• AI as judge - Overview, benefits, and costs