Context Matters: Evaluating MCP-Based Context-Aware AI — A Case Study of Email Communications in Nonprofit Organizations
The Model Context Protocol (MCP) enables large language models to dynamically integrate external organizational context, supporting more context-aware AI applications. We investigate its role in nonprofit email communication by introducing the Knowledge-Aware Nonprofit Tooling Architecture (KANTA), an MCP-based design that connects AI models to nonprofit data and tools, including mission statements, conversation history, and user-uploaded context. In a study with eight professionals from six nonprofits (22 scenarios), participants evaluated three context variants per thread: light (mission only), medium (mission plus current thread), and full (adds email history and uploaded context). They rated clarity, tone/personalization, and likelihood-to-use. Across measures, full-context replies were preferred and received higher satisfaction scores than light and medium. A linear mixed-effects analysis indicated significant improvements for full over lighter variants, and a nonparametric test showed a consistent, marginal trend in the same direction. Qualitative analysis showed participants perceived full-context outputs as “knowing what it’s talking about,” requiring less editing, and better matching relationship tone; suggestions emphasized asking clarifying questions when information was missing and maintaining professional formatting. We discuss how MCP operationalizes contextual grounding within existing nonprofit workflows, why combining MCP with interactive clarification and, where appropriate, retrieval-augmented generation may further improve usability, and limitations of this exploratory sample. Our results offer early evidence that richer, MCP-based context can measurably improve the usefulness of AI-generated nonprofit emails.