FairMAid: An LLM-based Fairness Metrics Recommender System
AI-based technologies increasingly influence everyday life, making it essential to ensure fairness in their responses. Fairness metrics are essential to detect potential bias in machine learning-based decision-making systems. However, many of these metrics are mathematically incompatible and cannot be satisfied simultaneously. As a result, fairness metrics must be carefully tailored to the specific application scenario and the system objectives. While Large Language Models (LLMs) can assist developers in selecting appropriate fairness metrics, their internal representations of fairness are neither curated nor systematically organized. This paper presents a preliminary evaluation of FairMAid (Fairness Metrics Aider), an approach that combines the interpretive and generative capabilities of a Large Language Model (LLM) with a curated formal knowledge base to deliver context-sensitive recommendations for task-specific fairness metrics. The approach involves two main aspects: (i) constructing a curated and extensive knowledge base of fairness metrics that facilitates the incorporation of fairness metrics emerging from diverse domains and (ii) integrating it with an LLM. Preliminary results show promising recommendation performance in common scenarios, although the responses are not always completely satisfactory in some specific contexts.