Machine learning (ML) is increasingly used in high-stakes areas like autonomous driving, finance, and criminal justice. However, it often unintentionally perpetuates biases against marginalized groups. To address this, the software engineering community has developed fairness testing and debugging methods, establishing best practices for fair ML software. These practices focus on training model design, including the selection of sensitive and non-sensitive attributes and hyperparameter configuration. However, the application of these practices across different socio-economic and cultural contexts is challenging, as societal constraints vary.

Our study proposes a search-based software engineering approach to evaluate the robustness of these fairness practices. We use first-order logic properties and socially critical datasets to find neighborhood datasets where the fairness properties fail. This is challenging due to factors like noise, faulty labeling, and demographic shifts. We use causal graph representations of datasets and search algorithms to explore perturbations of causal graphs, assessing the robustness of fairness properties. Our research on five causal graphs from fairness-sensitive applications shows that many best practices are not robust to small changes in the causal graph. This highlights the need for a more localized approach to ensure fairness in ML software development.