ISPY: Automatic Issue-Solution Pair Extraction from Community Live Chats
Community live chats contain rich sets of information for potential improvement on software quality and productivity. One of the important applications is to mine knowledge on issues and their potential solutions. However, it remains a challenging problem to accurately mine such knowledge due to the noisy nature of interleaved dialogs in live chat data. In this paper, we first formulate the problem of issue-solution pair extraction from developer live chat data, and propose an automated approach, named ISPY, based on natural language processing and deep learning techniques with customized enhancements, to address the problem. Specifically, ISPY automates three tasks: 1) Disentangle live chat logs, employing a feedforward neural network to automatically disentangle a conversation history into separate dialogs; 2) Detect dialogs discussing issues, using a novel convolutional neural network (CNN), which consists of a BERT-based utterance embedding layer, a context-aware dialog embedding layer, and an output layer; 3) Extract appropriate utterances and combine them as corresponding solutions, based on the same CNN structure but with different feeding inputs. To evaluate ISPY, we compare it with six baselines, utilizing a dataset with 750 dialogs including 171 issue-solution pairs and evaluate ISPY from eight Gitter communities. The results show that, for issue-detection, our approach achieves the F1 of 76%, and outperforms all baselines by 30%. For solution-extraction, our approach achieves the F1 of 63%, and outperforms the baselines by 20%. Furthermore, we apply ISPY on three new communities to extensively evaluate ISPY’s practical usage. Moreover, we publish over 30K issue-solution pairs extracted from 11 communities. We believe that ISPY can facilitate community-based software development by promoting knowledge sharing and shortening the issue-resolving process.