Several papers in the literature claim detection of bugs such as buffer overflow through machine learning techniques. Models get trained to report bugs at the function or file level, and reviewers of the results have to eyeball the code to determine whether there is a bug in that function or file, or not. Contrast this to static code analysers which report bugs at the statement level, showing the path to the bug being reported, allowing reviewers of the reports to easily determine if the statement has a bug or not.

Based on our experience with implementing scalable and precise bug finders in the Parfait tool, we experiment with machine learning to understand how close the techniques can get to a precise static code analyser. In this paper we summarise our finding in using ML techniques to find buffer overflow bugs in programs written in the C language. We treat bug detection as a classification problem. We use feature extraction and train a model to determine whether a buffer overflow has occurred or not at the function level. Training is done over labelled data used for regression testing of the Parfait tool. We evaluate the performance of the different classifiers using the 10-fold cross-validation, and the leave-one-out strategy. To understand the generalisability of the the trained model, we use it on a collection of unlabelled real-world programs and manually check the reported warnings.

Our experiments show that, even though the models give good results over training data, they do not perform that well when faced with larger, unlabelled data. We conclude with some of the open questions that need addressing before machine learning techniques can be used for detecting buffer overflow errors.