Journal First: Learning from Very Little Data: On the Value of Landscape Analysis for Predicting Software Project Health)
When data is scarce, software analytics can make many mistakes. For example, consider learning predictors for open source project health (e.g. the number of closed pull requests in twelve months time). The training data for this task may be very small (e.g. five years of data, collected every month means just 60 rows of training data). The models generated from such tiny data sets can make many prediction errors.
Those errors can be tamed by a landscape analysis that selects better learner control parameters. Our niSNEAK tool (a) clusters the data to find the general landscape of the hyperparameters; then (b) explores a few representatives from each part of that landscape. niSNEAK is both faster and more effective than prior state-of-the-art hyperparameter optimization algorithms (e.g. FLASH, HYPEROPT, OPTUNA).
The configurations found by niSNEAK have far less error than other methods. For example, for project health indicators such as C= number of commits; I=number of closed issues, and R=number of closed pull requests, niSNEAK’s 12 month prediction errors are {I=0%, R=33% C=47%} while other methods have far larger errors of {I=61%,R=119% C=149%}. We conjecture that niSNEAK works so well since it finds the most informative regions of the hyperparameters, then jumps to those regions. Other methods (that do not reflect over the landscape) can waste time exploring less informative options.
Based on the above, we recommend landscape analytics (e.g. niSNEAK) especially when learning from very small data sets. This paper only explores the application of niSNEAK to project health. That said, we see nothing in principle that prevents the application of this technique to a wider range of problems.
To assist other researchers in repeating, improving, or even refuting our results, all our scripts and data are available on GitHub at https://github.com/zxcv123456qwe/niSneak.
Wed 17 AprDisplayed time zone: Lisbon change
14:00 - 15:30 | Analytics & AIResearch Track / Journal-first Papers at Sophia de Mello Breyner Andresen Chair(s): Lingming Zhang University of Illinois at Urbana-Champaign | ||
14:00 15mTalk | DeepLSH: Deep Locality-Sensitive Hash Learning for Fast and Efficient Near-Duplicate Crash Report Detection Research Track Youcef REMIL INSA Lyon, INFOLOGIC, Anes Bendimerad Infologic, Romain Mathonat Infologic, Chedy raissi Ubisoft, Mehdi Kaytoue Infologic | ||
14:15 15mTalk | DivLog: Log Parsing with Prompt Enhanced In-Context Learning Research Track Junjielong Xu The Chinese University of Hong Kong, Shenzhen, Ruichun Yang The Chinese University of Hong Kong, Shenzhen, Yintong Huo The Chinese University of Hong Kong, Chengyu Zhang ETH Zurich, Pinjia He Chinese University of Hong Kong, Shenzhen | ||
14:30 15mTalk | Where is it? Tracing the Vulnerability-relevant Files from Vulnerability Reports Research Track Jiamou Sun CSIRO's Data61, Jieshan Chen CSIRO's Data61, Zhenchang Xing CSIRO's Data61, Qinghua Lu Data61, CSIRO, Xiwei (Sherry) Xu Data61, CSIRO, Liming Zhu CSIRO’s Data61 | ||
14:45 15mTalk | Demystifying and Detecting Misuses of Deep Learning APIs Research Track Moshi Wei York University, Nima Shiri Harzevili York University, Yuekai Huang Institute of Software, Chinese Academy of Sciences, Jinqiu Yang Concordia University, Junjie Wang Institute of Software, Chinese Academy of Sciences, Song Wang York University | ||
15:00 7mTalk | Toward Understanding Deep Learning Framework Bugs Journal-first Papers Junjie Chen Tianjin University, Yihua Liang College of Intelligence and Computing, Tianjin University, Qingchao Shen Tianjin University, Jiajun Jiang Tianjin University, Shuochuan Li College of Intelligence and Computing, Tianjin University | ||
15:07 7mTalk | Fair Enough: Searching for Sufficient Measures of Fairness Journal-first Papers Suvodeep Majumder North Carolina State University, Joymallya Chakraborty Amazon.com, Gina Bai North Carolina State University, Kathryn Stolee North Carolina State University, Tim Menzies North Carolina State University DOI Pre-print | ||
15:14 7mTalk | Representation Learning for Stack Overflow Posts: How Far are We? Journal-first Papers Junda He Singapore Management University, Xin Zhou Singapore Management University, Singapore, Bowen Xu North Carolina State University, Ting Zhang Singapore Management University, Kisub Kim Singapore Management University, Singapore, Zhou Yang Singapore Management University, Ferdian Thung Singapore Management University, Ivana Clairine Irsan Singapore Management University, David Lo Singapore Management University | ||
15:21 7mTalk | Journal First: Learning from Very Little Data: On the Value of Landscape Analysis for Predicting Software Project Health) Journal-first Papers DOI Pre-print | ||