How Well Static Type Checkers Work with Gradual Typing? A Case Study on Python
Python has become increasingly popular and widely used in many fields. Dynamic features of Python provide much convenience for developers. However, they can also cause many type-related bugs undetected until runtime, which increases the cost of maintenance. Static type checking is essential to find bugs early, and the introduction of gradual typing and type annotations makes it easier to perform static type analysis. However, it remains to be investigated how well gradual typing improves real bug detection. Therefore, we conducted a comprehensive study on three widely used checkers: MyPy, PyRight, and PyType. We used a benchmark containing 10 popular Python projects with 40 real type-related bugs. First, we performed static type checking on the projects with and without type annotations to evaluate the effectiveness of finding real bugs. Second, we manually analyzed the missing bugs and investigated the reasons. The results show that the three tools can detect 29 of the 40 studied bugs after annotating, while only 14 bugs are detected before annotating. We also found that type annotations can substantially improve the ability of static type checkers to detect real bugs. A detailed analysis of bugs missed by the checkers shows that: (i) the accuracy of type analysis is challenged when it comes to programs with complicated dynamic features, such as dynamically changing object’s attributes, even with annotations; (ii) the inaccurate type annotations can undermine the ability of static type checkers to detect real bugs; (iii) static type checkers have different checking strategies in some cases, which has an impact on real bug detection. Our study can not only enable developers to better understand static type checking and make better use of them but also guide future research.
Tue 16 MayDisplayed time zone: Hobart change
13:45 - 15:15 | Programming Languages, Types, and ComplexityDiscussion / Research / Replications and Negative Results (RENE) / Journal First at Meeting Room 106 Chair(s): Vittoria Nardone | ||
13:45 9mFull-paper | How Well Static Type Checkers Work with Gradual Typing? A Case Study on Python Research Wenjie Xu Nanjing University, Lin Chen Nanjing University, Chenghao Su Nanjing University, Yimeng Guo Nanjing University, Yanhui Li Nanjing University, Yuming Zhou Nanjing University, Baowen Xu Nanjing University | ||
13:54 9mFull-paper | Too Simple? Notions of Task Complexity used in Maintenance-based Studies of Programming Tools Research Patrick Rein University of Potsdam; Hasso Plattner Institute, Tom Beckmann Hasso Plattner Institute, Eva Krebs Hasso Plattner Institute (HPI), University of Potsdam, Germany, Toni Mattis University of Potsdam; Hasso Plattner Institute, Robert Hirschfeld University of Potsdam; Hasso Plattner Institute | ||
14:03 9mFull-paper | Path Complexity Predicts Code Comprehension Effort Research Sofiane Dissem Harvey Mudd College, Eli Pregerson Harvey Mudd College, Adi Bhargava Harvey Mudd College, Josh Cordova Harvey Mudd College, Lucas Bang Harvey Mudd College | ||
14:12 5mShort-paper | Revisiting Deep Learning for Variable Type Recovery Replications and Negative Results (RENE) Pre-print | ||
14:17 9mTalk | Programming language implementations for context-oriented self-adaptive systems Journal First Nicolás Cardozo Universidad de los Andes, Kim Mens Université catholique de Louvain, ICTEAM institute, Belgium Link to publication DOI Media Attached | ||
14:26 9mFull-paper | Improving Code Search with Multi-Modal Momentum Contrastive Learning Research Zejian Shi Fudan University, Yun Xiong Fudan University, Yao Zhang Fudan University, Zhijie Jiang National University of Defense Technology, Jinjing Zhao National Key Laboratory of Science and Technology on Information System Security, Lei Wang National University of Defense Technology, Shanshan Li National University of Defense Technology Pre-print | ||
14:35 9mFull-paper | Revisiting Lightweight Compiler Provenance Recovery on ARM Binaries Replications and Negative Results (RENE) Pre-print | ||
14:44 31mPanel | Discussion 7 Discussion |