Write a Blog >>
Wed 12 Oct 2022 10:30 - 10:50 at Gold A - Technical Session 11 - Analysis and Types Chair(s): Thiago Ferreira

The objective of pre-trained language models is to learn contextual representations of textual data. Pre-trained language models have become mainstream in natural language processing and code modeling. Using probes, a technique to study the linguistic properties of hidden vector spaces, previous works have shown that these pre-trained language models encode simple linguistic properties in their hidden representations. However, none of the previous work assessed whether these models encode the whole grammatical structure of a programming language. In this paper, we prove the existence of a \textit{syntactic subspace}, lying in the hidden representations of pre-trained language models, which contain the syntactic information of the programming language. We show that this subspace can be extracted from the model’s representations and define a novel probing method, the AST-Probe, that enables recovering the whole abstract syntax tree (AST) of an input code snippet. In our experimentations, we show that this syntactic subspace exists in five state-of-the-art pre-trained language models. In addition, we highlight that the middle layers of the models are the ones that encode most of the AST information. Finally, we estimate the optimal size of this syntactic subspace and show that its dimension is substantially lower than those of the models’ representation spaces. This suggests that pre-trained language models use a small part of their representation spaces to encode syntactic information of the programming languages.

Wed 12 Oct

Displayed time zone: Eastern Time (US & Canada) change

10:00 - 12:00
Technical Session 11 - Analysis and TypesResearch Papers / NIER Track / Late Breaking Results at Gold A
Chair(s): Thiago Ferreira University of Michigan - Flint
10:00
20m
Research paper
SA4U: Practical Static Analysis for Unit Type Error Detection
Research Papers
Max Taylor The Ohio State University, Johnathon Aurand The Ohio State University, Feng Qin Ohio State University, USA, Xiaorui Wang The Ohio State University, Brandon Henry Tangram Flex, Xiangyu Zhang Purdue University
10:20
10m
Vision and Emerging Results
Principled Composition of Function Variants for Dynamic Software Diversity and Program Protection
NIER Track
Giacomo Priamo Sapienza University of Rome, Daniele Cono D'Elia Sapienza University of Rome, Leonardo Querzoni Sapienza University Rome
10:30
20m
Research paper
AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models
Research Papers
José Antonio Hernández López Department of Computer Science and Systems, University of Murcia, Martin Weyssow DIRO, Université de Montréal, Jesús Sánchez Cuadrado , Houari Sahraoui Université de Montréal
Link to publication Pre-print
10:50
10m
Paper
Towards Gradual Multiparty Session TypingVirtual
Late Breaking Results
Sung-Shik Jongmans Open University of the Netherlands; CWI
11:00
20m
Research paper
Static Type Recommendation for PythonVirtual
Research Papers
Ke Sun Peking University, Yifan Zhao Peking University, Dan Hao Peking University, Lu Zhang Peking University
11:20
20m
Research paper
Prompt-tuned Code Language Model as a Neural Knowledge Base for Type Inference in Statically-Typed Partial CodeVirtual
Research Papers
Qing Huang School of Computer Information Engineering, Jiangxi Normal University, Zhiqiang Yuan School of Computer Information Engineering, Jiangxi Normal University, Zhenchang Xing Australian National University, Xiwei (Sherry) Xu CSIRO Data61, Liming Zhu CSIRO’s Data61; UNSW, Qinghua Lu CSIRO’s Data61
11:40
20m
Research paper
Jasmine: A Static Analysis Framework for Spring Core TechnologiesVirtual
Research Papers
Miao Chen Beijing University of Posts and Telecommunications, Tengfei Tu Beijing University of Posts and Telecommunications, Hua Zhang Beijing University of Posts and Telecommunications, Qiaoyan Wen Beijing University of Posts and Telecommunications, Weihang Wang University of Southern California