In the field of software engineering (SE) research, there has long been a focus on automating various development tasks in an attempt to facilitate or augment the abilities of developers. Research aligned with this objective typically aims to learn models from information mined from software repositories and then apply these models to automate a given SE task. The large majority of this work has focused on artifacts consisting of two main modalities of information — code and natural language. However, one information source which has been comparatively underutilized is the visual modality of software expressed via User Interfaces (UIs). UIs serve as an important medium of interaction between the logic of an application and users, and as such, they encode salient information about underlying program functionality into rich, pixel-based data representations. Given the latent information contained within UIs, and the rapid advancement of Deep Learning (DL) techniques for computer vision and natural language processing in recent years, there is a tremendous opportunity to leverage UI-related software artifacts to offer novel forms of software development automation. This talk will explore the exciting potential of mining and learning patterns from user interfaces to support mobile app development, and lay out some community challenges that, if addressed, could catalyze this area of research.

Kevin Moran is an Assistant Professor in the Department of Computer Science at George Mason University. He graduated with his B.A. in Physics and Computer Science from the College of the Holy Cross in 2013, his M.S. in Computer Science from the College of William & Mary in 2015, and his P.hD. in Computer Science from the College of William & Mary in 2018 advised by Dr. Denys Poshyvanyk. His main research interests include software engineering, maintenance, and evolution with a focus on mobile platforms. Additionally, he explores applications of data mining and machine learning to software engineering problems.