Direct manipulation interfaces are developed in many domains where objects have inherently visual representations. After prototyping phases, however, relying solely on direct manipulation can lead to repetitive copy-and-paste tasks and, furthermore, can make it difficult for expert users to manipulate complex content in reusable and composable ways. What is needed are software interfaces that allow users to freely mix between programmatic and direct manipulation, a combination we dub prodirect manipulation, each of which can be used for its distinct strengths.

As an example of this vision, we present the Sketch-n-Sketch prodirect manipulation editor for designing Scalable Vector Graphics (SVG) images. In Sketch-n-Sketch, the user first defines a program in a small functional language that generates an output SVG canvas, and then directly manipulates the output canvas while the system infers program updates, in real-time, to match the changes. To achieve this novel workflow, we employ a lightweight, trace-based program synthesis algorithm, together with a set of design decisions that mitigate ambiguities without user intervention. We demonstrate that Sketch-n-Sketch allows a variety of examples to be designed and maintained more easily than when using purely programmatic or purely direct manipulation tools alone.

(Joint work with Brian Hempel, Mitch Spradlin, and Jacob Albers)

I am an Assistant Professor at the University of Chicago. I enjoy developing and deploying programming language technology — type systems, synthesis algorithms, and other program analysis techniques — for applications in software engineering and human-computer interaction.