How do you create an interaction semantics?

Programming semantics are well-studied. What about direct manipulation semantics? Can they be in bijection?

I'm especially interested in, say, reflecting mathematical semantics in interactive diagrams. With Penrose, we've been asking, eg, what dragging a point outside a circle in a venn diagram means mathematically - it's an "edit" in the mathematical construction.

This is true for other domains that have a formal representation underlying a highly interactive one.

(UI Continuations)

Expressive Query Construction through Direct Manipulation of Nested Relational Results

Karger's lab has a disproportionate number of these kinds of papers!


(Fructure. BTW I'm not just talking about program editors, they're just a convenient example to point to.)

Direct manipulation (mouse, not keyboard)

What even is a program? You can make great Smalltalk (System Archaeology Seminar) programs without writing much code at all. I'm not talking about no-code/future-code, but rather, writing a trace of behavior by live (not premeditated) interactions on objects.

  • Self Language (and other Morphic environments) turns green empty "morphs" into rich interactive objects just through right-clicking

Structure editing (keyboard, not mouse)

Programming by demonstration

emacs’ context menu mode

SnapGene is very much “gene programming by demonstration” - all your plasmid edits get recorded and diffed into a step by step protocol

This is a tiny toy programming system inspired by several ideas:

Turtle graphics as in LOGO, but with a bidirectional user interface as in Sketch-n-Sketch, but using gradient-based updates as in ML systems.

Kartik Chandra's Differentiable Snap!

Figure 8 of

There's a very deep truth about noun-verb distinction in UI.

Figure 4 of

"Documents as user interfaces"

  1. Algebraic Interfaces