How do you create an interaction semantics?

Programming semantics are well-studied. What about direct manipulation semantics? Can they be in bijection?

I'm especially interested in, say, reflecting mathematical semantics in interactive diagrams. With Penrose, we've been asking, eg, what dragging a point outside a circle in a venn diagram means mathematically - it's an "edit" in the mathematical construction.

This is true for other domains that have a formal representation underlying a highly interactive one.

(UI Continuations)

Expressive Query Construction through Direct Manipulation of Nested Relational Results

Karger's lab has a disproportionate number of these kinds of papers!

fructure

(Fructure. BTW I'm not just talking about program editors, they're just a convenient example to point to.)

Direct manipulation (mouse, not keyboard)

What even is a program? You can make great Smalltalk (System Archaeology Seminar) programs without writing much code at all. I'm not talking about no-code/future-code, but rather, writing a trace of behavior by live (not premeditated) interactions on objects.

  • Self Language (and other Morphic environments) turns green empty "morphs" into rich interactive objects just through right-clicking

Structure editing (keyboard, not mouse)

Programming by demonstration

https://observablehq.com/@observablehq/data-wrangler

https://twitter.com/andy_matuschak/status/1435772997155971073?s=21

https://twitter.com/_paulshen/status/1463202129619808259

https://maryrosecook.notion.site/Scapula-9fc4b570f0234d569d982c022435433d

emacs’ context menu mode https://ruzkuku.com/texts/emacs-mouse.html

https://twitter.com/prathyvsh/status/1304420854135021575

https://twitter.com/yoshikischmitz/status/1232914839791493125

SnapGene is very much “gene programming by demonstration” - all your plasmid edits get recorded and diffed into a step by step protocol https://www.snapgene.com/features/#configured-for-automatic-documentation-and-easy-data-exchange

https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell

This is a tiny toy programming system inspired by several ideas:

Turtle graphics as in LOGO, but with a bidirectional user interface as in Sketch-n-Sketch, but using gradient-based updates as in ML systems.

Kartik Chandra's Differentiable Snap! https://kach.github.io/turtlegrad/

https://twitter.com/disconcision/status/1508912020380434439?s=20&t=gYgR5XSK1vgFHppPMZlIdg

https://faculty.washington.edu/ajko/papers/Ko2005Citrus.pdf

Figure 8 of https://profs.etsmtl.ca/mmcguffin/research/2020-mcguffin-VPLs/mcguffin-avi2020.pdf

There's a very deep truth about noun-verb distinction in UI.

Figure 4 of https://andrewhead.info/assets/pdf/augmented-formulas.pdf

https://twitter.com/rsnous/status/1531353260279402497?s=21&t=lenlmBapUWsoEj03jcBbbA

"Documents as user interfaces" https://dl.acm.org/doi/abs/10.1145/108844.108994


Children
  1. Algebraic Interfaces

Backlinks