How do you create an interaction semantics?
Programming semantics are well-studied. What about direct manipulation semantics? Can they be in bijection?
I'm especially interested in, say, reflecting mathematical semantics in interactive diagrams. With Penrose, we've been asking, eg, what dragging a point outside a circle in a venn diagram means mathematically - it's an "edit" in the mathematical construction.
This is true for other domains that have a formal representation underlying a highly interactive one.
Karger's lab has a disproportionate number of these kinds of papers!
(Fructure. BTW I'm not just talking about program editors, they're just a convenient example to point to.)
Direct manipulation (mouse, not keyboard)
What even is a program? You can make great Smalltalk (System Archaeology Seminar) programs without writing much code at all. I'm not talking about no-code/future-code, but rather, writing a trace of behavior by live (not premeditated) interactions on objects.
- Context menus are cool, actually
- UI Continuations - curried context menus!!!
- Histogram by Tomas Petricek - "We represent programs as lists of interactions"
- Wrangler: Interactive Visual Specification of Data Transformation Scripts
- You can model the domain of raytracing by interacting with InkScape layers
- Interaction Techniques in InfoViz (Ji Soo Yi et al)
- Brad Myers' TOPAZ
- Self Language (and other Morphic environments) turns green empty "morphs" into rich interactive objects just through right-clicking
“Verb-noun vs noun-verb” https://t.co/eNPRq3AqrC— TOoᴼᵒº˙⁰0₀ₒ.·ph (@tophtucker) July 1, 2020
Thanks to Omar https://t.co/WZk6BIbXX4 for reminding me of Amit’s excellent post; see also https://t.co/fdf7EPmvKW.
Anyone have any other good literature on subject-verb order in GUIs? https://t.co/ntvKPyO4cd pic.twitter.com/BCLu2D7WVH
Structure editing (keyboard, not mouse)
an interface as a thin veneer over a structured program, manipulable in a structured manner, always in bijection 🥰— Nintendo .DS_Store (@sliminality) February 13, 2020
- Fructure - "a structured interaction engine"
Programming by demonstration