How do you create an interaction semantics?
Programming semantics are well-studied. What about direct manipulation semantics? Can they be in bijection?
I'm especially interested in, say, reflecting mathematical semantics in interactive diagrams. With Penrose, we've been asking, eg, what dragging a point outside a circle in a venn diagram means mathematically - it's an "edit" in the mathematical construction.
This is true for other domains that have a formal representation underlying a highly interactive one.
Expressive Query Construction through Direct Manipulation of Nested Relational Results
Karger's lab has a disproportionate number of these kinds of papers!
(Fructure. BTW I'm not just talking about program editors, they're just a convenient example to point to.)
Direct manipulation (mouse, not keyboard)
What even is a program? You can make great Smalltalk (System Archaeology Seminar) programs without writing much code at all. I'm not talking about no-code/future-code, but rather, writing a trace of behavior by live (not premeditated) interactions on objects.
- Context menus are cool, actually
- UI Continuations - curried context menus!!!
- Histogram by Tomas Petricek - "We represent programs as lists of interactions"
- Wrangler: Interactive Visual Specification of Data Transformation Scripts
- You can model the domain of raytracing by interacting with InkScape layers
- Interaction Techniques in InfoViz (Ji Soo Yi et al)
- Brad Myers' TOPAZ
- Self Language (and other Morphic environments) turns green empty "morphs" into rich interactive objects just through right-clicking
“Verb-noun vs noun-verb” https://t.co/eNPRq3AqrC
— TOoᴼᵒº˙⁰0₀ₒ.·ph (@tophtucker) July 1, 2020
Thanks to Omar https://t.co/WZk6BIbXX4 for reminding me of Amit’s excellent post; see also https://t.co/fdf7EPmvKW.
Anyone have any other good literature on subject-verb order in GUIs? https://t.co/ntvKPyO4cd pic.twitter.com/BCLu2D7WVH
Structure editing (keyboard, not mouse)
an interface as a thin veneer over a structured program, manipulable in a structured manner, always in bijection 🥰
— Nintendo .DS_Store (@sliminality) February 13, 2020
Programming by demonstration
https://observablehq.com/@observablehq/data-wrangler
https://twitter.com/andy_matuschak/status/1435772997155971073?s=21
https://twitter.com/_paulshen/status/1463202129619808259
https://maryrosecook.notion.site/Scapula-9fc4b570f0234d569d982c022435433d
emacs’ context menu mode https://ruzkuku.com/texts/emacs-mouse.html
https://twitter.com/prathyvsh/status/1304420854135021575
https://twitter.com/yoshikischmitz/status/1232914839791493125
SnapGene is very much “gene programming by demonstration” - all your plasmid edits get recorded and diffed into a step by step protocol https://www.snapgene.com/features/#configured-for-automatic-documentation-and-easy-data-exchange
https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell
This is a tiny toy programming system inspired by several ideas:
Turtle graphics as in LOGO, but with a bidirectional user interface as in Sketch-n-Sketch, but using gradient-based updates as in ML systems.
Kartik Chandra's Differentiable Snap! https://kach.github.io/turtlegrad/
https://twitter.com/disconcision/status/1508912020380434439?s=20&t=gYgR5XSK1vgFHppPMZlIdg
https://faculty.washington.edu/ajko/papers/Ko2005Citrus.pdf
Figure 8 of https://profs.etsmtl.ca/mmcguffin/research/2020-mcguffin-VPLs/mcguffin-avi2020.pdf
There's a very deep truth about noun-verb distinction in UI.
Figure 4 of https://andrewhead.info/assets/pdf/augmented-formulas.pdf
https://twitter.com/rsnous/status/1531353260279402497?s=21&t=lenlmBapUWsoEj03jcBbbA
"Documents as user interfaces" https://dl.acm.org/doi/abs/10.1145/108844.108994
Children
Backlinks