CADtron: A gesture-based CAD environment← Back to Kevin's homepagePublished: 2022 July 14
CADtron is a gesture-based 2D CAD prototype exploring the idea that:
- Precision mouse clicking is annoying, so
- a CAD tool built on pen-input / gestures might feel faster, more fluid, and more fun
After three months of exploration, I’m quite optimistic — I’ll likely start dogfooding the prototype whenever I’m using laser cutters on the regular.
In the mean time, the best overview of the project is this talk I gave at the invitation of the Ink and Switch research group:
Timestamps: 0:00 intro and motivation 1:50 Autodesk Inventor demo 7:40 research hypothesis 9:30 CADtron demo 14:30 distinguishable gestures 16:20 creating arcs 17:50 integrated slicer 19:30 radial menu, bulk movement / layers 21:00 on symbolic pen interfaces
Jack Schaedler for the Back to the future of handwriting recognition explorable explanation; CADTron uses a modified version of Jack’s code for gesture recognition.
During the initial development I emailed weekly research updates to a handful of friends. These emails are collected below; note the video links.
2022 Jan 31 Week plan - CADtron constraints/dimensions
Add constraints and associated UI to CADtron
- dimension and angle input via on-screen keypad
- length, point-on-line, angle, parallel, and line-line equality constraints
- remove constraints (probably via tree/list showing all constraints)
Refactor gesture and interaction code into a statechart, so it’ll be more reliable and extensible
- handle when the cursor leaves the browser window
- show hints based on current state/mode (“tap again to add point”, “drawing lines”)
- (bonus) add spacemouse support for zoom/pan and undo button.
Very happy with CADtron progress, in particular throwing everything together quickly in ClojureScript and pushing hard on getting the SolveSpace C++ library compiled to web assembly so I can use that rather than getting sidetracked writing my own constraint solver.
Chrome devtools can be used as an interactive debugger for C/C++ compiled to WASM, showing source, allowing breakpoints, frame inspection, etc. Very neat, I’d use this even if I wasn’t deploying my final application to the web.
2022 Feb 7: CADtron design roadmap
Get CADtron to minimally usable state for basic 3D printing:
- Private deploy to my website so I can show people on calls
- Fix “undo”
- Show contextual hint messages consistently
- Export closed paths directly to slicer; some way to set extrusion depth
- Add circles
Document CADtron design roadmap. Initial spike has been good, but now I need to document and prioritize scope for polish vs. new conceptual stuff:
- How to visualize + manipulate constraints (as icons, annotation lines, implied construction geometry, color/stroke groupings, etc.)
- Gestures: continuous vs. discrete recognition; should I care at all about mouse usability?
- Layers/features: what should be possible within a single sketch and what should require other abstractions?
Got pretty much all of last week’s goals done: I refactored all of the gestures into a statechart, have a lil’ on-screen numpad, and got the constraint solver working.
Here’s a 1 min app demo video (no sound, lots of debug info showing).
It’s been fun to revisit statecharts and my old code from Subform. Performance has been fine thus far, but in the back of my mind I’m wondering if I’ll need to move geometry and rendering calculations into Rust. I’m curious to see how my friend’s Zaplib framework / company comes along, since it’s designed for exactly the sort of high-performance graphical web app use case.
2022 Feb 14: CADtron gesture language
Refine gestures for fluid geometry creation and manipulation:
- Refactor so the two most frequent gestures (selecting point and linework) are pretty much always recognized correctly; implement as non-modal, but make sure it’s possible to make modal later in case non-modal feels like “too many gesture types”.
- Interpret gestures within geometry creation context (e.g., when in line-drawing-mode, circling a point should create a new line segment over to that point)
- Make “undo” gesture easier to draw / more reliable to recognize (this is a common operation!)
Video overlay might be a really fun/useful feature and is worth a shot:
- Display QR code within the app; scanning from a phone/tablet will magically make that device’s video feed appear in the background.
- As far as I can tell, a server will be required to negotiate the initial connection between devices; if you have any clever ideas or know of good libraries / prior art, definitely let me know!
I successfully integrated a web-based 3D-printer slicer, including live updates. Shout out to Stewart at Grid.Space for chatting about options and making a few patches to his software to support my “I don’t want no stinking files” workflow.
My “design roadmap” thinking time went pretty much entirely into gestures, not much progress on the rest.
I’m continuing to punt on “proper” geometry handling and associated details.
E.g., rather than building a UI for specifying solids vs. holes, I’m just assuming the greatest-area closed-path is intended to be solid and everything else should be subtracted from that. Hopefully this will suffice for the next 6 months…
I looked at speakers/audio for establishing the desktop/phone network connection and found this spectacular library which provides 8–16 bytes/second over very cute R2D2 sounds. I don’t think I’ll be able to use it here but am absolutely looking for excuses to use it elsewhere.
2022 Feb 21: CADtron bugfixes, gesture recognition reliability
A month in, this project is starting to feel like what I imagined: 4m demo video.
This week I’m focusing on using the tool and fixing the many lil’ bugs and paper cuts:
- make sure there’s always a status message or other mode indication
- solver should be able to fail and recover gracefully when it diverages (maybe reject bad constraints?)
- highlight constraints and relationships of “selected” geometry
- some way to directly manipulate/drag geometry without precisely clicking on it?
Refactoring the semantics of the gesture system went really well. In particular, I’m happy with how I can now select geometry and constraints in arbitrary order, though this feels risky like any kind of leverage — great when it works as intended, but potentially very confusing/annoying if the system misinterprets what you were actually trying to say.
I was thrilled to get a proof of concept working for “scan this in-app QR code to make your phone’s video magically appear there” (quick demo video) but it’s not a crucial product feature so I’m tabling it for now.
The “select geometry / constraints in any order” problem led me down a pattern matching rabbit hole and I ended up reading most of Domain Modeling Made Functional, thinking about abstract data types and looking at F# (which, of course, compiles to WASM now!).
I also found Meander which is a fascinating lovechild between pattern matching, logic, and relational ideas. Trying not to nerd-snipe myself into sketching out what it’d look like with explicit types (exhaustiveness checking, optimal performance, etc.).
2022 Feb 28: CADtron core mechanics / immediate feedback
The app doesn’t feel completely fluid yet and it’s still too easy to get into a confusing state or perform the wrong operation. I’m going to walk back some UI complexity and avoid adding features until the core mechanics feel right.
- Don’t allow erasing while in geometry-drawing or geometry-constraining modes
- Still need to highlight constraints and relationships of selected geometry
- Immediate visual feedback when selecting points or lines (as soon as they’re circled or crossed)
I never hit my stride last week; after a lot of gesture parameter tweaking and quick fixes for one-offs, the codebase is a mess and the logic is smeared around. One thing I’ve learned is that the original architecture of classifying and dispatching a gesture on “pen up” is insufficient to support immediate contextual feedback while gesturing.
2022 Mar 7: CADtron arcs ‘n discoverability
Circular arcs are the last missing geometric primitive type I need for my CAD designs, and adding them raises two challenges:
- How to draw arcs? The app leans on familiar interactions for drawing lines (point-by-point) and circles (double-clicking), but it’s not clear how to make creating arcs both discoverable and fluid. I’ll try show them as ghosted “options” anytime you’re extending from a point; TBD whether this’ll make sense or be too visually noisy/confusing
- Constraint limitations. Arcs can be made tangent only to lines with which they share endpoints. This condition is much more specific than the type-based relationships I’ve implemented previously (e.g., horizontal applies to any two points, perpendicular to any two lines, etc.), so I’ll need to rework the constraint matching logic and associated feedback
Sorting these out and making sure arcs can be referenced correctly by all geometry creation and constraining operations will likely take all week.
6m video. The app feels much better after:
- Reimplementing gesture recognizer to use 8 cardinal directions rather than 4, which improves accuracy and allows faster-to-draw shapes (including a new undo gesture)
- Having some gestures take effect immediately. E.g., scribbling gesture exits modes as soon as recognized rather than waiting for pen-up. Gestures that latch to geometry also drop you into the mode as soon as selection is recognized.
- Highlighting stuff as soon as it’s touched.
- Recognizing smoothed/thinned paths if the original gesture is non-manifold (which is easy to do accidentally at the start/end of a gesture when manipulating the pen or mouse buttons)
I also added midpoint constraints and backfilled some direct manipulations around resizing circles, snapping lines to circles, etc.
I’ve been reading Sussman’s new book, Software Design for Flexibility, but it hasn’t gripped me quite yet. I actually find Scheme fairly difficult to read (too many parentheses) and extensible generic functions make me anxious. The book’s examples of extending arithmetic functions to support symbolic algebra and calculus are obviously powerful and impressive, but I wouldn’t want to debug that codebase…
Speaking of reuse, I find myself wanting to jump into statechart states like “manipulating arc” from multiple places (e.g., when first creating a new arc, but also when clicking on an existing arc to resize it). The “create a new arc” transition can either:
- synthesize a fake “you clicked on this arc” event and dispatch to the statechart
- know what the “you clicked on this arc” transition handler does to the extended variables and do this work itself to “set up” the manipulating arc state before jumping to it.
Neither of these feel great to me; fundamentally the need is to jump into the “manipulating arc” state only after setting up extended variables first (i.e., specifying the arc being manipulated). Curious if y'all have suggestions or ideas.
2022 Mar 14: CADtron live previewing and UI consistency
“Live” interactions like snapping feel great, but I’m not sure how far CADtron can go since geometric constraints can amplify seemingly minor changes. For example, in Illustrator/Figma you can always extend a line by dragging to snap it to something else. But in CADtron that line might have a fixed length constraint, in which case moving it might also move/change a bunch of other geometry — including the geometry you’re trying to snap to!
Existing CAD programs avoid this by just not previewing new constraints, but I’d like to try pushing a bit here; I can think of a few possibilities:
- If a lot of geometry would change, show that ghosted but continue to show original geometry too (and only use the latter for interpreting gestures)
- Only preview simple changes that affect only the geometry being directly manipulated: If the line can be extended to snap, then extend it; otherwise, preview only abstractly (“your current gesture will snap to this highlighted thing”)
Both of these cases “preview differently” depending on the exact geometric situation at hand, which makes me nervous from a UI consistency/predictability standpoint.
Finally, I’m not thrilled with my code architecture around interactions since it’s difficult to keep straight in my head. I don’t have a strong idea of how to improve the situation, but I want to keep trying while the domain is still (relatively) simple.
I removed “special case” geometry construction previews and behavior and consolidated into more general “manipulation” behaviors. This is for consistency: Everything should behave the same whether you are creating a “new” line or manipulating an existing one. The most reliable way to do this is to just use the same code for both situations.
I’m happy with this direction conceptually but given the live-preview concerns above it remains to be seen whether I can find a nice unified design or whether it was actually better to have distinct modes for “draw new geometry, unconstrained” and “manipulate existing, potentially constrained, geometry”.
My focus on drawing/manipulating modes kept me from getting to arc UI, and I’m not sure if I’ll get to that this week either.
While focusing on the core mechanics is important, I am wary about over-optimizing them at the expense of necessary-for-CAD features like reusable sub-sketches, repetition, symmetry, etc. However, I have no idea what it’ll take for the drawing and constraint-manipulation mechanics to “feel good enough” for me to move on. ¯\_(ツ)_/¯
2022 Mar 21: CADtron sketching gesture final touches
The core sketching and constraint gestures feel pretty solid, so I’d like to spend this week on final completeness and consistency things so that in April I can move onto new features and concepts. For this week:
- restore midpoint constraints, which will require reworking selection internals (since selecting a midpoint is really simultaneously creating a new constrained point, then selecting that point for future operations)
- restore the “counter-clockwise to select for extension” gesture. In hindsight I really liked how fast that felt; I can disable later if other folks are confused in testing.
- select segment and convert to arc (currently the convert-to-arc guides only show up when points are selected)
Last week I refined the direct manipulation logic. Initially I thought relying on the solver would work in general, but this turned out to feel very weird (see video), so I backed that out in favor of specific cursor-motion-to-geometry-translation routines that only fall-back to the solver when absolutely necessary. E.g., dragging a circle will just resize the diameter — but if the diameter cannot be changed then the solver will be invoked to translate the circle instead.
I also implemented a “convert to arc” guide that works pretty well and allows for fluid creation and direct manipulation of arcs in a single stroke.
Finally, I fixed up the live-preview glitching by only interpreting gestures in the context of the geometry as it existed at the start of the gesture. This avoids situations where snapping alternates frame-by-frame as a gesture keeps being reinterpreted against “changing” geometry.
For extreme cases where lots of geometry moves after a snap, it may be necessary to show the original geometry ghosted, but I’ll hold off on worrying about that until I actually experience such failure modes in practice.
2022 Mar 28: CADtron layers
A friend is moving house in May and thinks CADTron is pretty close to being usable by him to sketch and refine room layouts, so I’m going to adopt his use case as a milestone for April.
The two major features he’ll need are layers (which can be used to model alternative configuration and visibility toggled) and reusable components (so he doesn’t need to manually copy and keep synchronized between layers his, e.g., living room wall geometry or furniture footprints).
Layers are also a prerequisite for structured geometric computation (the CAD “feature tree”).
For this week:
- save/load. Not worried about long-term file format or breaking changes, but if he spends an hour making furniture footprints I want to make sure he won’t lose it.
- simple layers: no constraints/relationships between layers, but it should be possible to toggle layer visibility and move geometry between layers.
I got everything done and more. I started experimenting with bulk move/rotate functionality, which uses visual enclosure to scope rather than the typical approach of a lasso or repeatedly control clicking to explicitly define a selection set.
The move/rotate distinction is made by entering the mode through a radial menu, which itself comes up when right-clicking (mouse) or pressing the pen while holding one of its buttons. This is the first time I’m using any sort of secondary button, and I figured it’d be (lightly) preferable to, e.g., a specific gesture that raises the radial menu.
I don’t feel particularly strongly about the control scheme though, so it’s all subject to change. Worth noting that if I get CADTron working on a multitouch device, the buttons could be replaced with finger/pen distinction.
2022 Apr 11: CADtron layers
Apologies for missing last week, all is explained in the 5 min video update.
I’m not ready to bite off reusable-components yet, so I’m going to spend the week drawing stuff and patching up any awkward/broken functionality.
Off the top of my head, I know there’s stuff like:
- editing numeric constraints like pairwise distance, diameter, etc.
- scoping constraint menu to the selection / active layer
- breaking inter-layer constraints when moving geometry between layers
- bulk duplicate would be handy (and maybe eliminate the need for components)
Finished up basic mechanics of layers: show/hide, moving geometry between them, etc. Save/load is also finished.
I’m not sure how I feel about the “implicit selection” via spatial containment of the radial menu — I’ll experiment with a few other operating principles to see if another one “feels right”. (E.g., expanding to the transitive closure of points connected/constrained to the ones physically enclosed.)
I spent some time thinking about datalog and in-memory databases, notes here. CADtron isn’t hitting a performance wall yet, but I’ll start needing to care relatively soon — I’m on the fence about doing boring / super contextual stuff that’ll probably work (specialized caching, etc.) versus exploring a more general solution (reimplement datascript in Rust; differential datalog for materialized views).
2022 Apr 18: CADtron beta release
- add duplicate command
- bulk operations should expand to all connected geometry (including points contained within closed shape, rather than just closed shape)
- circles should be included in enclosure/selection calculations
Another friend (in addition to the one moving house) mentioned they wished they had CADtron recently while doing some laser cutting, so I’m adding a few features to make sure they can use it from their hackerspace’s computer:
- available online at public URL
- navigable via mouse, not just pen + apple trackpad
- probably some kind of scaling when doing SVG export; TBD given their laser cutter workflow
I hardcoded a fixed set of layers and used them to color geometry so it’s more visually apparent how things are partitioned. This got me on a bit of a visual design kick and I spent a while trying to make the colors consistent and otherwise tidy things up.
I also cleaned up the constraints list by hiding all of the point-point coincident constraints (which aren’t actually useful to care about in the abstract — easier to just erase line segments). I also scoped the menu to only show constraints on the currently active layer.