CADtron: A gesture-based CAD environment

← Back to Kevin's homepagePublished: 2022 July 14

CADtron is a gesture-based 2D CAD prototype exploring the idea that:

  1. Precision mouse clicking is annoying, so
  2. a CAD tool built on pen-input / gestures might feel faster, more fluid, and more fun

After three months of exploration, I’m quite optimistic — I’ll likely start dogfooding the prototype whenever I’m using laser cutters on the regular.

In the mean time, the best overview of the project is this talk I gave at the invitation of the Ink and Switch research group:

Timestamps:
 0:00 intro and motivation
 1:50 Autodesk Inventor demo
 7:40 research hypothesis
 9:30 CADtron demo
14:30 distinguishable gestures
16:20 creating arcs
17:50 integrated slicer
19:30 radial menu, bulk movement / layers
21:00 on symbolic pen interfaces

Thanks

Weekplans

During the initial development I emailed weekly research updates to a handful of friends. These emails are collected below; note the video links.

2022 Jan 31 Week plan - CADtron constraints/dimensions

This week

Add constraints and associated UI to CADtron

Refactor gesture and interaction code into a statechart, so it’ll be more reliable and extensible

Last week

Very happy with CADtron progress, in particular throwing everything together quickly in ClojureScript and pushing hard on getting the SolveSpace C++ library compiled to web assembly so I can use that rather than getting sidetracked writing my own constraint solver.

Neat stuff

Chrome devtools can be used as an interactive debugger for C/C++ compiled to WASM, showing source, allowing breakpoints, frame inspection, etc. Very neat, I’d use this even if I wasn’t deploying my final application to the web.

2022 Feb 7: CADtron design roadmap

This week

Get CADtron to minimally usable state for basic 3D printing:

Document CADtron design roadmap. Initial spike has been good, but now I need to document and prioritize scope for polish vs. new conceptual stuff:

Last week

Got pretty much all of last week’s goals done: I refactored all of the gestures into a statechart, have a lil’ on-screen numpad, and got the constraint solver working.

Here’s a 1 min app demo video (no sound, lots of debug info showing).

Neat stuff

It’s been fun to revisit statecharts and my old code from Subform. Performance has been fine thus far, but in the back of my mind I’m wondering if I’ll need to move geometry and rendering calculations into Rust. I’m curious to see how my friend’s Zaplib framework / company comes along, since it’s designed for exactly the sort of high-performance graphical web app use case.

2022 Feb 14: CADtron gesture language

This week

Refine gestures for fluid geometry creation and manipulation:

Video overlay might be a really fun/useful feature and is worth a shot:

Last week

2m video demo, including voiceover.

I successfully integrated a web-based 3D-printer slicer, including live updates. Shout out to Stewart at Grid.Space for chatting about options and making a few patches to his software to support my “I don’t want no stinking files” workflow.

My “design roadmap” thinking time went pretty much entirely into gestures, not much progress on the rest.

I’m continuing to punt on “proper” geometry handling and associated details.

E.g., rather than building a UI for specifying solids vs. holes, I’m just assuming the greatest-area closed-path is intended to be solid and everything else should be subtracted from that. Hopefully this will suffice for the next 6 months…

Neat stuff

I looked at speakers/audio for establishing the desktop/phone network connection and found this spectacular library which provides 8–16 bytes/second over very cute R2D2 sounds. I don’t think I’ll be able to use it here but am absolutely looking for excuses to use it elsewhere.

2022 Feb 21: CADtron bugfixes, gesture recognition reliability

This week

A month in, this project is starting to feel like what I imagined: 4m demo video.

This week I’m focusing on using the tool and fixing the many lil’ bugs and paper cuts:

Last week

Refactoring the semantics of the gesture system went really well. In particular, I’m happy with how I can now select geometry and constraints in arbitrary order, though this feels risky like any kind of leverage — great when it works as intended, but potentially very confusing/annoying if the system misinterprets what you were actually trying to say.

I was thrilled to get a proof of concept working for “scan this in-app QR code to make your phone’s video magically appear there” (quick demo video) but it’s not a crucial product feature so I’m tabling it for now.

Neat stuff

The “select geometry / constraints in any order” problem led me down a pattern matching rabbit hole and I ended up reading most of Domain Modeling Made Functional, thinking about abstract data types and looking at F# (which, of course, compiles to WASM now!).

I also found Meander which is a fascinating lovechild between pattern matching, logic, and relational ideas. Trying not to nerd-snipe myself into sketching out what it’d look like with explicit types (exhaustiveness checking, optimal performance, etc.).

2022 Feb 28: CADtron core mechanics / immediate feedback

This week

The app doesn’t feel completely fluid yet and it’s still too easy to get into a confusing state or perform the wrong operation. I’m going to walk back some UI complexity and avoid adding features until the core mechanics feel right.

Last week

I never hit my stride last week; after a lot of gesture parameter tweaking and quick fixes for one-offs, the codebase is a mess and the logic is smeared around. One thing I’ve learned is that the original architecture of classifying and dispatching a gesture on “pen up” is insufficient to support immediate contextual feedback while gesturing.

2022 Mar 7: CADtron arcs ‘n discoverability

This week

Circular arcs are the last missing geometric primitive type I need for my CAD designs, and adding them raises two challenges:

  1. How to draw arcs? The app leans on familiar interactions for drawing lines (point-by-point) and circles (double-clicking), but it’s not clear how to make creating arcs both discoverable and fluid. I’ll try show them as ghosted “options” anytime you’re extending from a point; TBD whether this’ll make sense or be too visually noisy/confusing
  2. Constraint limitations. Arcs can be made tangent only to lines with which they share endpoints. This condition is much more specific than the type-based relationships I’ve implemented previously (e.g., horizontal applies to any two points, perpendicular to any two lines, etc.), so I’ll need to rework the constraint matching logic and associated feedback

Sorting these out and making sure arcs can be referenced correctly by all geometry creation and constraining operations will likely take all week.

Last week

6m video. The app feels much better after:

I also added midpoint constraints and backfilled some direct manipulations around resizing circles, snapping lines to circles, etc.

Neat stuff

I’ve been reading Sussman’s new book, Software Design for Flexibility, but it hasn’t gripped me quite yet. I actually find Scheme fairly difficult to read (too many parentheses) and extensible generic functions make me anxious. The book’s examples of extending arithmetic functions to support symbolic algebra and calculus are obviously powerful and impressive, but I wouldn’t want to debug that codebase…

Speaking of reuse, I find myself wanting to jump into statechart states like “manipulating arc” from multiple places (e.g., when first creating a new arc, but also when clicking on an existing arc to resize it). The “create a new arc” transition can either:

  1. synthesize a fake “you clicked on this arc” event and dispatch to the statechart
  2. know what the “you clicked on this arc” transition handler does to the extended variables and do this work itself to “set up” the manipulating arc state before jumping to it.

Neither of these feel great to me; fundamentally the need is to jump into the “manipulating arc” state only after setting up extended variables first (i.e., specifying the arc being manipulated). Curious if y'all have suggestions or ideas.

2022 Mar 14: CADtron live previewing and UI consistency

This week

“Live” interactions like snapping feel great, but I’m not sure how far CADtron can go since geometric constraints can amplify seemingly minor changes. For example, in Illustrator/Figma you can always extend a line by dragging to snap it to something else. But in CADtron that line might have a fixed length constraint, in which case moving it might also move/change a bunch of other geometry — including the geometry you’re trying to snap to!

Existing CAD programs avoid this by just not previewing new constraints, but I’d like to try pushing a bit here; I can think of a few possibilities:

Both of these cases “preview differently” depending on the exact geometric situation at hand, which makes me nervous from a UI consistency/predictability standpoint.

Finally, I’m not thrilled with my code architecture around interactions since it’s difficult to keep straight in my head. I don’t have a strong idea of how to improve the situation, but I want to keep trying while the domain is still (relatively) simple.

Last week

7m demo.

I removed “special case” geometry construction previews and behavior and consolidated into more general “manipulation” behaviors. This is for consistency: Everything should behave the same whether you are creating a “new” line or manipulating an existing one. The most reliable way to do this is to just use the same code for both situations.

I’m happy with this direction conceptually but given the live-preview concerns above it remains to be seen whether I can find a nice unified design or whether it was actually better to have distinct modes for “draw new geometry, unconstrained” and “manipulate existing, potentially constrained, geometry”.

My focus on drawing/manipulating modes kept me from getting to arc UI, and I’m not sure if I’ll get to that this week either.

While focusing on the core mechanics is important, I am wary about over-optimizing them at the expense of necessary-for-CAD features like reusable sub-sketches, repetition, symmetry, etc. However, I have no idea what it’ll take for the drawing and constraint-manipulation mechanics to “feel good enough” for me to move on. ¯\_(ツ)_/¯

2022 Mar 21: CADtron sketching gesture final touches

This week

The core sketching and constraint gestures feel pretty solid, so I’d like to spend this week on final completeness and consistency things so that in April I can move onto new features and concepts. For this week:

Last week

4m video demo

Last week I refined the direct manipulation logic. Initially I thought relying on the solver would work in general, but this turned out to feel very weird (see video), so I backed that out in favor of specific cursor-motion-to-geometry-translation routines that only fall-back to the solver when absolutely necessary. E.g., dragging a circle will just resize the diameter — but if the diameter cannot be changed then the solver will be invoked to translate the circle instead.

I also implemented a “convert to arc” guide that works pretty well and allows for fluid creation and direct manipulation of arcs in a single stroke.

Finally, I fixed up the live-preview glitching by only interpreting gestures in the context of the geometry as it existed at the start of the gesture. This avoids situations where snapping alternates frame-by-frame as a gesture keeps being reinterpreted against “changing” geometry.

For extreme cases where lots of geometry moves after a snap, it may be necessary to show the original geometry ghosted, but I’ll hold off on worrying about that until I actually experience such failure modes in practice.

2022 Mar 28: CADtron layers

This week

A friend is moving house in May and thinks CADTron is pretty close to being usable by him to sketch and refine room layouts, so I’m going to adopt his use case as a milestone for April.

The two major features he’ll need are layers (which can be used to model alternative configuration and visibility toggled) and reusable components (so he doesn’t need to manually copy and keep synchronized between layers his, e.g., living room wall geometry or furniture footprints).

Layers are also a prerequisite for structured geometric computation (the CAD “feature tree”).

For this week:

Last week

I got everything done and more. I started experimenting with bulk move/rotate functionality, which uses visual enclosure to scope rather than the typical approach of a lasso or repeatedly control clicking to explicitly define a selection set.

The move/rotate distinction is made by entering the mode through a radial menu, which itself comes up when right-clicking (mouse) or pressing the pen while holding one of its buttons. This is the first time I’m using any sort of secondary button, and I figured it’d be (lightly) preferable to, e.g., a specific gesture that raises the radial menu.

I don’t feel particularly strongly about the control scheme though, so it’s all subject to change. Worth noting that if I get CADTron working on a multitouch device, the buttons could be replaced with finger/pen distinction.

2022 Apr 11: CADtron layers

Apologies for missing last week, all is explained in the 5 min video update.

This week

I’m not ready to bite off reusable-components yet, so I’m going to spend the week drawing stuff and patching up any awkward/broken functionality.

Off the top of my head, I know there’s stuff like:

Last week

Finished up basic mechanics of layers: show/hide, moving geometry between them, etc. Save/load is also finished.

I’m not sure how I feel about the “implicit selection” via spatial containment of the radial menu — I’ll experiment with a few other operating principles to see if another one “feels right”. (E.g., expanding to the transitive closure of points connected/constrained to the ones physically enclosed.)

Neat stuff

I spent some time thinking about datalog and in-memory databases, notes here. CADtron isn’t hitting a performance wall yet, but I’ll start needing to care relatively soon — I’m on the fence about doing boring / super contextual stuff that’ll probably work (specialized caching, etc.) versus exploring a more general solution (reimplement datascript in Rust; differential datalog for materialized views).

2022 Apr 18: CADtron beta release

This week

Continue polishing:

Another friend (in addition to the one moving house) mentioned they wished they had CADtron recently while doing some laser cutting, so I’m adding a few features to make sure they can use it from their hackerspace’s computer:

Last week

I hardcoded a fixed set of layers and used them to color geometry so it’s more visually apparent how things are partitioned. This got me on a bit of a visual design kick and I spent a while trying to make the colors consistent and otherwise tidy things up.

I also cleaned up the constraints list by hiding all of the point-point coincident constraints (which aren’t actually useful to care about in the abstract — easier to just erase line segments). I also scoped the menu to only show constraints on the currently active layer.