Some of my friends know me to have a near-obsession over the emerging NUI paradigm, known to most people through little things like Apple’s iPhone or Jeff Han’s famous video. When I first saw the video, it struck me, as many, that a big change was coming. Others have felt similarly to me, but some have failed to grasp the full picture, and dismiss this revolutionary development out of hand when they try to use it with their favorite software and find it less than usable. I too encountered this disappointment years ago, when I first tried the experiment. The touch hardware is only one side of the story, but this takes a while to realize.
I started recording music on a Tascam four-track in high school, and being the nerd that I am, quickly moved into DAW land before that term really had a meaning (it’s still rather nebulous), and have been recording for my own sake ever since. As music and technology advanced (and they both have), and as I grew, I began to tire of recording. I love to hate on Ableton, but their Live software really did shake up the game. Here, finally, was an application which truly transformed the computer into an interactive musical instrument. Recording had finally broken free of the linear-time it had been bound to ever since the days of tape (well, Robert Fripp might take exception to that assertion, but he himself is exceptional).
As I began to play more with the software, I of course became frustrated with the mouse interface. I was trying to make music, not check my e-mail, damn it! Taking my hands off the guitar, picking up a mouse, finding the cursor on the screen to know how to move, it, clicking a weird button… the ergonomics were just too frustrating to make continued use of in a studio environment, let alone in performance. Being a highly process-oriented individual, I began to think of alternate solutions. A foot controller like Behringer’s FCB1010 is of great utility, but lacks the dynamic feedback capabilities that make modern screens so beautiful. It became clear that a touch screen would offer the best of all worlds. So I bought one. Only it turns out, all that software really was designed for a mouse and keyboards. The icons were just too small for my fat fingers to accurately press! On top of that, things like rotary faders don’t have a consistent feel that maps to fingers. The mouse pointer can disappear when the button is clicked, but my finger stays put. Do I want to drag up and down with my finger, well away from the display of the rotary control, in order to rotate it, really?
Fast forward. We are on the verge of cheap, omnipresent multitouch-capable hardware, with operating system-level support. This will be most radically felt in the multimedia and artistic realms, as intuitive interface diminishes the barrier of entry caused by software learning curves. Design interfaces to respond to the expressive nature of gesture, and people lose their fear of experimentation. Create interesting parameter mappings between the physical input and the digital result, and configuration choices cease to be overwhelming.
All this is a bit long winded to get to what I really want to address. Steinberg’s VST audio plug-in architecture is a defacto standard with a mind-boggling assortment of third-party offerings. Most plug-ins also offer custom GUIs for editing their parameters. Unfortunately, the vast majority of these work extremely poorly with touch. How hard would it be to add some extra exports to a VST library to expose a NUI presentation in parallel to the common GUI? Would it be possible to retrofit a layer to fit over existing VSTs? Perhaps some recommendations, if not a formal specification, for the pieces of VSTGUI to avoid in a NUI case– for instance, CCursorType doesn’t really make sense at all, and neither does CMouseWheelAxis. But there are more subtle things: what knob mode works best for touch screens? In my experience, linear click drag (kLinearMode) doesn’t seem to make much sense for touch, but for kCircularMode, care must be taken that the knob’s representation is sized large enough for the fat fingers to accurately manipulate. After all, they don’t get any of the tactile feedback of a real physical knob, so proper placement is pretty much all visual. In any case, it would be great to see some guidance on retrofitting the myriad VSTs out there to transition them from WIMP to OCGM in a timely fashion. I suppose, as always, it’ll just take some time for the adjustment to soak into the collective consciousness. And, as always, I’ll be patient but eager.