This is the record of an ongoing attempt to build a virtual reality front end for VCV Rack, a free and open-source software modular synthesizer.
There are two components: a virtual reality application that reinterprets the VCV Rack user experience in 3d, built with Unreal Engine, and a dedicated module for VCV Rack that handles introspection of the running rack state and communication with the Unreal application.
I’ve been doing a lot of thinking since the—honestly, strangely serendipitous—discovery that I had to either switch game engines (because I was probably most likely violating Unreal’s EULA), or find my way around some things I had written off as impossible. Both felt daunting. Discouraging. But I’m thinking I might actually manage the latter. I’ve been thinking of what that might mean, being able to more clearly define what has become a very murky boundary between the two halves of this project. I think it’s time to get focused.
With texture acquisition maybe kind of solved, the next hurdle comes from file handling. In my early experiments, I couldn’t figure out a way to get my module in Rack to load new patches without crashing Rack itself. I eventually gave up, and awkwardly handled loading by signaling to Rack to shut down, waiting, and opening it back up with the patch path as an argument. Besides being awkward, loading patches this way is problematic for two reasons…
Time for a demo of the current state of the project. We have a library, context menus, super wiggly cables, and all the other stuff we’ve seen up to this point. Here it is, all coming together. I have a hard time believing this is all my work. I am beside myself, and so proud of what I’ve managed to accomplish in the last year.
Went off on a little tangent today and figured out how to semi-reliably pull the “main” color from an SVG, to use as the color of the non-textured faces of the mesh the SVG is painted on (ie the back and sides of a module, as seen above). Here’s how it works, for the curious…
No more putting it off. Let’s get this thing running in VR. A partial to-do list: give some thought to performance, support a head-mounted display and motion controllers, adapt my existing controls to two independent hands, and…
One of the biggest questions I had coming into this project was how I’d get the graphics from Rack into Unreal. On the Rack side, by default, everything is vector graphics. That’s handled by loading SVG files or drawing graphics dynamically with nanovg. Unreal, however, does not natively support vector graphics…
Before complicating things with VR motion controllers, I want to make sure I have the basic logic behind user interaction in place using only a mouse pointer…
As I begin working on user interaction I quickly find that I have merely arranged the component parts in space, and nothing is actually attached to anything else. Initial attempts to rectify this situation lead to some amusing outtakes…
Now that I’m confident I’m getting data into Unreal in a consistent and performant way, I’m adding some simple geometry sketches to get started on user interaction…
For the next step in sending data from Rack to Unreal, we’re going to start getting real-time with light updates. First, I need to get the position and size data for the lights over to Unreal like I’m doing for everything else, and while I’m at it…
My custom module is now sitting in the virtual rack with the other modules, and—after a thousand debug statements and recompiles to figure out how and where to find the data—it is introspecting the state of the running Rack instance. It is gathering data about the other modules…
For as long as I can remember I have been fascinated by synthesizers and electronic music. I had seen modular synthesizers, and I must have intuited that their purpose was to give an artist lower-level control over the sounds they were making, but they mostly remained a mystery to me. Recently, at the yearly Bandcamp employees offsite, one of the special guests they flew out to talk with us was Suzanne Ciani…
