Patchbaying .NET, part 2

It’s been some time, but I’ve been continuing with the visual patching system for .NET that I wrote about last time. After playing around with Cairo more and more, I decided that the component model offered by WinForms, while inferior in flexibility and aesthetics, gave enough of an upper hand in terms of developing controls that I would go ahead using that. Mono’s support for WinForms seems to be increasing all the time as well, so at least for now, I’ve decided to stick with it, and focus more on the gritty details of actually getting the system working well.

I still have a long way to go with the infrastructure, but I’ve gotten to the point now where I can realistically start developing targeted packages. My first focus is on getting an OpenGL framework going, for which I’ve found the Tao Framework quite handy. Unfortunately Tao still leaves a few things to be desired. For instance, OpenGL enumerations are simply wrapped as integers, which while true to the spirit of the underlying specification, is still a royal pain compared to the much more semantically strong .NET enumerations. Overall, though, Tao is proving itself to be a quick and easy way to get the functionality I need into this framework. It’s definitely nice to not have to deal with writing the interop code myself.

So far I’ve got a primitive mesh class written, which reads in WaveFront/Alias .OBJ formatted mesh data and renders it, currently using immediate mode. Obviously this is slow and silly, but it serves its purpose as a prototype. I foresee good things happening soon on this route.

Of course the development and brainstorming for all of this has led me to realize how much potential a program like this actually has. The concept of dataflow programming has somehow managed to escape the larger consciousness of the so-called computer science community. Certainly there are many people familiar with the concepts, and even a few working on such languages (many of the visual languages, in fact, lend themselves to dataflow naturally) but overall it seems to be much neglected, especially considering the prodigious benefits it offers. Things like parallelism, which are commonly viewed as difficult to handle in traditional languages, reduce to simple graph traversal problems in the dataflow paradigm. Long unconnected paths in a DAG of the dataflow system can be automatically parallelized by a computer. Sorting the operational flow by dependencies with a topological sorting gives the computer ample opportunity to optimize the problem. This stuff is cool, and it’s a shame it isn’t used more widely in the computing community. My hope is that, by bringing some of these concepts to the .NET world, where interop is so blindingly easy, many problems can be ameliorated. Exporting patches as C# source or even emitting assemblies directly are planned features.

Patchbaying .NET

As I’ve explored more and more audio and visual software, I’ve had my interest piqued by patchbay style graphical languages, such as Pd and Max/MSP. Particularly in the VJ community, softwares such as Salvation and VSXu have made it easy to assemble powerful sound-reactive visuals, and tweak them on the fly using the same interface.

My last attempt to explore this subject dealt with visualizing Python object graphs for making visuals. I consider it a qualified success, but Python is a funny language. The weak typing and late binding make coding extremely flexible and mutable, but unfortunately also leads to a dearth of useful metadata. Contrasting this to .NET, which has metadata oozing out everywhere, I decided to try a similar experiment with that platform.

Objects in .NET tend to store references to other objects, forming a graph in memory. Connected portions of this graph are traversed at various times, such as during serialization, but being able to see a visual depiction of the graph could be really nice for some high level control. The best part is, due to the metadata, the visual depictions can automatically deduce what controls and connections of the underlying objects should be displayed.

My prototype uses WinForms, simply because it is convenient. A list displays several .NET types which can be instantiated by dragging them onto the main canvas. In this primitive version, only classes with default constructors can be created, but it’s enough to get a picture of the potential. Objects which implement a custom IModule interface can also have custom WinForms controls added to their visual module.

All the public properties of a particular object are retrieved via reflection and listed on the left edge of the module. These can be connected to other modules, as long as the types are compatible, and these connections will be displayed via arrows. When the connection is made visually, the system calls the property’s setter (again via reflection) with the linked object as the value parameter. A properly designed object should then have a reference to the linked object, which it can internally call for whatever it wishes.

There are a few caveats. Most notably, only reference types will properly ‘connect’ currently, since value types are simply copied. Therefore, for my beginning experiments, I created a class Scalar, which is basically just a referencable holder for a float. Furthermore, Scalar can be subclassed in order to implement IModule in different ways: for instance, a value controlled by an on-screen linear slider, or one controlled by a text box. However, it would still be nicest if regular value types could be used directly. Why is this a problem?

There seem to be two fundamental approaches to module-wire computation systems. The method which I’ve used so far in my systems, in which the module objects store their own data, including connections to other modules which they can call at any time, could be termed a data pull system. The method embraced by Pd and others takes a different route, forcefully pushing the data from into a module, getting the result, and transferring that to the next module. The difference can also be thought of as where the data is kept: in the former, it’s stored in the modules, while the latter it’s (conceptually) in the wires themselves.

The push method clearly obviates the problem with value types mentioned above, as they are constantly being copied to their successors in the signal chain. Unfortunately, just when these copies should be made is another unanswered question. One of my big frustrations with Pd was its inherent update rate tied to the audio frequency. This is great for audio, and solves the question of when to push data for that specific domain, but it essentially amounts to a magic number for any other type of system. If I want to create a visualization system that has nothing to do with audio, why should the audio driver rate matter at all? Figuring out an appropriate time to push value types remains an open problem for me.

2870The WinForms prototype was a fun experiment, and verified .NET’s feasibility to me for this sort of project. I personally hate coding for WinForms, though. It’s convenient for the standard Windows interfaces, but patching connections between modules isn’t such an application. The custom control rendering is hackish, sloppy, and slow, because it doesn’t really fit in with the traditional application model. Additionally, I love Linux and Mono’s support for WinForms, while rudimentary, still leaves plenty to be desired. Its underlying drawing layer, though, known as Cairo, holds much promise. After some hacking around I got a nice little test application rendering some vector graphics onto an OpenGL texture. My plan for the next step is to redevelop the patcher using Cairo to draw the main interface on top of an OpenGL window which can be used render 3d scenes, controlled by the patchbay interface in real time. It should be cool.

Unified text-based and visual programming

I’ve tried quite a few graphical programming languages over the years, such as Pure Data (pd), but having experience with more traditional text-based languages, was always left frustrated by the seemingly roundabout way of data entry. The same things that attract me to vim and Dvorak made me long for more convenient methods.

At the same time, there are some things which are much better suited to graphical programming environments. It’s easier to keep track of variables, since they just sit right in front of you. Controls can model traditional interfaces such as knobs for tweaking values, making real-time manipulation much more friendly.

Obviously, both systems have their advantages and disadvantages. While typing ‘x = y + z*2’ is much quicker and more concise than navigating multilevel menus to create discrete operators and operands in a visual system, finding exactly that right shade of indigo is much nicer with a typical palette finder rather than guess and checking with RGB triplets. Both systems are equally capable, but some tasks lend themselves more naturally to one system than the other. Being able to pick and choose which to use at any given time led me to attempt a parallel model.

In Canvasthesia I’m using Python to implement much of the higher level functionality. It’s quite a versatile language, and has the somewhat unusual ability to interpret code from an interactive console. I have a few classes which represent various types of entities, such as a Renderable entity, which startlingly enough, renders something in a scene. Adding a custom Python descriptor (EntityConnection) to a class derived from one of these base entity classes lets the system know that a particular type of entity can be connected to it.

class Test(vj.Renderable):
testlink = vj.EntityConnection(vj.EntityType.Renderable)
def __init__(self):
def Render(self):

This simple renderable entity does nothing of interest, as evidenced by its minimal Render method. However, it does contain a link to another renderable entity via its testlink attribute. Because it is an EntityConnection descriptor, it is automatically added to any instance’s corresponding visual control.

A few lines typed into the console creates a couple objects and adds them to the scene:

test = Test()
test.testlink = Test()

Although only a single Test instance was directly attached to the scene, a simple depth-first search reveals that a second actually exists. Additionally, it is evident that there is a connection between them via the first’s testlink attribute. The patchbay shows this intuitively:Patchbay test

Likewise, a new entity could be created in the visual editor and subsequently accessed by the Python console. They are simply parallel frontends manipulating the same backing objects, allowing whichever is convenient to be used at any time. It’s the best of both worlds.