browser-controlled circular sequencer
This is a continuation of some things I’ve been thinking about for a while. These things include:
- uniting HTTP and OSC in some unholy manner
- experimenting with different kinds of interfaces
- making some sort of “instrument” that a group of people could play together
- making some horrible noise
My goal going into this was to create a system that would allow each user to create a circular sequencer in a web browser (most likely running on a smart phone) by dragging elements (like samples, and the sequencer itself) around. The sequencers would all play through a common sound system, and would all be displayed together in some sort of global view.
From the user’s perspective, what you have is a sequencer. You create your own sequence, other people create their own sequences, everybody sees all the sequences up on one big display. This is an instrument that people can play together. Like the world’s worst piano duet. On sequencers. With bad samples. As the video shows, I’ve gotten this to work with the iPhone, so you could, in theory, entice a room full of people with iPhones to lay down some self-organizing beats. I haven’t done much testing on the Palm Pre or Android, but I think it should work, assuming they support the standard browser mouse events. The user first clicks somewhere in the red box to draw the initial circle sequencer. She can then play with the circle. She can click on the edge of the circle and drag to change its size (which changes the amount of time that it takes to get around the circle, which effectively changes the speed of the sequence). She can click in the middle of the circle to move it. She can click outside the circle and drag to the edge of the circle to add a sample (more on that in a minute). And she can click on a sample and drag it to a different part of the circle. There are four samples that can be used, and the user chooses the sample to put on the circle by clicking in outside of the circle in one of four quadrants and dragging to the desired location on the circle. Imagine that the center of the circle is the point where two perpendicular lines meet, giving you four quadrants. The samples break down thusly:
- upper-right = noise
- lower-right = buzz
- lower-left = beep
- upper-left = tweet
So, lots of people can create these little sequencers using their smart phone web browsers. And then there’s a big screen that shows everybody’s sequencers, and shows the current location in each sequence with a little white circle. And there’s some sort of PA that’s blasting out the sequences. And the crowd goes wild.
From a technical perspective, this is an example of how to use a bunch of different technologies to cobble together a demo that explores some random ideas.
Let’s start at the browser. We have your run-of-the-mill HTML and JavaScript. Nothing too fancy here. I’m taking advantage of some of the iPhone’s touch-specific event handlers, but that’s about it. Oh, and I’m using JQuery, which in this day and age is pretty routine. So my HTML and JavaScript handle events and draw pretty pictures. And they send messages to the HTTP server. I’m using some JQuery AJAX and JSON functionality to send messages to the server when things (like circles or samples) get created or moved.
The web server is a bit of custom Python. If you’re familiar with BaseHTTPServer then you know that Python makes it very easy to write your own web server. If not, let’s just assume I’m awesome and I write web servers in between sky diving and fighting terrorists. You can see me starting the server in the video with this command:
sudo python ../polosc/polosc.py 80 cs.not cs.map
The server runs on port 80, and takes two types of input files. The .map files map HTTP request paths to actual files on the system. The .not files tell the server that when it receives a POST request for the specified path it should look at the posted JSON data and use it to send an OSC message (using simpleOSC) to the specified server. The message format is pretty simple. It looks like this:
{"address":ADDRESS, "data":[DATA1, DATA2, DATA3, ...], "type":[TYPE1, TYPE2, TYPE3, ...]}
The “type” list is optional, but it is used to get around the following facts:
- the JavaScript JSON code sends values that look like ints even when I want them to be floats
- the Python JSON libary sees these ints and treats them like ints
- Chuck needs separate handlers for OSC messages that take different arguements, even if they go to the same address
So, now we have an OSC-to-HTML bridge. But where do these OSC messages go?
They go to the program that runs this whole show, circlesequencer.ck, which is written in Chuck. This program takes the OSC messages describing creation and update events, and uses them to construct a model of what’s going on. It then uses this model to play the sequences from each circle sequencer. And it updates the global view, which will be discussed shortly. I chose Chuck because I like the way it handles timing and audio. I probably could have used another language and imported a timing library, but the Chuck timing model is a joy to use. This almost makes up for the fact that Chuck’s data structures are horribly anemic. Almost. Between its insistence on exact type matches for OSC messages and a weak hash-map interface, I spent a lot of time fighting with Chuck to get it to do what I wanted. But I finally got it to work, so I guess I’ll consider that a victory. Earlier I called the sounds “samples”, but they’re actually just really simple waveforms with simple attack-decay envelopes. Chuck is powerful enough that you could put almost any kind of audio that you wanted in here, but I stuck with some minimalist sounds for the demo.
When a change occurs in the representation of the world, the Chuck program sends an OSC message to the global view server. “Global view server” is just a way of saying that I wrote a program that displays all the circle sequencers in one place and shows a little white circle for the current position of each sequence. The global view server is written in Processing and it knows how to handle the OSC messages that the Chuck program sends out.
Finally, to get the video working I had to use PD, because my video recording software wants to use Soundflower to capture application audio, but Chuck and Soundflower weren’t cooperating. So I piped the audio from Chuck out to Jack, got that audio in PD, then sent it out to Soundflower, which was picked up by the video software. And thus I had a video with sound, and lots of audio latency. I eventually got everything cleaned up in iMovie.
I’ll post a little postmortem and post some code later.
(later)
Implementation
I’m going to present this code as-is. It was exploratory, so there are lots of global variables and functions that should have been methods floating around. Things came up, I threw in a quick solution. If you dig around the code, you’ve been warned.
The best way of understanding the system is to think of it as following an MVC pattern. The web frontend acts as the controller, the Chuck layer acts as the model, and the Processing program acts as the view. Things are actually a little messier than that, but thinking in terms of MVC helps orient the discussion.
Controller
I’m not much of a web programmer, so I’ve probably done lots of stupid things here. At a high level, the controller takes mouse or touch events (which we can generally call “up events” and “down events”), figures out where they are relative to the canvas element, decides what to do, and redraws the canvas. The “decides what to do” can be broken down like this:
- If no circle sequencer exists and a down event occurs, create one where the event occurred.
- If a down event occurs within the circle and a little way from the edge, move the circle to the point where the up event occurs.
- If a down event occurs near the edge of the circle but not an a sample marker, change the radius to the point where the up event occurs.
- If a down event occurs near the edge of the circle on a sample marker, move the marker to the point on the circle point where the up event occurs.
- If a down event occurs outside of the circle, place a sample marker on the circle where the up event occurs.
At the end of each of these actions, some JSON-encoded data is send to the HTTP server using an XMLHTTPRequest call, which is taken care of by JQuery’s ajax() function.
I generate a random circle ID every time the page loads. A better solution would be to generate the ID the very first time the page loads and store that in a cookie, then reuse it. An even better solution would be to have the server generate the ID and store that as a cookie. Maybe later.
HTTP-OSC Bridge
Before talking about the model itself, I will take a moment to talk a little more about the HTTP server and the way it converts JSON data to OSC messages. I wrote the HTTP server because I felt like I was constantly rewriting different versions of it anyway for my various web projects, and I thought it might be nice to abstract everything out into an HTTP-OSC bridge. It is written in Python and takes advantage of lots of powerful modules that already exist. Mostly my code just glues these modules together. Just to be up front, it requires Python 2.6 because it uses the json module. You might be able to use an older version of Python and include the stand-alone json module, but I can’t really offer any help there.
I talked a little bit about the format of the JSON messages earlier, but here’s a refresher on what they look like:
{"address":ADDRESS, "data":[DATA1, DATA2, DATA3, ...], "type":[TYPE1, TYPE2, TYPE3, ...]}
Let’s say I wanted to the following OSC message (broken into plain text parts so as to avoid confusion):
/circlesequencer/circle/create,sfff "A7CC9D8" 20.0 80.0 30.0
This would tell the model to create a circle with an id of "A7CC9D8" and a radius of 20.0 at location (80.0, 30.0). To do this, I would send the following JSON data:
{"address":"/circlesequencer/circle/create", "data":["A7CC9D8", 20, 80, 30], "types":"sfff"}
I will talk a little more about the OSC messages when I describe the model.
The OSC forwarding behavior is specified in a .not file (short for "notification"). The file consists of comma-separated entries that look like this:
/command,127.0.0.1,6464
This tells the server that whenever a HTTP POST has a path of "/command", it should generate an OSC message from the posted data and send it to the server "127.0.0.1" on port 6464. The server uses a similar mechanism to specify how to map request paths to filesystem paths.
This is a general-purpose program. The only thing that ties it to the circle sequencer system is the configuration files. So if you want to build something else and you need a way of turning HTTP messages into OSC messages, I would encourage you to consider using this. You know, like if you wanted to play a set using Ableton Live, and you wanted to give the audience control over some synth parameters from their iPhones (using the OSC functionality of the coming-really-soon-but-not-yet Max/MSP integration).
Model
The model is the brains of the operation. It is written in chuck, and it keeps track of all the sequencers and is responsible for updating the current sequencer position. It also generates the sequencers' sounds. When it receives an OSC message it update's it's version of the world. It accepts the following OSC messages:
- /circlesequencer/circle/create,sfff circle_id radius x y
- /circlesequencer/circle/update,sfff circle_id radius x y
- /circlesequencer/marker/add,ssfs circle_id marker_id marker_angle marker_type
- /circlesequencer/marker/update,ssf circle_id marker_id marker_angle
It also sends the messages on to the view after it has updated itself. If the update fails, no message is sent to the view. For example, a client might send it a message saying that a marker has been added to circle "XYZ", but the message will be ignored if the model has not received a "/circlesequencer/circle/create" message for circle "XYZ". In addition to the creation and update OSC messages, it sends messages to the view whenever the sequencer's position has changed (it's position in the sequence, not it's position on screen).
View
The view displays all of the sequencers that the model has told it about. It is written in Processing and uses the oscP5 library (yes, that appears to be blinking text on the page) to handle the OSC messages coming from the model. The implementation is pretty simple.
Postmortem
It Worked!
I started off with a somewhat vague idea, and just started writing code. This was a good exercise in exploratory programming. The only part that was really well thought out beforehand was the HTTP-OSC bridge, and that was a consequence of having written more or less the same code several times in other projects. I knew what I wanted, and I knew how to get it. As I said before, most of these things grew organically, based on immediate needs, so it lacks organization. But ugly code that works is better than a bunch of UML diagrams.
Order of Implementation
I started off writing the HTTP-OSC bridge. In fact, you could say that this project was mostly a test-bed for that. Once I had that in place, I wrote the model. I figured that would give me a good shot at figuring out what kinds of data and operations I needed. I tested the model using curl to send HTTP messages to the HTTP-OSC bridge and simpleOSC and the Python shell to send OSC messages directly. With the model and the HTTP-OSC bridge in place, I started working on the controller. Once I had a first pass at the controller, I moved on to the view. Then I went back and refined the controller.
Repetition
One problem with this approach was that I had to implement the circle and marker classes in JavaScript, Chuck, and Processing (which is really just Java). I know that there are systems out there that let you specify a class in one language and then export to others, but I'm not sure any of them support Chuck. And I wasn't really interested in exploring that for this project. So I went ahead and repeated myself a few times. Not a big deal with a fairly simple set of classes, but a few more layers and a little more complexity might have made it very painful. But by the time I got to the view I more or less knew what I needed, so I was able to code it up quite quickly.
Separation of Concerns
In retrospect it might have made sense to move the code that handled playing the sounds into another system. Basically it would have been another "view" only in this case it would have been for hearing rather than seeing. I probably would have just sent OSC messages from the model to this view whenever I needed to play a sound. Who knows, maybe I would have played around with SuperCollider or something like that.
As an alternative I could have written the model and the view (and even the HTTP-OSC bridge) in the same language and run them as one big executable. This consolidation could have saved me some of the time that I spent re-implementing classes and functions. On the other hand, I would have had to find a language that was expressive in the necessary domains, and I'm not sure that one exists. Chuck handles audio and time very well (and OSC slightly less well), but has no way to handle HTTP or video. Processing does a decent job of drawing pictures, but it lacks Chuck's audio prowress. Ultimately if this was a commercial product I might take what I've learned and rewrite the whole thing in Java or C, but that would be down the road a bit.
The Right Tool for the Job(?)
JavaScript (and JQuery)
If you're writing something to run in a mobile web browser, JavaScript is more or less the ONLY tool for the job. And I don't have a lot of experience with JavaScript, so I don't know that I'm qualified to say much about it. What I can say is that JQuery provides powerful wrapper around the XMLHTTPRequest object. If you're starting out with JavaScript, go ahead and take a look at JQuery.
Python
I've used Python enough that its idioms creep into my general programming. As a result, I didn't have any problem using it to write the HTTP-OSC bridge. The "batteries included" philosophy means that if you want to do something there's probably a pretty easy way to do it using the standard modules. I'll admit that at this point I'm probably too biased to offer an objective opinion.
Chuck
As I've said before, Chuck gives you lots of powerful features. It's fairly easy to send and receive OSC messages. The concept of time is built into the language. And it gives you lots of ways to create and manipulate audio. But there are some things that could be better.
The OSC message receiving system insists on knowing exactly how many arguments each message will have, and the type of each argument. This can make it difficult to use with languages that convert easily between floats and ints and allow that conversion to impact the way they generate the OSC messages that you send to Chuck. I had problems when Python would send an int instead of a float because it got a number from the HTTP-OSC bridge that didn't have a decimal point. You could argue that this is actually a problem on the Python side, but most other OSC systems that I have used allow you to set up a handler on an address and then read out the message arguments one at a time. This at least lets you set up handlers that show you when the program has received a message, rather than just ignoring them.
It would be nice if Chuck let you specify associative arrays using a syntax like this:
{"key1": value1, "key2": value2 . . .}
Other languages let you do this, and it makes it much easier to use them. Also, why doesn't array.size() return the number of elements in an associative array? I had to create a list of array keys and then loop over that to retrieve the values from an associative array. This seems ugly. Maybe there's a good implementation reason.
Processing
Processing gives you some easy ways to draw things in a window. But once you get past the syntatic sugar, you're working with Java. This gives you all the power of Java, but also all the verbosity. As with Chuck, there is no easy way to create things like associative arrays. At least Java gives you lots of powerful datatypes.
I actually tried to get Jython working with Processing. I figured that would give me the best of both worlds, the flexibility of a scripting language with the UI goodies of Processing. Unfortunately I had some problems getting all the classes to load, so I gave up. Maybe I should have tried a little harder.
OSC
OSC is lightweight, fairly flexible, and supported on a number of different platforms. I really can't complain too much.