....an interactive sound installation made in collaboration with Tom Hill

The Commission
This work was created in response to a call for proposals by a group called Artfully Sighted. The group is made up of a handful of NRSB members with a range of visual impairments. The group wanted a piece of work that would allow them to experience the gallery space in Nottingham Castle in a different way.

The chosen space for our work was the Long Gallery. As the above picture shows, this space is long and tall. Hard flat walls and a high vaulted ceiling may be perfect for displaying paintings, but the acoustics in the room make having a conversation at normal volumes difficult, your voice seems to just trail off into a blur of reflections. This makes the space disorientating, more so for someone who relies upon sounds and words to make sense of their environment.
We wanted to improve the space through sound, using it not only to make the room less aurally confusing, but also to describe the images it contains.

Tom and I have worked together many times over the years on music projects. In this project we took very clear roles. Tom created sounds, composed the music and did all the design work for the show, while I concentrated on the technology that would allow these sounds to be triggered. This post concentrates on my side of the work. If you would like to read about the music and sound design for the piece you can do so on Tom's blog: here

The Technology

blockwerk orchestra works by tracking the movement of "blocks" around the gallery space. These blocks have special patterns known as fiducial markers printed on them. The markers are a little like a barcode, they can be read at a distance from cameras mounted in the ceiling. Each marker is individual, so the computers attached to the cameras not only knows where a marker is in the room, but also which one it is.

The Grid

The diagram above shows the layout of the room. The operating area of the piece is limited by camera angles, to extend it I used 2 cameras. The regions each camera covers are marked here in red and blue, with each of these being split into a grid of 15 cells. When a block is "seen" in a cell it triggers a sound specific to both that block and the cell it is in, in this way each block can be played much like a giant keyboard by people just walking around the room with them. Each of the 7 blocks has its own unique sound set. For example the blue block controls the sounds of a piano, while the pink block plays the sounds of a cello being plucked. The sounds are played from 4 speakers that mark out the 4 corners of the operating area, the sounds always come from the nearest speaker to the user and in this way give a better sense of the location of particular paintings.

Duets

As well as each block acting as a controller for an instrument (except the black one, more on that later), they can also be used in combinations to trigger small steps in a composition made up of 3 duets. Snippets of these play whenever certain blocks inhabit the same cell. For example if the green (harp) and pink (cello) blocks are seen to be in the same cell on the grid then the computer will play the first note of a duet between celesta and dulcimer. The next step in this sequence will play the next time the two blocks meet in any cell. There are 3 pairings like this, each of them plays its own duet but they will all always stay in time as their positions within the sequence are updated across all 3 duets no mater which is triggered. This means that its possible to play different parts of a duet, with different pairings, it also means that if 6 people were to walk in close pairs of the correct grouping the full 6 instrument piece of music would play, and sound like this:

Blockwek Orchestra Duets by origamibiro

The guide block

The black block is the guide block. It works slightly differently to the others by triggering sound effects that relate to certain pictures around the gallery. This done in 2 ways. We wanted there to be some long ambiences, that would act as kind of summaries of groups of paintings, so Tom created 4 such files to go with the four stretch's of wall covered by the cameras. These were then made into 2 stereo files, allowing me to pan form one ambience to the next dependant on where the block is in the room. So, for example, if you were to stand in the middle of the room you may hear trafic coming from one side, and birds singing from the other, if you stepped closer to the birds then the trafic would slowly fade out leaving you with just the sounds of the wall you are nearest. The second layer of the guide block is in individual sound effects, there are 12 of these that are played when the block enters the cell nearest to the painting they were inspired by. For example if the block is seen in the cell infront of a painting of a ship at sea then the sound of a creaking deck plays from the nearest speaker.

API's

The blockwerk orchestra project would not be possible without a fair amount of technology, and a lot of good open source work created by others. The markers themselves and the method for recognising them is the work of reacTIVision . They were originally developed for use in multitouch screens. The software I have used to actually track these is CCV, created by the NUI group I made a few slight changes to it to help me with tracking the markers over such a long distance. The information from CCV is then passed to MAX/MSP via UDP. I made the max patch myself, it interperates the messages from CCV, defines the grid, and triggers the sounds. It contains quite a lot of very specific tricks to deal with the inaccuracies of tracking this type of marker from a long distance which i am only to glad to share with anyone who wish's to contact me.