The basis for any tangible interface that interacts with a computer or electronics is the sensor. Sensors track a change in themselves or the environment and convey this change to the computer in the form of a number. A sensor could track anything from a touch to a stretch or a shake. Using this changing number they output, we can manipulate parameters in a computer. In the case of a musical instrument this could mean the volume, pitch or any number of characteristics that can be manipulated in some way. While thinking through what types of sensors lend themselves to gestural musical interfaces I have come up with the following list. The first group seems a little more straightforward and easier to implement because they only output one value. The second group is a little more complicated to implement and use effectively.
The Simple
Force Sensitive Resistor – This sensor can measure the amount of force being applied. The output value will be different depending on how hard they are pressed. It could form the basis for what’s called a velocity sensitive button. This output could be mapped to any parameter but is often mapped to loudness. The harder the button is pressed the louder the sound is.
Soft Potentiometer – This sensor is a strip that is either linear or circular. The output value will be different depending where on the strip a finger is placed. If a finger is moved up on down the length of the sensor the output value will slide up or down with it. This kind of gesture lends itself to a classic mod wheel style of expression commonly found on synthesizers. However it could also be used to determine where a button is being pressed so the sound is different depending on where it is pressed.
Vibration Sensor – These come in a couple different varieties but essentially do the same thing. When triggered by a vibration this sensor will send a signal to the computer. Think of it like a switch. When still it is off. When it is shaken it is on. Imagine the act of rhythmically shaking a shaker. A very similar kind of motion could be used to trigger a sample or instrument.
Stretch Sensor – The output of this sensor changes depending on how much it is stretched. I could imagine these being used in two ways. The first would be in a pull and push motion. Think about how an accordion is pulled and pushed to generate sounds. Second perhaps as a plucked string. As the sensor is pulled and released the change in output could trigger a sound.
Air Pressure Sensor – The output of this sensor changes depending on the air pressure. Like a horn or a flute this sensor could trigger a sound when someone blows through it. Alternatively it could also be triggered by someone sucking on it. The output value also depends on how forceful one is applying their breath, because of this it could also be used to control how loud the sound is or any other number of parameters.
Flex Sensor – The output of this sensor changes depending on how much it is bent. Two ideas immediately spring to mind when I think of this gesture. The first would be if you’ve ever seen anyone play a saw with a bow. The musician has to blend or flex the saw in order to change the pitch of it. Secondly I could see it being used in a glove like instrument to manipulate a parameter depending on how the user bends their fingers.
The More Complicated
Gyroscope / Accelerometer – This sensor are used to measure position and orientation. This means they output a different number for three axes. This makes the process of interfacing it with a computer more complicated than the above sensors. I could see it being used for many types of gestures including shaking, swaying and leanings. Imagine playing a instrument that as you lean forward, backwards, and to the side the sound changed with those movements.
Capacitive Touch Sensor – How these work is a little complicated to explain, but for our purposes their output value will change when a part of the body like a finger is applied. The value will change depending on what exactly is touching it so two fingers will be different then one. This sensor could be used like a touch screen to know where it is being touched but also what part/s of the body are touching it. It seems obvious how this could be used in a gestural musical interface. Capacitive touch sensors come in many shapes and sizes
On their own each of these sensors can provide an engaging and tangible way for a user to interact with and manipulate sound but I think the most interesting types of interfaces begin to emerge when different types of sensors are combined. Imagine if you will a button or key that uses a Force Sensitive Resistor and a Soft Potentiometer. This button could track how forcefully and where exactly it was being pressed. Essentially a velocity sensitive button that knows where your finger is on it.