Noise Machine update

Me: This vocabulary exercise would be so much easier if we could show pictures of sound!
John: You’d be pretty fucked if you were a blind music technology student.
Me: Wait but you would still be able to hear sound and you could turn pictures into sound and then listen to the visual representation of sound which would…oh.
John: Yea.

My noise machine will be something like a barcode scanner except it turns image data into sound. There’s a lot of software out there that accomplishes this, eg Photosounder, Coagula, and Metasynth (and SPEAR for non-raster spectrogram), but I don’t think any of them does it in real-time (actually there probably is a way with Max but it’s not as flexible/efficient as I want it to be). It will be retrofitted onto a 2d rangefinder that my roommate and I hacked together for Build18, so it will be able to determine, not only color, but distance as well. So a given datapoint would contain information about color, distance, and horizontal (or vertical if the scanner is oriented vertically) position (in other words, a 5d vector: <r,g,b,d,x> or <h,s,l,d,x> or <h,s,v,d,x>)

I know, for certain, that low x will correspond to low frequencies, and high x will correspond to high frequencies, similar to a piano. It would also probably be in a logarithmic scale to correspond to pitches rather than linear frequency, since that is more intuitive/interesting to listen to. That being said, I’m uncertain about the mappings of all the other dimensions. I’m aiming for a machine that people can play around with, learn about sound in the frequency domain, and potentially be able to create rudimentary music with. One idea that came to mind was having the scanner be static and having a flexible whiteboard attached to a conveyor belt with a crank so people can draw sounds and intuitively play them back. Another idea is having distance mapped to amplitude so that frequencies would be louder the closer something is, allowing for playing around with physical objects and body parts.

Again, my noise machine is mostly based off of the Spectral Tablature, with some additional inspiration from my objective description of reduced listening and how I was having some difficulty explaining frequency analysis and spectrograms.


7 thoughts on “Noise Machine update

  1. This project sounds fantastic and it sounds like you have everything thought out so it’s difficult to think of suggestions. Having distance be tied to amplitude makes intuitive sense to me, as does the horizontal being mapped to frequency. It would be very interesting to me if lines could be drawn to make glissandi.

    If this turns out to be a static scanner on a conveyor belt, perhaps the beginning of the piece could have some kind of coded data that explains what kind of sound to use. For example, you could begin the roll of images and at the very top where it begins, there is information that tells the computer that pure red corresponds to a sine wave (rudimentary example) but blue corresponds to a triangle wave. Maybe this way, the composer could decide how he wants to encode the music. Just a thought.


  2. Hi David,

    I’m very curious about your eventual solution to mapping in this context, and how that will affect people’s use of the machine. (Or, alternatively, how the desired interaction will affect your mapping decisions.)

    Initially when you mention a bar-code scanner, I envision a person essentially scanning an image (a visual space) and then waiting for their sonic polaroid, if you will (I feel like I’m dating myself with the polaroid….) Another mode I imagine is more interactive – where someone is essentially improvising sonically, and the visual input is really just source material. I’m not sure if the distinction is clear, or really that important, but in my mind one is more static, and one is more dynamic. Which mode is more interesting to you?

    Anyway, the article below is somewhat relevant if you are more interested in the dynamic mode, where the machine would become more of an instrument.

    In my experience, mapping is always the hardest part. Good luck!

  3. Hi David,

    This idea is so brilliant and profitable! I cannot give you any suggestions on the software and hardware components (since I do not know them as well as you do), but it does seems like you have everything thoroughly planned out. I am interested in how you are going to make the “context” of the picture available through sound. I understand that you can transform pixels and colors to different frequencies to signify their differences, creating a relationship between the visual and the audio. For me, images are more contextually and conceptually based, having settings and/or subjects that have their own symbolizing sounds. As for your “noise machine”, it reveals the images themselves by breaking down their visual sensory components. Hence it will be hard for the audiences to combine all the “noises” back into a complete image, which I think is part of what the machine is trying to do.

    – Ruby

  4. Sup! Awesome idea! I’m glad you’ve checked Metasynth~
    I’ve been thinking of doing something similar for a while now. Last year, around this time, I wrote a piece of software myself too.

    It pulls photos from flickr tags, and makes music from them. You’ll notice the code isn’t the best – that’s cuz I’m not a big programmer 😀 But my suggestion would be to spend at least a week thinking the mapping. It changes everything. I guess my questions are :

    – I can see that you want to use the rangefinder you’ve built. Respect. But for the sake of saving time : Why not use Kinect ?
    – If you’re thinking of converting the data to “sound” are you planning to make the software that makes the sound ? Or are you planning to make a midi/osc controller that spits out data based on the color/depth etc. and use that data in a DAW with your plugins to make the sounds?
    – Is it going to be for windows only ? (pleaseee make mac too – or use OFX or PD or something so everyone can use it. Try to make it platform independent)
    – Also have you seen what Jorge’s trying to do ? Perhaps you guys could work together.

    • A major detail that I’ve forgotten to specify is that all of this is happening on the Raspberry Pi. The program directly extracts distance/color information and I’m unfamiliar with the Kinect (and no color). I will probably use some DSP library to transform the information into audio using the inverse FFT, ie synthesizing rather than sending control data. Unfortunately, since this will be tightly coupled to the hardware itself, it won’t be cross-platform.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s