.jpg "Don't Lose The Music")
Synopsis
For this project I decided to focus on the observation of daily life in the city. Like the music I’ve put together, everything builds up to a climax, e.g. daytime. Using the fader controls on the iPhone, every time a faders value is increased a piece of audio and imagery appears at the same time. I started the video off slowly with some ambient music that is accompanied by a image of canary wharf, showing the awakening of the scene. Then using the middle rotary control, at the bottom of the iPhone’s interface, movement begins to develop. As all the fader controls are increased, the whole scene comes to life.
Evaluation
The above video is a pre-recording of a live performance I did at home. I used my iPhone to control Modul8 and Ableton Live together to make a live audio and visual performance. Due to having lots of trial version software, doing a live performance at the crit most definitely wouldn’t have gone to plan. I was using this application called Osculator, which sends out MIDI signals from the TouchOSC on the iPhone to any audio software. The trial version of Osculator has a limited time until it gets interrupted by a message saying that it needs to be activated. I have tried purchasing a serial from the company to activate it, but the activation codes didn’t seem to work so I was still cursed with same issue. However, the pre-crecorded video demonstrates my idea perfectly well.
This project was quite challenging to begin with. When I think back to what my initial ideas were, I feel the final outcome is quite different… Visually that is. The idea of converging both visual performing and audio performing together has been my main focus point of this project, which I feel I’ve achieved.
I started out wanting to make an iPhone application that would allow you to control both audio and visuals at the same time. I later discovered that there was an application called TouchOSC, which allows you to create your own interface, which can then be mapped to any audio software, just like a normal mixer. Before I discovered the TouchOSC application, I found myself looking and making my very own custom made multi-touch trackpad. This was achieved by using a decent sized cardboard box, which was roughly the height and with of an width of an A4 piece of paper. I used a basic picture frame, which would act as the interface. I used a thin sheet of plain A4 paper to go underneath of the glass of the frame. I then installed a webcam inside the box so any movement made on the frame glass will be picked up from the camera and the data will be sent to this piece of software called Community Core Vision.
The brief entailed that the overall piece should combine both sound and visuals that are generated by a live performance. I’m not a musician and I’ve ever used any professional audio software until now. I thought this project would be the best opportunity to get to learn a bit of the basics of creating a simple track in Abelton Live. Phil gave me a good incite into the overall interface of Ableton and I took from him what I thought would be relevant for my idea. Abelton can be used for two purposes, mixing or recording. For me I was mixing… to a certain extent. I basically gathered 6 audio tracks I bought from a CD containing lots of royalty free liquid drum and bass loops. I think mixed them together to make one piece of music that was on a loop. The reason I did this was because it linked into my idea really well. The building up of the start of a day, links in with the building up to a piece of music.
For the visuals I was introduced to another piece of software called Modul8. From the beginning of my initial idea, I was trying to work out if it was possible to use After Effects and be able to use that as a tool to make a live performance with. The reason I was fixated on using After Effects was because of the flexibility and the varied effects it has to offer. However, I later discovered I could use Modul8 to my advantage. After experimenting with bits of footage I had, I realized Modul8 was quite a flexible piece of software and there were lots of things I could do with video and still images.
One of the difficulties I had was mapping the midi controls from Abelton into Modul8. After doing some quick research into this I found out that the IAC Driver, which is already installed on macs, needs to be enabled live so that the data from Ableton can be sent to Modul8; therefore the two programmes become in sync with each other. It was a very simple solution in the end… like most things.
There are definitely ways in which I could improve this project. The audio is one thing I could improve on upon. Throughout the video I’ve used a mixture of looped samples, so if I had to do this again I would possibly look at mixing in more detail to get a varied production in the music. The visuals could also be improved upon by experimenting with more layers and mapping each to different obscure sounds, just to make a scene more interesting. I think if I had a bit more time I’d like to have thought more about how the visuals could be displayed as a performance. The initial idea throughout this project is that they would be displayed on a backdrop/screen behind a DJ in a club environment.
After showing Phil my initial video test, he felt the idea I was trying to portray didn’t reflect from the visuals I had used. I could see where he was coming from because the video didn’t really give any connotations about time passing by, which is what I’m exploring in this video.
I’ve now added different visuals that will hopefully give the audience a sense of what the piece about. The image of Canary Wharf is still going to be included in this, as the main focal point of the visuals, but I’ve enhanced the image with a time-lapse video of clouds sailing through the sky. To do this I used After Effects.
I decided to mask out the sky background of the image to help boost the appearance of backdrop visuals. This was also so I could be in control of that particular area of the image, where I want to embed the time-lapse into.
I imported the video into After Effects and then dragged into composition, on top of the image layer.
Then next thing was to use the Luma Matte effect, which embeds the video into the image like in the screenshot below.
Video Preview
I have got another backdrop for the canary wharf scene. It’s a time-lapse video, which I found in my files, of the clocks by canary wharf… ironically. The whole pice is about time passing and this video fits the description perfectly. Now I wanted to make this backdrop blend well without it spilling too much into city landscape image. I thought if I used the masked layer I used to cut out the city image with in After Effects, it should prevent any of the backdrop video showing through the city image.
I dragged the video into the top half of the composition and since it was a mask, the bottom half where the city image hides the rest of the video showing through.
I like to make the edges smooth when blending two bits of video together, so I used the feather tool to make the edges denser. This made a small proportion of the video spill over into the city image area, but it should work well nonetheless.
Video Preview
I rendered a short clip in After Effects with both layers combined to see what the outcome would look like. I’m pretty happy with how it looks, but it may turn out slightly different in Modul8… but hopefully it will look better!
I’ve managed to map the TouchOSC controls on my iPhone to Modul8 as well as in Ableton. I did have an issue with mapping some of the channels in Ableton to the channels in Modul8. I was only able to assign one channel in Modul8, until I realised at the top of the screen there were three menu tabs, Focus, Position and Layerset and Position. I was in the focus menu, which only allows you to assign one channel to a MIDI control, but you are able to assign controls to the effects on that particular channel.
I then switched to the Position menu, which gives you more flexibility about what Midi controls you can assign to and access to all the layers in Modul8. After I discovered this, I was able to map all the controls I needed to on my TouchOSC editor.
Test Video
Above is a short extract I did, after mapping TouchOSC to Modul8, of the iPhone controlling Modul8 and Ableton Live at the same time. I’m really pleased with how it’s turned out!
I’ve now pretty much got the audio sorted for this project and the visuals are coming together. I’m now at the stage where I need Modul8 and Ableton to talk to teach other. After some research I discovered that the IAC driver on the mac is the tool to use for this process. It’s basically the communicator, which enables you to send data from one software to another so they become connected.
The IAC driver can be found in Applications, Utilities, MIDI Setup and Show MIDI Window. When I opened the window I double clicked on the IAC Driver Icon, which brought up another window (IAC Driver Properties).
I made sure the device was online. In Ableton’s preference window, under the MIDI tab, the IAC Bus 1 Driver was now showing in the MIDI ports.
Then I went into Modul8′s preferences and clicked on the Key/Midi Mapping tab. I enabled both the Osculator and IAC Driver device.
Then this allows the midi output data from Ableton to sent to Ableton. From here, I was able to start adding the midi controls in Modul8 on my TouchOSC interface. I am using the same controls as I was in Ableton. I am linking the sounds with certain visuals and effects.
Just been playing around with some visual effects in Modul8. On the way back home from Bournemouth, I recorded some video footage of the landscapes whilst sat on the train. I then imported these clips into Modul8 and composited them over the Canary Wharf image.
The image above shows the output result of the video and the still image composited together, using the Luma Key. Once you’ve applied the Luma Key to each layer, it gives you more freedom to experiment with other effects. Below is the output result with each layer being extruded to create the illusion of depth.
In Modul8 I’ve looked at using the Extrusion effect, which duplicates layers of an image and pulls them apart creating a sense of depth. Below are a couple of videos I recorded of the visuals I made in Modul8.
Without Effects
With Effects
I don’t want to create anything majorly abstract. I really love city life and I want to show the observation of the build up of high energy that a city emits. When you watch time lapse videos of cities landscapes, they are often describes as micro chips processing information all the time. That’s pretty much what I’m trying to get across here. I like to think of a city as a big powerful computer that is constantly processing information.
I’ve never made music or used any audio software before, but I’ve had a small desire to learn a little bit. For this unit I’ve made a liquid drum and bass track using royalty free samples and loops in Ableton. It’s a very basic track combined of five bits of looped audio ranging from a drum beat to ambient sounds. I wanted to create a an urban sort of sound because I want to set my visuals in the outskirts of London.
The screen shot above shows the sessions/mixing viewer in Ableton. This is where you drag all your audio clips of samples and loops into categories and also select the effects you want to use. The left of the screen is where all the music library is stored. Alot of Dj’s use this window in Ableton because it’s designed and laid out for living mixing. I’ll be using this window because that’s the aim of this project to produce a live performance.
Above is a screen shot of the six audio loops that make up the whole track. The controls I’m going to be working with are the Volume/Fader controls and Solo buttons of each of each tracks. They will be synced to TouchOSC so I can control them on my iPhone.
Audio Effects
The Dry/Wet controls in Ableton will synced to the Rotary controls on TouchOSC. This one control determines how much of the effect is produced.
The Warm Reverb Long effect is self explanatory. It creates an echo effect on the music, creating the illusion of density of the music being played in an actual environment.
The Autowah effect is kind of a strange distorted effect like the sound is drowned. I can imagine, visually, the images magnifying according to the beat of the sound being washed in and out.
Filed under: Performance Video
With the music that I’m putting together in Ableton, it’s going to start off with one audio track, followed by another audio until the whole piece comes together. This is how I want the visuals to be portrayed.
I’ve started using Modul8 for the first time. It’s actually quite user-friendly after playing around with it. I was unsure about using it as I felt it wouldn’t execute the idea I had in mind, but Phil told me to try it out and see if what I want to create is possible to do using the software. After some experimenting, I have managed to create certain effects that i was wanting and I also have a clearer vision of how I want the piece to look.
Below are some small screen shots of the effects I did in Modul8.
I’ve made a drum and bass track in Ableton, using royalty free sampled loops. I always associate fast paced visuals with a liquid drum and bass track. Like the photos I’ve used, I want to make a time lapse video of Canary Wharf at night or late evening.
Filed under: Performance Video
After testing out OSC with Ableton, I felt I needed to tweak the design layout of the iPhone’s interface for it to work for my idea. This also may need altering as I progress further, but it’s an improvement of my previous layout.
OSC Output
Filed under: Performance Video
I finally managed to upload a video of a test I did with TouchOSC using my phone to control Ableton.
I used the custom layout I made in TouchOSC to control a track I put together in Ableton. In the video I’m using 5 fader controls, which are linked up to the 5 different audio loops I’ve used in Ableton. This then gives me the freedom to slowly build up the track and break it down.
Syncing Custom Layout to iPhone
After creating the layout for the interface, I needed to get the interface onto the iPhone. First thing I did was created a local network.
Then I went to the System Preferences on my iPhone and connected it to the network I’d just created. After that I opened up TouchOSC and selected Network. I then had to type in the name of the network in Host with .local at the end. I set the outgoing port to 8000 and the ingoing port to 9000.
I then used a program called Osculator. This is a program which receives OSC data from TouchOSC then changes this data into MIDI CC data, which is something that most music software can easily understand e.g. Ableton. When I opened up Osculator, I set the ingoing port to 8000 and hit enter. After a moment a new section should be added in TouchOSC saying found host 1 displaying the computer network name:8000 (Osculator).
This now means both programmes, Osculator and TouchOSC, are communicating with each other. Now that this worked I was able to load my custom interface layout to my iPhone. I opened up my layout in OSC Editor on my mac, then clicked on the sync tab. I then went back to the TouchOSC on my iPhone and went to Layout, Add layout and then the name of my computer appeared.
I clicked on the name and the layout was automatically transferred from the computer to the iPhone. All that was left to do was to head back to them main menu in TouchOSC and click on the Done tab. This then brought my layout on the iPhone, ready to be mapped to Ableton!
Mapping iPhone To Ableton
This is what Osculator looks like when you open it up.
It’s a small box that appears in the middle of the desktop. Now that the iPhone and Osculator are synced up, I am able to any of the controls on my iPhone and it will send OSC data to Osculator and it will show up in the blank sections. I moved all the controlls one by one so that they all show up on the screen.
When they were all displayed in Osculator, I locked the data so that the data isn’t accidentally duplicated if I touch one of the controls on the iPhone. I assigned all the controls to MIDI CC and selected channel 16 for the MIDI data to be sent on. Channel 16 is probably the best to use to avoid any confliction with any other hardware. The value of each MIDI CC needed to be unique so since there are 12 controls I used the numbers 1-12.
Once that was done I was then opened up Ableton, where I was ready to complete the final step in this set-up. I had already put together a track in Ableton so all I needed to do was assign the controls on my iPhone to the controls in Ableton. I went into Preferences to make sure that iPhone was being picked up as a MIDI device.
I enabled the Track and Remote columns for OSCulator Out and OSCulator In interfaces. This will enable us to record events and send control events to OSCulator as well.
At the top right hand corner of the screen, there is a tab called MIDI. I clicked on that and the whole interface of the program is highlighted blue to signify it’s in MIDI mode. This enabled me to link up the controls in Ableton to my iPhone. To link up a control on the iPhone to Ableton, all you do is select a control in Ableton which will display 4 corner frames, to show its ready to synced up. Then select the control you want it to be synced by moving the control on the iPhone.
If you click the volume fader in Live, you will notice OSCulator will automatically detect a MIDI message and will convert it to an OSC message that it sends to TouchOSC running on your iPhone. That means that TouchOSC and Live are synchronized both ways.
Filed under: Performance Video
I have decided that I’m going to use the iPhone as a MIDI device to control sound effects in Ableton. After looking through various ways of trying to map my inexpensive Multi-touch pad to Ableton, I felt I wasn’t really going to achieve my goal with the prototype. I’ve managed to get the tracking data of my finger movements on the interface of my touch pad from Community Core Vision into Osculator. However, I’ve been constantly thinking ahead about how well the touch pad would work in terms of it’s interface and I could see some problems arising not far off. I felt I was limited to how much I could control in Ableton with a blank interface. There isn’t really a clear structure for me to map my touch pad to Ableton. I think a lot of detail would have been needed in that area; therefore I wouldn’t have time to work on making a visual piece for this project as well.
Looking through all the tutortials about how to get Osculator to send MIDI data through to Ableton, pretty much all the examples were showing either, the iPhone or iPad as the MIDI devices to control the software. There are loads of examples on youtube of people using there iPad’s or iPhones as MIDI devices. Here’s one of someone using their iPhone to control some effects in Ableton on a looped track.
I started thinking maybe I should use my iPhone for this project. I mean, why not? The TouchOSC application is out there for people to create their own MIDI devices on their iPad, iPod Touch or iPhone. I feel I’ll get a better result with using the TouchOSC application, plus it’s also very flexible. If I need to make any amendments to the interface, I can easily do that.
Hopefully, once I’ve got the application up and running with Ableton I should then be able to map another software as well to work in conjunction with the visuals.
Installing TouchOSC on iPhone
TouchOSC is a universal iPhone / iPod Touch / iPad application that lets you send and receive Open Sound Controlmessages over a Wi-Fi network using the UDP protocol. It’s basically similar to Community Core Vision, but is a specific application.
The application allows to remote control and receive feedback from software and hardware that implements the OSC protocol such asApple Logic Pro/Express, Renoise, Pure Data, Max/MSP/Jitter, Max for Live, OSCulator, VDMX, Resolume Avenue 3, Modul8, Plogue Bidule, Reaktor, Quartz Composer, Vixid VJX16-4, Supercollider, FAW Circle, vvvv, Derivative TouchDesigner, Isadora and others.
The interface provides a number of different touch controls to send/receive messages:
- Faders
- Rotary controls
- Push buttons
- Toggle buttons
- XY pads
- Multi-faders
- Multi-toggles
- LEDs
- Labels (New!)
It supports full multi-touch operation, five controls can be used at the same time. Additionally the program can send Accelerometer data. The application comes with five default layouts that are organized in multiple pages but custom layouts can be constructed using the TouchOSC Editor application.
TouchOSC Software Editor/Layout Design
The TouchOSC software editor allows you to custom design the layout of the interface for which will control the audio software.
Above is a screen shot of the program. You have the option to design a layout for the iPad, iPhone or iPod Touch, from the layout list. I’m going to be designing a layout for the iPhone. Designing a layout is relatively simple as long you know exactly what you want to use. I’m new to all this so I’m going to go to design a simple layout that reflects the layout in Ableton.
In Ableton I have 5 audio tracks so therefore I’m going to use 5 fader controls on my layout page.
Above is a screenshot of the layout I’ve made in TouchOSC. I’ve added some extra controls which I’ll use for controlling effects to the track I’ve made in Ableton. The next stage will be to sync this design to the iPhone and mapping it to Ableton using Osculator.
