Sensing stuff...

OK, here's another set of methods of getting information into the computer, other than my webcam experiments.

A short while ago I read a very inspirational book by Brendan Dawes (http://www.brendandawes.com/) called 'Analog In , Digital Out' (at Amazon UK). He's one of the gurus on Flash and all things multimedia. His book focusses on the ways that analog or real-life sources of data, like light and sound, can be converted into useful and meaningful data on a computer.

His experiments include making a doorbell ring when someone visits his website, and gathering colour, size and shape data from a webcam trained on a piece of playdough - where the playdough was the user interface...!

All sounds strange but these two examples alone have given me lots of ideas and influence on my project.

The doorbell example was created using the following kit:
http://www.makingthings.com/products/KIT-MAKE-CTRL/
This electronics kit connects to the computer using either USB or ethernet and can have an array of sensor devices attached to it. These sensors include those for light, infrared, pressure and a series of switches. It's also possible to send signals out to the device and control devices like motors and LED lights.

The data is received into Flash using a custom actionscript class as a stream of numerical data. It's then moreorless up to the designer to use the data as they see fit.


The playdough example was created using a combination of a webcam and a piece of software called Cycling74 Max 5. The software company originally began by making audio-plugins for editing suites on the Apple Mac. Using the program's video capabilities, it's possible to detect an object's size, colour and whole load of other variables...

Lastly, a lot of the examples in the book employ a programming language called Processing, which is derived from JAVA. The description from the website says...

"Processing is an open source programming language and environment for people who want to program images, animation, and interactions. It is used by students, artists, designers, researchers, and hobbyists for learning, prototyping, and production."

Processing is an incredibly versatile language that makes programming specifically for graphics a complete cinch. Many of the classes you would have to explicitly call if using a traditional JAVA compiler are readily available. I downloaded a special class that enabled me to use my webcam within the Processing environment. The example I tested consisted of around only 50 lines of code. It enabled the user to wave their hand (or any other object) in front of the camera and attract a virtual tennis ball to their fingertips... I'll provide an example on my website soon!


These are the 3 main programming environments I shall research and employ during the course of my project...


MA

Jean Michel Jarre - An Introduction...

The first port of call for my research into unconventional user-interfaces begins with Jean Michel Jarre and his amazing laser harp! First, watch the video below...



For those who haven't heard of Jean Michel Jarre, he is one of the most innovative electronic music producers of our time (well, in all fairness, ANY time as he's only been producing since the mid 60s) in the way he pioneered the use of the first synthesisers and continued to do so.

He's also renowned for his live performances which include:

Rendezvous Houston (text shamelessly from http://en.wikipedia.org/wiki/Jean_Michel_Jarre)

In 1986, NASA and the city of Houston asked him to do a concert to celebrate NASA's 25th anniversary and the city of Houston's 150th anniversary.

During that concert, astronaut Ronald McNair was to play the saxophone part of Jarre's piece "Rendez-Vous VI" while in orbit on board the Space Shuttle Challenger. It was to have been the first piece of music recorded in space, for the album. After the Challenger disaster of January 28, 1986 which killed McNair, the piece was recorded with a different saxophonist, retitled "Ron's piece" and the album dedicated to the seven Challenger astronauts.

The Houston concert entered the Guinness Book of Records for the audience of over 1.5 million. During the concert, Houston native Kirk Whalum performed Ron McNair's saxophone part on "Ron's Piece". The concert featured giant projections of photographic images and laser patterns onto the buildings of downtown Houston, including a gigantic white screen on the front face of the Texaco Heritage Plaza building, which was under construction at the time. Due to vehicles stopping on the freeway passing the concert venue the freeways had to be closed down for the duration of the concert.


Twelve Dreams of the Sun

On 31 December 1999 Jarre held a spectacular music and light show in the Egyptian desert, near Giza. The show, called The 12 Dreams of the Sun, celebrated the new millennium and 5,000 years of civilization in Egypt. It also offered a preview of his new album, Metamorphoses.

The concert which started on new years eve and followed all the way through to the dawn of the new millennium in a 12 hour spectacular which including many performances from local artists and musicians, the concert used the backdrop of the great pyramids to project images onto, but fog during the evening concert by Jarre caused the projections on the facades of the pyramids to be blocked from view. Jarre played for around two hours during the build up to the new millennium with a countdown at midnight and spectacular firework display and then returned on stage in the early morning to perform a second slot to see in the first sunrise of the new millennium.

The following is a clip from Twelve Dreams of the Sun. It shows the countdown to the new millennium - forget London or NYC for the year 2000, this is amazing...



Anyway, I'm slightly off the point here... the focus of my research into Jean Michel Jarre is mainly on the innovative controllers for his synthesisers which include the laser-harp featured in the first video...

...more later!


MA

Project Ideas

One of the aims of this blog is to document my work on my final year project for university... It's a facility for me to write down all the interesting bits and pieces I happen to discovery along the way!

At the moment my ideas are still coming together for my project. I'll be focussing on the following topics over the next few weeks:

1. The way computer games utilise music.
2. Unconventional user interfaces in - a. Computer games, b. Professional music software, c. Studio/Live artist performances.
3. Various programming langauges that support such interfaces.

4. ...whilst all the time creating/programming experiments, which demostrate some findings of the above points!

Just thought a small bit of clarification was needed before I plunge ahead (for me as much as you, the reader!)


MA

Rez - Gameplay videos...

For those interested, the following links are complete gameplay videos, from Youtube - wish I was good enough to get past Area 3... C'est la vie...

Area 1
"Buggie Running Beeps 01" / Keiichi Sugiyama
http://uk.youtube.com/watch?v=7hh9fZkr1jk

Area 2
"Protocol Rain" / mist
http://uk.youtube.com/watch?v=A4EFNWe4mCc

Area 3
"Creation The State Of Art (Part 1 - 6)" / Ken Ishii
http://uk.youtube.com/watch?v=yzoGoAo0ZMA

Area 4
"Rock Is Sponge" / Joujouka
http://uk.youtube.com/watch?v=cCh6WlK027I

Area 5 / Last Boss
"Fear" / Adam Freeland
"Boss Attacks" / Coldcut
http://uk.youtube.com/watch?v=3nq_oMDlrtM (an awesome 26 minutes long!)

Enjoying watching!


MA

Rez

Rez is one of the only games centering around music that can truly get away with it's abstract graphics. I suppose the developers got around having to explain the surreal graphics by saying it's set inside a computer - where allegedly anything can happen :P

The blurb on the back of the PlayStation 2 (I can't account for the Dreamcast box-art) release is wonderfully vague:

Discover a new world
A world of sound, visuals and vibrations
Enter the network and awaken 'Eden'
Release your instincts
Open you senses

(...and in the smallest writing but the most important!)

Experience synesthesia (http://en.wikipedia.org/wiki/Synesthesia)

Without proceeding too much further, have a look at the video below...



The ONLY idea of this game is to aim and shoot. The player doesn't have to worry about navigating around the level or running out of ammunition. This game is known as an 'On The Rails Shooter' which essentially means that the computer has complete control and all you need to do is shoot - simple. Though, not hugely simple(!) If you don't shoot fast enough and/or accurately enough you'll 'die'...

OK - the musical aspect of the game. There are 5 'areas' that make up the game. Each area has it's own piece of music. These are tracks that have been composed by 'real-life' artists. While each track is a proper piece of music in itself, they've been stripped down piece by piece and split into stages to match the pace of the game. That is to say, that at the beginning of an area, it's generally a slow start in terms of having things to shoot at, therefore, the music only consists of a basic drum rhythm. Each area is split into 10 stages. At the height of the action, the music will sound very 'busy' with all the usual parts in place (drums, bass, synths, percussion...)

After harping on about all the shooting and the music, the main part of the game is the experience that the player gets. Everytime they shoot something, a percussive noise, synthesiser note or sound effect is played in harmony with the rest of the music. When an enemy has been dispatched off the virtual coil, there is a small light-show explosion. It's difficult to explain but the way the enemy explodes visually matches the exploding sound... if that makes sense at all (see the Wikipedia link above, yes I KNOW Wikipedia is genrally a no-no, but in this case it's actually a good pool of information, so there...!)

So, to conclude...

The graphics and overall aesthetic of the game will play a huge influence in the way I think about the way I design my application - whether it turns out to be a game - or not...

Links...
Official site of the original Dreamcast/PS2 game - http://www.sonicteam.com/rez/e/news/index.html
Official site of Rez HD for the Xbox360 - http://www.thatgamecalledrez.com/
Wikipedia entry for Rez - http://en.wikipedia.org/wiki/Rez... There's a couple of pieces of information that may prove interesting...


MA

(Rez shall return in this blog...)

DJ Max Portable meets My Webcam - Mockup

This is roughly what I'm looking at making. I'll be using code from my previous webcam examples and merging that with the beginnings of code that will judge whether the user ('player' is such a strong word at this stage!) has made a motion at the correct time and in the correct place on the screen.

The example shall work as follows:
1. The blocks fall down the screen the image of the user.
2. The user shall jab upwards into the picture with their right or left hand when the corresponding block reaches the thick line at the bottom of the screen.
3a. If the user responds early, late or not at all then nothing will register (- in DJ Max Portable terms, this would count as a failure and would reduce the chances of finishing the song)
3b. If the user judges the timing correctly (there will be a reasonable margin for error here) then the game continues...

Watch this space...


MA

DJ Max Portable on the PSP...

Well, here we have DJ Max Portable for the PSP. I'm not trying to write a review of the game, rather explain how it works.

The simple idea of the game is to press the correct buttons on the PSP in time to the music. The blocks falling down the screen (see the screenshots) indicate which buttons to press. When the player presses the correct button then they've contributed to the song by triggering a sound. Throughout each song in the game, the button presses add everything from rhythm parts to synthesiser sounds. The more accomplished the player, the more pleasing the song will sound.

If the player makes too many mistakes then the game will exit the level/song and display a message that instructs the player to get more practice. Because of this, the game has quite a steep learning curve.

The difficulty of the game is determined by several factors.

1. The speed/complexity of the song that's playing. The faster or more parts that the song has, then the more blocks will fall down the screen.

2. The number of buttons the player has to use. The player can select to play using 4, 6 or even 8 buttons on the PSP. Personally, playing with 4 buttons so far has proven to be quite tricky!

The music you play to is graded in terms of difficulty, so initially the gameplay feels quite sedate, though after the first couple of songs are out the way the pace quickens somewhat dramatically!

So far this game has proved quite influential in terms of my project. I'll be trying to build a small concept project where the player will have to move in front of a webcam to the rhythm of the music rather than pushing buttons. So far the accuracy of detecting motion on specific parts of the screen using a webcam (see my examples if you can!) have been a bit sketchy so I'll be trying the DJ Max Portable equivalent of '2 button mode' for the time being...

So, the view the player will have is of themselves on the screen with blocks falling down over their image. When the blocks reach the bottom of the screen, the player will make a motion (I'll figure the precise motion out at some point!) over a specified area of the screen to 'clear' the block and enable them to progress through the game...
...hopefully some of that makes sense!

Videos of more talented people than me playing DJ Max Portable 1 & 2 on the PSP
http://uk.youtube.com/watch?v=2pxsq-nA6o4
http://uk.youtube.com/watch?v=09VLs3nqE7M
http://uk.youtube.com/watch?v=zfMlAv7ABEg
(the last link will give you an idea of how frantic the gameplay gets!)

From here:
Mock-up images to demonstrate this idea - pictures are a thousand words which means there will be less stuttering from me!


MA

Summary: Games centering around music

Some of the most surreal and original computer games over the history of console and PC gaming have involved mixing musical elements with engaging visuals and/or artwork.
I'm in the process of investigating a selection of games that involve the player with music in some way.

Some of the games have the player actually contributing to the music - with each button press triggering a sound... example: DJ Max Portable/Guitar Hero

...all the way through to the music merely being used as an indicator of the player's progress through the game where the music 'builds'... example: Rez

Finally, there are some games where music creation is the only purpose. I won't be focussing too much on this sort of music game - because there isn't really a 'game' element. The player doesn't progress in any way and there aren't any goals or challenges.

Detecting motion in certain areas...with a webcam - part deux

This example is similar to the other area detection example - however it's more refined, uses 'slightly' more sophisticated code to detect the certain areas, AND is more fun to play with.

When you start the example, you will see three blinking dots towards the bottom of your picture. These are the hitpoints where the motion is sensed.

Area Detection example 2

I've found that raising your hand up relatively quick up into the picture works quite well! It's way more useable than the first example - in my little honest opinion


MA

Detecting motion in certain areas...with a webcam

Next - it would be handy to detect motion in certain areas of the screen. The following example detects motion in each corner of the camera's picture - like a 2x2 grid.

Each corner plays a different sound when you wave your hand in front of it. Now - it's not the best sort of detection by any stretch of the imagination, but have a bit of a play and see what you think...

Area Detection example 1

Have fun playing many tunes...!

Simple motion detection using a webcam

Well, I want to make some kind of music-based program or even a game. I haven't quite decided what I'd like to make though, however I'm going to be mucking around and experimenting with wild and wacky user interfaces. Using a webcam as an input device seems a good place to start - the mouse and keyboard are boring :P

So the following link will take you to my first experiement - motion detection. You obviously need a webcam for this otherwise nothing will happen(!). If you're using an Apple Mac with a built-in webcam you may need to select the correct camera device - for some reason there are a few in a list to choose from!

Motion Detection Example

The program senses a certain level of motion in front of your webcam. I guess it depends on which webcam you have - I have the Microsoft VX-1000 and it seems sensitive enough... Let me know how you get on...

At some point I'll upload videos of me waving manically at my computer screen - good times...


MA

Glide Labs' new home!

This is the home of the news for Glide Labs - the nickname for my final year uni project!

Here will be all the research, programming experiments and all kinds of other things...

Back with 'all kinds of other things' in a moment...


MA