Glide Labs

IR Dot to Dot

2009-02-17T12:35:00.005Z

“One of the next tasks in this iteration of the gesture application is to investigate a reliable method of using the coordinate data of more than one IR blob. Not only would this method increase the sensitive area of the Wiimote, it would also make the motion much smoother (similar to my averaging method to smooth out the accelerometer readings).”

Well, I've managed to do this now – as you will have seen from the video in the 'intro' post of the same title.

First, view the video below that shows the latest iteration of the application

videohere

For the purposes of development, I've decided to show the positioning of the infrared 'blobs' that the Wii Remote's camera tracks, along with their index numbers. These index numbers are assigned by the Wii Remote itself, along with the calculations to determine the position of each indexed blob. This in itself is actually quite interesting (believe it or not!) and is due to a limitation of Bluetooth:

“...the camera is a high resolution camera and works incredibly fast (on 100 Hz). This generates a lot of data which has to be analyzed. It is way too much to send by Bluetooth to the Wii, for Bluetooth has a maximum capacity of 1 Mbit/s. Assumed that one pixels takes one byte, sending 100 images of 1024x768 pixels take 630 Mbit/s, which is way too much for Bluetooth to handle.”

Smit, J.B.. (2008). Affordable Head Tracking. University of Groningen. (chapter 2), p17.

Consider that figure of 630 Mbit/s when you can have up to 4 Wii Remotes connected to a PC (or indeed Wii) at once – that's an awful lot of bandwidth!

Instead of very long explanations, I'll reel off a list of features that the application now has, associated with infrared. (Still probably long explanations!)

1. Can use 1 to 4 infrared dots to determine the position of the Wii Remote

2. When 2 dots are detected (this is where my Bluetack comes out again to cover up all but 2 LEDs) then the distance between them in pixels (relative to the resolution of the application) is calculated. From this, it's possible to track backward and forward motion and make an approximate representation for the 3D model's movement. This is a significant step to adding the ability to recognise gestures in three-dimensions

I've limited it to using only 2 dots at the moment because then it's a simple Pythagoras calculation to get the distance between the dots. If the Wii Remote is detecting 4 blobs then it's a question of working out the greatest distance between all 4 points. This is potentially a lot of calculations that could slow the program down.

Aside from that, if the remote is really close to a cluster of LEDs at one end of the sensor-bar then it it would appear that the remote is far away due to the small distance between the LEDs. Therefore, the LEDs must be at either end of the sensor-bar.

3. If there aren't any IR sources detected by the Wii Remote then the 3D model shader changes to represent it as 'off-line'. The application stops registering movement rather than registering lots of strange directions. When the remote is moved out of range then the remote freezes in it's last position. When the remote is moved back into range then the movement from the last position continues.

Next up will be the explanation of the Levenshtein algorithm!

IR Dot to Dot - intro

2009-02-16T20:51:00.005Z

Consider the following video - more detailed explanation and other videos on the way!

(higher quality available here)

Using the IR Sensor Bar = Cheating?

2009-02-08T12:03:00.014Z

Well, when this project has set its sights on using the accelerometer in the Wii Remote for motion sensing then the answer is a resounding 'yes!'. One of the main reasons why I've shied away from the use of the sensor bar is that I wanted to make a gesture recognition application with as FEW artefacts as possible – i.e. just the Wii Remote and no other hardware save for a computer with Bluetooth capability. However, the data gathered whilst employing the IR sensor-bar, the scope for my project has somewhat widened...

After discovering the limitations of the Wiimote's accelerometer I decided to delve into the possibilities of the other sensor in the Wii Remote – it's camera. For those who don't know, that sensor bar sitting on top of the television isn't actually sensing anything at all. It's merely emitting a number of infra-red sources. The Wii knows where the player is pointing because the camera in the Wiimote is tracking the position of the infra-red sources. In fact, the camera in the Wiimote is capable of tracking 4 infra-red points (known as 'blobs') at 100Hz (100 times per second).

The Director Xtra I'm using for the purposes of my project supplies the coordinates (X and Y values between 0 and 1) of each blob in a handy array. So, thankfully, the potentially fiddly process of getting to the data has been dealt with – it's just a question of interpreting the data to get something meaningful from it.

One blob to rule them all...

Armed with my Wiimote, el-cheapo battery powered sensor bar and a few lumps of Bluetack I set to work.

Firstly, my sensor bar isn't as good as the actual Nintendo one, but then it wouldn't be as it cost a whole $6. It has 6 infra-red LEDs, 3 at each end of the bar and arranged in such a way that they 'fan out' to create a spread of infra-red – the theory being that the Wii console will appear more sensitive or reactive. The genuine Nintendo sensor-bar has a total of 10 LEDs in it making it far superior. For the purpose of the program I'm writing, I only need one (there are downfalls to this approach that I'll explain shortly) so I covered up the rest of the LEDs with Bluetack!

I've adapted the program I wrote for the accelerometers to use the readings from the Wiimote's camera instead.

The 3D model of the Wiimote is pointing forwards and doesn't rotate with the movement of the real Wiimote. Obviously there will be a certain amount of rotation when the real-life Wiimote is moved, however one has to remember that the Wiimote's camera must be pointed in the direction of the infra-red. This is so the application can recognise the movement. If the application relied on the accelerometer for movement as well, there would have to be some consideration taken for the occasions that the Wiimote's camera would be out of view of the infra-red (i.e. when pointed straight up or down).

This is where the downfall of using one infra-red LED for positioning becomes apparent. LEDs in general do not have a very wide emitting angle. It's very unlikely that you would see any light – or in this case, detect any infra-red – if looking at the LED from the side. Once the Wiimote's camera has moved out of effective viewing range of the LED then the readings either become unreliable or non-existent (in this case the Wiimote returns a value of -1).

Accelerometer & IR side by side...

This version of the application incorporates a slightly different method of movement recognition. The versions of the application that rely on accelerometer data as a means of control use – for want of a better term – a state logic. The Wiimote has to be swung to a certain position for it to register as a different movement. For example, if the user swings the Wiimote to the right, the application will register that the Wiimote is in the 'right' position. Now, if the user swings the Wiimote to the left, the application won't register any leftwards motion until the accelerometer is reading that the Wiimote is in the 'left' position. It is better to think of the application as recognising a series of pre-set positions that the Wiimote can be held in rather than detecting the direction of the motions themselves.

The version of the application relying on IR dot positioning works differently. There are no pre-set positions that the user can hold the Wiimote. This time, the application senses the direction of the motion. For example, the Wiimote is motionless in the user's hand – the 3D model on the screen is also stationary. The user moves the Wiimote up – as soon as this happens, the application moves the 3D model of the Wiimote up the screen and registers this as an upward movement.

This method is much more effective and natural-feeling than the previous applications using the accelerometer data. Whenever the Wiimote is stationary this registers as an 'idle' state – unlike the accelerometer application where 'idle' meant holding the Wiimote in a specified position.

The method also makes the 'game' I created seem a bit easier and less frantic. When the game demanded an 'up-down' motion, the accelerometer-based version would see the player having to swing the Wiimote far enough to get it into the correct position. It's confusing for the game-player who is probably wondering why it took so long for their motions to be recognised. Now just the slightest movement in the correct direction would register as a movement in a certain direction – I suppose some may argue that this takes the challenge out of the game, but it opens the door for more complex gestures that the player would not have had the time (or patience) to make.

Another advantage to this method of sensing is that it's easier to perceive the motion one's making when gesturing at a 2D surface such as a television or computer monitor...

As an example scenario; one of the gestures the application may recognise is a 5 pointed star. You are given a whiteboard pen. There are two whiteboards in the room. One is mounted on the wall and the other is mounted on the ceiling. You are told that you have to look in front of you while drawing, so no neck tilting! First you draw the star on the whiteboard mounted on the wall - you can see the board and the shape you're drawing so it's really easy. Now, when it comes to draw the star on the ceiling whiteboard it's not as easy because you can't really see what you're doing and the star looks distorted.

OK, that may be quite a long way of trying to explain the benefits of the IR dot method, however it nicely highlights its benefits. In programming terms, it makes the process of designing gestures easier and enables more complex methods of determining gestures to be integrated into the application – such as the Levenshtien Distance algorithm.

Next steps...

One of the next tasks in this iteration of the gesture application is to investigate a reliable method of using the coordinate data of more than one IR blob. Not only would this method increase the sensitive area of the Wiimote, it would also make the motion much smoother (similar to my averaging method to smooth out the accelerometer readings).

Returning to the accelerometer-based version of the application. I may try to alter the code to sense change in direction rather than finite states. The accelerometer data was too 'jittery' the last time I tried it, however, that was before I started using averaged data over time rather than real-time values. It might be possible! However, bear in mind that the problems highlighted in my whiteboard scenario earlier will still be an issue.

Once the aspect of getting the user's input has been dealt with, the various methods of checking for valid gestures will be the next priority. In terms of my project proposal, the investigation into forms of gesture alphabets shall begin. This will represent another iteration in the application, where we move away from the gaming aspect and more toward the place that gesture recognition has in the everyday computing world...

...and now for the annoying bit - Accelerometers

2009-02-07T09:29:00.006Z

After hours of testing and gazing hypnotically at a spreadsheet application, one hand on the mouse, the other clutching my Wii Remote, I've come to the conclusion that it's not possible to measure displacement using data from the Wii Remote's accelerometers.

I have searched through forums associated with non-gaming-exclusive applications of the Wiimote and found that the general opinion of members is that it's not possible (http://www.wiili.org/forum/tracking-distance-and-3d-position-without-a-sensor-bar-t3763.html). Another search under general applications of accelerometers (not associated in any way with the Wiimote) revealed algorithms that take accelerometer data and several other determinable variables to calculate displacement.

Technically I 'should' be able to swipe the Wiimote 1 metre to the right and collect the data output from the accelerometers. Then I could integrate these readings to retrieve the velocity at a given time, and therefore the displacement (or distance). The final result should have some resemblance to 1 metre.

I performed the same gesture a number of times, however the displacement calculations returned wildly different results. A slight deviation in the results could be acceptable however when moving the Wiimote over the distance of 30 centimetres I returned results between 4 and 40 centimetres.

At this point, it should be noted that the Wiimote doesn't actually output values that can be interpreted as acceleration. The values it returns are actually measured in g's (g-force) which is the amount of acceleration in relation to the Earth's gravity. 1g is roughly equivalent of 9.8m/s/s or 9.8ms^-2so the data from the Wiimote can be converted to real acceleration data. (http://en.wikipedia.org/wiki/G-force)

In the experiments I conducted using the Wiimote, the accelerometer data was captured at anything from 100 times per second to as little as 10 times per second. Therefore, 9.8ms^-2 has to be divided by the number of samples per second.

All the calculations are perfectly sound on paper, however the source of the data – the accelerometer – isn't reliable enough to produce consistent, predictable results.

So, why not?

Well, first, I had a look at the specification of the Wiimote's accelerometer – an Analog Devices ADXL330 (http://www.analog.com/en/mems-and-sensors/imems-accelerometers/adxl330/products/product.html). The g-force readings of the accelerometer are accurate to +/-3g. If the Wiimote is shaken hard or swung sufficiently fast then readings outside +/-3g can easily be produced. Any readings outside the specification of accelerometer should be disregarded, however, in some cases, this would mean completely disregarding an entire set of data!

Next, I found that the Wiimote can be held at a certain orientation and provided the user has a steady enough hand, they can move the Wiimote through the air without altering the output of any of the axis on the accelerometers.

Before writing off the Wiimote's capabilities to measure displacement using its accelerometer data, it should be noted that a programmer and car-enthusiast (http://forums.vwvortex.com/zerothread?id=3362586) successfully strapped his Wiimote to his car and managed to measure his 0-60mph time, and therefore distance traveled. It should be noted that perhaps periods of smooth acceleration can be accurately recorded with the Wiimote. This is because the g-force readings from the accelerometer would be constantly increasing (or decreasing). If one was to plot the readings on a graph of acceleration against time, they would see a smooth line. If one was to plot the readings of a rapid hand gesture on a similar graph, the line would resemble more that of a 'heart-rate monitor'. It's obvious that the former readings would be more accurate.

The problems I have experienced don't appear to just be the preserve of programmers using the Wiimote for non Nintendo Wii software.

It also appears that more observant gamers have noticed the shortcomings of the Wiimote as well. The 'real-life' movement that was promised by Nintendo with the release of the Wii console doesn't match the now-apparent limitations of the system.

(http://www.gamasutra.com/view/feature/1494/wheres_the_wiimote_using_kalman_.php)

Using the Tennis game in Wii Sports as an example, more specifically the way the player serves the ball to the opponent, one can see that there are many ways of playing the game. The method of serving in Wii Sports Tennis is to hold the Wiimote down by your side, raise it up quickly and bring it down again. A player who may have played tennis before or at least knows what a tennis serve looks like may try to serve the ball using a real-life serving technique. From personal experience, I've seen adults try to serve by throwing a non-existent ball into the air, and then try to hit it using the Wiimote. Annoyingly for them, using a real-life technique in Wii Sports Tennis doesn't actually work. The serve must be carried out using the rather childlike method described above.

Other gestures like those required in the bowling game in Wii Sports appear to be more realistic in terms of the motion the player makes with the Wiimote. However, this is purely due to the fact that the real-life bowling action is within the capabilities of the Wiimote's accelerometers. Compare this to where the action had to be altered by the game's developers for the tennis game to a gesture that the Wiimote can understand.

In terms of the gaming aspect of the Wiimote's technical ability, it seems suitable to simulate some types of motion that occur on very definite axis lines – i.e. in straight lines rather than a diagonal swing across all axes. For other, more complex gestures, there is a certain amount of difference between the motion that the game's developers tell the player to make (on-screen instructions) and the action motion that the game picks up (the code running the game). For example, a game may tell the player to perform an elaborate swishing gesture with the character's 'sword', but in fact, the game's code doesn't care how elegant or 'swishy' the gesture is. It will only concern itself with a very small portion of the movement to produce a distinct gesture.

Possibly the most irritating aspect of the Wiimote for consumers is that it isn't the all-singing all-dancing control device that Nintendo promised. All the people who go and buy the various sword fighting games out there (Red Steel etc) and games from the Star Wars franchise must have a sour taste in their mouths when they realise that the Wiimote isn't quite the sword or Lightsaber they first thought. This is due, once more, to the Wiimote's inability to sense it's position in space – it's not possible using accelerometers.

Nintendo are shortly to release a device called Wii Motion Plus that bolts onto the Wiimote (where the Nunchuck is usually connected) that contains a gyroscope. This device will enable game developers much more scope when programming the physical aspect of their games.

"Nintendo's upcoming Wii MotionPlus accessory for the revolutionary Wii Remote controller again redefines game control, by more quickly and accurately reflecting motions in a 3-D space."

Hopefully, this will see less use of the IR Sensor Bar to detect distance from the TV and more use of the Wiimote on its own.

While the purpose of this article wasn't to moan able the shortcomings and downfalls of the Wiimote, I feel that some of the text is necessary due to the fact that I myself came to the Wiimote thinking that so much more was possible! However, it's not all bad – there are ways around most of the problems I'm encountering thus far.

Now, for the purpose of this investigation, I shall suspend the research into calculating displacement with the Wiimote. The main reason for this is that it doesn't appear to be possible to obtain accurate readings from the types of gestures I wish to record and recognise.

Wii Gesture Project - "Wii Swing To Music"

2009-02-02T06:06:00.011Z

OK, this is a pretty big step up in the versions, so I'm going to stop giving releases a version number or we'll be up in the hundreds before too long I shouldn't wonder...

What would have been version 5 was the application's ability to recognise simple gestures. Whenever a gesture was recognised, the application played a lightsaber sound (yeh, I know, too tempting right?)

Now (for technically version 6) the application has morphed into a simple game - just for the purposes of demostration - where the player has to make the correct gesture displayed on the screen within a certain amount of time. All this is happening in time to a piece of music, so the general idea is that the player should wave the Wii Remote in the correct directions in time with the music...

...and it 'kind of' works!

Anyway, take a look at the video below. There isn't any audio present at the moment due to some syncing problems I had with the screen recorder, so at some point my trusty camera will come out again and I'll have to film myself playing it...! For the time being, picture it being played to Lady Gaga's 'Just Dance'.

Higher quality video available here...

That's all for the moment.... Over n out...

Wii Gesture Project - version 4

2009-01-29T12:53:00.006Z

(higher quality video available for viewing here!)

"What you'll probably notice is that the movement is quite stuttery, so even when the Wiimote is in a stationary position there may still be some jitters. This is due to the fact that the 3D model of the Wiimote takes it's rotation properties directly from the ever-changing data of the real Wiimote's accelerometers."

This is fixed! If you compare the movement of the Wii Remote in this video to the other 3 I've recorded so far, you'll notice that the motion much smoother. It's difficult to tell from the video, but now it's easier to select the direction you want to point the Wii Remote. Before, if the Wii Remote was held on the cusp of a different direction, for example between 'left' and 'idle', the chances were that there would be lots of 'left' directions generated as the accelerometers in the Wii Remote flit between 'left' and 'idle' positions.

The smooth movement is gained by sampling the last 15 accelerometer values for each axis (x, y and z) and calculating an average of each. This average is used in place of the raw accelerometer readings. This method irons out the little 'in-between' vibrations from the accelerometer.

The slight (and the benefits of averaging 99.9% outweigh this) problem is that there is a slight delay between moving the Wii Remote and the program reacting. This is because the latest movement is being averaged alongside the last 14 movements could be much lower or higher - so the numbers have to effectively 'catch up'. In fact, as the application is running at 30 frames per second, the movement is up to half a second behind. The delay can be reduced by taking fewer samples for the average.

...the work continues.

Wii Gesture Project - version 3

2009-01-29T10:12:00.003Z

"There is a problem with this method of data collection and that is that a complete gesture must fall perfectly within each group of 8 movements."

Well, not anymore! The method of populating the array has been modified slightly so that the oldest of the 8 movements is pushed off the back of the array allowing the newest movement in at the front.

e.g:

movementList = [1, 5, 7, 3, 7, 3, 2, 1]

A new movement comes in - number 5 (which is 'down')
All but the first value from the movementList array is copied into a temporary array...

temp_movementList = [5, 7, 3, 7, 3, 2, 1]

...then the latest movement is tacked onto the end so we end up with...

temp_movementList = [5, 7, 3, 7, 3, 2, 1, 5]

...finally the contents of temp_movementList are copied into movementList and then processed into a single string...

levenshteinString = "57373215"

This process happens after every movement the user makes with the Wii Remote.

As ever, a higher quality video can be viewed here.

That's all folks!

LEDs, Camera, Tilt!

2009-01-28T10:44:00.005Z

Whilst all the tilting is happening with the 3d model on the computer monitor, the Wiimote is also providing feedback to the user. Every one of the 9 positions (8 tilts and 1 'idle' position) has a different light sequence. I'm particularly happy with the way the LEDs zoom left and right when the Wiimote is held in 'idle' position - slightly reminiscent of a certain car...

Have a look at the video (apologies for the shoddy focus of my camera!)
There is a high(er!) quality version of the video here.

The picture below shows how the LEDs will behave when the Wiimote is tilted in different directions - hopefully clearer to understand than the video!
(note: the glowing circles denote LEDs that flash)

The LED sequences are merely there, at the moment, to provide the user with extra feedback as to their positioning of the Wiimote.

(it also looks rather good as well!)

Wii Gesture Project - version 2

2009-01-27T15:58:00.004Z

...one step further to having some gesture recognition!

Have a look at the video.

(Once again, if the video quality is too iffy for you, go here)

You will see that the application records the user's gestures in groups of 8. The movements, each direction represented by a number, are stored in an array (or 'list' as Lingo would have us call them!). At the end of every group of 8, the array is condensed into a single string. This forms the basis of a Levenshtein Distance comparison string - I'll explain this in another post, but it's a core part of the application nevertheless!)

For example, the application may output:
movementList = [1, 2, 1, 2, 3, 5, 1, 5]

...therefore...

movementListLev = "12123515"

We take each number...
1=up, 2=up-right, 3=right, 4=down-right, 5=down, 6=down-left, 7=left & 8=up-left

So we know from 'movementListLev' that the user has moved the Wii Remote up, up-right, up, up-right, right, down, up & down.

There is a problem with this method of data collection and that is that a complete gesture must fall perfectly within each group of 8 movements. The next stage in the development will involve creating the array/list so that newest movement numbers push out the oldest numbers, that way the 8 most recent movements are ALWAYS shown...

Wii Gesture Project - version 1

2009-01-26T20:25:00.006Z

Here is the first iteration of an application that will hopefully be able to recognise a gesture made with a Nintendo Wii Remote. So far, the program can mimic the rotational movement of the Wiimote and therefore, can determine which direction the Wiimote is pointing.

Watch the video above (if you haven't already!) to see how it works.

(If you don't like the Blogspot video quality, have a look at a higher resolution version here)

What you'll probably notice is that the movement is quite stuttery, so even when the Wiimote is in a stationary position there may still be some jitters. This is due to the fact that the 3D model of the Wiimote takes it's rotation properties directly from the ever-changing data of the real Wiimote's accelerometers. Perhaps the next thing to do is add a method that 'rounds' the accelerometer data to a certain (lesser) degree of accuracy - although less accurate just for the visual appearance as opposed to the inner workings of the application.

The work continues...

Movement purely on accelerometers!

2009-01-23T20:26:00.005Z

Finally back on track after months of mucking about...!

Today, I altered my Wiimote program in Director to make a 3D object move around based on the data gathered from the Wii Remote's accelerometers. The control is actually pretty accurate. So far, you have to hold the remote in a fairly 'wooden' fashion. By this I mean that you can't swish the remote left or right as one might if playing a real Wii game (Wii Sports Tennis comes to mind here).

So, after some faffing with the Lingo in Director I have the following degree of control over my 3D ball...

note: the 'deadzone' for movement is attained when you hold the remote straight up in the air.

Move left: swing the remote to the left
Move right: swing the remote to the right
Move down: pitch the remote forwards (the end of the remote is moved in an arc away from your body)
Move up: pitch the remote backwards (the end of the remote is moved towards your body, the amount of movement is limited for this action)

At the moment, the movement of the ball isn't directly related to the accelerometer data in terms of the actual numbers... So if the accelerometer hints that the Wii Remote is being swung to the left then the ball is moved left by a finite amount.

That's all for the moment, however everything is well on the way to more exciting things!

How the axes work on the Wii Remote

2008-10-27T16:40:00.003Z

If you move the Wii Remote along each of these axes then the values from the accelerometer along each axis will vary.

So moving the remote rapidly from side to side will cause change in the 'x' axis.

The sensitivity of the accelerometer in the remote (and indeed the Wii Nunchuck - they're the same spec) generally means that it's very difficult move along one axis at a time.

Now it's time to work out how to get this information recorded in Director!

Squeezii Schmiizii Diizii "Wiigee"

2008-10-26T20:27:00.003Z

Ohhhhh, yes, the titles can and shall get worse than that...

In my quest for all things interactive and Wii, I stumbled apon this rather nice gesture-recognition library based in Java called Wiigee. It's being developed by several German university students and from the demo video it looks amazing.

I compiled all the code using Eclipse but, unfortunately, my Bluetooth stack ('Blue Soleil') and Bluetooth dongle aren't compatible with the requirements of the Java Bluetooth library, 'Bluecove' which requires 'L2CAP'.

To save the waffle, check out the following video on the Wiigee site:

http://www.wiigee.org/demovideo/demovideo.html

Links

Wiigee site: http://www.wiigee.org/index.html
L2CAP definition: http://en.wikipedia.org /wiki/Bluetooth_protocols#L2CAP_.28Logical_Link_Control_and_Adaptation_Protocol.29
Bluecove site: http://code.google.com/p/bluecove/
Eclipse site: http://www.eclipse.org/
Blue Soleil site: http://www.bluesoleil.com/

Using the Wii Remote with Director

2008-10-23T15:38:00.005+01:00

Proof Of Concept time folks!

Using an example from my module for 3D graphics using Director and the example code for the Wii Remote Director Xtra, I've managed to demonstrate how it is indeed possible for the Wii Remote to change variables in Macromedia Director MX (note: the Xtra only seems to work for MX, those with the new Adobe Director probably won't get anything working, tough :P).

Obviously, this isn't exactly the first time anyone's done this before (yeh, obviously!) but this stage of my research moreorless defines the direction in which my project is likely to take - for the programming portion of it at least.

The screenshot below shows what I came up with... It's the quickest and dirtiest method, but I just needed to see for myself that it's possible to manipulate 3D graphics, albeit on a basic level, using the Wii Remote.

I set the program up to make the cube rotate about the x-axis and the sphere rotate about the y-axis. In terms of the Wii Remote this means that pointing it up or down will make the cube move and tilting it from side to side will make the sphere spin from side to side.

The program is very responsive with no delay between making the movement with the remote and seeing the result of the movement reflected on the screen.

When I placed the remote on a flat surface and flicked it to make it rock, even these small rocking movements were registered. All in all, this test was a complete success! The next test will be to import a terrain/model from 3dsMAX that's slightly more complicated, just to see if the smoothness of this test can be maintained.

One thing to note for all this - this is just taking the numbers from the Wii Remote Xtra and using them without performing any calculations. I also didn't have much regard for HOW the 3D models were moved/rotated, I just used the first method that came to hand, therefore the test doesn't respond as expected sometimes...!

The next task will be to have the 3D models moving in exact relation to the way the Wii remote is pitched or rolled. i.e. the changes aren't relative, they should be absolute! Another task will be to officially begin my research into specialised user-interfaces and methods of control. This will all hark back to my original ideas that involved the use of webcams - who knows, this method of control may crop up again in my project!

Wii & Director MX

2008-10-23T15:15:00.001+01:00

Well, the software I'm using to make my project has changed since the hazy summer days when Flash seemed like the ideal option.

There are a few reasons for this!

One - Director MX has an Xtra that you can download for free that enables seamless integration between the Wii Remote and Director using very little Lingo code to get the data into the software...

Two - It's possible to import 3D models from 3dsMAX and then traverse them in Director - again with relatively little effort compared to using a 3D-engine with Flash (like Papervision 3D)

Three - The method of getting the data into Flash, using WiiFlashServer, required the use of AS3, some of which is radically different to AS2 and I personally do not have the time to update my knowledge of AS. As I have to learn Director's Lingo language for another module, it's easier really!

Next up is the proof-of-concept!

cheers...

Individual Project Proposal - Wiiiiiiii

2008-10-23T14:49:00.003+01:00

Oh, it's been a while again...

Here's my project proposal - it's not the greatest prose in the history of English but it sort of does the job!

My project will centre around researching intuitive methods of interacting with a computer – be it a workstation, laptop or another bespoke computer system such as a games console or installation-art.

The first element of my project will be made up of research into the existing intuitive control methods or user interfaces that are available on the market today, and several from the past. Many user interfaces or methods of control aren't intuitive or natural and may require training to use. I shall be finding out why these methods of control have been successful, or not.

The bulk of the research will involve finding methods of interacting with the computer system that feel more 'human' or natural.

Over the course of the overall research into user interfaces, I shall also be investigating the many possibilities there are for capturing motion information onto a computer and then using it in a meaningful way. For example, the Nintendo Wii remote controller can be linked to a computer via Bluetooth and used with a variety of programming environments ranging from Flash, Director all the way through to many open-source developments.

The second part of the project will involve using the research to produce an application based in a 3D environment.

The control device that I will be using is the Nintendo Wii remote. This is due to the fact that it has a whole array of sensors (accelerometers) that send a continuous stream of information to be interpreted by the computer.

The aim is to give the user the ability to traverse the 3D environment without using any button presses, but rather using a series of different movements or gestures – the latter being much more natural. There have been many inventive uses for the sensors in the Wii remote, which is apparent from looking at the choice of games that are available for the Wii. Another of my aims when creating this application is to create a series of gestures that need to be carried out to move through the world (such as waving the remote in a circular motion or moving it very quickly, miming the swinging motion of a baseball bat). There may also be a game-playing element to this application.

That's it but it will be revised shortly...!

Project Idea #1

2008-08-16T01:02:00.002+01:00

It's been a while again, but here's a semi-decent idea for a project! So, point by point below...

1. 'Rez in Flash'
2. ...but in a faux-3D world.
3. The player flies into the screen.
4. Exclusively uses a Nintendo Wii Remote as a means of control.
5. Move the player left and right by altering the 'roll' of the Wii Remote.
6. Aim using the Wii Remote functioning in the same way as a mouse.
7. Use of gestures - Tracing different shapes using the Wii Remote to unlock areas and proceed through the game.

That's all folks - hope it's easy to make!

MA

Another Dose Of Wii - asblabs.wiimote

2008-07-23T08:31:00.004+01:00

(yeh, the awful puns continue...)

Found an incredibly useful set of classes from here:

http://www.asb-labs.com/blog/flash-gaming-classes-wiimote-wiiflash/

Amongst other fun things you can:

1. Point the Wii Remote at the screen and use it to move a cursor object around the stage.
2. Control individual LEDs on the Wii Remote - a bit pointless, but very very fun I suppose.
3. Register if a certain combination of buttons has been pressed. Intended for use in game programming, however, there are a dozens of other uses for this ablity apart from gaming...

...back later!

MA

3 Shades Of Wii... and Johnny Lee

2008-07-21T22:56:00.006+01:00

(that's pun #2 up there... guaranteed to get much worse as this project progresses)

Below is a list - which I shall build up - of all the software that I've found, and will subsequently fiddle with and document in a fair amount of detail... but very briefly -

(note: if the software is for use with a PC then I'll assume that BlueSoleil is running and a connection between the Wii Remote and PC has been established)

glovePIE - script driven software that runs on a PC that senses all the movements from the Wii Remote including the accelerometers. The most flexible solution with a high learning curve.
(http://carl.kenner.googlepages.com/glovepie)

WiiCade - this is a Flash API that allows you to create a Flash application with Wii Remote control other than mere pointing and clicking, more buttons are supported, intended to be run through the Wii Internet Channel.
(http://labs.wiicade.com/index.html)

WiiFlash - Probably the most exciting out of the lot because it allows you to directly access the Wii Remote from Flash using actionscript. It's also possible to connect and differentiate between up to 4 remotes and nunchucks. The major downside is that there isn't currently any access to data from the accelerometers on the Remote. You also have to run a piece of software called WiiFlash Server that may require you to update your .NET framework, although it runs transparently and doesn't need any configuration.
(http://www.wiiflash.org/)

Johnny Lee - OK, so he's not a piece of software but it would be stupid to be writing this without mentioning his research. Lee has come up with many ideas for implementing the technology inside the Wii Remote including an interactive whiteboard and many small 3D experiments that have to been seen! Anyway, his work has become quite an inspiration for my project - if only in terms of the innovations...
(http://www.cs.cmu.edu/~johnny/)

Thaaaaaat's all folks...

MA

Wii Things Amuse Wii Minds...

2008-07-10T20:33:00.009+01:00

Phew! After a little while, I'm back with my account of my Wii little adventures (that's pun #1...)

In the past, there have been many systems, gizmos and gadgets released that have incorporated some sort of pointing device. To cut to the chase, none have proven so sucessful, at least on such a widespread scale as the 'Wiimote' used as the primary control device for the Nintendo Wii games console.

There are sensors to detect movement in all directions, including the speed (or rather acceleration) of those movements. The remote can also sense it's tilt and pitch. The main use of the remote however, is to point at the screen and interact with various elements presented on the television. To achieve this, a series of infrared LEDs are placed above or below the television. The remote has a camera built in to it's point that senses the position of the infrared LEDs and interprets this information as a position for a cursor etc. on the screen. Yes, that so-called 'Sensor Bar' that is placed on your TV doesn't sense a single solitary atom of information... all the communication is between the Wiimote and Wii using Bluetooth!

Anyway - the point of me waffling about the Wii Remote is that it's possible to use all it's functionality with a PC!

The best part is that it's actually a relatively simple process to make the connection between Wiimote and PC.

You need the following (links at the end of the post)...

1. a Wii Remote
2. an actual Nintendo Wii OR a wireless sensor bar - you need this for the pointing aspect of the remote. However, you could even use several candles if you wanted to!
3. Bluetooth connectivity on your computer (I'm using a PC with Windows XP for the record...)
4. BlueSoleil - this is the software that will actually detect and pair your Wii remote to your computer
5. glovePIE - finally, this is the software that reads the information from the Wii remote's sensors and provides appropriate feedback - in this case, through mouse movements and mouse clicks!

The only thing that remains is to tell glovePIE what you want it to sense from the Wii remote... The most reliable (but by no means the only) code I've found for controlling the mouse so far resides here:

http://www.wiili.org/index.php/GlovePIE:Carl%27s_IR_Mouse

Just paste all that code into the glovePIE code view and press the 'run' button at the top! ...and Bob should be your mother's brother...

I shall return when I've managed to write some of my own code for glovePIE :)

Links...
http://www.wiiprojects.org/bluetooth.html - useful for setting up BlueSoleil
http://carl.kenner.googlepages.com/glovepie_download - glovePIE download page
http://www.wiili.org/ - everything else you'll need to know for research and scripts for glovePIE

Sensing stuff...

2008-06-11T13:47:00.016+01:00

OK, here's another set of methods of getting information into the computer, other than my webcam experiments.

A short while ago I read a very inspirational book by Brendan Dawes (http://www.brendandawes.com/) called 'Analog In , Digital Out' (at Amazon UK). He's one of the gurus on Flash and all things multimedia. His book focusses on the ways that analog or real-life sources of data, like light and sound, can be converted into useful and meaningful data on a computer.

His experiments include making a doorbell ring when someone visits his website, and gathering colour, size and shape data from a webcam trained on a piece of playdough - where the playdough was the user interface...!

All sounds strange but these two examples alone have given me lots of ideas and influence on my project.

The doorbell example was created using the following kit:
http://www.makingthings.com/products/KIT-MAKE-CTRL/
This electronics kit connects to the computer using either USB or ethernet and can have an array of sensor devices attached to it. These sensors include those for light, infrared, pressure and a series of switches. It's also possible to send signals out to the device and control devices like motors and LED lights.

The data is received into Flash using a custom actionscript class as a stream of numerical data. It's then moreorless up to the designer to use the data as they see fit.

The playdough example was created using a combination of a webcam and a piece of software called Cycling74 Max 5. The software company originally began by making audio-plugins for editing suites on the Apple Mac. Using the program's video capabilities, it's possible to detect an object's size, colour and whole load of other variables...

Lastly, a lot of the examples in the book employ a programming language called Processing, which is derived from JAVA. The description from the website says...

"Processing is an open source programming language and environment for people who want to program images, animation, and interactions. It is used by students, artists, designers, researchers, and hobbyists for learning, prototyping, and production."

Processing is an incredibly versatile language that makes programming specifically for graphics a complete cinch. Many of the classes you would have to explicitly call if using a traditional JAVA compiler are readily available. I downloaded a special class that enabled me to use my webcam within the Processing environment. The example I tested consisted of around only 50 lines of code. It enabled the user to wave their hand (or any other object) in front of the camera and attract a virtual tennis ball to their fingertips... I'll provide an example on my website soon!

These are the 3 main programming environments I shall research and employ during the course of my project...

MA

Jean Michel Jarre - An Introduction...

2008-06-11T11:31:00.012+01:00

The first port of call for my research into unconventional user-interfaces begins with Jean Michel Jarre and his amazing laser harp! First, watch the video below...

For those who haven't heard of Jean Michel Jarre, he is one of the most innovative electronic music producers of our time (well, in all fairness, ANY time as he's only been producing since the mid 60s) in the way he pioneered the use of the first synthesisers and continued to do so.

He's also renowned for his live performances which include:

Rendezvous Houston (text shamelessly from http://en.wikipedia.org/wiki/Jean_Michel_Jarre)

In 1986, NASA and the city of Houston asked him to do a concert to celebrate NASA's 25th anniversary and the city of Houston's 150th anniversary.

During that concert, astronaut Ronald McNair was to play the saxophone part of Jarre's piece "Rendez-Vous VI" while in orbit on board the Space Shuttle Challenger. It was to have been the first piece of music recorded in space, for the album. After the Challenger disaster of January 28, 1986 which killed McNair, the piece was recorded with a different saxophonist, retitled "Ron's piece" and the album dedicated to the seven Challenger astronauts.

The Houston concert entered the Guinness Book of Records for the audience of over 1.5 million. During the concert, Houston native Kirk Whalum performed Ron McNair's saxophone part on "Ron's Piece". The concert featured giant projections of photographic images and laser patterns onto the buildings of downtown Houston, including a gigantic white screen on the front face of the Texaco Heritage Plaza building, which was under construction at the time. Due to vehicles stopping on the freeway passing the concert venue the freeways had to be closed down for the duration of the concert.

Twelve Dreams of the Sun

On 31 December 1999 Jarre held a spectacular music and light show in the Egyptian desert, near Giza. The show, called The 12 Dreams of the Sun, celebrated the new millennium and 5,000 years of civilization in Egypt. It also offered a preview of his new album, Metamorphoses.

The concert which started on new years eve and followed all the way through to the dawn of the new millennium in a 12 hour spectacular which including many performances from local artists and musicians, the concert used the backdrop of the great pyramids to project images onto, but fog during the evening concert by Jarre caused the projections on the facades of the pyramids to be blocked from view. Jarre played for around two hours during the build up to the new millennium with a countdown at midnight and spectacular firework display and then returned on stage in the early morning to perform a second slot to see in the first sunrise of the new millennium.

The following is a clip from Twelve Dreams of the Sun. It shows the countdown to the new millennium - forget London or NYC for the year 2000, this is amazing...

Anyway, I'm slightly off the point here... the focus of my research into Jean Michel Jarre is mainly on the innovative controllers for his synthesisers which include the laser-harp featured in the first video...

...more later!

MA

Project Ideas

2008-06-10T18:10:00.006+01:00

One of the aims of this blog is to document my work on my final year project for university... It's a facility for me to write down all the interesting bits and pieces I happen to discovery along the way!

At the moment my ideas are still coming together for my project. I'll be focussing on the following topics over the next few weeks:

1. The way computer games utilise music.
2. Unconventional user interfaces in - a. Computer games, b. Professional music software, c. Studio/Live artist performances.
3. Various programming langauges that support such interfaces.

4. ...whilst all the time creating/programming experiments, which demostrate some findings of the above points!

Just thought a small bit of clarification was needed before I plunge ahead (for me as much as you, the reader!)

MA

Rez - Gameplay videos...

2008-06-09T10:20:00.005+01:00

For those interested, the following links are complete gameplay videos, from Youtube - wish I was good enough to get past Area 3... C'est la vie...

Area 1
"Buggie Running Beeps 01" / Keiichi Sugiyama
http://uk.youtube.com/watch?v=7hh9fZkr1jk

Area 2
"Protocol Rain" / mist
http://uk.youtube.com/watch?v=A4EFNWe4mCc

Area 3
"Creation The State Of Art (Part 1 - 6)" / Ken Ishii
http://uk.youtube.com/watch?v=yzoGoAo0ZMA

Area 4
"Rock Is Sponge" / Joujouka
http://uk.youtube.com/watch?v=cCh6WlK027I

Area 5 / Last Boss
"Fear" / Adam Freeland
"Boss Attacks" / Coldcut
http://uk.youtube.com/watch?v=3nq_oMDlrtM (an awesome 26 minutes long!)

Enjoying watching!

MA

Rez

2008-06-08T08:35:00.013+01:00

Rez is one of the only games centering around music that can truly get away with it's abstract graphics. I suppose the developers got around having to explain the surreal graphics by saying it's set inside a computer - where allegedly anything can happen :P

The blurb on the back of the PlayStation 2 (I can't account for the Dreamcast box-art) release is wonderfully vague:

Discover a new world
A world of sound, visuals and vibrations
Enter the network and awaken 'Eden'
Release your instincts
Open you senses

(...and in the smallest writing but the most important!)

Experience synesthesia (http://en.wikipedia.org/wiki/Synesthesia)

Without proceeding too much further, have a look at the video below...

The ONLY idea of this game is to aim and shoot. The player doesn't have to worry about navigating around the level or running out of ammunition. This game is known as an 'On The Rails Shooter' which essentially means that the computer has complete control and all you need to do is shoot - simple. Though, not hugely simple(!) If you don't shoot fast enough and/or accurately enough you'll 'die'...

OK - the musical aspect of the game. There are 5 'areas' that make up the game. Each area has it's own piece of music. These are tracks that have been composed by 'real-life' artists. While each track is a proper piece of music in itself, they've been stripped down piece by piece and split into stages to match the pace of the game. That is to say, that at the beginning of an area, it's generally a slow start in terms of having things to shoot at, therefore, the music only consists of a basic drum rhythm. Each area is split into 10 stages. At the height of the action, the music will sound very 'busy' with all the usual parts in place (drums, bass, synths, percussion...)

After harping on about all the shooting and the music, the main part of the game is the experience that the player gets. Everytime they shoot something, a percussive noise, synthesiser note or sound effect is played in harmony with the rest of the music. When an enemy has been dispatched off the virtual coil, there is a small light-show explosion. It's difficult to explain but the way the enemy explodes visually matches the exploding sound... if that makes sense at all (see the Wikipedia link above, yes I KNOW Wikipedia is genrally a no-no, but in this case it's actually a good pool of information, so there...!)

So, to conclude...

The graphics and overall aesthetic of the game will play a huge influence in the way I think about the way I design my application - whether it turns out to be a game - or not...

Links...
Official site of the original Dreamcast/PS2 game - http://www.sonicteam.com/rez/e/news/index.html
Official site of Rez HD for the Xbox360 - http://www.thatgamecalledrez.com/
Wikipedia entry for Rez - http://en.wikipedia.org/wiki/Rez... There's a couple of pieces of information that may prove interesting...

MA

(Rez shall return in this blog...)

DJ Max Portable meets My Webcam - Mockup

2008-06-07T22:27:00.005+01:00

This is roughly what I'm looking at making. I'll be using code from my previous webcam examples and merging that with the beginnings of code that will judge whether the user ('player' is such a strong word at this stage!) has made a motion at the correct time and in the correct place on the screen.

The example shall work as follows:
1. The blocks fall down the screen the image of the user.
2. The user shall jab upwards into the picture with their right or left hand when the corresponding block reaches the thick line at the bottom of the screen.
3a. If the user responds early, late or not at all then nothing will register (- in DJ Max Portable terms, this would count as a failure and would reduce the chances of finishing the song)
3b. If the user judges the timing correctly (there will be a reasonable margin for error here) then the game continues...

Watch this space...

MA

DJ Max Portable on the PSP...

2008-06-07T21:35:00.005+01:00

Well, here we have DJ Max Portable for the PSP. I'm not trying to write a review of the game, rather explain how it works.

The simple idea of the game is to press the correct buttons on the PSP in time to the music. The blocks falling down the screen (see the screenshots) indicate which buttons to press. When the player presses the correct button then they've contributed to the song by triggering a sound. Throughout each song in the game, the button presses add everything from rhythm parts to synthesiser sounds. The more accomplished the player, the more pleasing the song will sound.

If the player makes too many mistakes then the game will exit the level/song and display a message that instructs the player to get more practice. Because of this, the game has quite a steep learning curve.

The difficulty of the game is determined by several factors.

1. The speed/complexity of the song that's playing. The faster or more parts that the song has, then the more blocks will fall down the screen.

2. The number of buttons the player has to use. The player can select to play using 4, 6 or even 8 buttons on the PSP. Personally, playing with 4 buttons so far has proven to be quite tricky!

The music you play to is graded in terms of difficulty, so initially the gameplay feels quite sedate, though after the first couple of songs are out the way the pace quickens somewhat dramatically!

So far this game has proved quite influential in terms of my project. I'll be trying to build a small concept project where the player will have to move in front of a webcam to the rhythm of the music rather than pushing buttons. So far the accuracy of detecting motion on specific parts of the screen using a webcam (see my examples if you can!) have been a bit sketchy so I'll be trying the DJ Max Portable equivalent of '2 button mode' for the time being...

So, the view the player will have is of themselves on the screen with blocks falling down over their image. When the blocks reach the bottom of the screen, the player will make a motion (I'll figure the precise motion out at some point!) over a specified area of the screen to 'clear' the block and enable them to progress through the game...
...hopefully some of that makes sense!

Videos of more talented people than me playing DJ Max Portable 1 & 2 on the PSP
http://uk.youtube.com/watch?v=2pxsq-nA6o4
http://uk.youtube.com/watch?v=09VLs3nqE7M
http://uk.youtube.com/watch?v=zfMlAv7ABEg
(the last link will give you an idea of how frantic the gameplay gets!)

From here:
Mock-up images to demonstrate this idea - pictures are a thousand words which means there will be less stuttering from me!

MA

Summary: Games centering around music

2008-06-04T08:51:00.000+01:00

Some of the most surreal and original computer games over the history of console and PC gaming have involved mixing musical elements with engaging visuals and/or artwork.
I'm in the process of investigating a selection of games that involve the player with music in some way.

Some of the games have the player actually contributing to the music - with each button press triggering a sound... example: DJ Max Portable/Guitar Hero

...all the way through to the music merely being used as an indicator of the player's progress through the game where the music 'builds'... example: Rez

Finally, there are some games where music creation is the only purpose. I won't be focussing too much on this sort of music game - because there isn't really a 'game' element. The player doesn't progress in any way and there aren't any goals or challenges.

Detecting motion in certain areas...with a webcam - part deux

2008-06-03T20:08:00.001+01:00

This example is similar to the other area detection example - however it's more refined, uses 'slightly' more sophisticated code to detect the certain areas, AND is more fun to play with.

When you start the example, you will see three blinking dots towards the bottom of your picture. These are the hitpoints where the motion is sensed.

Area Detection example 2

I've found that raising your hand up relatively quick up into the picture works quite well! It's way more useable than the first example - in my little honest opinion

MA

Detecting motion in certain areas...with a webcam

2008-06-03T20:03:00.001+01:00

Next - it would be handy to detect motion in certain areas of the screen. The following example detects motion in each corner of the camera's picture - like a 2x2 grid.

Each corner plays a different sound when you wave your hand in front of it. Now - it's not the best sort of detection by any stretch of the imagination, but have a bit of a play and see what you think...

Area Detection example 1

Have fun playing many tunes...!

Simple motion detection using a webcam

2008-06-03T19:41:00.001+01:00

Well, I want to make some kind of music-based program or even a game. I haven't quite decided what I'd like to make though, however I'm going to be mucking around and experimenting with wild and wacky user interfaces. Using a webcam as an input device seems a good place to start - the mouse and keyboard are boring :P

So the following link will take you to my first experiement - motion detection. You obviously need a webcam for this otherwise nothing will happen(!). If you're using an Apple Mac with a built-in webcam you may need to select the correct camera device - for some reason there are a few in a list to choose from!

Motion Detection Example

The program senses a certain level of motion in front of your webcam. I guess it depends on which webcam you have - I have the Microsoft VX-1000 and it seems sensitive enough... Let me know how you get on...

At some point I'll upload videos of me waving manically at my computer screen - good times...

MA

Glide Labs' new home!

2008-06-03T19:34:00.000+01:00

This is the home of the news for Glide Labs - the nickname for my final year uni project!

Here will be all the research, programming experiments and all kinds of other things...

Back with 'all kinds of other things' in a moment...

MA