Using the IR Sensor Bar = Cheating?

Well, when this project has set its sights on using the accelerometer in the Wii Remote for motion sensing then the answer is a resounding 'yes!'. One of the main reasons why I've shied away from the use of the sensor bar is that I wanted to make a gesture recognition application with as FEW artefacts as possible – i.e. just the Wii Remote and no other hardware save for a computer with Bluetooth capability. However, the data gathered whilst employing the IR sensor-bar, the scope for my project has somewhat widened...

After discovering the limitations of the Wiimote's accelerometer I decided to delve into the possibilities of the other sensor in the Wii Remote – it's camera. For those who don't know, that sensor bar sitting on top of the television isn't actually sensing anything at all. It's merely emitting a number of infra-red sources. The Wii knows where the player is pointing because the camera in the Wiimote is tracking the position of the infra-red sources. In fact, the camera in the Wiimote is capable of tracking 4 infra-red points (known as 'blobs') at 100Hz (100 times per second).

The Director Xtra I'm using for the purposes of my project supplies the coordinates (X and Y values between 0 and 1) of each blob in a handy array. So, thankfully, the potentially fiddly process of getting to the data has been dealt with – it's just a question of interpreting the data to get something meaningful from it.

One blob to rule them all...

Armed with my Wiimote, el-cheapo battery powered sensor bar and a few lumps of Bluetack I set to work.

Firstly, my sensor bar isn't as good as the actual Nintendo one, but then it wouldn't be as it cost a whole $6. It has 6 infra-red LEDs, 3 at each end of the bar and arranged in such a way that they 'fan out' to create a spread of infra-red – the theory being that the Wii console will appear more sensitive or reactive. The genuine Nintendo sensor-bar has a total of 10 LEDs in it making it far superior. For the purpose of the program I'm writing, I only need one (there are downfalls to this approach that I'll explain shortly) so I covered up the rest of the LEDs with Bluetack!

I've adapted the program I wrote for the accelerometers to use the readings from the Wiimote's camera instead.



The 3D model of the Wiimote is pointing forwards and doesn't rotate with the movement of the real Wiimote. Obviously there will be a certain amount of rotation when the real-life Wiimote is moved, however one has to remember that the Wiimote's camera must be pointed in the direction of the infra-red. This is so the application can recognise the movement. If the application relied on the accelerometer for movement as well, there would have to be some consideration taken for the occasions that the Wiimote's camera would be out of view of the infra-red (i.e. when pointed straight up or down).

This is where the downfall of using one infra-red LED for positioning becomes apparent. LEDs in general do not have a very wide emitting angle. It's very unlikely that you would see any light – or in this case, detect any infra-red – if looking at the LED from the side. Once the Wiimote's camera has moved out of effective viewing range of the LED then the readings either become unreliable or non-existent (in this case the Wiimote returns a value of -1).

Accelerometer & IR side by side...

This version of the application incorporates a slightly different method of movement recognition. The versions of the application that rely on accelerometer data as a means of control use – for want of a better term – a state logic. The Wiimote has to be swung to a certain position for it to register as a different movement. For example, if the user swings the Wiimote to the right, the application will register that the Wiimote is in the 'right' position. Now, if the user swings the Wiimote to the left, the application won't register any leftwards motion until the accelerometer is reading that the Wiimote is in the 'left' position. It is better to think of the application as recognising a series of pre-set positions that the Wiimote can be held in rather than detecting the direction of the motions themselves.

The version of the application relying on IR dot positioning works differently. There are no pre-set positions that the user can hold the Wiimote. This time, the application senses the direction of the motion. For example, the Wiimote is motionless in the user's hand – the 3D model on the screen is also stationary. The user moves the Wiimote up – as soon as this happens, the application moves the 3D model of the Wiimote up the screen and registers this as an upward movement.

This method is much more effective and natural-feeling than the previous applications using the accelerometer data. Whenever the Wiimote is stationary this registers as an 'idle' state – unlike the accelerometer application where 'idle' meant holding the Wiimote in a specified position.

The method also makes the 'game' I created seem a bit easier and less frantic. When the game demanded an 'up-down' motion, the accelerometer-based version would see the player having to swing the Wiimote far enough to get it into the correct position. It's confusing for the game-player who is probably wondering why it took so long for their motions to be recognised. Now just the slightest movement in the correct direction would register as a movement in a certain direction – I suppose some may argue that this takes the challenge out of the game, but it opens the door for more complex gestures that the player would not have had the time (or patience) to make.

Another advantage to this method of sensing is that it's easier to perceive the motion one's making when gesturing at a 2D surface such as a television or computer monitor...

As an example scenario; one of the gestures the application may recognise is a 5 pointed star. You are given a whiteboard pen. There are two whiteboards in the room. One is mounted on the wall and the other is mounted on the ceiling. You are told that you have to look in front of you while drawing, so no neck tilting! First you draw the star on the whiteboard mounted on the wall - you can see the board and the shape you're drawing so it's really easy. Now, when it comes to draw the star on the ceiling whiteboard it's not as easy because you can't really see what you're doing and the star looks distorted.

OK, that may be quite a long way of trying to explain the benefits of the IR dot method, however it nicely highlights its benefits. In programming terms, it makes the process of designing gestures easier and enables more complex methods of determining gestures to be integrated into the application – such as the Levenshtien Distance algorithm.

Next steps...

One of the next tasks in this iteration of the gesture application is to investigate a reliable method of using the coordinate data of more than one IR blob. Not only would this method increase the sensitive area of the Wiimote, it would also make the motion much smoother (similar to my averaging method to smooth out the accelerometer readings).

Returning to the accelerometer-based version of the application. I may try to alter the code to sense change in direction rather than finite states. The accelerometer data was too 'jittery' the last time I tried it, however, that was before I started using averaged data over time rather than real-time values. It might be possible! However, bear in mind that the problems highlighted in my whiteboard scenario earlier will still be an issue.

Once the aspect of getting the user's input has been dealt with, the various methods of checking for valid gestures will be the next priority. In terms of my project proposal, the investigation into forms of gesture alphabets shall begin. This will represent another iteration in the application, where we move away from the gaming aspect and more toward the place that gesture recognition has in the everyday computing world...