Further to my first entry here, Chris and I have been hard at work making our software compatible with the rest of the world.  In order to do this, we have made a stand-alone application which reads data from the Wiimote and then goes on to generate TUIO and Windows 7 touch events.

This software can be downloaded in binary or source format from our Google Code project: http://code.google.com/p/wiituio/

In summary:

This project aims to improve the stability of the IR sources captured by the Wiimote using some thresholds and spatio-temporal classification. The application generates TUIO and Windows 7 Touch events using these stabilised contacts.  This enables people to turn their existing Wiimote whiteboards into fully functioning Wiimote multi-touch whiteboards that are compatible with most multi-touch applications out there.

Each raw IR source captured by the Wiimote is either assigned to the best existing tracked source or generates a new tracker. This means that events can be generated from stable data without the jitter (namely, false-positives generated between two IR sources and the unordered source buffer) that occurs when trying to use the Wiimote to capture true multi-touch IR.

We hope that by releasing this to the public we will make it easier for people to start hacking around and making multi-touch fully collaborative (Single Computer-Multi User) applications without inuring the need to start with the large cost of professional hardware!

For your viewing pleasure I have uploaded a small YouTube video which demonstrates this software in a nut-shell.  The video was taken at the behest of some of the folk over at the fantastic Natural User Interface forums.

http://www.youtube.com/watch?v=rbMafItax6Y

- A video demonstration which compares the previously available software to ours before going on to demonstrate the integration with Windows 7 (MS Surface) technology.

“Hello World”

Recently Christopher Bull and I have been putting together a low-cost multi-touch interactive table for a project we are working on.  Whilst shelling out a few thousand pounds for one of the fancy FTIR (Frustrated Total Internal Reflection) devices was not out of the question, I am an avid fan of both leg-room and trying to make things on a shoestring budget.  This inevitably led us to the fantastic work of Johnny Chung Lee and Brian Peek.  Peek’s managed Wiimote Library for C# enabled Johnny Chung Lee to use some clever maths to transform the infra-red light spots sensed by the Wiimote from the real world onto a virtual digital rectangle.  From here he took the virtual positions and used them to control the mouse.  Very elegant in it’s simplicity.

At this point, Christopher and I thought we were onto a sure thing for our own cheap multi-touch table.  Rather than just taking the one point to control the one mouse, we just take all the other points and map them onto our own application.  However there are couple of problems with this:

• Windows XP (the target platform for our project) does not support multi-touch.  This was not the end of the world because of projects like TUIO and MultitouchVista. I cannot commend these projects enough!  However required use of the MT:Controls and the TUIO layers introduced too much extra stuff into the software stack for my liking at this early stage.  It was all getting rather complicated rather quickly.
• The WPF (another target platform) prior to version 4 did not support multi-touch.  This meant that applications designed for it were only able to recieve input from a single mouse.  This is largely due to controls being able to ‘capture’ the events triggered by an input device – a necessity in most UI implementations.

These two issues alone were small and relatively easy to deal with.  In fact, by using both the TUIO and MultitouchVista projects we were able to get a simple proof of concept off the ground.  The chief drawback of this combination though is the Wiimote itself.  The problem lies in how it reports the sensed infra-red light back to the user:

So here, because each tracked input (e.g number 1) takes up a buffer position.  In the first state there are 3 IR lights being tracked.   If one is lost (say input number 2) then it is impossible to know which inputs are left because they are not defined (or rather indexed) by their position in the buffer.

This means that the user experiences strange behaviour characterised by inputs jumping around when more than one user uses the table and begins/ends clicking.  Bad times for us because it means that we cannot reliably implement ‘input capturing’ which, as mentioned before, is very important to user interface libraries.

What next?

The solution to this problem is relatively simple.  You keep a history of all the tracked points for a given time and check all the new ones to the old ones; pairing them up based on some thresholds and distance.  The code to do this is pretty tricky because you end up mapping N resources to M destinations – naturally, an in-place solution is _much_ harder to write than the alternative and given the small numbers if IR lights we are dealing with is almost completely unnecessary from a performance perspective.

The algorithm looks something like this:

So far it seems to be working a treat.  I will post up some code and a test application soon.  This algorithm should be able to be dropped straight into a project like Wiimote TUIO to make it behave a bit more naturally.

There are several things to keep in mind:

• The Wiimote isn’t perfect and there are still artefacts.  I am working on removing these with some more advanced maths (for instance, it sometimes generate extra tracking points along a line between two inputs).
• Sometimes the line of sight between the IR sources and the Wiimote gets interrupted or the Wiimote momentarily looses tracking of the source.  The algorithm above will not correct for that as it stands, but it is possible to use some temporal metrics to work out how likely it is that it was accidental or a purposeful flicker.
• Rather than just using the euclidean distance between raw positions as a basis for ranking the inputs, I used the average of the predicted input position and the smoothed actual position to help add bias that helps the algorithm select the correct tracker when the distances are the same.
• This system allows the developer to assign their own unique ID’s to each input, thus allowing higher level applications (such as user interfaces) to capture events from specific inputs.

The next step from here is to create a set of simple bindings which can be connected into existing WPF controls at runtime (not compile time like the MT:Controls) such that they can respond to muli-touch events!

I hope this is of some interest!  Post if you want some code!