Wow. That video yesterday did have some impact, my page hits went up exponential (and the number of emails in my inbox). Yesterday's post was mostly motivational, here's the hard facts.

Hardware setup and restrictions

The hardware I use at the moment is a MERL DiamondTouch my supervisor organised with MERL as part of MERL's university loan program. Such a table has a number of horizontal and vertical antennas, by touching the table a circuit is closed and the table can tell where the touch happened (see the original paper). It supports up to 4 users, each user has to touch a different conductive pad. This way, the device can differentiate between users.

The kernel driver (written by MERL) spits out a 900 byte package everytime the table delivers it (a few times a second). I've written an X driver that disassembles the package, extracts the blobs and passes them up to the X server. That's all the driver does. The X server does pointer emulation and the rest.

Now - the DT is good for detecting multi-touch from different users but not good at detecting multitouch from the same user. There's only one set of antennas for x and y, so when I press two fingers down, there are 4 possible touch-points. There's ways to work around that (as long as x/y of two touches aren't overlapping) and you may have seen me using two hands in the video.

The DT also doesn't give me real bitmaps for the touch, a rectangular bounding box is all I can get out of it.

If you have a FTIR table or some other vision-based system you won't have these problem and nothing stops you from doing true multi-touch. You may have issues identifying different users though.

The DiamondTouch X driver was the first real X input driver I've ever written, so there are still some bugs in it. Need to check with MERL whether I can put it online, one of the header files I used is from them.

Software

The applications in the video are google earth, firefox and gnome-calculator. All unmodified of course.

The drawing app is a quick test app I wrote a few days ago. It's less than 400 lines of C code, and that includes debug messages and stuff to open/display the window. It's also one of the worst apps I've ever written and until I cleaned up the code a bit I'm too ashamed to put the source online.

It simply adjusts the line width to the dimensions of the touch bounding box and draws a line from the last point to the current. Originally I used the bitmap as the line shape, but as I said before, my hardware doesn't give me usable bitmaps.

The window manager is my hacked-up window manager I always used to test MPX. It doesn't know about touch events.

So during the video you can see touch applications, standard single-user apps and an MPX app simultaneously working. This is the really novel thing. Not the blob drawing app...

The in-betweens

So what's actually happening between a touch and the client doing stuff. Easy: the driver fetches the bounding box of the touch, selects a hotspot (in my driver this is the center of the bounding box) and creates the bitmap. This information is passed on to the X Server. The X server creates a BlobEvent from this information and also emulates a core pointer event and a X Input pointer event on the hotspot.

During the event processing stage, the X server checks where the event needs to go. If it's a standard app, all events except the core event are discarded. If it's a touch appliation, the blob event is delivered. This is just normal stuff, the same thing X does with pointer and keyboard events anyway.

An app can listen to both pointer and blob events of course.

Now one of the reason why the whole thing actually works is XGE. This is an extension I've written a while ago that allows the X server to send events longer than 32 bytes (something that isn't in the original specs). We need that for the bitmaps after all.

Pointer emulation?

Pointer emulation is central to the whole principle. There are a few applications out there that only know about pointers and keyboards. If you show firefox the middle finger as a touch gesture, it will ignore it. It doesn't know anything but pointer and keyboard events. So we can either say "Hey, there's this great new touch-toolkit, rewrite all your apps. And btw. you can't use your email client or web browser until you've rewritten it to understand touch events". Good luck with that.

I don't have time to rewrite tens of thousands of apps. So what MPX does is give those apps the environment they know. Multi-user apps that know about multiple pointers can use additional features. Apps that know about blob events can use those.

So stop whining about pointer emulation and rewrite your apps.

Or maybe not. Not everybody has a touch table, but everybody has a mouse. So your app has to now support both mice and touch screens. And multiple users. And hotplugging. Welcome to hell :)

Choose what your app needs. Do you need touch-support in a terminal? probably not. In google earth? Well, maybe.

btw. a lot of UI elements perform an action on a button release event (clicking links for example). It's actually not easy to press and release on the same tiny button with a input device the size of your finger. That's why sometimes nothing happens although you think you've pressed a button.

When is it available?

I'm still cleaning up and checking the code. I will hopefully merge it into the mpx branch next week.