MPX: The Multi-Pointer X Server -

Last day at Uni - shifting the blog

peter — Mon, 30 Jun 2008 16:32:01 +0930

Today was my last day at uni. Working here was great, since the team in the lab was not only quite knowledgable, they were also a lot of fun to hang out with. Likewise, thanks to the University I was able to publish all the code I wrote during my PhD candidature as open source, which will eventually benefit (or annoy) a lot of people. Thanks to all who helped me over the years.

As part of cleaning out my desk I am shifting this blog to
http://who-t.blogspot.com.

Consider this blog to be discontinued.

Likewise, in the foreseeable future, my email peter@cs... will be shut down. Please update to peter.hutterer, at the domain who-t.net.

DiamondTouch 0.3.0

peter — Wed, 25 Jun 2008 18:00:53 +0930

xf86-input-diamondtouch 0.3.0 is out.

This is basically just to tie up some loose ends, the driver was broken since I had to orphan the blob branch. 0.3.0 builds against current master, all blob stuff is deactivated.

The driver now supports multi-device hotplugging through the option "NumDevices". This enables users to let the DT be hotplugged through HAL, but still create up to 4 devices in the X server to represent the required number of users.

Look at the HOTPLUG.README to get hotpluggy goodness.

Get it while it's fresh.

http://people.freedesktop.org/~whot/diamondtouch/xf86-input-diamondtouch-0.3.0.tar.bz2
MD5: 5b6f3ae84cfd38bca2c3e6fdf9057ee1 xf86-input-diamondtouch-0.3.0.tar.bz2
SHA1: b42e1dd23ecbcf833899d36027435b120aa65e71 xf86-input-diamondtouch-0.3.0.tar.bz2

http://people.freedesktop.org/~whot/diamondtouch/xf86-input-diamondtouch-0.3.0.tar.gz
MD5: a4e97ff3407efb907b1a699949745e8d xf86-input-diamondtouch-0.3.0.tar.gz
SHA1: d05a6db882c6113ab8076ffe0f8f9ef70b1ca1e0 xf86-input-diamondtouch-0.3.0.tar.gz

Multi-touch? or multi-point?

peter — Thu, 12 Jun 2008 23:13:10 +0930

Increasingly I get as whether MPX allows multi-touch. The answer is neither yes nor no.

There is a bit of confusion out there as to what constitutes multi-touch. The simple answer "deals with multiple touch points" isn't quite that simple.

What is "touch"? In user interfaces, it usually means being able to manipulate an object by touching a sensitive surface. This can be done indirectly (touchpad on the laptop) or directly (touch screens).

To the X server, whether it's direct or indirect manipulation is irrelevant. What does matter though is the data the touch-device provides. Many devices reduce the touch point to a simple x/y coordinate (e.g. Wacom tablets, most touchscreens in public places, and AFAIK even the iPhone do essentially that [1]). From the X server's point, there is no difference between a mouse and such a touch device. MPX is multi-point, i.e. if your device supports multiple touch points, or you have multiple such devices, you have already won. Use them. Write software for them.

But - and here's the big difference: some devices can detect the area of touch, rather than a single point. And here's where it gets interesting, as this allows gestures. Touch with a flat hand is different to the side of hand, is different to the thumb, is different to a finger. This is true "touch support", and by far not commonplace yet. One of the prominent examples that supports true multi-touch is MS Surface.

MPX doesn't do multi-touch. But it turns out that multi-point is the hard thing and multi-touch is quite easy (once you have multi-point) [2]. The only difference is that you have to get the information to the clients. The X server doesn't have appropriate events to send the data and last year, I dabbled with this for a few weeks [3]. The result were the BlobEvents [4], but that branch hasn't seen many updates since, mainly due to lack of time. It will come back, once I find the time to merge the blob branch into current master, clean it out, and get some public review.

But for now - multi-point. Not multi-touch.

[1] there are exceptions, some provide pressure on the point, wacom tablets provide tilt, etc.
[2] From a technical perspective. Semantically, it's quite hard to get it right.
[3] http://wearables.unisa.edu.au/mpx/?q=node/86
[4] http://wearables.unisa.edu.au/mpx/?q=node/88

So you want to build a multi-device aware application?

peter — Thu, 05 Jun 2008 10:38:41 +0930

The use of Xlib for XInput has changed a bit, but not that much. This mini-tutorial shows you how to get the list of devices and how to register for events. For many applications, this is pretty much all you need.

Let's get started. The two most important changes to previous versions of XInput are:

applications have to announce support for XI 2 (MPX).
the list of input devices contains multiple core devices.

Announcing support for XI 2 is easy, just issue an XQueryInputVersion() before you use any other XI requests. The reason why this is necessary is simple: the behaviour of XInput has changed, and the server must know which clients suports XI 1.x, and which clients support XI 2. This way, it can change some replies accordingly to make XI 1.x clients not break. In practice, your code should look something like this:

int main (int argc, char* * argv) {
    Display * dpy;

    dpy = XOpenDisplay(NULL);
    XQueryInputVersion(dpy, XI_2_Major, XI_2_Minor);

    /* do stuff */

    XCloseDisplay(dpy);
    return 0;
}

Next, we want to know which devices are actually available. XListInputDevices() does this for us. It returns an array with XDeviceInfo structs, each of which specifies the name, id and the capabilities of the devices. In addition, the XDeviceInfo also contains a "use" field, which is of some importance. The use field can have one of 5 values:

IsXPointer for any master pointer (i.e. cursor).
IsXKeyboard for any master keyboard (i.e. keyboard focus)
IsXExtensionPointer for any physical pointer device.
IsXExtensionKeyboard for any physical keyboard device.
IsXExtensionDevice for any physical device that is neither pointer nor keyboard (this is fairly uncommon).

As a rule, you never register for events on the physical devices. Really. The cursor and the keyboard focus are the user's input points, so focus on them. Only special applications like the GIMP or configuration apps have to worry about the others. Again some code:

XDeviceInfo* info;
int ndevices;
int i;

info = XListInputDevices(dpy, &ndevices);

for (i = 0; i < ndevices; i++) {
    XDeviceInfo* current = &info[i];
    printf("Found device: %s (%d)\n", current->name, current->id);
    switch(current->use) {
        case IsXExtensionPointer:
        case IsXExtensionKeyboard:
        case IsXExtensionDevice:
                /* foo() */
                break;
        case IsXPointer:
        case IsXKeyboard:
                /* bar() */
                break;
    }
}

XFreeDeviceList(info);

Try calling XListInputDevices before and after a call to XQueryInputVersion().You'll notice that the device list is significantly shorter if the server thinks you do not support XI 2.

Ok. Now we know the devices, let's start using them. Writing most multi-device apps is incredibly easy: Open the device, register for events, wait for events to roll in.

XDevice *pointer1;
XDevice *pointer2;
XEvent ev;

pointer1 = XOpenDevice(id1); /* get the ID from ListInputDevices */
pointer2 = XOpenDevice(id2);

/* register for events, see below */

while(1) {
   XNextEvent(dpy, &ev);
   printf("Event type %d received\n", ev.type);
}

Now, I left out the registering for event for now because this is historically painful. You see, the X Protocol specification requires that extension event codes are dynamically assigned at runtime. That is, while the core protocol's type ButtonPress is always the same, the XI type DeviceButtonPress can change from server to server (and even when you restart the server). So we need to get the type. In addition, because we only want to register for certain device's events, we need to get the EventClass. It's easiest to think of the EventClass as the mask to register for a specific event on a specific device. Xlib provides some macros for us here:

int type_bpress;
XEventClass cls[2];

DeviceButtonPress(pointer1, type_bpress, cls[0]);
DeviceButtonPress(pointer2, type_bpress, cls[1]);

Now, type_bpress is set to the event type for DeviceButtonPress events. We can pass the same variable in both times since the type remains static for the lifetime of the server. The class however is device-specific. (Oh - and by the way, the macros re-use the arguments twice, so don't try to pass in something like (evclass++), it can be pain to debug...)

Now we can complete the above example:

XDevice *pointer1;
XDevice *pointer2;
XEvent ev;
int type_bpress;
XEventClass cls[2];

pointer1 = XOpenDevice(id1); /* get the ID from ListInputDevices */
pointer2 = XOpenDevice(id2);

/* register for events */
DeviceButtonPress(pointer1, type_bpress, cls[0]);
DeviceButtonPress(pointer2, type_bpress, cls[1]);
XSelectExtensionEvent(dpy, myWindow, cls, 2);

while(1) {
   XNextEvent(dpy, &ev);
   printf("Event type %d received\n", ev.type);
   if (ev.type == type_bpress)
   {
        XDeviceButtonEvent* bev = (XDeviceButtonEvent*)&ev;
        printf("Press received by device %d\n", bev->deviceid);
   }
}

Here's another important change in XI 2. Events always contain coordinates in the screen's coordinate system, only valuators are device-specific. So for most apps, you don't have to worry about coordinate conversion, just use x_root, y_root or x/y of the XDeviceButtonEvent structure.

That's it for now. There's a number of other macros such as DeviceMotionNotify, all to be found in XInput.h. All the functions described here have man pages that explain the parameters better than I can.

Click here to see the full source.

MPX has been merged.

peter — Tue, 27 May 2008 14:32:08 +0930

http://lists.freedesktop.org/archives/xorg/2008-May/035641.html

For those of you watching xorg-commits, you may have noticed that MPX has been
merged into master [1].
The client-side libraries and protocol headers have been merged quietly last
week already, this time is just the xserver itself.

A change in the previously proposed schedule [2]: the XKB rework didn't make
it. With the recent SSL issue and other workloads, Daniel wasn't able to get
it ready in time. We agreed on merging MPX without it, as it should not affect
XKB much anyway.

A number of changes went in since the announce, the most important being the
return of input device cordinate scaling. Tablets can thus be used. Thanks to
Magnus Vigerloef for his help on this matter.

The input ABI has been bumped and all the input drivers on fdo have been
updated to compile with the new API. You will need to get the latest version
however. I don't think the ABI will remain stable until the next release, but
I figured input modules not loading encourages people to fetch the latest
release rather than complaining about them not compiling anymore.

The video ABI has not been bumped yet, but you _must recompile_ your video
driver. No API changes, a recompile is enough.

As previously mentioned, existing setups should not be affected and anything
that changes is most likely a bug. Unless you create additional cursors/foci,
in which case the border between bug and broken UI paradigm is somewhat
blurry.

For the impatient, run the following commands:
Update xorg/app/xinput to the most recent version.

xinput --list --short
xinput --create-master "foobar"
xinput --reattach "MyMouse" "foobar pointer"
xinput --reattach "MyKeyboard" "foobar keyboard"

It is at this point you have a second set of devices that work independently.
It is at this point that you may realise that neither GNOME, KDE, nor any
other environment actually works decently with multiple devices. So start
fixing them right now.

To go back to a standard setup:

xinput --reattach "MyMouse" "Virtual core pointer"
xinput --reattach "MyKeyboard" "Virtual core keyboard"
xinput --remove-master "foobar pointer"

Have fun. Input bugs that result from the MPX merge can be assigned to me.
So long, and thanks for all the mice.

Cheers,
Peter

[1] http://lists.freedesktop.org/archives/xorg-commit/2008-May/016662.html
[2] http://lists.freedesktop.org/archives/xorg/2008-May/035225.html

Tablet support is back

peter — Sat, 24 May 2008 11:42:00 +0930

With the great help of Magnus Vigerlof, plenty of emails back and forth to understand it, and finally Pippa and Ben Avery who lent me their wacom tablet over the weekend I'm happy to announce that tablet support (i.e. scaling of input coordinates) is back. This was the last big chunk missing.

The coordinate handling to the client is:
rootx/y and eventx/y of a DeviceXXXEvent are now always in screen coordinates. This eases the development of XI clients that don't really care about device coordinates.

valuators0...32 are always in device coordinates, and the coordinates depend on the device mode. If the device is mode relative, the valuators are relative, if the device is mode absolute, the coordinates are absolute.

One thing that we need to think about is whether we introduce a ReportingMode in addition to the device mode. ATM, it is not possible to have a device report relative coordinates if it is in absolute mode.

MPX merge coming up

peter — Mon, 12 May 2008 09:47:12 +0930

The MPX merge has been announced.
http://lists.freedesktop.org/archives/xorg/2008-May/035225.html

After over two years of development, MPX will be merged into master in the
week following the 25th of May.

Any yays and nays need to be expressed ASAP.

=== Stability ===
I'm using it on my laptop and on my box at home. Most of the recent patches
were cleanups, and I haven't had any issues with it.
With the most recent patch to HW-render the cursor, I cannot tell the
difference between master and mpx.

== Schedule ==
daniel is working on an xkb cleanup right now. mpx will be merged into this
xkb cleanup, and the combination will be merged into master.

xkb-mpx merge to happen in the week of the 19th to 25th, drop onto master
after testing.

This includes changes to x11proto, xextproto, inputproto, libXi, libX11,
libXext, xinput, and xserver.

I also need to get a simple patch into libxcb. This must be done first, w/o
the libxcb patch I cannot merge.

== Changes and possible issues ==
If you only ever use one cursor and one keyboard focus, you should not notice
any differences. If you do, please file a bug.

The story is different for multiple input devices. Now, as far as I can tell
this is the first time ever anyone has had native support for multiple
independent input devices on the GUI. So a number of basic UI-paradigms just
curl up and weep when you start juggling three mice and four keyboards.

I spent a lot of time trying to make the server behave well, even if the
application doesn't support multiple input devices. But, nothing is perfect
and there are some cases where the behaviour may seem weird. These behaviours
however should be deterministic and thus you can get used to them until the
applications are fixed up and actually support XI.

One thing that will get lost for the time being is support for tablets. Magnus
is working on that, and it may be in mpx before the merge happens.

== XI v2.0 ==
Due to the changes in server behaviour, XI will be bumped to v2.0. XI 1 is
still supported, albeit the server does some tricks here to not screw up your
settings.

Any comments - please put them forward NOW before it is too late!

HW rendered cursors are back.

peter — Wed, 07 May 2008 13:58:22 +0930

Pushed a simple but quite important fix yesterday.

As you may know, cursors are rendered in HW these days. One benefit of this is that the cursor doesn't flicker when it is moved around. Another one is that the cursor is responsive even when the computer is under load. (This has to do with the server implementation, not with the hardware. HW cursors are updated during SIGIO handling, not during event processing, hence the responsiveness)

Unfortunately, current hardware can't render arbitrary numbers of cursors, and although anholt claims that he has ideas of how to render two cursors in HW, I understand we'll be stuck with SW rendering for a while (cursors, DRI works nonetheless). SW cursors are bad, since they can be laggy. I've been using it for a while at home, and it can be quite annoying.

The solution: MPX now switches between HW and SW rendering. As long as we have only a single cursor, we use HW rendering. When the second cursor is created, the server switches to SW cursors, and then back again to HW when only one cursor is left.

The side-effect: I could now drop MPX onto your machine and you wouldn't even notice that it's there until you start creating new devices. Yay.

GIMP support is back...

peter — Mon, 28 Apr 2008 15:18:05 +0930

Well, that's just one of the changes, but the most visible one.

In the past, the GIMP didn't work. Simply because the new device semantics state that if you grab an attached slave device, it will be detached and set to floating automatically until you ungrab again. Now, the GIMP always grabbed all extension devices, leaving you with nothing to actually control the cursor.

So, one of the changes I did over the last weekend was to get some versioning support into XI. The previous one was weak, as it didn't allow the client to tell the server what it supported. A new XQueryInputVersion() call in libXi can do that now, unless this call is issued we assume that the client supports XI v1.x.

For those clients that only support 1.x, we don't return the full device list but rather only the virtual core pointer, the virtual core keyboard and finally all floating extension devices. This essentially resembles a XI setup pre-MPX. As a result, the GIMP only tries to open extension devices that are already floating, leaving you with the ability to control your cursor normally.

This isn't a perfect solution, as some other semantics change too but it works alright so far.

Other than that, it was mainly cleanup. Including the transition of libXi to docbook xml based manpages, which should make it easier to actually write manpages now.

To update, pull inputproto, libXi and xserver.

Fixing some issues

peter — Tue, 15 Apr 2008 16:13:40 +0930

Had some fun over the weekend. Turns out the wireless desktop I bought the other day is ... well, special. The evdev driver doesn't just register a mouse, the multimedia keys on the keyboard are registered as key class on the mouse. When you press any of them, the X server thinks the mouse just pressed a key.

Now as it happens, MPX wasn't really set up to cater with devices that are both mice and keyboards simultaneously. Even worse, the class copying MPX needs to do each time you change devices was quite expensive. So I've implemented some workarounds for all that and MPX now works smoothly and stable on my home machine.

I think it's getting close to being merge-able now, so please give the current version a good testing.