I had a bit of time on the weekend to give something new a try: x2x.

For those that don't know it, x2x is a program that allows you to control the cursor/keyboard on a remote X server (using XTEST). Turns out aside from a few bugs MPX worked exactly like expected: I could export a single cursor to another box while the others are still working locally and likewise I could export the cursor from another box into one specific cursor on the MPX box. While of course using the connected devices to control separate cursors.

This is neat, as it removes the borders between the screen. It doesn't matter which box a device is connected to, you can just interact on any machine that is within reach. Think of a meeting with a shared digital whiteboard. You work on your laptop, then move over to the whiteboard to work together with others, then move back again.

Some bugs are left, grabs don't always work and I'm not sure about keyboards right now. But it's still exciting to see that it works in principle.

VNC works on a similar principle. I connected vncviewer to x11vnc on the MPX box and had vnc control a different cursor than the local one. Pity that vncviewer doesn't render both cursors though.

Aside from that, not much time to work on anything. The thesis is eating up all my time.

After quite a while I finally had time to look at the blob event stuff again, and a couple of new ideas have been implemented. (btw. this breaks existing APIs and ABIs, so make sure you alter the lot if you have written a driver or app).

Changed/new fields in the XBlobEvent:

subdevice ... removed. see identifiers
rotation ... if your device can detect what a blob is, then you should be able to also detect rotation. (new)
intensity ... use this for pressure, light intensity data, etc. (new)
elevation ... the distance between the source of the blob and the actual blob. On a touchscreen, you could use this for a hover information. On a IR-remote based device, you could use it for the distance between the wall and the light source.
identifiers ... replaces the subdevice. The way I originally envisioned the subdevice (bitflags or constants for e.g. Middle finger right hand) would not be flexible enough. It is replaced by identifiers, which are simply Atoms following the event on the wire.

This is a lot more flexible, as your device can now define what an atom means itself. Over time I hope we can define a couple of standard atoms ("Right hand", "Thumb", "Middle finger", etc.) that are used by most drivers. A combination of these atoms would then specify what the actual blob represents. This also leaves more room for future devices.

Reporting the capabilities

I also finally implemented the BlobClass stuff in the server. So now you can query the server for the list of input devices and it will list the blob device's capabilities. Furthermore, the device must now also report resolution and min/max value for its axes.

The following capabilities can be reported by the device (from XI.h):

/* Blob capabilities */
#define BC_Elevation   (1 << 0) /* can detect elevation                   */
#define BC_Rotation    (1 << 1) /* can detect rotation                    */
#define BC_Intensity   (1 << 2) /* can detect intensity                   */
#define BC_Identifier  (1 << 3) /* can identify objects                   */
#define BC_Data        (1 << 4) /* can provide bitmap data                */
#define BC_Multiblob   (1 << 5) /* multiple blobs simultaneously          */
#define BC_Multibutton (1 << 6) /* multiple button presses simultaneously */

The capabilities reported are a bitmask of the above. For example, my DiamondTouch driver reports BC_Multiblob. A FTIR table could report (BC_Data | BC_Multiblob). MS Surface, with it's ability to recognise a few objects could do (BC_Elevation | BC_Rotation | BC_Data | BC_Multiblob | BC_Identifier).

This gives a client the possibility of figuring out whether a device can actually deliver the type of blob events needed. Upon initialisation, a driver also has to provide a list of identifiers it will use later. So if the driver supplies the atom for "Right Hand" here, you know that blob events can be tagged with this atom lateron.

Just a word of warning: I ran out of time, so testing was short. I haven't ported my drawing app to the new blob events yet, so far all the testing was done without blob-aware clients.

xinput [0] has a blob branch now, use it to list your device's capabilities.

ok, really gotta go now.

[0] git://people.freedesktop.org/~whot/xinput

If you are in/around Adelaide next Tuesday, I'll be giving a presenation of and about MPX at the LinuxSA monthly meeting.

snippet from Geoffrey's announce email

  When:   7:00pm-9:30pm (doors open 6:45pm) on
          Tuesday, 20th November, 2007
  Where:  Senior Secondary Assessment Board
          of South Australia (SSABSA)
          Boardroom (1st floor)
          60 Greenhill Road
          Wayville SA
  Cost:   FREE
  Who:    Anyone and everyone.
          No pre-registration necessary.

More info about venue etc at www.linuxsa.org

I just merged my mdsd branch into mpx and pushed the whopping 54 commits (sorry, should have probably pushed earlier but I didn't know if it would all work out). MPX now has a device hierarchy. Let's explain how it works:

Slave devices (SD): These are your physical devices, the ones plugged in to your computer.
Hotplugging supported through HAL, as in the standard X server. An SD can be attached to a master device.

Master devices (MD): These are virtual devices inside the server. The master devices are your actual interaction points, represented by a cursor and a keyboard focus. There's an even number of MD at all times (N pointers, N keyboards). New MDs can be created and removed at any time.

By default, you get one set of master devices (Virtual core pointer, Virtual core keyboard). All SDs are attached to these MDs. So unless you do something with the attachment, everything is exactly like the standard X server.

Download the xinput tool from git://people.freedesktop.org/~whot/xinput, then checkout the mpx branch and compile it.

$> xinput list

Lists all your devices. A device that has the "use" field set to IsXPointer or IsXKeyboard is a master pointer or keyboard respectively. SDs have IsXExtensionPointer or IsXExtensionKeyboard.

$> xinput create-master foo

You have now created two new devices: "foo pointer" and "foo keyboard" (say they have device ids 10 and 11). A new cursor is on the screen and you have a new keyboard focus. Let's say "xinput list" tells you that device 2 is your keyboard and device 3 is your mouse.

$> xinput reattach 3 10        # attach SD 3 to MD 10
$> xinput reattach 2 11

Now your mouse controls the second cursor, and the keyboard feeds into the second keyboard focus, with all the MPX goodness we had before.

$> xinput reattach 3 0         # 0 is the virtual core pointer
$> xinput reattach 2 1         # 1 is the virtual core keyboard
$> xinput remove-master 10 

The new master device has been deleted, the cursor disappeared, and your devices control the original cursor again.

$> xinput float 3 

Your mouse is now floating. It is not attached to a master. It will still send events (see below), but you won't see a cursor anymore and you have to hand-control it. You won't need this too often, but it's possible.

Rinse. wash. repeat.

What happens internally?

Each time you move the mouse, the SD generates an event. This event is sent to clients (if applicable). The event is also duplicated and sent through the master device to the clients. A core event is emulated by the master and sent as well. A floating SD doesn't sent through a master.
So you have the choice to listen for core events (you'll always get events from MDs), for XI events from MDs (this is what you should do) and to XI events directly from the SDs. You should never have to listen directly to the SDs, unless you have a really, really good reason to do so. I can't think of any right now though.

So basically nothing has changed: you as developer listen to XI events just like before, except that you ignore SDs.

A DeviceHierarchyChangedEvent is sent whenever a SD or MD is created, reattached, floated, removed, etc.

Flexible device classes

It gets more interesting when you have devices with different capabilities. The MD will always adopt the state of the last SD. If you XListInputDevices, you can see each device has classes. Each time an SD routes an event through a MD, the MD adopts all the classes of the SD.

If this happens, you will get a DeviceClassesChangedEvent. Further events from this MD represent the capabilities as in the given classes. When a different SD sends an event, the MD flips again.

So in the worst case, where you have a wacom tablet and a usb mouse attached to the same master, the master may change state between reporting in absolute device coordinates and reporting in relative screen coordinates with each event. Whoopee.

The cool thing about this MDSD hierarchy is that it allows MPX to look exactly like the normal X server. Unless you create a new MD and reattach a device, you won't notice it's there. Yay.

btw. one bug (which may be a feature, hard to tell) is that keyboard layouts are somewhat weird now. If you have en english layout on SD1 (attached to MD1) and a german keyboard layout (SD2, MD2) and you type on both at the same time, each time the keyboard changes internally, clients will get a mapping notify. Clients usually query the keyboard map, but because they don't tell you which keyboard to use the map for the ClientPointer's keyboard is returned. This means that if your CP isn't set properly, you may lose your keyboard layout depending on the application. (e.g. xterm 1 always thinks english layout, xterm 2 thinks all come from a german layout). This feels a bit weird and I'll probably change this.

I finally got a day of coding again after having to to lots of other stuff over the last week. Finally pulled from master (the first time since the libpciaccess merge). Quite a whopping number of commits and in the meantime master has adopted some changes from MPX. I hope the pull didn't break anything but so far it looks ok.

I got the master/slave device hierarchy partly sorted out, maybe I can merge a half-decent working version into MPX by the end of the week. I don't expect it to be finished anytime soon though. Routing events through multiple devices is causing major headaches with grabs.

I talked to Laszlo Bacsi over ICQ a while ago about blob events. Last week he came back to me and pointed me to this video: http://www.youtube.com/watch?v=jylEzf01dPE

Guess that answers whether you can couple MPX and FTIR. Here's the email he sent me with more information about what he did:

Hello Peter,

here's how I did the MPX+multitouch stuff:

- I did a standard Ubuntu 7.04 install under Parallels (everything
runs on a Macbook).
- Downloaded all the necessary stuff from the Xorg repository and yours.
- Compiled everything. This was quite easy following the guide on the
MPX webpage plus some common sense.
- Half of my work consisted of solving a video driver problem which
was caused by Parallels. (parallels official driver doesn't work with
the development version of Xorg and the vesa driver had some problems
on bigger resolutions) This is was spesific to Parallels so on a two
machine setup this might not be a problem.
- Our multitouch screen is an FTIR screen (google Jeff Han) and
finger tracking is done by a modified version of recTIVision which
uses the TUIO protocol for sending touch events. I wrote a simple
converter which sends this data to the xf86-input-blob driver in the
required format.
- and that's it basically. the result can be seen on youtube: http://
www.youtube.com/watch?v=jylEzf01dPE

I don't know if I covered everything, so if there are other questions
I'll be happy to answer them.

Regards,
LacKac
http://lackac.hu
http://kitchenbudapest.hu

Now from what I can tell, the app (CatanTouch) is only using one touchpoint at a time. But since the screen is hooked up through blob events there isn't really a reason why you couldn't use multi-touch as well.

Thursday, Jan 31 2008. 2.30 pm: Redefining Input in X.

Be there!

For the first time, I did get to play with a surface table this week, including listening to a keynote by Robert Levy (Surface SDK Program Manager).

BTW: People, stop claiming that MPX is a Surface killer or the other way round! There is overlap, but they have very different goals (see the end of this post).

The apps they had running were some very simple demo apps including two "debugging" apps. The trackedimage from my hand was good, and it was an uncalibrated setup. A few times applications didn't react to my touch though, although this problem didn't occur in the debugging app. Maybe I just didn't click in the right place.

Funnily enough, with one difference the data they extract from the blobs seems to be the same as the XBlobEvents. So I guess I didn't do too bad then.

The apps were - well, just demos. One was creating waves as you touch a virtual water surface, another one was a basic puzzle, etc. Good enough to show off interaction with a touchscreen but not overly exciting.

Robert Levy's talk was more interesting. He started with explaining the technology then went on to demonstrating the SDK. Writing applications is easy, with lots tools to make it easy to build GUIs, define interactions, animations etc. He started with a typical "hello world", then went on to modify it here and there to make it a photo browsing tool.
They have a bunch of widgets that are "surface-aware", so just using them in your application will give you all the features. Writing apps seems really simple, but I can't speak for complex apps. I asked Robert about this, he said they're working with the companies that build apps to get any problems out of the way.

None of the widgets (right now) have true touch support, they are all using the fingers as a fancy way to get x/y coordinates. This is for the surface-aware widgets, if you want to, you can hook into the blob events and do everything manually. No hovering info right now either, something which is in the pipe for MPX BlobEvents.
But they did a good job of packaging everything into a consistent infrastructure.

The main difference between surface and MPX in technical terms:
Surface doesn't run standard apps. All apps are full-screen and only do multi-touch, but not mouse + keyboard.
MPX on the other hand is all about multiple devices in any application. And once you have mice and keyboards sorted out, touch-support comes more or less for free.

Also Surface comes as a complete package including hardware, system and toolkits. MPX covers a single layer in the stack: the windowing system.
Once GTK, Qt and all the other toolkits start adopting MPX, we'll get closer to a complete package. But either way, it is different and it will remain different.

Do I like it? Yes, but I want to see some apps that go beyond novelty factor.

Apologies to Robert if I misunderstood anything.

I just pushed a set of changes to fix up DGA. Haven't tested anything but xmoto yet but in theory games using SDL should work now (one pointer only though).
Also included a change to switch the core keymap, so keyboard layouts work properly again.

One more change to come (incorrect focus events being delivered in some cases), then we should be ready for a mpx 0.5 and a new set of packages for people to try.

After a fairly tough week of coding, I could finally put some decent results up. Just pushed a number of changes, the last one the most significant one.

MPX faced the problem that the DIX would create a core event and an XI event for each device state change. These events were then processed completely disjunct, with the server having no knowledge that they belong together.
This resulted in the device state sometimes being set twice, etc. Major source of grief, also one of the reasons why XKB was practically broken in MPX. (the other reason being XKB practically broken anyway, but let's not get into that).

So, with the last commit, core events are not generated directly anymore, they are emulated while processing XI events. And suddenly a lot of problems are gone.

Give this version a try, I tested it with wmaker, gnome and MPWM. Seems to work ok, even CAPS LOCK works now. whoopey.