I am aware that this is an old post now, and so my input may be irrelevant. However, I notice that in the 3D acceleration portion you mention cross compiling. If you have a Debian system running on your device – why not install GCC on the device and compile the driver on the device itself? To my mind this would save a few complications. Of course, it will take longer to compile – but I doubt this would be a huge problem.

I hope my input helps,
Dom

For keyboard, one idea for the keyboard might be to display a virtual
keyboard full screen on top of all the other windows, but make the keyboard
transparent (50% or something). That way there is enough space for typing, and one still can see what letters appear in the focused application.

I am trying a setup using an chroot image and vnc. But unfortunately, vnc doesnt seem to support transparency…… (any ideas?)