I made a little digital toy to play with hand movement and computer graphics:

It's called Magic Hands, and you can have a play with it at magichands.forestinthetree.com.

Computers as Theatre

Magic Hands is a remix of open source fluid simulation (beautifully crafted by Pavel Dobryakov), and MediaPipe's Machine Learning gesture and hand recognition.

I was introduced to the fluid simulation from the Sandspiel making of blog post¹ (a recommendation from TodePond), and was completely mesmerised by the magical feel of it. I'd also played with pose recognition using Tensorflow and ages ago made a communal drawing, art piece using custom physical hardware (based on wiimote tracking), and it all came together with this little toy.

When I first connected the hand tracking to the fluid simulation, and saw the direct control my hands had over the graphics, it genuinely made me feel giddy with joy 😄.

I'm a fan of the idea of Computers as Theatre, and I think theatre is quite an accurate metaphor of what computers are and what they can be. It captures the interconnected relationship between the creator, the audience and the artefacts created, where the computer program is a medium to communicate ideas and create experiences. By connecting visuals to hand movement, it felt like a form of puppetry, and I'd like to explore what else can be done with this medium.

Sparking joy

This little hack, tapped into my sense of wonder I had as a kid, and it's the kind of thing I'd like to do more of. I'm very much inspired by works of pop art like Jeff Koon's Puppy, where anyone coming to see it, gets an immediate sense of joy without much to explain. It's just fun!

Being a Fullstack Web Developer by trade, I also really enjoyed pushing the limits of what the Web Platform can do nowadays. WebGL + Machine Learning gesture recognition were definitely not available when I started my career, and there's so much there to play with.

A Brief Rant

Another influence of Magic Hands is the hands part of Bret Victor's Brief Rant on the Future of Interaction Design written in 2011. Bret starts with a video about the future of interaction design, created by a big tech company, where pretty much all the futuristic interactions are screen based swiping gestures. The rant goes on to question why, in the future, we wouldn't use the full spectrum of capabilities of what the hand can do.

At this present day, to make things on the web, the main way to interact with the computer is still keyboard and mouse - we haven't even gotten to screen based gestures replacing the day to day making task yet, let alone the hand gestures above. What if you could directly manipulate what was on the screen with your hands? What if it were more tactile? What if you could pluck, sculpt, squeeze the web? I highly recommend reading Bret Victor's article for more of a call to action on why you should explore interaction design outside of screens, looking at the full capabilities of the human body.

Maybe Augmented reality (AR), Virtual Reality (VR), or Mixed Reality (MR) is going to create these opportunities and new forms of interaction, but a lot of these solutions require you to buy new hardware and it is going to be a while before they become readily available to the masses. Magic Hands is a form of Augmented Reality and works on open web technologies, where all you need is a web browser and an attached camera. You don't even need a web server, so it continues working without the internet, after your initial download of the webpage. Sure, it's only very basic and doesn't quite run fast enough on all hardware, but the web is a really nice low barrier to entry platform to explore the ideas of what more expressive controls can do for human computer interaction.

If you want to play around with what hand tracking in the browser can do right now, you can look at the MediaPipe JavaScript library (demo), which Magic Hands uses, or JavaScript libraries that make hand tracking easier to use, including:

• Handtrack.js (Tensorflow)
• Handsfree.js (Tensorflow/MediaPipe)
• YoHa (Tensorflow)
• Manitas (MediaPipe)
• Maybe I'll create one too some day 🤷🏼‍♂️

Have a play 🙌🏼

Magic Hands is very much a starting point, as a creative outlet to explore the ideas of more intuitive and humane ways of controlling the computer. There are still more questions to be answered in terms of user experience and what else it could be, and I'll be keen to find the theatrical spark amongst the hand tracking and computer graphics.

Have a play, and let me know what you think on my discussion page, or feel free to say hello.

If you are curious about the code, you can check it out at forestinthetree/magic-hands (although, ⚠️ it's all very much work in progress).

Links

• 🙌🏼 Play with Magic Hands
• 💬 Discussion
• 🛠 See the code

¹ You can play with the brilliant sand game at sandspiel.club, or the new studio.sandspiel.club #