After TADHack Mini in London, I realised that the quality of the hacks produced at these events was really high and that having a shot at winning a prize at TADHack Global was not going to be an easy task: people would really have “to up their game, be in (mental) shape, know the APIs they’ll be using on the back of their hands, have a WOW factor, and potentially a business case for it.”

So we decided to build a small team (of two) to make the magic happen: Mike Dauphinais to the rescue!

Mike and I are colleagues at Voxbone. He is a Java Developer and all-around tech enthusiast. He was going to help us “up our game”. So did we do it?

1. “Be in (mental) shape”

That was pretty hard given the fact that the weekend of June 12th was Festas Dos Santos in Lisbon (the city’s biggest folklore event of the year).

2. “Know the APIs we’ll be using”

We spent a full two weeks looking at the sponsors and trying to decide what we were going to work on. We finally managed to find an idea on Friday afternoon right before boarding the plane. We downloaded all the Matrix.org documentation and started reading on the plane through specs, starter guides, FAQs, etc.. I’ve never read a manual for that long.

3. “Have a WOW factor”

We brought all the electronic gadgets we owned that were sitting around at home (LittleBits, Arduino, Leap Motion, Raspberry Pi) just so we wouldn’t miss this point!

4. “The potential business case”

So what did we build?

After the “Eureka” moment on the cab to the airport, we decided to build a tool that could translate sign language to instant messages and pass these messages to all participants in a chatroom. Basically, it’s the equivalent of Speech-to-Text for people that, unfortunately, can’t speak. As Randy Resnick puts it: ASL-to-text (we called it SignChat).

How did we build it?

SignChat uses Leap Motion to detect hand gestures (signs) and translate them to text. Then, we used Matrix.org home servers to create the instant messaging environment that receives the translated text and propagates it to the other participants. Finally, we used the Voxbone VoxSMS API to add SMS discussion capabilities to the application, say, for people on the go.

Here’s a snapshot of the structure:

Screen Shot 2015-06-30 at 10.56.34

More info here.

We split the work in two: The Messaging and the Translating.

Mike took care of the messaging part by setting up a Matrix home server and event listeners that could interact with the other devices (Leap Motion and SMS handset)

I took care of the Translating part and built the interface that detects the movements, replicates them on the screen, and sends them to Mike’s Matrix home server.

Here’s the final result!

Though the projects built at TADHack Global were extremely impressive, we ended up winning the Matrix prize which consisted of a TON of Tessel modules (which will add to our collection of electronic gadgets sitting at home waiting for the next TADHack edition).

This was Mike’s first TADHack and my second and it was definitely a success (again!). Big thanks to Alan Quayle for the perfect organisation, Matthew Hodgson and the Matrix.org team for all their support!