Concept
Real Fast Draw began with a simple impetus: to add interactivity to our lectures and presentations. Interactivity is central to everything we do at Second Story, so it felt strange and almost hypocritical to stand before a group of people and simply orate. In order to better align medium with message, we decided to make an interactive deck of slides that audience members could play with while a speaker presented.
Constraints
Real Fast Draw had two eponymous constraints:
- It had to be built Real Fast. We had very little time and resources with which to build it: two people, one week.
- It had to work Real Fast. Namely, users needed to be able to interact with the canvas in real time, over a regular 3G network.
Constraints, of course, can be motivators as much as they can be impediments. In this case, they led us to use two relatively new "HTML5" web technologies: WebSockets and WebGL.
Drawing on Earlier Drawing Applications
To get real faster we decided to use a previous Second Story project as a point of reference. The project that best fit our goals was Infinite Creativity, a multiuser drawing experience. Infinite Creativity allowed users to draw shapes on their mobile devices or multi-touch screens and then send them to a shared canvas. Beautifully designed, it was exemplary of Second Story's work. Its multi-user model was a perfect fit for the sort of "interactive presentation" we aspired to:
WebSockets: node.js and socket.io
Unlike Infinite Creativity, Real Fast Draw needed to be in Real Time: that is, rather than have users compose on their devices and subsequently send their compositions to the canvas, we would treat devices like mouse pads. As users moved their fingers on the screen, they would see immediate feedback up on the presentation screen.
To this end, communicating via standard HTTP would not suffice. Nor did we have a backend developer to create the necessary services for this to happen. These two constraints made it quite clear that the WebSockets were the way to go.
Our first step was to set up a server with node.js. Node.js makes it incredibly easy to build multi-user applications like Real Fast Draw. It is featherweight, simple to install, and a breeze to use.
Next we installed socket.io, a framework for making WebSocket connections between browser clients and our node server. Socket.io even supports browsers that can't run HTML5 or WebSockets, by falling back on Flash and other technologies. The logic for our whole server is as simple as the following:
io.sockets.on('connection', function (socket) {
socket.broadcast.emit('ca', { id: String(socket.id) });
// TouchDown socket.on('d', function (data) { data.id = socket.id; socket.broadcast.emit('cd', data); }); // TouchStill (touch is still down, but not dragging yet) socket.on('s', function () { socket.broadcast.emit('cs', { id: socket.id }); }); // TouchMove socket.on('m', function (data) { data.id = socket.id; socket.broadcast.emit('cm', data); }); socket.on('disconnect', function () { socket.broadcast.emit('cr', { id: String(socket.id) }); }); });
Using node.js and socket.io, we were able to get our canvas and clients talking in real time in just a couple hours, with no backend development effort. The application was working Real Fast, Real Fast.
An early prototype looked like this:
The Canvas, or not
At this point we were ready to move on to the presentation screen itself. We landed on a layout with the "canvas" occupying the top half of the screen, and our presentation slides at the bottom. Everything would be done within a browser, using Javascript. The slides would be unique pages loaded into an iFrame, while the canvas would be, of course, an HTML5 canvas.
In order for users to be able to tell themselves apart, each was assigned a unique shape and color upon connecting to the canvas. This shape would serve as a cursor on their device, and correspond to the shape being drawn on the canvas. These shapes and colors were taken wholesale from Infinite Creativity.
This worked great in early iterations: we had recreated the basics of the Infinite Creativity experience within a web browser, in real time.
As we began to test with multiple users, however, it soon became clear that the canvas would not be able to draw the hundreds (or even thousands) of shapes that users would be making every second. One swipe of a finger could create upwards of a hundred unique points. Multiply that by thirty or forty audience members and our presentation would be brought to its knees.
Real fast, we decided to switch to WebGL.
WebGL Sprites
Reimagining the canvas in WebGL was much less arduous of a process than it could have been, thanks in no small part to the three.js Javascript library. In fact, WebGL provided a perfect excuse to leave the two-dimensional canvas of Infinite Creativity and pursue a more 3D approach to drawing. In developing a reactive media wall for Coca-Cola last year, we became captivated with the way that a seemingly two-dimensional surface could suddenly be rotated to reveal the parallax happening between layers. What would Infinite Creativity look like with similar parallax between each of the users' shape trails?
The first step was to create a camera that would stay focused on the center of the drawing area while constantly changing its field of view.
Next, we took what had been simple shapes drawn into a canvas and made them GL sprites. The boost in performance was immediately palpable. We were now moving and rotating thousands of sprites at once while maintaining a reasonable frame rate. The look and feel of the application was much more exciting than it had been in 2D, as well:
WebGL Particles
At this point we were very close to what we wanted, although five or more users would begin to bring the frame rate down considerably. We needed to make one last (real fast) change.
We have been using OpenGL particles a lot lately, in experiences like The Bubble-izer and Protecting the Secret. While sprites allowed us to animate thousands of objects, particles pushed that number into the hundreds of thousands.
The trick with particles, as opposed to Sprites, is that you can't be constantly creating and destroying them. Your particle system must be a VBO of a fixed length, which gets pushed to the graphics card en masse. To deal with this, we simply create a VBO of 3000 particles for every new user, and make them all invisible. Whenever the user draws on their screen, we simply take invisible particles, move them into position, and make them visible.
Using particles, we were able to add as many users that we want without slowing down that canvas whatsoever. At last, we were drawing Real Fast:
