Stormin' The Castle

You want the users of your web app to be able to use their webcam to take their own profile shots and you’d like them to be able to personalize or stylize the picture before uploading it your weberver. Or maybe you’ve seen one of those fun photobooth-type apps that takes photos, applies cool image effects to them and lets you share them… wondering how they worked.

Imagine if your HTML5 app could programmatically access your webcam, take snapshots and modify the image data procedurally. You could then let your user’s upload or share their self portraits.

In this article, I’m going to teach you how to do that and more. Building on some articles that I’ve shared in the past, I’ll walk you through a sample application that displays the video from your webcam, applies image processing effects to that video, lets you take still snapshots of the filtered video and lets you serialize the snapshots so that they can be uploaded to a web server.

In one of my earlier articles, “Lights, Camera, Action!” I showed you the minimal amount of code needed to access you webcam and display the video into an HTML5 video object.  In this sample, we use the same code which lets us display the original video from the webcam.  But in order to do something more interesting with the video, we need to get the image data into a canvas object so that our Javascript code can manipulate it.  In fact in the sample app, we actually hide the HTML5 video object itself by setting it’s display style to none and the filtered video is actually rendered by a canvas object.  As it turns out a HTML5 video object can be used in place of a source image in a drawImage call to a 2d canvas context. Unfortunately the HTML5 video object doesn’t provide a frame accurate way of accessing the image data for a given frame. So in order to acquire the frame data we’ll leverage the requestAnimationFrame method to execute our callback on a recurring basis so that it can repeatedly acquire video frames. The following code is used to alias the vendor prefixed variants of requestAnimationFrame in order to make the code more portable across a range of browsers.  Credit: This requestAnimation polyfill was written by Paul Irish.

// Alias the vendor prefixed variants of requestAnimationFrame so that 
// we can access them via window.requestAnimationFrame fallback to 
// setTimeout at 60hz if not supported.
window.requestAnimationFrame = (function(){
  return  window.requestAnimationFrame       || 
          window.webkitRequestAnimationFrame || 
          window.mozRequestAnimationFrame    || 
          window.oRequestAnimationFrame      || 
          window.msRequestAnimationFrame     || 
          function( callback ){
            window.setTimeout(callback, 1000 / 60);
          };
})();

When requestAnimationFrame is invoked it will invoke the provided callback in approximately 1/60th of a second.  We could have also just used the setTimeout method.  But there are several advantages to using requestAnimationFrame over setTimeout.  The primary advantage is that if the browser tab is not currently visible or if our browser is minimized then requestAnimationFrame (unlike setTimeout) will not invoke the callback, thereby saving cpu cycles when our page is not visible.  Our requestAnimationFrame callback function is called update and is shown below:

function update(){  
  //...
  processVideoFrame();
  requestAnimationFrame(update); 
};

In the update function, processVideoFrame is called to acquire a single frame of video from the video object and process it.  A call to requestAnimationFrame is then made to queue our update function to be called again approximately 1/60th of a second later (Assuming that rate can be maintained by the CPU).  This function serves as the “pump” that will continually capture video frames and process them with any selected image filters.

function processVideoFrame() {
  // We have to check for the video dimensions here.
  // Dimensions will be zero until they can be determined from
  // the stream.
  if (context && video.videoWidth > 0 && video.videoHeight > 0) {
    // Resize the canvas to match the current video dimensions
    if (canvas.width != video.videoWidth) 
      canvas.width = video.videoWidth;
    if (canvas.height != video.videoHeight) 
      canvas.height = video.videoHeight;

    // Copy the current video frame by drawing it onto the 
    // canvas's context
    context.drawImage(video, 0, 0, canvas.width, canvas.height);
    processImage(canvas);
  }
}

In the processVideoFrame function, the canvas object is sized to match the dimensions of the video object. A call to the drawImage method will copy the pixel data from the current frame within the video object onto the canvas’s context.  A subsequent call to processImage is made to apply an image filter if one is currently selected.

function processImage() {
  if (canvas.width > 0 && canvas.height > 0) {
    if (imageFilter) {      
      context.putImageData(imageFilter.apply(null, [context.getImageData(0, 0, 
        canvas.width, canvas.height)]), 0, 0);
    }
  }
}

In another one of my articles, “How You Can Do Cool Image Effects Using HTML5 Canvas”, I demonstrated how to apply image filters to the pixel data contained within a canvas object. We’ll be doing the same thing here on each frame of captured video.  In the sample app, the following image filters can be applied to the video: grayscale, sepia, red, brighter, darker.  Please refer to my earlier article for more detail on image filters.

The next thing I want to cover is how you can take a snapshot of the image currently in our canvas, so that it can be uploaded or shared. The following function is used to take a snapshot.

function takeSnapshot() {
  var url = canvas.toDataURL();
  console.log(url);
  // Set the src of the image url to the data url
  document.querySelector("#snapshot").src = url;
  // Display the DOM elements that contain the snapshot
  document.querySelector("#snapshotdiv").style.display="inline-block";
}

The real work is done by the canvas’s toDataURL method.  This method takes the image data currently contained within the canvas, copies it, compresses it and returns a data URL of this image. So just what is a data URL?  A data URL (or more correctly… a data URI) is a way to represent a serialized chunk of data as a “stringified” URL.  An example is shown here:

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==

This example data URL encodes a PNG image of a red dot. To try this out for yourself, you can take this example and paste it into the URL bar of your browser and you’ll see a small red dot rendered. If you’d like to learn more about data urls, check out the wikipedia article.  Since this URL is a base64 encoded version of the image file itself. This string can be uploaded to your webserver and saved. It will just take a bit of string manipulation and a base64 decoder in your backend code to convert it back to a binary image file. Turning our attention back to the takeSnapshot function, once we’ve obtained the data url for our image it can be assigned the src property of an image object to display the snapshot.

Get My Free Bonus Article on Color Matrix Filters!  I’ll send you a link to a bonus article for my data-driven Color Matrix Filter and sample application when you join my mailing list.  A Color Matrix Filter can do almost an infinite number of different color filter effects.  You won’t want to miss adding this to your toolchest!  Just drop your email in the form below:

That’s all for this go ‘round. Hope you’ve learned a few things and enjoyed the article. You can try out the sample app here and get the full source code here.