How we built interactive heatmaps using Solr and Heatmap.js

One of the things we obsess over at Shutterstock is the customer experience.  We’re always aiming to better understand how customers interact with our site in their day to day work.  One crucial piece of information we wanted to know was which elements of our site customers were engaging with the most.  Although we could get that by running a one-off report, we wanted to be able to dig into that data for different segments of customers based on their language, country, purchase decisions, or a/b test variations they were viewing in various periods of time.

To do this we built an interactive heatmap tool to easily show us where the “hot” and “cold” parts of our pages were — where customers clicked the most, and where they clicked the least.  The tool we built overlaid this heatmap on top of the live site,  so we could see the site the way users saw it, and understand where most of our customer’s clicks took place.  Since customers are viewing our site in many different screen resolutions we wanted the heatmap tool to also account for the dynamic nature of web layouts and show us heatmaps for any size viewport that our site is used in.

Heatmap on Shutterstock.com
Shutterstock’s heatmap tool running on our home page

The main technologies used to build our interactive heatmap tool were our click tracking system, Lil Brother,  Apache Solr, and Heatmap.js.  Lil brother is able to track every click a user makes on our site, along with the x,y coordinates of the cursor, the page element clicked, and some basic info about the customer (country, language, browser, and a/b test variations).

Solr provided the means to filter and aggregate our click data.  The way in which we were using Solr is described in more detail in our post Solr as an Analytics Platform .   In this case, we indexed each click event as a separate document in solr along with all the customer metadata linked to it.

 Our schema.xml file contained the following fields:

<field name="mouse_x_y" type="string" indexed="true" />
<field name="page_url" type="string" indexed="true"/>
<field name="country" type="string" indexed="true"/>
<field name="language" type="string" indexed="true"/>
...

Once we generated our solr index, we needed to build a query to get the data for our heatmap.  To do this we ran a facet query on the mouse_x_y field.  This gave us a histogram of the number of clicks in each position on the page (we rounded off the coordinates to the nearest 10 pixels in order to group clicks into reasonably sized buckets). Once we had the number of clicks per bucket from Solr, we passed that data to heatmap.js which rendered the heatmap in the browser.

In order to run Heatmap.js on all the pages on our production site we launched the app through a bookmarklet that loaded up the javascript, and ran AJAX requests against Solr. The bookmarklet also exposed controls for us to select other parameters like country, language, and a/b variations so that we could drilldown into specific groups of customers. As a bookmarklet, the tool was able to detect settings like the browser viewport size and display the heatmap based on those dimensions.

Since we developed the heatmap tool, our designers and product specialists have been using it to identify elements of our site that could be optimized – either by changing or removing some elements – to better serve customers’ needs.  Knowing that nearly all of our customers interact with the search bar helped to steer our design to make it the most prominent element on the page above the fold.  Knowing that many of the links lower down on the page were not used as often helped us make the decision to redesign that area and put more valuable discovery paths there for customers.

In order to help out folks who are interested in building out an interactive heatmap tool for their sites, we’ve open sourced the Shutterstock Heatmap Toolkit.  The toolkit allows you to run a solr instance and web server, and includes a batch of sample data to try it out on.

You can run the tool on your own data too by creating a JSON file with individual click events — where each event includes the mouse x/y coordinates, and any other attributes such as the page element clicked, and information about the user (the toolkit itself contains a sample set of data that you can base it on).  A script is also included to start solr, build an index, and run the web server that powers the heatmap app itself.  Follow the steps in the README on Github to try it out on the example data.

Being able to visualize and dig into our customers’ interactions with our site has provided valuable insight for our designers, product specialists, and developers.  Having the ability to navigate and dig into different slices of this data in realtime is highly valuable if you want your product team to be nimble and have answers to questions as quickly as they can ask them.