Happy Tax Day! Now, where did your tax dollars go?

Like it or not, Americans have a date with the IRS today. In a few hours, our tax forms will have been sent in… yet most of us will still be left wondering, “How will the government spend our money?”

In February, we launched the Data Viz Challenge to answer that very question. The challenge was a five-week competition that asked developers to visualize how our federal income tax dollars are spent. We received more than 40 thought-provoking entries that each take a unique approach to making this data more accessible to taxpayers.

Take a look at the variety of visualizations in this short video:

The entries ranged from quirky and fun to serious and academic. Some were as simple as a receipt, others showed us how many minutes or days we work to cover public spending. One even rendered the data as a cityscape and invited people to climb the towers virtually.

In the end, our jury selected Anil Kandangath’s “Where Did My Tax Dollars Go?” as our Grand Prize winner. As the jury noted, his entry is information-rich but elegantly designed, and at no point while interacting with the visualization do you lose the “big picture.” Anil’s entry is a great example of how data visualization can take boring, complicated, but critically important information and make it accessible to everyone. Congratulations, Anil.

Thank you to all of the entrants and our partners Eyebeam and Whatwepayfor.com for making this possible. We hope these submissions help you better understand where your tax dollars are spent—and even inject a little bit of fun into your tax day this year. Happy filing!

Bing Maps: Heat Mapping Crime Data and HTML5 Canvas

Crime data is frequently presented using “heat maps”. See, for example:

imagehttp://www.bbc.co.uk/truthaboutcrime/crimemap/ image http://www.zubedpi.com/

One of the reasons why heat maps are used to visualise crime data is that crimes are typically recorded as individual incidents, occurring at one particular location, yet we tend to want to think of areas at high- or low- risk of crime. We therefore need some way of aggregating those individual crime occurrences into regions that can be analysed.

It doesn’t necessarily make sense to create a thematic map of crimes based on geographic or administrative divisions – a burglar probably does not have much regard for whether he commits a crime in one polling district/ward/street of a city or another, for example, and we should not attempt to fit the distribution of crimes to such arbitrary regions. A heatmap creates a way of summarising and presenting a set of point data with the only influencing factor being the underlying geographic distribution of the points – a greater number of points in the same proximity leads to a “hotter” region.

Seeing as the government has just launched the http://www.police.uk/data portal providing access to a dataset of all reported crime in England and Wales (at a ridiculous cost of £300,000 for 40 basic CSV files, updated monthly), I thought that this would be a good opportunity to update my old heatmapping code to use Bing Maps AJAX v7. What’s more, seeing as Internet Explorer 9 now supports the element, I thought I’d try creating heat maps dynamically in the client-side browser rather than rendering them as images on the server-side, as I typically have done in the past.

To start with, I created an array of locations from which my heatmap would be created. I used all of the “anti-social behaviour” crimes reported by Norfolk constabulary during December 2010 (3,185 records). Having converted each record in the CSV file to a Microsoft.Maps.Location, my array looked like this:

[php]var locations = [
new Microsoft.Maps.Location(52.672802972383, 0.943923796417529),
new Microsoft.Maps.Location(52.624236498992, 1.29493956651563),
new Microsoft.Maps.Location(52.6218783667792, 1.29080178760268),


I then created a element in my HTML page and placed it so that it exactly covered the

element in which my Bing map would be created:
[php]var canvas = document.createElement(‘canvas’);
canvas.id = ‘heatmapcanvas’
canvas.style.position = ‘absolute’;
canvas.height = 800;
canvas.width = 1024;
canvas.style.zIndex = 1;

Note that, as Bing Maps v7 doesn’t have an AddCustomLayer() method as in the previous v6.x versions, I’m having to manually add the canvas element into the correct place in the DOM using appendChild(). This method is not perfect as it places the canvas on top of the map controls as well as the map tiles – I’m going to be looking at this issue later.

The next step is to loop through each of these locations and add a radial gradient at the appropriate pixel location on the canvas (which, because the canvas is placed exactly over the map, can be determined by the tryLocationToPixel() method). The appearance of each heat point produced is affected by two parameters – radius determines the extent to which heat spreads out around a point, and intensity determines how hot it is at the centre of each point.

[php]// Create the Intensity Map
for (var i = 0; i < locations.length; i++) {
var loc = locations[i];

// Convert lat/long to pixel location
var pixloc = map.tryLocationToPixel(loc, Microsoft.Maps.PixelReference.control);
var x = pixloc.x;
var y = pixloc.y;

// Create radial gradient centred on this point
var grd = ctx.createRadialGradient(x, y, 0, x, y, radius);
grd.addColorStop(0.0, ‘rgba(0, 0, 0, ‘ + intensity + ‘)’);
grd.addColorStop(1.0, ‘transparent’);

// Draw the heatpoint onto the canvas
ctx.fillStyle = grd;
ctx.fillRect(x – radius, y – radius, x + radius, y + radius);

To start with, I used values of radius 20 and intensity 0.25. This meant that the centre of any single point would be plotted as black colour with 0.25 transparency. The visibility of each point would gradually diminish to be fully transparent 20 (pixels) away from the centre. However, the heat of any points closer than 20 (pixels) to each other would create an additive effect, and produce a heat intensity map like this:


Having created a single-channel heat intensity map as above, you can then colourise it by creating a linear gradient, and mapping the alpha values of each pixel on the canvas to the appropriate value from the gradient.

The following code demonstrates a linear colour gradient that progresses from cool colours to hot colours, as typically used in a heat map:

[php]function createColourGradient() {
var ctx = document.createElement(‘canvas’).getContext(‘2d’);
var grd = ctx.createLinearGradient(0, 0, 256, 0);
grd.addColorStop(0.0, ‘magenta’);
grd.addColorStop(0.25, ‘blue’);
grd.addColorStop(0.5, ‘green’);
grd.addColorStop(0.75, ‘yellow’);
grd.addColorStop(1, ‘red’);
ctx.fillStyle = grd;
ctx.fillRect(0, 0, 256, 1);
return ctx.getImageData(0, 0, 256, 1).data;

And this colour gradient can be used to colour the intensity map as follows:

[php]      var dat = ctx.getImageData(0, 0, canvas.width, canvas.height);
var pix = dat.data;
var ColourGradient = createColourGradient();
for (var p = 0; p < pix.length;) {
var a = pix[p + 3]*4; // get the alpha value of this pixel
pix[p] = ColourGradient[a]; // get the red value from linear gradient that corresponds to this alpha
pix[p + 1] = ColourGradient[a + 1]; //get the green value based on alpha
pix[p + 2] = ColourGradient[a + 2]; //get the blue value based on alpha
p+=4; // Move on to the next pixel
ctx.putImageData(dat, 0, 0);

// Populate the canvas
ctx.fillRect(0, 0, 1024, 800);[/php]

The preceding code might need a bit of explanation…

  • Each pixel in the array returned by getImageData() is represented by 4 bytes of data, those bytes representing the Red, Green, Blue, and Alpha values of that pixel.
  • So, in the for loop above, the line pix[p+3]*4 is used to return the alpha value only of each pixel in the pix array. (Remember that the canvas was created by placing radialgradients around each point, which varied from an alpha of intensity at the centre to alpha of 0 at a distance of radius from the centre).
  • I then look up the pixel that lies at the equivalent position in the ColourGradient spectrum, and copy the first 3 bytes from it into the pixel array, replacing the purely black image with the Red, Green, and Blue values.

The result of colourising the map using the linear gradient defined above is as follows:


Of course, HTML5 makes it very easy to colour the heat map using any colour gradient you want. Changing the ColorStops in the linear gradient as follows, for example:

[php]grd.addColorStop(0.0, ‘black’);
grd.addColorStop(0.5, ‘red’);
grd.addColorStop(0.9, ‘pink’);
grd.addColorStop(1, ‘white’);[/php]

leads to the following:


There’s still a few issues I’m having with my code – namely trying to get the element inserted at the right place in the DOM to be above the map tiles but underneath the navbar (it seems that all elements in the AJAX v7.0 control are displayed at z-index:0, so I can’t use this as a method top position it). Also, whilst the code above works great in IE9 and Google Chrome, Firefox often gives me the “slow script” warning… I’m not sure what element it’s unhappy with, or whether it’s just that Firefox is not as HTML5-compliant as it claims to be… (ahh the joys of creating cross-browser web apps!)

If I get these issues ironed out, and if anybody shows any interest, then I might package up the complete heatmap code into a reusable class and pop it up on codeplex.

Data Mash-Ups and the Future of Mapping: JISC Report

Over the past few months we have been working with colleagues here at CASA, University College London and at the University of Nottingham, in association with the Joint Information Systems Committee (JISC) to write a report on Data mash-ups and the future of mapping. We are pleased to say the report has just been released and is available to download.

Report by Suchith Anand, Michael Batty, Andrew Crooks, Andrew Hudson-Smith, Mike Jackson, Richard Milton, Jeremy Morley

Data Mash-Ups and the Future of Mapping
Executive Summary
The term ‘mash-up’ refers to websites that weave data from different sources into new Web services. The key to a successful Web service is to gather and use large datasets and harness the scale of the Internet through what is known as network effects. This means that data sources are just as important as the software that ‘mashes’ them, and one of the most profound pieces of data that a user has at any one time is his or her location. In the past this was a somewhat fuzzy concept, perhaps as vague as a verbal reference to being in a particular shop or café or an actual street address. Recent events, however, have changed this. In the 1990s, President Bill Clinton’s policy decision to open up military GPS satellite technology for ‘dual-use’ (military and civilian) resulted in a whole new generation of location-aware devices.Around the same time, cartography and GIScience were also undergoing dramatic, Internet-induced changes.
Traditional, resource intensive processes and established organizations, in both the public and private sectors, were being challenged by new, lightweight methods. The upshot has been that map making, geospatial analysis and related activities are undergoing a process of profound change. New players have entered established markets and disrupted routes to knowledge and, as we have already seen with Web 2.0, newly empowered amateurs are part of these processes. Volunteers are quite literally grabbing a GPS unit and hitting the streets of their local town to help create crowdsourced datasets that are uploaded to both open source and proprietary databases.
The upshot is an evolving landscape which Tim O’Reilly, proponent of Web 2.0 and always ready with a handy moniker, has labelled Where 2.0. Others prefer the GeoWeb, Spatial Data Infrastructure, Location Infrastructure, or perhaps just location based services. Whatever one might call it, there are a number of reasons why its development should be of interest to those in higher and further education. Firstly, since a person’s location is such a profound unit of information and of such value to, for example, the process of targeting advertising, there has been considerable investment in Web 2.0-style services that make use of it. Understanding these developments may provide useful insights for how other forms of data might be used. Secondly, education, particularly research, is beginning to realize the huge potential of the data mash-up concept. As Government, too, begins to get involved, it is likely that education will be expected to take advantage of, and indeed come to relish, the new opportunities for working with data.
This TechWatch report describes the context for the changes that are taking place and explains why the education community needs to understand the issues around how to open up data, how to create mash-ups that do not compromise accuracy and quality and how to deal with issues such as privacy and working with commercial and non-profit third parties. It also shows how data mash-ups in education and research are part of an emerging, richer information environment with greater integration of mobile applications, sensor platforms, e-science, mixed reality, and semantic, machine-computable data and speculates on how this is likely to develop in the future.
There are two versions for download: the first is an optimised version (900Kb) and the second is the one with full resolution graphics (14Mb)