Vote 2012 Map Center: The innards

(Published Nov. 12, 2012)

For most of the time I've been working at the PBS NewsHour, I've been working on our Vote 2012 Map Center, which is a collection of interactive maps showing demographic data and election results of interest to people following the 2012 presidential election.

Last week, that election finally happened, so now that all of that's out of the way, here's a (probably overly detailed) overview of a lot of the technologies and techniques that ended up being used in this project. The Map Center itself is here, and most of the client-side code (and the server-side code for the second-screen part) is available on GitHub.

Client-side overview

The client-side code in the Map Center represents my first JavaScript project of any appreciable size, so it ended up being pretty weird in an architectural sense. I started off depending more on jQuery since I was relatively new to JavaScript and then used less of it as I worked on other parts of the project.

Different parts of the Map Center, including different data sets, appear on different HTML pages; this allowed for more straightforward analysis of user habits using our existing analytics tools and practices and provided for simpler customization of layouts and other runtime variations between pages and environments.

The code is split into several modules which individually can be included on a given map page as needed; because I wasn't familiar with manipulating the global (window) object at the time, though, I ended up using Mark Ziesemer's namespace function to reassure all modules that their dependencies did in fact exist. The core functionality lives in one global object, which is pretty nice, and the module-specific code for the most part lives in closures (event handlers), so there's not much pollution of the global namespace. Some configuration options do end up using global variables, but I did make sure to give all of those a Map Center-specific prefix so they'd be unlikely to collide with anything else.

Once the document has loaded and its graphics canvas is ready for drawing, a basic publish/subscribe system (using jQuery custom events on the document object) takes over to control the rest of the Map Center code's execution.

Graphics architecture

We decided to go with maps that were based on normal SVG images, which was great in the sense that it didn't care about what projections we used or whether we had Alaska and Hawaii in any particular place relative to the continental United States. Of course, since those didn't have any sense of geography left about them, they ended up being difficult to work with when it came to things such as adding cities or people's requests for showing candidate travel and other point-specific (as opposed to area-specific) information. (I've since seen the Washington Post do some interesting stuff with reprojecting boundary data on the client and rendering it in Leaflet, which might help with providing the best of both worlds — but I didn't know about that when I was planning all of this out.)

Originally, I was rendering these with Raphaël, but I ended up switching to Dojo for two reasons. The major reason was performance: When drawing many paths at once (for example, all of the more than 3,100 counties and county-equivalents in the country in one national map), Dojo ended up being considerably faster in our iPad rendering tests. The other reason was Dojo's fallback to rendering graphics in a <canvas> element when both SVG and VML are unavailable — a situation that mainly applied to Android 2.x devices.

The performance advantage lessened somewhat when iOS 5 came out and might be even less since Raphaël 2.0's release, and Android's stock browser has supported SVG since Honeycomb, so if I were starting this project from scratch right now, I'd probably go with Raphaël instead; Dojo's a bit too heavy for my tastes, and the event handling is just weird — understandably so, since there are so many fallbacks for the graphics objects, but still weird.

Client-side wish list

As I said earlier, this project was in many ways my real introduction to JavaScript as a language, so I learned more as I went along about many best practices within the language. If I were to do it again, I'd like to think I'd make at least some of these changes:

Make the map view system itself more general to allow for simpler addition of other kinds of views. As it exists now, the Map Center only knows how to handle a national map of states and a national map of counties as special cases; anything else is assumed to be a statewide map of counties. This made it difficult, for example, to add maps of congressional districts — enough so that we decided against it by the time the Census Bureau finally released some interim shapefiles for those district boundaries.
Use a more rigorous dependency management system such as RequireJS or some other AMD-compliant loader.
Implement some sort of build process for client-side code to automatically concatenate and minify the code and therefore reduce network overhead. In Mike Pennisi's work on the second-screen system, he introduced a grunt-based build process for server-side code and all of the client-side code he wrote, but I didn't get around to using this for the previously existing Map Center functionality.
Make more use of actual templating systems such as Mustache and more use of storing data within normal variable instead of within DOM elements; both of these would reduce the ridiculous amount of DOM manipulation I ended up doing in some places.

Embedded maps

We also got several requests from PBS member stations and other organizations to make it easier to embed parts of the Map Center. To help them do so, we created a page (using PHP for templating and therefore containing some of the ugliest code I've ever written) with a responsive design that allowed the embedded map to fit reasonably well in whatever width was available. (Calculating the necessary height for the resulting <iframe> element was an interesting exercise in trial and error.)

I don't have much in the way of good advice in putting together a responsive design, though, other than to use Respond.js if you intend to support IE 8 and earlier. A lot of the rest was just trial and error in terms of seeing what design changes would work at different page sizes.

Data processing (census, etc.)

The majority of the data in the Map Center came from the U.S. Census Bureau. That organization's American FactFinder service isn't the simplest thing around to navigate, though, and it mostly provides data in CSV format, which isn't all that easy to handle in JavaScript. To get them into a more useful form, I put together a quick Python script to take a column of one of those CSVs and turn that into JSON.

Some other interesting data sources I ended up using for other maps:

For the popular vote from the 2008 general election, the National Atlas obtained state-level data from the Federal Election Commission, and the U.S. Geological Survey got county-level data from each state's election authority. (This was all kinds of fun originally (before I added the FEC stuff at the state level) because of the number of people who insisted that my crediting USGS for election data must be a mistake.) It turns out Alaska reports election results by state House district instead of by county-equivalent (borough or census area, really, since it's Alaska), which also was a fun special case to work with.
The monthly unemployment statistics played a pretty big role in this election. Laziness is a virtue for programmers, so after a few months of really painful search-and-replace on this big text file, I wrote a Python script instead to fetch the latest numbers for me and give them to me in the JSON I wanted.
Religion came up as a topic relatively often, so to find out where evangelical Protestants were more or less common, I used the U.S. Religion Census from the Association of Statisticians of American Religious Bodies.
Our Listen to Me project included videos from people across the country about their opinions on the current political system; Blaine Sheldon helped out with the Map Center's interface to those videos.

I love the fact that I was doing a project like this so soon after a decade ended; between this and the U.S. decennial census, I had all kinds of relatively recent data in great detail that I could use. I'm not sure how I'd approach it in other years when these kinds of products are less current. (Of course, the American Community Survey keeps running, but for county-level data everywhere, you have to use five-year estimates, which isn't exactly ideal.)

AP results

The Associated Press provided our results data for the primaries and caucuses, the Wisconsin gubernatorial recall election and the general election.

That data comes in several semicolon-delimited text files, which we then had to process into XML for our broadcast graphics and JSON for the Map Center. Because we had multiple applications needing to access the same files, we had one script pulling the AP files to an internal HTTP server that both the broadcast and the Map Center update scripts could pull from as needed.

That AP pull script and the Map Center update script (both available here, with credentials redacted) both used Python's excellent logging module with a small logging server I found in order to provide a centralized place for me to keep track of everything's operation during tests and on election night itself.

Traffic problems

On election night, our website (which is hosted as flat files on a PBS Apache server) started having problems dealing with the unusually high amount of traffic.

As I described above, users' browsers originally were requesting updated results files from that same PBS server every 15 seconds. We changed that timing for new users to 45 seconds to try to alleviate some of that load — but obviously existing users' browsers didn't know about that change (and might not have even after a refresh, if their browsers had cached the results page's code), so after some discussion with some PBS Interactive folks, I started mirroring the results JSON files on S3 and changed the code for new users so they'd start looking for it there, and PBS added a redirect so existing users would start requesting the files from S3 as well. (It also ended up breaking our results pages for primaries and caucuses, which I found out the following week.)

We had no reason to believe there would be any traffic problems, especially because this is the same setup we used for all of the primaries and caucuses earlier in the year, but obviously if I were to do this again, I'd use S3 for the results from the beginning. Flat files are useful no matter where they're stored, and I had them set up as JSONP from the beginning anyway just in case such an infrastructure change became necessary — so I might as well have made the change sooner. (Of course, as with any overhaul of a deadline project, it comes down to what we felt we had time for in the weeks leading up to the election, but it would have been nice to have fewer major changes to make on that actual night.)

Second-screen

As part of a grant we received along with the Mozilla Foundation, we partnered with Bocoup for some work on a way to have users' browsers automatically update to whatever maps were being shown on air at the same time as our initial attempt at some sort of more immersive second-screen experience. Mike Pennisi did a lot of great work on that part of the platform (and helped rewrite some of the innards of our Electoral College calculator).

Because we were taping the segments that used the Map Center ahead of when they actually aired (so it would be easier to schedule the interviews with public media reporters who participated in those segments), I ended up creating a quick little control panel from which I could send updates as the segments actually aired. While this was great in terms of helping us to be more flexible with when things were broadcast, it also meant we had to shut down the service early so I could fight the traffic fires I described earlier.

All that said, it still was a rather promising platform, and Mike built it in such a way that we should be able to use it for other second-screen efforts with little (if any) adaptation needed. It should be fun!

Conclusion

Again, this was my first major JavaScript project, so it definitely has its warts; that said, I'm still pretty proud of it, especially considering how much it was able to do by the time the election rolled around. Hopefully this information helps anyone out who's interested in how it worked or who's interested in coming up with something like this in the future!