For the baseline summary I considered using a color-coded map (like CQ, CNN) but I care more about aggregates than individual districts or states. Instead I created five non-map visualizations of the same vote share data, using R and ggplot2. Each visualization helped me better understand my data and refine my assumptions and expectations, even if I eventually discarded the output. The interactive nature of R allowed me to experiment and iterate very quickly until I got what I needed. The R code and data are available at the end of the post.

Figure 1: Democratic Vote Share US House 2002-2008 (scatterplot)

Figure 2: Democratic Vote Share US House 2002-2008 (scatter + jitter)

The scatter plots helped me realize what I really wanted was a summary of the distribution of the democratic vote share, not the raw values themselves. That lead me to the following:

Figure 3: Democratic Vote Share US House 2002-2008 (histogram)

Since a histogram was too raw I decided to switch back to a box-and-whisker plot.

Figure 4: Democratic Vote Share US House 2002-2008 (box and whisker plot)

This lead me to use the established seats-votes plot from theoretic political science literature (Kastellec, Gellman, Chandler (2006), and Jackman, etc (PDF).

Figure 5: Democratic Vote Share USE House 2002-2008 (Seats-Votes plot)

It is no surprise the seats-votes plot proved to be the most useful for my purposes since it was specifically designed, by very smart social and political scientists, to look at this type of data. The seats-votes plot is very versatile and can be adapted to a single election by looking at all precincts within a single district. I performed this type of analysis in Aggregate electoral targeting blog post: Democratic vote share, by precinct in VA HOD 13.

Even though the other plots aren’t as useful, they do provide some diagnostic information. The box and whisker plot is probably easier to read if you only cared about median vote share, and the histogram plot was excellent in finding uncontested seats. For fewer than 435 points even the scatter plots could be very useful. Please email me or comment with ideas or alternative visualizations of vote share data.

R code
CSV file

The traditional source for campaign finance summaries is the Candidate Summary File, which is a single set of summary statistics for a campaign for an entire electoral cycle. But a campaign files a new F3 at least quarterly, and before and after every election they participate in. Each F3 provides insight into where a campaign stands, and with access to this intra-cycle data we can better compare campaigns and perform more sophisticated analysis. The House Data file is built from these F3 reports, all 41,050 reports from 2,241 candidates for the US House since 2002. Campaigns often update previously filed reports with amendments, so the file contains only the latest summary provided by a campaign.

The file is compiled automatically using FECHell into a zipped CSV format. New releases will be made within 3 days of a new batch of electronic filings, according to the FEC 2010 Filing Deadline Schedule (pdf warning).

Here is a simple example of a quarterly summary of total receipts and total disbursements made by all house campaigns in 2008:

The House Data Project is live today, with more examples and a data dictionary. The latest version can always be downloaded from http://offensivepolitics.net/data/housedata-latest.zip.

If you have any questions, comments, or suggestions about the house data file please don’t hesitate to contact me.

Targeting for turnout is a three-step process:

1. A turnout table is created for a previous election similar to the target election;
2. A scoring procedure is implemented with regression, clustering, or some other statistical process and
3. Every voter is scored with a likely turnout percentage.

Depending on his or her turnout percentage – high, middling, or low – a voter will be ignored, targeted for persuasion, or targeted for get-out-the-vote (GOTV) efforts by a campaign. Targeting for turnout, along with almost every other type of political targeting, is explained in detail in Political Targeting by Hal Malchow (2008).

In this post, I recreate parts of the regression analysis from Chapter Nine (Targeting for Turnout) of Political Targeting (Malchow 2008), using the free R Project for Statistical Computing. R is a programming environment that excels at data manipulation and statistical analysis, making it an interesting alternative to traditional statistical tools, like SPSS or web-based voter management software. The analysis will be performed against the full voters list from Ohio’s 1st congressional district, with the intention of predicting turnout for the 2010 congressional midterm elections. This analysis is similar or identical to what a candidate for Ohio’s 1st district would perform throughout the election year. The R code for each step in the analysis will be provided inline so a reader can perform the same operations.

OH-01 Voter File

A voter file is a list containing electoral and demographic data on registered voters, maintained by state boards of elections, political parties, PACs, or private companies. My voter file was downloaded from the Ohio Secretary of State in late 2009 and contains: name, address, age, registration date, voting history, and party affiliation (primary voters only).

To better simulate what a political campaign would use, I’ve appended the following fields:

• Gender: Using birth data from the Social Security Administration, I matched each voter’s first name to a probable gender. About 9% of names were unable to be matched and coded as an empty string.
• Age Group (2010): Using the birth year, I calculated age as of 2010, and then assigned each voter to an age group: 18-21,22-29,30-39, 40-49, 50-59, 60-69, 70-79, 80-89, and 90+.
• Age Group (2006): Using the birth year, I calculated age as of 2006, and then assigned each voter to an age group: 18-21,22-29,30-39, 40-49, 50-59, 60-69, 70-79, 80-89, and 90+.
• Household: I grouped voters into discrete households using the full street address and zip code.
• Marriage status: Using the household variable, I performed a very simple marriage determination: people living in the same household with a difference in age < 15 years were flagged as married.
• Last4 (2006): Measures participation in the last 4 major elections prior to 2006: 2004 Primary and General, and 2002 Primary and General. Range 0-4.
• Last4 (2008): Measures participation in the last 4 major elections prior to 2008: 2006 Primary and General, and 2004 Primary and General. Range 0-4.
• Last4 (2010): Measures participation in the last 4 major elections prior to 2010: 2008 Primary and General, 2006 Primary and General. Range 0-4.

Email me here for the code used to scrub and augment the voter file.

R Setup

I am using the R environment to perform this analysis. To download and install R, go to CRAN homepage and follow the instructions for your platform. Once R us up and running, execute the following to get the required libraries installed and load the voter file into memory:

# install plyr, ggplot2, and RColorBrewer 
# ggplot2 loads plyr as a dependency
install.packages(c("ggplot2","RColorBrewer"))	
# load required libraries (ggplot2 loads plyr as a dependency)
library(ggplot2)
library(RColorBrewer)
# load the voter file into vfs variables
vfs <- read.csv("voterfile.csv")

Now the dependencies are installed and the voter file is read into the vfs variable.

Turnout Table

According to Political Targeting (Malchow 2008), the strongest indicators of participation in a future election are age and previous participation. Malchow also says participation tends to be consistent between similar elections in different years. I am looking at the 2010 General election, a congressional midterm, so I used the 2006 General election as my guide. The first step is to generate a turnout table.

Political campaigns use a tool called “last 4″, which measures a voter’s recent participation. A voter’s last 4 score represents how many of the previous four elections he or she cast a ballot in. A standard is to use both the primary and general elections for the previous two major election years. My data set contains last 4 calculations for the 2010, 2008, and 2006 elections.

The 2006 last 4 calculation looks at elections as far back as the primary in 2002, but a percentage of voters in the list weren’t eligible or registered to vote in some or all of these elections. These voters have an incomplete last 4 score, and need to be evaluated separately so their scores don’t influence voters with a complete history. As such I created two turnout tables: one for voters eligible for all elections (full), and one for voters eligible for at least one of the last four elections (partial). The turnout tables below show a turnout percentage for every combination of age group and participation score for 2006 voters:

# find voters registered before the 2002 primary
ele.full <- which(vfs$reg.date <= '2002-05-07')
# find voters registered after the 2002 primary but before the 2006 general
ele.partial <- which(vfs$reg.date > '2002-05-07' & vfs$reg.date <= '2006-11-07')	
# show the turnout table for full eligible voters
turnout.full <- ddply(vfs[ele.full,],c("age.2006","last4.g2006"), function(x) length(which(x$turnout.g06 == "X")) / nrow(x) )	
# show the turnout table for partial voters
turnout.partial <- ddply(vfs[ele.partial,],c("age.2006","last4.g2006"), function(x) sum(x$turnout.g06 == "X") / nrow(x) )

Turnout percentage for turnout.full:

	Age group	18-21	22-29	30-39	40-49	50-59	60-69	70-79	80-89	90+
last4	0	0	8.59	10.06	11.82	13.31	13.64	11.23	8	5.76
	1	0	32.63	43.84	50.86	54.71	54.92	49.82	36.08	21.98
	2	100	60.14	72.32	79.61	82.32	83.8	79.95	69.28	55.86
	3	72.58	84.35	88.96	91.29	92.24	90.02	82.74	65.23	100
	4	100	78.43	88.88	93.43	94.94	95.66	94.66	90.25	79.41

Turnout percentage for turnout.partial:

	Age Group	18-21	22-29	30-39	40-49	50-59	60-69	70-79	80-89	90+
last4	0	21.38	13.07	15.85	16.97	19.63	27.97	29.8	36.08	29.41
	1	27.7	27.38	33.96	36.38	41.7	51.69	51.95	42.25	28.57
	2	52.66	55.3	66.13	66.48	72.99	79.05	73.83	70	20
	3	64.14	78.63	79.46	91.51	88.37	85.29	94.74	100	0
	4	0	0	0	0	0	0	0	0	0

For each table, we see that turnout percentage increases as previous participation increases for every age group, but it is pretty difficult to compare more than two age groups at once using this table. There are also several anomalous groups with 100% turnout, indicating a small population in that group. We’ll use the R library ggplot2 to create a simple visualization of each table to help interpret the turnout values:

# turnout-full visualization
qplot(last4.g2006,V1,color=age.2006,group=age.2006,data=turnout.full,geom=c("point","line"),
  main="OH-01 2006 General Turnout by Age Group, Last 4 (Full)",xlab="Last 4",ylab="Turnout %") + 
  scale_colour_hue(name="Age Group")
# turnout-partial visualization
qplot(last4.g2006,V1,color=age.2006,group=age.2006,data=turnout.partial,geom=c("point","line"),
  main="OH-01 2006 General Turnout by Age Group, Last 4 (Partial)",xlab="Last 4",ylab="Turnout %") + 
  scale_colour_hue(name="Age Group")

Figure 1: 2006 Turnout Percentage by Age as a Function of Previous Participation (full)

Figure 2: 2006 Turnout Percentage by Age as a Function of Previous Participation (partial)

Figure 1 shows turnout for voters with a complete last 4 score, and tells us that for all age groups except 18-21, turnout increases with previous participation, until turnout reaches a maximum of 85%-90%. The rate at which turnout increases is similar between age groups, suggesting previous participation may have more predictive value than age. Figure 2 is a representation of all voters who were registered in time for the the 2006 general but not for the 2002 primary. Figure 2 shows a relationship between turnout and previous participation, but there is substantially more noise than in Figure 1. Taken together we can verify the hypothesis put forth by Malchow that age and previous participation seem to have a positive influence on future participation.

Regression analysis

The 2006 turnout tables are useful but they don’t represent a formal model of turnout the 2006 election. A formal model will measure the interactions between the predictor variables (participation & age) and the intended outcome (turnout) of 2006 voters. This model can be applied to 2010 voters to project turnout.

The model can include the other voter file variables with potential predictive qualities like gender, party affiliation, and martial status. A campaign will traditionally build a linear regression model to project turnout, but linear regression doesn’t support categorical variables and can produce values that don’t make sense for turnout, so I won’t be using that type of regression here.

Instead, I’ll use a generalized linear model to perform a binomial regression with a logit link function (logistic regression). Logistic regression estimates a binary variable given an intercept and a number of independent continuous or categorical predictor variables. R has terrific support for defining and evaluating these models using the base glm package.

The goal is to fit a logistic regression on voter data from 2006, and then use that regression to project turnout for 2010. I actually create two regressions, one for voters with at least a 4-year voting eligibility (full model), and one for all other voters (partial model). This is identical to the segmentation used when creating the turnout tables. The output of these regressions is the probability that a voter will turn out in the given year. A campaign can use this figure to estimate total turnout in an election, and to allocate resources to different geographic and demographic segments.

The R function glm is used to create two models of 2006 turnout based on last 4 participation, age group, gender, party affiliation, and martial status.

# create temporary variables inside the data frame for 2006 values
vfs$last4 <- vfs$last4.g2006
vfs$age <- vfs$age.2006
# create a model for voters with at least 4 years of voting history
full.lr <- glm(turnout.g06 ~ last4 + age + gender+party+married,data=vfs[ele.full,],family=binomial)	
# run ANOVA against the full table to test for term significance 
anova(full.lr,test="Chisq")
# create a model for voters with less than 4 years of voting history
partial.lr <- glm(turnout.g06 ~ last4 + age + gender+party+married,data=vfs[ele.partial,],family=binomial)
#  run ANOVA against the partial table to test for term significance 
anova(partial.lr,test="Chisq")

Now that I have fitted models, I’ll use the predict function to capture the model output. The output is the likelihood that a voter turned out in 2006 given his last4.2006 score, age group, gender, martial status, and party affiliation. Then I can compare the predicted turnout probability with the actual turnout to determine the effectiveness of each model. This isn’t a valid statistical measure of accuracy but merely a smell test.

# create a new column in the vfs data frame
vfs$pred.g06 <- c(0)
pred.g06.full <- predict(full.lr,type="response")
pred.g06.partial <- predict(partial.lr,type="response")
# apply the full model to voters with at least 4 years of registration
vfs[names(pred.g06.full),]$pred.g06 <- pred.g06.full
# apply the partial model to voters with less than 4 years of registration
vfs[names(pred.g06.partial),]$pred.g06 <- pred.g06.partial
# take the number of correct predictions divided by the number of voters  
full.correct <- sum((vfs[ele.full,]$pred.g06 > .5) == (vfs[ele.full,]$turnout.g06 == "X")) / nrow(vfs[ele.full,])
# value is .797 = ~80% accurate for the full model
# take the same for the partial model
partial.correct <- sum((vfs[ele.partial,]$pred.g06 > .5) == (vfs[ele.partial,]$turnout.g06 == "X")) / nrow(vfs[ele.partial,])
# value is .776 = ~77% accurate for the partial model

The prediction rates for our regressions aren’t spectacular: 80% for the full model and 77% for the partial model. Given the limited information in our voter file, though, they aren’t that bad. Additionally, a political campaign would have access to other data like detailed demographics, financial data, and more accurate lifestyle or ideological information. Extending the regression with these variables might increase the predictive power of the system.

2010

Now I’ll apply the regression equations to project turnout in 2010. First, I determine which regression (partial or full) to apply to current voters by their registration date:

# find voters registered before the 2006 primary (328594 voters)
ele.full2010 <- which(vfs$reg.date <= '2006-05-02')	
# find voters registered after the 2006 primary but before the 2008 general (68863 voters)
ele.partial2010 <- which(vfs$reg.date > '2006-05-02' & vfs$reg.date <= '2008-11-04')

Next I prepare the data and project 2010 turnout for each model using the predict function:

# assign the last4 and age model variables to values calculated for 2010
vfs$age <- vfs$age.2010
vfs$last4 <- vfs$last4.2010	
# call predict for the full model 	
pred.g10.full <- predict(full.lr,newdata=vfs[ele.full2010,],type="response")	
# predict based on the partial model
pred.g10.partial <- predict(partial.lr,newdata=vfs[ele.partial2010,],type="response")		
# turnout % for 2010 
pred.g10.turnout.full <- sum(pred.g10.full > .5) / length(ele.full2010)
# 63% predicted turnout
pred.g10.turnout.partial <- sum(pred.g10.partial > .5) / length(ele.partial2010)
# 18% predicted turnout	
# save the predictions into the vfs data frame
vfs$pred.g10 <- c(0)
vfs[names(pred.g10.full),]$pred.g10 <- pred.g10.turnout.full
vfs[names(pred.g10.partial),]$pred.g10 <- pred.g10.turnout.partial

According to pred.g10.full and pred.g10.partial, OH-01 will see 63% overall turnout for voters from the full model and 18% overall turnout for voters from the partial model. To determine the validity of the 2010 projections, I plotted 2006 actual turnout against the 2010 projected turnout for every age group. As stated in the introduction, Malchow says the turnout rates for 2010 should be similar to the 2006 election, so I expect no large unexplainable deviations in the chart. A separate chart is created for each participation model (full, partial):

# which voters have a > 50% chance of turning out in 2010
turnout.g10 <- vfs$pred.g10 > .5	
# which voters turned out in 2006
turnout.g06 <- vfs$turnout.g06 == "X"
# build a summary of voters who turned out in 06 or 10 based on age
to <- rbind( ddply(vfs[ele.full,],"age.2006",function(x) data.frame(age=x$age.2006[1],n=nrow(x),series="G06"))[,c(2:4)],
                ddply(vfs[ele.full2010,],"age.2010",function(x) data.frame(age=x$age.2010[1],n=nrow(x),series="G10"))[,c(2:4)])					
qplot(x=age,y=n,data=to,fill=series,stat="identity",geom="bar",position="dodge",main="OH-01 2006 Turnout vs 2010 Projected Turnout (full)",xlab="Age", ylab="Count") +  
  scale_fill_brewer(pal="Paired","Election")
 
to <- rbind( ddply(vfs[ele.partial,],"age.2006",function(x) data.frame(age=x$age.2006[1],n=nrow(x),series="G06"))[,c(2:4)],
		 ddply(vfs[ele.partial2010,],"age.2010",function(x) data.frame(age=x$age.2010[1],n=nrow(x),series="G10"))[,c(2:4)])			
qplot(x=age,y=n,data=to,fill=series,stat="identity",geom="bar",position="dodge",main="OH-01 2006 Turnout vs 2010 Projected Turnout (partial)",xlab="Age", 
  ylab="Count") + scale_fill_brewer(pal="Paired","Election")

Figure 3: OH-01 2006 Actual Turnout and 2010 Projected Turnout (full)

Figure 4: OH-01 2006 Actual Turnout and 2010 Projected Turnout (partial)

Figure 3 suggests larger 2010 turnout for all groups as compared to 2006. The projected increase in the 22-29 age group seems unlikely, but can probably be explained by the higher turnout among younger voters in 2008. In addition to 2008 being a presidential election year, the Obama for America campaign focused on registering new voters and activating dormant voters, both of which increased turnout among younger voters. That higher 2008 turnout inflated the last 4 measure for new voters, which pushed up the projected 2010 turnout. Figure 4 exhibits a similar projected increase for the younger age groups, which is probably due to the same increase in 2008 turnout. Before using these models in an actual election, the projections would need to be scaled based on some other turnout estimate. Despite the inflated values, however, this is a strong system for turnout prediction and could be used by almost any congressional campaign.

Model Improvements

In addition to scaling, the regression models would need to be improved in several ways before being put into production. The predictor variables are currently considered independently, which effectively discounts any interactive effects that may exist. Turnout for younger married females or unmarried democrats may be better modeled using compound variables, for example. Also, the model makes no use of demographic or opinion survey information available to political campaigns. Finally, the projection isn’t limited to two regressions; a campaign could create regressions by county or school district, or based on marriage status, or any other combination.

Other Visualization Examples

While not specifically related to turnout, I produced several simple visualizations that explore the rest of the voter file. The full power of R can be applied using the same voter data from the turnout projections.

## precinct summary
# summarize turnout in 2008 & registered democrats, by precinct
pct <- ddply(vfs,"precinct.code",function(x) data.frame(turnout.g08=sum(x$turnout.g08 == "X") / nrow(x),dem.pct=sum(x$party=="D")/nrow(x),nvoters=nrow(x)))
# visualize
qplot(turnout.g08*100,dem.pct*100,data=pct,geom="point",size=nvoters,alpha=I(0.4),main="OH-01 Precinct Turnout/Registration Summary",
  xlab="Registered Democrats (%)",ylab="2008 General Turnout")
 
## 2008 turnout by gender + age
qplot(age.2010,data=vfs[which(vfs$turnout.g08 == "X"),],geom="bar",fill=gender,position="dodge",main="OH-01 2008 General Turnout by Gender, Age",xlab="Age",   
  ylab="Count") + scale_fill_brewer(pal="Set1")
## 2008 newly registered voter counts by age
qplot(age.2010,data=new.08,main="OH-01 2008 Newly registered voters",xlab="Age",ylab="Count")
## 2008 newly registered voter turnout by age
qplot(age.2010,data=new.08,fill=turnout.g08,position="dodge",main="OH-01 2008 Turnout for newly registered voters",xlab="Age",ylab="Count") + 
  scale_fill_brewer(pal="Paired")

Figure 5: OH-01 2010 Precinct Summary	Figure 6: OH-01 2008 General Turnout by Gender, Age
Figure 7: OH-01 2008 Newly Registered Voters by Age	Figure 8: OH-01 2008 Newly Registered Voters by Age, Turnout

Figure 5 shows the 2008 Democratic turnout percentage and 2008 general turnout percentage for each precinct, and each bubble is scaled to the registered voter population of the precinct it represents. Figure 6 shows 2008 general turnout by gender and age group. Figure 7 is the raw count of voters registered between the day after the 2006 general election and the day of the 2008 general election, broken down by age group. Figure 8 is a variation of Figure 7, showing 2008 turnout of voters registered after the 2006 election. None of these charts took more than 5 minutes to create from concept to output, and ggplot did almost all of the heavy lifting. I believe the ease with which these charts were built highlights the utility of having your data analysis tool also be your visualization tool.

Summary

In this short example I’ve analyzed, visualized, and modeled electoral data using R and a few add-on packages. These are standard techniques used by any congressional campaign, but they are usually performed by some by combination of Excel, SPSS, or SQL. By using R, I avoided the compatibility issues usually encountered when transferring data between tools. R would have also allowed me to perform clustering, component analysis, or Bayesian inference on the same data from the same R interface. All together, these reasons make R a good addition to the political analysis toolbox for a campaign or campaign consultant. If you would like to discuss how advanced statistical analysis can help your Democratic campaign model turnout, increase fund-raising, or benchmark field operations, please don’t hesitate to contact me.

Click here (14MB) to download the R scripts and data associated with this post. The voter file data has been scrubbed to remove the VoterID, name, and address components.

• Speed improvements – Schedule and field names names are matched by compiled regular expressions instead of brute-force string matching, resulting in a ~25% speed increase for large files.
• DEF file fixes – The field definition files FECHell uses have been heavily updated to remove bad fields and standardize field names for most schedules
• FECForm library – FECHell now includes an optional library called FECForm. FECHell returns an hash indexed by field names. This is problematic because field names are case sensitive, version dependent, and some field names are difficult to understand. FECForm tries to alleviate some of these issues by providing a consistent interface for one form (F3) and several schedules (SA,SB,SC,SC1) using named parameters. This is especially helpful when parsing very old FEC files where name and address components were delimited by a ^ instead of being an individual field, COMMITTEE was called CMTTE, etc.
  Example:

  fec_version, original_form_type, form_type, values = h.header_lines("12345.fec")
  # old way 5.2-6.4
  puts "Committee: #{values['COMMITTEE NAME']}"
  # old way 3.0-5.1
  puts "Committee: #{values['FILER NAME']}"
  # new way
  f = FECForm.schedule_for(form_type, fec_version, values)
  puts "For: #{f.committee_name}"

  Other major forms (F3X,F1,F2,F6) are in progress, as are the outstanding schedules (SC2).

• Monolithic unit tests – FECHell / FECForms finally comes with some unit tests! An example file for every form and every schedule, for every supported version, is included in the distribution. Tests are included for every form and field supported by FECForms.

Visit the FECHell page for more information, installation instructions, and more examples.

One concern was my using arbitrary-sized bins for the range comparison. Revo blog commenter Mike Lawrence suggested 3 equal sized bins with a neutral color for near zero values. Given the variance in election returns between townships (sd=28 points in 2010,sd=20 in 2008) I’m going to choose 10 bins instead of 3, but otherwise this is an excellent suggestion. Initially we’ll be measuring the difference between the Republican and Democratic vote percentage in each township for the 2008 and 2010 elections. This tells us not only which party won, but by how much.

Here is the code to recreate the MA 2010 senate returns map with breaks of 15% and a different color palette showing neutral middle colors.

  library(maptools)
  library(sp)
  library(Hmisc)
 
  labels = c("54-70% Dem","41-55% Dem","26-40% Dem","11-25% Dem","0-10% Dem","0-10% Rep","10-24% Rep","25-39% Rep","40-54% Rep","55-70% Rep")
  cuts=c(-.7,-.55,-.4,-.25,-.1,0,.1,.25,.4,.55,.70)
 
  masen <- read.csv('masen2010.csv')
  masen$Total <- masen$GOP + masen$DEM + masen$LIB
  masen$Offset <- (masen$GOP - masen$DEM) / masen$Total
  shp <- readShapeSpatial('tl_2009_25_cousub')
  shpA <- shp[match(masen$place,shp$NAME),]
  masen$Col1 <-cut2(masen$Offset,cuts=cuts)
  colors <- rev(brewer.pal('RdYlBu',n=10))  
  plot(shpA,col=colors[as.integer(masen$Col1)],border=grey(.85))
  title("Major Party Winning Vote Percentage MA Sen 2010")
  legend("bottomleft", legend=labels,fill=colors)

Which gives us the following output:

I believe this map more accurately represents the results. As a companion to the 2010 major party results I’d like to recreate the 2008 presidential results, using the same scale and bins.

gen2008 <- read.csv('mapg2008.csv')
gen2008$Offset <- (gen2008$rep - gen2008$dem)/  gen2008$total
gen2008$Col1 <-  cut2(gen2008$Offset,cuts=cuts)
shpA <- shp[match(gen2008$place, shp$NAME),]
plot(shpA,col=colors[as.integer(gen2008$Col1)],border=grey(.85))
legend("bottomleft",legend=labels,fill=colors)
title("Major party win percentage MA Presidential 2008")

Now the two graphs can more or less be compared apples to apples, and we can draw some very interesting conclusions from that comparison. The nine townships that were heavily democratic in 2008 actually stayed heavily Democratic in 2010, white most all other townships showed an increase in Republican support. These townships didn’t necessarily flip from being won by a Democrat to being won by a Republican, but every township showed an increase.

The least successful map I had previously produced was attempting to illustrate the degree of increased Republican support. What I really wanted to show was the increase in Republican support between the elections 2008 and 2010 elections.

masen <- masen[order(masen$place),]
magen <- gen2008[order(gen2008$place),]
masen$RepP <- (masen$GOP / masen$Total)*100
magen$RepP <- (magen$rep / magen$total)*100
gain <- masen$RepP - magen$RepP
gr <- cut2(gain,cuts=seq(0,30,by=5))
colors <- brewer.pal("Reds",n=6)
shpA <- shp[match(magen$place,shp$NAME),]
plot(shpA,col=colors[as.integer(gr)])
title("Increase in Republican vote percentage 2008 to 2010")
legend("bottomleft",legend=c("0-4%","5-9%","10-14%","15-19%","20-24%","25-30%"),fill=colors)

From this map we can see that from 2008 to 2010 the Republican vote percentage actually increased in every single township in MA. This is kind of shocking, especially when you consider that Democrats enjoy a 15 point registration advantage statewide(source: warning PDF). If I were a campaign manager for Richard Neal (2nd Cd) or Jim McGovern (3rd CD) I’d take a long look at what the Coakley campaign did to so many democratic voters to either stay home or cross party lines in the 2010 special election.

jjh

First look at the results from the MA 2010 Senate special election.

MA Senate 2010 Results

Now compare that to the results of the 2008 presidential primary in MA, blue gradients for Obama and red gradients for Clinton.

Democrat 2008 Presidential Primary results

Using the same color scale and data cut-points we can see the same results that DavidNYC came up with, namely that winning townships for Clinton in 2008 tracked decently to winning townships for Scott Brown in the 2010 Senate race.

Obama lost the MA presidential primary in 2008, but he won in the general. Here is a map showing the vote margins from the 2008 presidential general:

MA 2008 Presidential results

Compared to the MA 2010 Senate race last week these 2008 numbers are astounding. The democratic vote margin in 2008 was 26 points (62% to 36%), while the 2010 senate race was -5 points (47-52), with 30% less turnout in 2010 than in 2008. I decided to create another map showing the decline in democratic vote support

Democratic vote change 2010 to 2008

Does this mean voters have turned against democrats in MA? Maybe, but it is interesting that Scott Brown got more votes in the special election (1,168,107) as McCain did in the 2008 general (1,108,854), even though 900,000 less people voted in the special. So Brown was able to do two things: 1) Activate republican McCain voters, 2) Cause democrats to cross party lines. I don’t think this spells certain doom for congressional democrats in the midterms, but it does show that Democrats will stay home on election day, even in a historically democratic state, for the wrong candidate.

You can create the maps yourself or play with the data by downloading the source + data files here. With a recent installation of R you can recreate the maps above by running the the following command:

R CMD BATCH draw.R

Thanks.

Electoral targeting is the process of quantifying the partisan bias of a single voter or subset of voters in a geographic region. Bias can be calculated using an individual’s demographic and voting behavior or by aggregating results from an entire election precinct. Targeting is traditionally performed by national committees (e.g., National Committee for an Effective Congress, National Republican Congressional Committee), state political parties, interest groups (e.g., EMILY’s List, National Rifle Association), or campaign consultants. Targeting data is consumed by campaign managers and analysts, and it is used along with polling data to build strategy, direct resources, and project electoral outcomes.

While aggregate electoral targeting can build a sophisticated picture of a district, the mathematics behind targeting are very simple. Targeting can be performed by anyone with previous electoral data, and calculations can be done using 3×5 note cards, with simple spreadsheets, or high-end software packages like SPSS. The targeting methods discussed in this post are taken from academic publications on electioneering: Campaign Craft (Burton, Shea 2006) and The Campaign Manager (Shaw, 2004).

Although targeting data is usually usually inexpensive or free, a down-ballot campaign or a primary challenger might not have the connections or support of a PAC or party to obtain the data. In these cases, a campaign will probably purchase one of the books listed above to perform its own analysis. Even an established campaign may run its own analysis, possibly to test different turnout theories or to integrate additional data. This post is directed towards these groups.

Together, we will assume the role of campaign consultant and perform an aggregate electoral analysis on the 13th House of Delegates seat (HOD#13) in the Commonwealth of Virginia. In HOD#13, the 18-year Republican incumbent Bob Marshall is being challenged by Democrat John Bell. This analysis will compute and visualize turnout, partisan bias, and a precinct ranking based on projected turnout and historical Democratic support.

The analysis of HOD#13 will be performed using R, an open-source computing platform. R is free, extensible, and interactive, making it an ideal platform for experimentation. The R package aggpol was created specifically for this tutorial, and it contains all the data and operations required to execute an aggregate electoral analysis. Readers can execute the provided R code to reproduce the analysis or simply follow along to learn how it was performed. Readers unfamiliar with R should read Introduction to R, which is available on the R project homepage.

The electoral and registration data used were compiled from the Virginia State Board of Elections using several custom written parsers and two different PDF-to-text engines. Please contact me for source data or more information at: jjh@offensivepolitics.net.

Prerequisites

This section only applies to readers interested in recreating the analysis and graphics produced in this tutorial. To completely recreate this analysis you will need the following:

1. The latest version of the R statistical computing environment. Binaries, source, and installation instructions can be downloaded from R homepage.
2. Additional R packages. This analysis requires several packages that provide additional functionality on top of the existing R system. Install the appropriate R environment for your system and run the program.
   plyr, ggplot2, RColorBrewer. To install these packages execute the following in your R environment:

   1	install.packages(c("plyr", "RColorBrewer", "ggplot2"))

   Next you’ll need to install the aggpol package for calculating aggregate political statistics. You will need to download the latest version: For Unix-style systems or For Windows systems. Installation of local packages is detailed in the R Manual on package installation.

Getting Started

Now that the prerequisites are installed we can get started with our data analysis. Start up your R environment and load the required libraries by typing in the following commands:

1
2
3
4

library(plyr)
library(aggpol)
library(ggplot2)
library(RColorBrewer)

We need to attach the VAHOD data set that comes with aggpol. This data set contains precinct-level electoral returns for state and federal elections in the Commonwealth of Virginia from 2001 to 2008. Since we are focusing on HOD#13, we’ll need to select just the records that have to do with that seat.

1
2

data(VAHOD)
hd013 <- vahod[which(vahod$seat == "HD-013"),]

The data set contains precinct-level electoral results for the following races: U.S. President, U.S.Senate, U.S.House of Representatives, Virginia Governor, Senate of Virginia, and Virginia House of Delegates. This breadth of electoral returns allows us to build a very detailed profile of the partisan bias of a district.

We will first determine the historical partisanship in HOD#13. Since partisanship can fluctuate over the years and different seats have different turnout expectations, we’ll first need to see the major party support for every seat in each election for precincts in HOD#13. We can use the historical.election.summary function from the aggpol package to group the precinct results into district results, and then break them down by seat and year.

1

esum <- historical.election.summary(hd013)

esum now contains:

We now have major-party turnout for every election in our data set. To best visualize the results we’ll build a bar graph comparing major-party turnout in each seat over time. We first need to transpose the election summary object (esum) from a summary format to an observation format, one line per distinct year+district+party. The plyr package makes this task extremely simple.

1
2
3

elx <- ddply(esum,c("year", "district_type"), function(x) 
  rbind(  data.frame(party="REP",turnout=x$rep.turnout.percent),
  data.frame(party="DEM",turnout=x$dem.turnout.percent)))

We will now use the powerful ggplot2 package to view the Republican and Democratic support for each election, in each seat, for our subset:

1
2
3
4

ggplot(elx,aes(year,turnout,fill=factor(party))) + 
  geom_bar(stat="identity") + 
  facet_wrap(~district_type,scales="free_x") + 
  scale_fill_brewer(palette="Set1")

Result:

This graphic gives us a decent understanding of district-level electoral trends. For U.S. federal elections (figs.: PVP, USH, USS), we can see a distinct drop in Republican support moving towards 2008; the results for U.S. House (USH) and U.S. Senate (USS), in particular, show a strong increase in Democratic support. This growth correlates to statewide trends that resulted in the election of two Democratic Senators representing Virginia for the first time since 1970. General Democratic gains notwithstanding, the House of Delegates (fig.: HD) results aren’t as promising for a Democratic challenger. The incumbent Del. Marshall saw more than 60% support in three of the last four elections and saw no challenger at all in 2003. While the district may be trending more Democratic over time, the voters of HOD#13 are obviously big fans of Del. Marshall.

Now that we understand the historical partisanship of this district we need to understand historical turnout, allowing us to project of the number of votes required to win. We will utilize the historical.turnout.summary function from the aggpol package to produce a summary of turnout for this district.

1

historical.turnout.summary(hd013, district.type="HD", district.number="013", years=c(2001,2003,2005,2007))

Looking at this table one can see some data collection problems in the 2001 HD elections. In recent years, precincts belonged to only one House of Delegates seat, but in 2001 and somewhat less so in 2003 some precincts are split and some have duplicate names and now information on how to allocate results from different races to precincts. The turnout numbers are slightly affected by these problems, but the aggpol attempts to correct this by substituting alternate years or even races if possible.

The take away from from the previous table is that turnout for the last four House of Delegates elections has hovered around 30%. This makes some political sense, because Virginia holds state elections in odd-numbered years with no federal elections to drive up turnout. This leaves a lot of registered voters to be activated, but we need to delve down to the precinct level to find them.

We use the district.analyze function of aggpol to aggregate all electoral results into a summary for each precinct.

1

hd013s <- district.analyze(hd013)

hd013s is a data frame with columns calculated for every precinct; several values for each major party and other values for the precinct as a whole. Those statistics are:

• Aggregate base partisan vote – The lowest non-zero turnout for a major party, in all electoral years.
• Average Party Performance – The average percentage of the vote a party receives in the closest 3 elections in recent years.
• Swing vote – The part of the electorate not included in the aggregate base partisan vote.
• Soft-partisan vote – The average worst a party has performed, minus the actual worst.
• Toss-up – The portion of the electorate not included in the Aggregate base or soft-base partisan vote.
• Partisan base – The combined aggregate-base and soft-partisan vote for each major party.
• Partisan swing – The combined major party swing vote.
• Projected turnout – The portion of the electorate that is projected to turn out given previous turnout and current registration data.

These variables can be visualized with the following graphic, adapted –along with definitions above– from Campaign Craft (Burton, Shea).

The actual columns in the data frame returned from from district.analyze are:

• proj.turnout.percent – The projected turnout percent of for a hypothetical next election.
• proj.turnout.count – The projected number of voters who will turn out for a hypothetical next election.
• current.reg – Current number of registered voters in a precinct.
• partisan.base – The combined aggregate-base and soft-partisan vote for both major parties ( Partisan base ).
• partisan.swing – All non-base voters (1.0 – partisan.base).
• tossup – The portion of the electorate not in the base or soft support of either major party.
• app.rep – The average party performance of a Republican candidate in this precinct.
• base.rep – The aggregate base partisan vote for a Republican candidate in this precinct.
• soft.rep – The soft partisan vote for a Republican candidate in this precinct.
• app.dem – The average party performance of a Democratic candidate in this precinct.
• base.dem – The aggregate base partisan vote for a Democratic candidate in this precinct.
• soft.dem – The soft partisan vote for a Democratic candidate in this precinct.
• partisan.rep – Combination of aggregate base and soft vote percentages for the Republican.
• partisan.dem – Combination of aggregate base and soft vote percentages for the Democrat.

The most useful statistic above is the Average Party Performance (APP), which is an average of major-party turnout in the 3 closest recent elections. The APP describes supporter levels for a best-case scenario in a close election. We’ve already calculated the APP of each major party (app.dem, app.rep), but when a race doesn’t have a third party candidate what we’ll usually visualize is the share of the combined partisan performance that each party receives. We’ll add these variables to our summary data frame generated previously, one for each major party.

1
2

hd013s$dem.share <- hd013s$app.dem/(hd013s$app.dem+hd013s$app.rep)
hd013s$rep.share <- hd013s$app.rep/(hd013s$app.dem+hd013s$app.rep)

Now that we have the APP and partisan vote share for each party, we can visualize the precinct-level terrain for the Democratic challenger Mr. Bell. This visualization should show us the democratic support for each precinct and give us an idea whinc precincts could be competitive. We’ll produce this visualization using a density plot + 1d histogram, adapted from the seatsVotes plot in the pscl package. We’ll also draw a cut-line down the 50% vote mark to to help find competitive precincts.

1
2
3
4

qplot(dem.share, data=hd013s, geom=c("density","rug"),
    xlab="Dem Vote Share",
    main="Democratic vote share, by precinct")  + 
  geom_vline(xintercept=.50)

We can see a lot of precincts are between 48% and 53% Democratic, which means those precincts could potentially go for either candidate. We need to classify these results into something more solid. Let’s say precincts with less than 48% Democratic share are Safe Republican, 48-52% are Tossup, and greater than 52% are Safe Democrat. This is a simple representation but can be refined later. We’ll add a seat classification to our data frame using the cut function:

1

hd013s$cl <- cut(hd013s$dem.share, breaks=c(0,.48,.52,1), labels=c("Safe Rep", "Tossup", "Safe Dem"))

Now we need to visualize how many precincts fall into which classification, using a histogram this time instead of a density curve.

1
2
3
4
5

ggplot(hd013s, aes(x=dem.share)) + 
  geom_bar(aes(fill=cl),binwidth=0.01) + 
  scale_fill_brewer("Precinct Rating", palette="RdYlBu") + 
  scale_x_continuous("Democratic Vote Share") + 
  scale_y_continuous("Frequency")

From the histogram we see that not only does a Republican candidate enjoy more “Safe” precincts, but even the majority of the tossup precincts have less than 50% Democratic share. While the precinct breakdown looks bad, a Democratic win in this district is theoretically possible if these tossup precincts are held. A Democratic candidate will face a tough challenge, so the next step will be identifying Democratic and Democrat-leaning precincts to target.

To make this target precinct list we’ll need a method to prioritize the precincts so that we can reach the most persuadable voters while spending the least resources. A popular method to identify a precinct as high-value is to sort precincts by lowest projected turnout with highest Democratic vote share. Lower turnout means there are registered voters waiting to be convinced to show up, and high Democratic vote share means more of those voters will be Democrats.

Since we measured both of these values (turnout%, democratic vote share), it is very easy to order our data by turnout (ascending) and democratic average party performance (descending) using R.

1

hd013s[order(hd013s$proj.turnout.percent,-hd013s$app.dem),c(1:2,20:21),]

This sorted list is our critical intelligence to finding persuadable voters, but we need a better way to visualize the output. Since we have two scalar variables (turnout %, democratic vote share) we can use a scatter plot with the Democratic vote share on the Y axis and Turnout % on the X. We’ll also color each precinct with its seat classification we defined earlier (Safe Republican, Tossup, Safe Democrat):

1
2
3

ggplot(aes(x=proj.turnout.percent, y=dem.share), data=hd013s) + 
  geom_point(aes(colour=cl,title="a")) + 
  labs(x="Projected Turnout %", y="Democratic Vote Share %",colour="Seat Type")

This chart echoes what we’ve seen previously: the Democratic challenger faces an uphill battle, but there is room for a win. We see a single “Safe Democract” precinct with very low turnout, and five “Safe Republican” precincts that run the board in turnout. Given the high number of “Tossup” precincts, and the fact that they run the gamut as far as turnout is concerned, we’ll need to incorporate additional information into our prioritization. If we also rank precincts by current voter registration, we can focus on precincts where we stand to gain the most ground.

Before we continue, we need to make sure there is enough difference in precinct-to-precinct registration to have an impact. Let’s look at some statistics for the current registration in this district.

1
2

mean(hd013s$current.reg)
sd(hd013s$current.reg)

There are on average 2,970 current registered voters in each precinct, but the standard deviation is 1,014 voters. A standard deviation that high tells us we need to take into account registration if we want to focus on the precincts with 4000 people and not 1000 people. A histogram of current registration will help us clarify this finding:

1
2

qplot(current.reg, data=hd013s, geom="bar",binwidth=500,xlab="Current Registration") + 
  scale_y_continuous("Frequency")

The standard deviation was correct: we see some very small precincts and some large precincts, but the majority are somewhere in the 2000-4000 range. The difference looks to be large enough to include current registration in our ranking.

We need to look at the Democratic Vote Share vs Turnout % scatter plot again, but with the points scaled to the current precinct registration.

1
2

qplot(proj.turnout.percent,dem.share,size=current.reg, data=hd013s,colour=cl)+ 
  labs(x="Projected Turnout %", y="Democratic Vote Share %",colour="Seat Type",size="Current Registration")

Democratic vote share by Turnout %

This plot is almost complete and ready to be analyzed. The last job is to label the points with ther precinct names. Our current precinct_name variable is actually a unique identifier with a FIPS county code, a precinct code, and a name, and it is too long for a point label. We’ll shrink it down to just the name and then we’ll recreate the scatter plot with the label:

1
2
3
4
5
6
7

# replace the fips code and precinct number w/ an empty string
hd013s$precinct.label <- sub("^[0-9]+ - [0-9]+ - ",'',as.character(hd013s$precinct_name))
# plot the previous graph again but this time use precinct.label as the label
ggplot(hd013s, aes(x=proj.turnout.percent, y=dem.share,label=precinct.label)) + 
  geom_point(aes(colour=cl,size=current.reg)) + 
  geom_text(size=2.5,vjust=1.5,angle=25) + 
  labs(x="Projected Turnout %", y="Democratic Vote Share %",colour="Seat Type",size="Current Registration")

From the chart we can see that a Democrat in the HD#013 will want to focus contact efforts on the precincts in the upper-left hand corner of the plot and will want to target larger precincts before smaller. Integrating the current registration into our previous sort command leaves us with the following sort order:

1

hd013s[order(hd013s$proj.turnout.percent,-hd013s$app.dem,hd013s$current.reg),c(1:2,4,20:22),]

Now that we have our ranking, we can figure out how much each precinct might offer. Let’s first see the number of votes required to win the seat, the number of votes we’re projected to receive given the calculated APP, previous turnout, and current registration. The district.summary function will provide us will all this information:

1

district.summary(hd013s)[,c(1,2,9,10,11)]

We can see that the projected turnout (proj.turnout.count) is about 25,401, so the votes projected to win this district is only 12,702. Using the Democratic APP, we can project Democratic turnout at 12,074, so we need to find 628 votes to win. How do we find these votes?

Lets go back to our sorted precinct list and take the top 30% and call them our target.precincts.

1
2

sorted.precincts <- hd013s[order(hd013s$proj.turnout.percent,-hd013s$app.dem,hd013s$current.reg),]
target.precincts <- sorted.precincts[1:(nrow(sorted.precincts)/3),]

We’ve got our target list, and we know we need 628 votes from them to bring our total to 50% + 1. Adding a small buffer to that number, we’ll take 640 target votes and allocate them across our target precincts, proportional to the number of registered voters in the precinct. Hopefully, this will set more realistic goals for larger and smaller precincts.

1
2
3

target.precincts$inc <- as.integer(640 * target.precincts$current.reg/sum(target.precincts$current.reg))
 
target.precincts[,c(2,3,17,23,18,20:22,24)]

The final column in the result is the target increase for that precinct (column: ‘inc’). With this information in hand the campaign field operations can devise a contact strategy to bring these voters to the polls on election day.

Conclusion

Playing the role of campaign consultant, we have analyzed previous electoral outcomes in the 13th seat of the House of Delegates in Virginia. We have shown how a Democratic candidate can leverage increasing Democratic support and low turnout to make this race competitive. We have also created a precinct targeting methodology that provides a high-level blueprint for resources planning. The analysis we performed performed is very standard, but using R makes our methodology unique. A down-ballot or primary-challenger campaign taking advantage of this methodology will spend less money and can experiment more on their targeting, potentially leading them to a win.

Are you a Democrat running for the Virginia House of Delegates who would like to see the same data for your race? Or, are you a Democratic congressional candidate preparing for the 2010 cycle? Contact me at jjh@offensivepolitics.net for robust targeting data or other analysis.

Follow Offensive Politics on twitter

I used several packages (XML and plyr) to fetch and extract reasons, and then tm to filter stop words and identify commonly used terms. Finally, I used ggplot2, the invaluable ggplot2 blook, and a helpful post from the R-help mailing list to perform the visualization.

R code

library(XML)
library(plyr)
library(ggplot2)
library(tm)
 
# fetch & parse the HTML
doc <- htmlParse("http://gop.com/index.php/learn/republican_faces/",isURL = TRUE)
# pull the matching A elements of CSS class tipz
nodes <- getNodeSet(doc, "//a[@class='tipz']")
# extract the 'title' attribute 
titles <- sapply(nodes, function(x) xmlAttrs(x)[["title"]])
# clean up the title attribute 
titles <- sub("^[^:]+::","",titles)
# create the corpus and doc term matrix
co <- Corpus(VectorSource(titles))
tdm <- DocumentTermMatrix(co, control=list("tolower", removeNumbers=TRUE, stopwords=TRUE))
# extract the tags at each level
levels <- c(1,2,3,4)
df <- ldply(levels, function(x) data.frame(freq=x,term=findFreqTerms(tdm,x,x))) 
#assign random non-repeating coordinates to the terms
df$x <- sample(1:nrow(df),nrow(df), replace=F)
df$y <- df$freq + rnorm(nrow(df))
 
# clear standard graph options (thanks mike lawrence on r-help)
clear <- opts(
         legend.position = 'none'
         , panel.grid.minor = theme_blank()
         , panel.grid.major = theme_blank()
         , panel.background = theme_blank()
         , axis.line = theme_blank()
         , axis.text.x = theme_blank()
         , axis.text.y = theme_blank()
         , axis.ticks = theme_blank()
         , axis.title.x = theme_blank()
         , axis.title.y = theme_blank()
 )
 
p <- ggplot(df,aes(x=x,y=y,colour=freq,label=term,size=freq)) + geom_text() + coord_polar()+ clear 
ggsave("because.png",p,dpi=72,scale=1.3)
ggsave("because.pdf", p)

And the output:

Click for a page-sized PDF, or the raw terms and frequency counts.

The most common term is ‘freedom’, followed by ‘equal’, and ‘pro’. After those come ‘personal’, ‘government’, ‘people’, ‘school’, ‘family’, and ‘believe’. A more robust analysis could use term extraction (pro family, pro life, anti government) or stemming, and then feed the results into a better visualization. That would take more than the 10 minutes I spent so far, so I’m leaving that as an exercise to somebody else.

As it is I have the most common answer as to why GOP.com visitors are Republicans: freedom. I think that’s probably why anybody belongs to any political party, but without a corpus from other parties I suppose we’ll never know.

In Part 1 of this tutorial we introduced the fechell library by extracting all itemized contributions from individuals made to the Obama For America campaign in 2007 and 2008. In Part 2 of the tutorial we will summarize that data set by importing it into a MySQL database and aggregating contributions by week and zip code. Next we’ll visualize the contribution data amounts on a map, week by week. Below is a sample image:

This visualization contains two separate measures: the top portion is a map of the continental US, marked with a dot at the geographic center of every zip code that had individuals who made contributions to Obama for America. The dot is colored according to how much money in total was raised from that zip code since the 2nd week of 2007, with the colors going from a dark blue to bright red indicating amounts. The second measure is a vertical bar plot showing how much money was raised, per week, from all zip codes combined. The current week is highlighted and annotated with the amount raised for that week. Taken together these measures can show us where the first monetary support for the Obama for America campaign came from and how it progressed geographically and in volume. We will use the free R statistics and visualization package to producing the weekly image we shown above. Looking at still images week by week is informative but not very exciting. After all the images are created we’ll use MEncoder to string together the images into a movie to demonstrate the growth of individual financial support pledged to Obama for America.

Installing the database

To follow along with this exercise you will need to download MySQL 5.1 for your platform. English instructions for installation are available in the MySQL 5.1 Reference Manual. After installation you will also need to create a database user and grant rights to create databases, create indexes and perform LOAD DATA LOCAL functions. All of these administration tasks are covered in the MySQL 5.1 Reference Manual Section 5.5 (MySQL User Account Management). Keep the user name and password of the account you created handy as we’ll use it in the next few steps to access the database.

Creating the database

We will be using two different tables to represent the itemized individual contributions: the transactions table will hold all 2.8 million transaction values, and the transactions_summary_weekly table will hold the sum of all contributions for every combination of year, week, and 5 digit zip code that exists in the data.

To populate these tables, download and save the transaction table creation script create-transactions.sql available in the Resources sub-section below to a directory. Next download the output of the first part of the post into that same directory (obama-data-F3P-2007-2008.csv), available as ZIP archive in Resources sub-section below. If this CSV file isn’t in the same directory as the create-transactions.sql script you’ll need to change line #4 to reference the exact location.

create-transactions.sql looks like this:

1
2
3
4
5

CREATE DATABASE fechell;
USE fechell;
CREATE TABLE transactions (contribution_date date,contribution_amount int, zipcode char(5)) engine 'MyISAM';
LOAD DATA LOCAL INFILE 'obama-data-F3P-2007-2008.csv' INTO TABLE transactions FIELDS terminated BY ',' LINES terminated BY '\r\n' IGNORE 1 LINES (contribution_date,contribution_amount,zipcode);
ALTER TABLE transactions ADD INDEX  ix_zipcode (zipcode);

We are simply creating a new database (‘fechell’), and creating a new MyISAM table called ‘transactions’ on that database. We chose MyISAM since large data imports are dramatically faster with MyISAM than the default InnoDB, at least in the default setup we’re using. You can execute this script by running the following command in the directory where you saved create-transactions.sql and obama-data-F3P-2007-2008.csv. Replace YOURDBUSERNAME with the user you created during the MySQL installation.

mysql -uYOURDBUSERNAME < create-transactions.sql

Since we are loading in 2.8 million records and then adding an index to the zip code field this command might take quite a while to execute depending on your processor speed. After it is finished you should have a large database of all individual itemized transactions in the fechell.transactions table.

Next we will create and populate the transactions_summary_weekly table, which will contain aggregated itemized individual fund raising totals by zip code by week and year.
The script create-transactions_summary_weekly.sql available in the Resources sub-section below looks like this:

1
2
3
4

USE fechell;
CREATE TABLE  transactions_summary_weekly (zipcode char(5) NOT NULL,contribution_year int(10) UNSIGNED NOT NULL,contribution_week int(10) UNSIGNED NOT NULL, total int(10) UNSIGNED NOT NULL) engine 'MyISAM';
INSERT INTO transactions_summary_weekly(zipcode,contribution_year,contribution_week,total) 
(SELECT zipcode,year(contribution_date),week(contribution_date),sum(contribution_amount) FROM transactions GROUP BY zipcode,year(contribution_date),week(contribution_date));

You can run the script with the following command:

mysql -uYOURDBUSERNAME < create-transactions_summary_weekly.sql

After the script is finished executing you should have two rather large tables populated with the information we’ll need to visualize the data.

Installing R

Before we can create the visualizations we’ll need to make sure the R toolkit and several add-on packages are installed correctly. If it isn’t installed already, you can download an R installer from the R mirrors list. The R Frequently Asked Questions page contains instructions for Linux, Unix, Mac, and Windows platforms.

Installing R Packages

After R is installed you will need to install several add-on packages that may or may not be included with your distribution: sp, maps,maptools, and RMySQL. All are available via CRAN and platform-dependent instructions for installing add-on packages fcan be found on the R Wiki under How do I install a package?.

Creating the visualization part 1 – images

Before you can create the visualization you will need to download the R script to draw the frames, and a support file containing latitude and longitude pairs for every zip-code in the US. Both are available in the References sub-section below as draw.R and zips.zip. Unzip the zips.zip file and move the newly created zips.csv file into the same directory as draw.R. You will need to make 1 change to the draw.R script before you can run it: line 37 creates the connection to the database we created and populated earlier. You’ll need to change the host, user name, and password arguments to match your database setup.

The draw.R file is pretty simple, so we wont walk through the code line by line. Most of the heavy lifting is done by the fantastic sp and RMySQL packages, with a little bit of help from a lat/long database of zip codes.

Once you’ve edited the draw.R to match your configuration you can can create the images with the following command line, assuming the R bin directory has been correctly added to your PATH:

R CMD BATCH draw.R

Assuming the packages were installed correctly, the zips.csv file was in the same directory, and the database configuration was modified you should now see 96 PNG files in your directory. The files are named o4a_year_week.png.

Creating the visualization part 2 – video

To create a FLV movie from the weekly images you will need to install MPlayer/MEncoder if it isn’t installed on your system already. You can find download and install instructions on the MPlayer download page. Once the installation is complete you can create a FLV movie with the following command line (some options taken from a page on http://jeremychaman.info/

mencoder.exe mf://*.png -mf w=1024:h=768:type=png -ovc lavc -lavcopts vcodec=flv:mbd=2:mv0:v4mv:cbp:last_pred=3:trell:keyint=50:vbitrate=300 -o output.flv -fps 2 -ofps 5 -of lavf

After the command is complete you will have a FLV file called output.flv. The final file is embedded below, and available in the Resources section:

Problems with this visualization

While this visualization serves our purpose of tracking the geographic spread and growing amount of individual contributions to the Obama for America campaign, it is not perfect. The entire process, from extraction to visualization was created in about half a day, so there is obviously more work that could be done. Also the specific process used in these tutorials was to be a demonstration of several strategies to perform your own analysis, not necessarily to build the best visualization of independent contributions.

One problem with the current visualization is the representation of an entire zip code by a dot at its geographic center. This is problematic because zip codes represent regions that are vastly different in size, so using single dots will end up in decisions being skewed. Based on our maps alone, we could conclude Obama for America enjoyed very little support in places like Montana and Colorado since they have very few dots within their borders. In truth the campaign received more than $11M of support from Colorado and just over $1.2M in itemized donations from Montana, but this is very difficult to compare this to Philadelphia or New York City where the map is a smear of dots. A better way to display information by zip code could involve drawing the ZIP boundaries and filling with a color, instead of just coloring a single point. This would work fine for places like Montana, Idaho, and Colorado where a zip code might refer to a very large area, but would fail in more populous areas. This is especially true in New York City, where a single zip code could represent an area as small as a single square mile and would be almost impossible to view on a reasonably sized graphic. Merging of zip code areas and averaging their fund raising totals could make this procedure more useful for national analysis or local analysis of heavily populated metro areas.

Additionally, the output format of the visualization could be improved by making it interactive. Combining our weekly summary data with a tool like the Google Motion Chart Gadget along with an Animated Timeline would provide a much greater user experience.

Finally, the national analysis of Obama for America is interesting by itself but would be much more useful if it included data from other candidates. Being able to overlay zip code summaries from primary challengers like Hillary Clinton for President, and general election opponent McCain-Pailn 2008 Inc would allow a much greater depth of analysis to be performed. Including demographic information, primary dates, and election returns on the same visualization would be the best.

I intend to address these topics -as well as analyzing congressional races- in a future tutorial.

What the visualization tells us

Despite the flaws listed above after viewing the final output of our visualization we can draw several conclusions about the fund raising success of the Obama for America campaign. First, the campaign was truly national and had financial support from across the country by the end of the campaign. But the campaign didn’t start out national – during the critical first two months of the campaign Obama for America drew financial support from several metro areas including Chicago, New York City, and DC/Northern Virginia. After several months of heavy development of the grassroots network the campaign started seeing donations coming in from across the US.

Using the weekly summary data we can try to attribute large spikes in total receipts to campaign events. Looking closely we see the campaign saw a large increase of contributions in the last week of March, which potentially could be attributed to popular support of candidate Obama’s More Perfect Union speech or as a result of the large new donor push at the beginning of that month. We can also look at weeks 2008/4 and 2008/5(beginning of February 2008) and see week to week receipts jump by $5M to around $11M for two full weeks. This spike coincides with the Super Tuesday Democratic primaries where candidate Obama nearly split the day with then-front runner Hillary Clinton.

Conclusion

Using free disclosure data and a few open-source tools, like Ruby, fechell, R, and MySQL just about anybody can perform their own professional analysis on federal campaign finance data.

Resources

Database creation script
Database population script
R script to draw images draw.R
Zip codes
FLV file
CSV extract from Part 1

fechell is a ruby library used to extract data from electronically filed Federal Election Commission reports, saving you from the hell of parsing them yourself. These reports are filed by all political action committees, presidential and House candidates; at this time Senate rules don’t require electronic disclosure. Depending on the form type and filer, a report may contain summaries of receipts and disbursements, a 48-hour notice of independent spending, a statement of candidacy, and other finance or organizational notifications. In the 9 years of available data, the format of each of these reports has changed several times making it complex to uniformly parse data across multiple years and versions. The fechell library, using the field definition files provided by Watchdog.net, allows simple pragmatic access to the full library of FEC data. To demonstrate the features of the library we will use fechell to extract all itemized donations made to the Obama for America campaign in 2007 and 2008 by individuals. We will then analyze these data by importing them into a database and visualizing contribution amounts aggregated by zip code over time. Plotting this data on a map will show us clusters of financial support from individuals, as well as how those clusters grow and change over time.

Code books & formats for electronic FEC reports

The terminology detailing electronically-filed FEC reports is sometimes confusing. A report is a plain text file, usually with an extension of ‘fec.’ Every report contains a header line detailing the FEC format version and the software used to file the report, with values separated by a version-dependent separator character. After the header, each line corresponds to either a form (ex: F1,F3P,F24), or a schedule attached to the form (Schedule A, Schedule I). The first column of every line tells the reader what type of form or schedule is contained on that line. Every form and schedule is detailed in a workbook named FEC_Format_v6.4.xls available from the FEC as part of their free Vendor Tools package. You will want to download this package and have the FEC_Format_v6.4.xls (format workbook) available when processing reports with fechell. The format workbook contains one worksheet for each form or schedule, and each worksheet gives a column-by-column description of each value contained in the report. The description contains a standard name, as well as sample data and implementation notes. The fechell library uses this standard name to return values from a form/schedule to the calling program. The library will determine the correct separator character and field definition to use at run-time so all these details are hidden from the user.

The use and sale of data from these reports for commercial purposes, especially identifying information of individuals giving to campaigns, is regulated by the FEC. See http://fec.gov/pubrec/publicrecordsoffice.shtml#using.

Installation

the fechell library is hosted on github and is available as a gem.

You will need to add GitHub to your gems source if you haven’t already done so.

sudo gem sources -a http://gems.github.com/

Now install the fechell gem.

sudo gem install offensivepolitics-fechell

Download FEC reports

Using the FEC electronic filing search tool we can find raw reports as uploaded by candidates and committees.

In this case we’re looking for committee id C00431445, as this is the primary campaign committee for Obama For America. Contributions are itemized on form F3P, “Summary of Receipts and Disbursements by an authorized committee (pres / vice pres).” Using those query parameters you will see close to a hundred filings but we’re only interested in 15 of them.
The combined size of all Obama For America F3P filings is about 473 megabytes, and you can download the filings individually using the following URLs:

359390.fec- period 01/01/2007-03/31/2007, filed 08/22/2008 – APR QUARTERLY
359395.fec- period 04/01/2007-06/30/2007, filed 08/22/2008 – JUL QUARTERLY
359397.fec- period 07/01/2007-09/30/2007, filed 08/22/2008 – OCT QUARTERLY
360400.fec- period 10/01/2007-12/31/2007, filed 08/30/2008 – YEAR-END
360372.fec- period 01/01/2008-01/31/2008, filed 08/29/2008 – FEB MONTHLY
360401.fec- period 02/01/2008-02/29/2008, filed 08/30/2008 – MAR MONTHLY
362085.fec- period 03/01/2008-03/31/2008, filed 09/15/2008 – APR MONTHLY
358077.fec- period 04/01/2008-04/30/2008, filed 08/18/2008 – MAY MONTHLY
358076.fec- period 05/01/2008-05/31/2008, filed 08/18/2008 – JUN MONTHLY
406266.fec- period 06/01/2008-06/30/2008, filed 03/03/2009 – JUL MONTHLY
405793.fec- period 07/01/2008-07/31/2008, filed 02/27/2009 – AUG MONTHLY
405795.fec- period 08/01/2008-08/31/2008, filed 02/27/2009 – SEP MONTHLY
405796.fec- period 09/01/2008-09/30/2008, filed 02/27/2009 – OCT MONTHLY
405794.fec- period 10/01/2008-10/15/2008, filed 02/27/2009 – PRE-GENERAL
406271.fec- period 10/16/2008-11/24/2008, filed 03/03/2009 – POST-GENERAL

Save all these files into a directory.

F3P fields

fechell will give you access to any field in a report, but you need to know specifically which fields to look for. Using the format worksheet (FEC_Format_6.4.xls), we can easily find the correct field names and their possible values. In the previous step we downloaded 15 different filings, all type F3P “Summary of receipts and disbursements by an authorized committee (pres / vice pres).” A F3P filing contains the normal FEC header line, a summary line detailing fund raising for the current period, followed by any number of schedules. Using the Specification Requirements document (FEC_Format_6.4.pdf) that comes with the ‘FEC Vendor Tools’ download we can quickly determine which schedules belong to which forms. In Appendix C, page 22 of Specification Requirements we see that form F3P can contain Schedules A, B, C, C1, C2 and D. Using the format workbook as a reference for each schedule, we see that Schedule A (‘Sch A’) contains an itemized receipt from an individual or a committee. In our analysis, we’ll look for Schedule A filings and ignore everything else.

We want to visualize contribution amounts from individuals by zip code and contribution date, so we’ll extract 3 fields from each Schedule A: contribution amount, contribution date, and zip code of contributor. Schedule A can contain contributions from several types of entities, each identified by the “ENTITY TYPE” field. Looking through the “Sch A” worksheet of the format workbook we see we want column #6 (“ENTITY TYPE”), column #19 (“CONTRIBUTOR ZIP”), column #20 (“CONTRIBUTION DATE”), and column #21 (“CONTRIBUTION AMOUNT”). The names of these columns are all we’ll need to start extracting data.

Extract the data

Now we’re ready to load the FEC data files and extract individual contribution data.
First we need to load and initialize the FEC library

require 'rubygems'
require 'fechell'
 
h = FECHell.new

Next we’ll use fechell to load each file we downloaded. We will pass a filename to the process function, and fechell will return the schedule type for that line and all the values keyed by their name from the FEC worksheet for the corresponding schedule.

Note: You’ll need to change the Dir["./obama-fec/*.fec"] path to the directory with your downloaded FEC reports from step 2.

require 'rubygems'
require 'fechell'
 
h = FECHell.new
Dir["./obama-fec/*.fec"].each do |filename|
  h.process(filename) do |line|
    schedule = line[0]
    values = line[1]
  end
end

We want to process Schedule A lines (SA) , and we only want contributions from the IND (individual) entity type . We first check the schedule value returned by fechell to make sure we’re processing a “Schedule A” (SA) line. If we aren’t, then we move to the next schedule. To check the entity type that made the contribution, we look in the values hash with the key ‘ENTITY TYPE’ field we identified at the beginning of this step. Referring again to the FEC worksheet we see that possible values for column #6 (“ENTITY TYPE”) are “[CAN|CCM|COM|IND|ORG|PAC|PTY]“. In this case we’re only interested in contributions from individuals (IND) so we’ll skip processing the line if that value doesn’t match.

require 'rubygems'
require 'fechell'
 
h = FECHell.new
Dir["./obama-fec/*.fec"].each do |filename|
  h.process(filename) do |line|
    schedule = line[0]
    values = line[1]
 
    next if schedule != 'SA'
 
    next if values["ENTITY TYPE"] != "IND"
  end
end

Now we know we’ve got a Schedule A line for an individual so we can extract the actual data we care about. Again, the field names used below were taken from the FEC worksheet for Schedule A: “CONTRIBUTION AMOUNT”, “CONTRIBUTOR ZIP”, and “CONTRIBUTION DATE.”

require 'rubygems'
require 'fechell'
 
h = FECHell.new
Dir["./obama-fec/*.fec"].each do |filename|
  h.process(filename) do |line|
    schedule = line[0]
    values = line[1]
 
    next if schedule != 'SA'
 
    next if values["ENTITY TYPE"] != "IND"
 
    amount = values["CONTRIBUTION AMOUNT"].to_f
    fullzip = values["CONTRIBUTOR ZIP"]
    date_str = values["CONTRIBUTION DATE"] 
 
    next if fullzip.nil? == true
    next if date_str.nil? == true
 
    zip5 = fullzip[0..4]
 
    puts "On #{date_str} we received #{amount} from zipcode #{zip5}"
  end
end

Now just printing out the individual contributions is a great way to verify the library is working, but it doesn’t really do a lot for us as far as analysis is concerned. Instead of just printing information we’ll create a CSV file of all the individual contributions, suitable for import into a database. We next do some small cleanup and data verification, as well as adding some simple error handling since FEC data is notoriously poorly formatted or incomplete. This leaves us with the final version of our individual contribution extraction code:

require 'rubygems'
require 'fastercsv'
require 'fechell'
 
csv = FasterCSV.open("obama-fec-SA-2007-2008.csv","w")
csv << ["contribution_date","amount","contributor_zip"]
 
h = FECHell.new
Dir["./obama-fec/*.fec"].each do |filename|
  h.process(filename) do |line|
    schedule = line[0]
    values = line[1]
 
    next if schedule != 'SA'
 
    next if values["ENTITY TYPE"] != "IND"
 
    amount = values["CONTRIBUTION AMOUNT"].to_f
    fullzip = values["CONTRIBUTOR ZIP"]
    date_str = values["CONTRIBUTION DATE"] 
 
    next if fullzip.nil? == true || fullzip == ''
    next if date_str.nil? == true || date_str == ''
 
    date_val = Date.strptime(date_str,'%Y%m%d')
    next if date_val.nil? == true
 
    date_formatted = "#{date_val.year}-#{date_val.month.to_s.rjust(2,'0')}-#{date_val.day.to_s.rjust(2,'0')}"
 
    zip5 = fullzip[0..4]
 
    csv << [date_formatted,amount,zip5]
 
  end
 
end
 
csv.close

Running this code on the Obama for America FEC files produces around 2.8 million individual itemized contributions over two years. The output file is about 67 megabytes and is ripe to be imported into a database for further analysis. Which is exactly what we do in Part 2(link).