On Tuesday January 3rd 2012 the Iowa Republican party held it’s presidential caucuses, with Mitt Romney beating Rick Santorum by 8 votes as of noon on Jan 4th. This was an exciting race with multiple lead changes and entrance polling showing many late undecideds and large gaps in candidate support by age and income. In a nice twist for those of us who like playing with election data (and who doesn’t?), the IA GOP has published their election results using Google Fusion Tables. This allows us to skip the usually messy data scraping and piecing together that is par for the course.

In this post I’ll create heat maps to illustrate candidate support geographically, as well as differential heat maps to compare results between candidates. Everything is done in R and all the code and data are available from the github repository for this post. Pull requests for new analysis or fixes are greatly appreciated.

Data Wrangling

Electoral data is usually published as some combination of PDFs, HTML files, CSV files, ESRI shapefiles, . There is usually a fair amount of work getting names and IDs to match up between formats, but not this time. Thankfully the IA GOP published their caucus results as a Google Fusion Table. This allows easy online viewing, and export to other formats. You can ignore this section if you just want to hack on the provided result files. Otherwise, to produce the data files for this post I did the following:

1. Exported a CSV copy of the IAGOP election results with File->Export in the Fusion Table.
2. Exported a KML file with Iowa county outlines:
   1. View the Iowa election map with Visualize->Map
   2. Export the KML file by clicking the “Export to KML” link on the map page
3. Loaded the KML file into Quantum GIS and exported the layer as a Shapefile.

That looks like a lot of work, and it was sort of complex, but it is miles easier than taking shape and result data from different sources, or performing all those steps in code.

R code

The R code makes use of several awesome packages: maptools, ggplot2, RColorBrewer, and gpclib. I took code for preparing and plotting shape files w/ ggplot2 from the gglpot2 wiki. I lifted some ggplot2 theme code from Osmo Salomaa on the ggplot2 mailing list. This work would have been substantially more difficult without the packages and links listed above; thank you to their authors for making their work available for free.

Support Heat maps

First we’ll create heat maps of the vote percentage each candidate received by county in the Iowa GOP Caucuses in 2012. These maps show us strong and weak geographic areas for each candidate. The scales are identical for all candidates so comparing maps should be quite easy. In alphabetical order, and click through for a much larger version:

Bachmann

Gingrich

Paul

Perry

Romney

Santorum

Take away

What do these maps tell us? Bachmann, who finished last, performed poorly across the entire state. Perry, who finished 5th, had some strong counties in the south west but overall performed poorly as well. Gingrich, the front runner until two weeks ago and the 4th place winner, had no really strong counties anywhere. Ron Paul finished 3rd and enjoyed several counties with large support in the the north east of Iowa, and was a contender almost everywhere. Rick Santorum had one huge win and several other large county wins in the west, while Romney had more success in the eastern side of Iowa. Romney also had more counties where he won 30-40% of the vote than Santorum did.

Relative Heat maps

The raw support heat maps above are OK, but they aren’t terrific for comparing two candidate’s returns. Next we’ll create relative heat maps that show the difference in support by county for each of the top 3 candidates. We’ll also create a histogram of the same values since mapping the differences can sometimes distort the overall distribution of values.

Romney vs Santorum

Positive values show a greater Romney percentage than Santorum, and negative values show the opposite. For example: If Romney won a county with 25%, and Santorum won the county with 50%, the Romney – Santorum value would be -25.

Analysis:

The relative heat map between the first and second place candidates in the IA GOP 2012 caucus shows us a few interesting items. Santorum cleaned up in the upper northwest of Iowa, less so in the southwest, and mixed results everywhere else. The histogram gives us a clearer picture of the overall breakdown of the results, showing us few large percentage wins for either candidate and a spike of smaller percentage wins for Santorum. The number of Santorum wins were probably offset by the size of the counties that Romney won, which isn’t shown on either of these graphs.

Romney vs Paul

Analysis:

The first and third place candidates dont have a dramatically different map or distribution than the first and second. The map shows a few Paul strong points, but the vast majority of the counties were very close races. The histogram backs up this hypothesis, and shows larger win percentages for Romney in more counties.

Santorum vs Paul

Analysis:

Aside from the large win the northwest, it looks like Paul actually had more winning counties than Santorum. Both candidates did well, and were competitive throughout the state.

Wrapup

The heat maps and differential maps give us an unbiased view into how the Iowa Republican 2012 caucuses turned out. Each of the top 3 candidates was a contender in almost every county and had one or two things gone differently than anybody could have won the caucus. Please feel free to download and hack the code for this article from my github page. Thank you for reading, and please direct any questions or comments to me using the comment form below or via email at: jjh@offensivepolitics.net.

In this post I’ll perform an analysis of the voter file for Wake County, NC. I’ll follow the same processes as a political consultant, but I’ll be using R instead of traditional tools. The data and code for this post are all available on my GitHub repository, so a reader should be able to follow along and expand on this analysis.

What is a voter file?

A voter file is a list of every registered voter for some political area, like a county or a congressional district. The voter file is purchased from state parties, 3rd party vendors, PACs, or municipal offices. The file usually contains the name, address, and voting history of every registered voter. Campaigns can purchase extra data (“appends”) like phone numbers, email addresses, demographic information, or organization membership information. All these data are used to sort, group, and prioritize contact of a voter.

Who uses a voter file?

The voter file is used by candidates for political office or issue campaigns to understand their target constituents. An good analysis of the voter file will reveal strategic demographic and turnout information that may otherwise be hidden. Armed with this information a campaign will assemble a voter contact plan and build turnout projections, both of which are essential campaign processes. A candidate without a strong understanding of demographic and turnout information for his election will most likely waste campaign resources talking to the wrong people.

How would this normally be done?

• A small campaign will probably use MS Excel or Access to perform a precinct analysis and build lists and counts.
• For larger campaigns, the CRM system (NGPVAN, Aristotle) all have cross-tabs, lists, and counting, but are primarily used for contact and fundraising compliance.
• Very sophisticated campaigns will use something like the Q-tool from Catalist. This is a voter analysis tool providing data-mining and modeling capabilities, along with the standard counting. Q-Tool is extremely impressive.

Why use R?

From the R-project home page: R is a free software environment for statistical computing and graphics. The R programming language allows a wide range of analysis and visualization not available in traditional political tools. Data manipulation is easier on the messy and disjoint data we deal with in political analysis. The visualization tools (like ggplot2 and lattice) let us easily generate graphics suited to our exact data. Finally, R has excellent support for basic politics statistics like clustering and regression analysis, to say nothing of more advanced statistical tools multilevel modeling and simulation.

The Voter File

We will be exploring the voter file for Wake County, NC. Besides being free and updated regularly, this voter file has several unique features including voter gender, party registration, and absentee voting information. This level of data is almost unheard of in free lists, so our analysis will be very realistic. Wake County has both a strong absentee and vote by mail program, as well as early voting and same day registration. Wake County also allows voters not affiliated with a major party to vote in primary elections, but for only one primary per election season. The Wake County Board of Elections is very far above the average with elections administration and data disbursement in all areas save one. Why would they distribute the voter file as a self-extracting zip archive?

The toolkit

I’ll be using the Wake Count voter file downloaded from WakeGOV.com on Nov 21, 2011. I’ll be using 2.14 of R, along with the following packages: plyr, ggplot2, gmodels, and RColorBrewer. All code and a slightly trimmed-down voter file can be downloaded from my GitHub repository.

Cross tabs

The initial analysis will focus on understanding the demographic makeup and voting history of registered voters in Wake County. We’ll perform simple counts (cross-tabs) on different segments of the file to better understand the demographic makeup and voting history of registered voters in Wake County.

Voting Status

CrossTable(vf$status,prop.c=F,prop.chisq=F,format="SPSS",max.width=10)

Total Observations in Table:  595741 

          |   Active  | Inactive  |
          |-----------|-----------|
          |   533011  |    62730  |
          |   89.470% |   10.530% |
          |-----------|-----------|

Our first cross-tab tells us that 10.5% of the voters on our list are inactive. Wake County considers a voter inactive is mail has been returned from this address. Determining voter status ahead of time is usually an expensive or impossible task but Wake County has helpfully done this for us. Without this information a campaign would waste money sending mail or door knocking ad the wrong address.

Party Affiliation

CrossTable(vf$party,prop.c=F,prop.chisq=F,format="SPSS",max.width=10)

          |      DEM  |      LIB  |      REP  |      UNA  |
          |-----------|-----------|-----------|-----------|
          |   246641  |     1577  |   178676  |   168847  |
          |   41.401% |    0.265% |   29.992% |   28.342% |
          |-----------|-----------|-----------|-----------|

The party breakdown in Wake County shows a 12 point Democratic registration advantage, which is a solid lead. The nearly 1/3rd of voters who aren’t affiliated with a party could make for some very close elections as each major party aggressively courts the unaffiliated.

Gender

CrossTable(vf$gender,prop.c=F,prop.chisq=F,format="SPSS",max.width=10)

          | Female    | Male      |  Unknown  |
          |-----------|-----------|-----------|
          |   316958  |   273696  |     5087  |
          |   53.204% |   45.942% |    0.854% |
          |-----------|-----------|-----------|

The gender cross-tab tells is a majority of Wake County voters are female, and by a healthy 15% margin. This information will inform every level of communication by the campaign.

Age Group

CrossTable(vf$age.group,prop.c=F,prop.chisq=F,format="SPSS",max.width=10)

Total Observations in Table:  595741 

          |   [17,30) |   [30,40) |   [40,50) |   [50,60) |   [60,70) |
          |-----------|-----------|-----------|-----------|-----------|
          |   114391  |   122686  |   132329  |   108995  |    67758  |
          |   19.201% |   20.594% |   22.213% |   18.296% |   11.374% |
          |-----------|-----------|-----------|-----------|-----------|
          |-----------|-----------|-----------|-----------|-----------|
          |   [70,80) |   [80,90) |  [90,100) | [100,110) | [110,120) |
          |-----------|-----------|-----------|-----------|-----------|
          |    30337  |    15469  |     3599  |      172  |        5  |
          |    5.092% |    2.597% |    0.604% |    0.029% |    0.001% |
          |-----------|-----------|-----------|-----------|-----------|

The Wake County file has age information, which we've binned into roughly 10-year sized buckets. As much as gender, age of registered voters will play role in determing the policy goals and communication method used by a campaign. For example: a precinct with many older voters is a good candidate for an afternoon or early evening canvas since many of your targets will be home. But a precinct on a college campus or otherwise full of younger voters may be better contacted through alternative forms like social media or email.

Party affiliation, by Gender

CrossTable(vf$gender,vf$party,prop.c=F,prop.chisq=F,format="SPSS",max.width=10)

|-------------------------|
|                   Count |
|             Row Percent |
|           Total Percent |
|-------------------------|

             | vf$party
   vf$gender |      DEM  |      LIB  |      REP  |      UNA  | Row Total |
-------------|-----------|-----------|-----------|-----------|-----------|
           F |   146512  |      627  |    88295  |    81524  |   316958  |
             |   46.224% |    0.198% |   27.857% |   25.721% |   53.204% |
             |   24.593% |    0.105% |   14.821% |   13.684% |           |
-------------|-----------|-----------|-----------|-----------|-----------|
           M |    98497  |      932  |    89645  |    84622  |   273696  |
             |   35.988% |    0.341% |   32.753% |   30.918% |   45.942% |
             |   16.534% |    0.156% |   15.048% |   14.204% |           |
-------------|-----------|-----------|-----------|-----------|-----------|
           U |     1632  |       18  |      736  |     2701  |     5087  |
             |   32.082% |    0.354% |   14.468% |   53.096% |    0.854% |
             |    0.274% |    0.003% |    0.124% |    0.453% |           |
-------------|-----------|-----------|-----------|-----------|-----------|
Column Total |   246641  |     1577  |   178676  |   168847  |   595741  |
-------------|-----------|-----------|-----------|-----------|-----------|

The first two-way cross-tab is somewhat intimidating at first but is easy enough to read with a key. Gender is along the left side, and party affiliation is along the top. The three numbers in each box represent the raw count of voters, the percentage of voters in that row, and the percentage of voters overall. The first box tells us that there are 146,512 registered Female Democrats; that Democrats make up 46% of female voters; Female democrats are 24.5% of the total electorate.

This crosstab is full of useful information that would be invaluable for campaign planning in Wake County: The Democratic registration advantage is 19 points for women, but only 3 points for men. Me are more likely to be unaffiliated than women, indeed the unaffiliated voters almost make up a 3rd party by themselves. The Wake County Board of Elections says unaffiliated voters may still vote in a partisan primary, but only one party's primary per election. This quirk will make the unaffiliated voters huge targets of opportunity during primary elections.

Graphics

As we've shown Cross-tabs are very powerful tools but can get quickly cause information overload. This is where data visualizations (fancy word for charts) come in. We'll use the powerful ggplot2 library to do some quick visualizations on the Wake County voter file. There are many other R packages for data visualization, but the ggplot2 library works very well for our purposes.
Voter Registration, by Age Group

qplot(age.group,data=vf,type="histogram",main="Wake County Registered Voters, by Age Group ")

Wake County Registration By Age Group
Click To Enlarge

2010 Turnout by, Age Group

 qplot(age.group,data=vf[vf$regdate <= "2010-11-04",],type="histogram",fill=g2010.v,position="dodge",main="Wake Count 2010 Turnout, by Age Group") + scale_fill_brewer(name="Voted 2010", pal="Set1")

Wake County 2010 General Turnout by Age Group
Click To Enlarge

Turnout in 2010 vs 2008, by Gender

gender.turnout <- ddply(vf,"gender",function(x) { data.frame(total=nrow(x),turnout=c(sum(x$g2010.v),sum(x$g2008.v)),election=c("2010","2008")) })
qplot(gender,turnout / total, data=gender.turnout, geom="histogram", stat="identity",fill=election,position="dodge", main="Turnout in 2010 vs 2008 Wake County, by Gender") + scale_fill_brewer(name="Election cycle",pal="Set1")

  gender  total turnout election
1      F 316958  146812     2010
2      F 316958  227910     2008
3      M 273696  127999     2010
4      M 273696  188888     2008
5      U   5087    1470     2010
6      U   5087    2157     2008

Wake County Turnout 2008 and 2010 by Gender
Click To Enlarge

Previously we saw women were much more likely to be Democrats than Republicans, but didn't tell us anything about their turnout propensity. This chart shows us turnout percentages by gender, for the 2008 and 2010 general elections. We see turnout was much higher for both men and women in 2008 than 2010. Also that women turned out at a higher rate than men in 2008, but closer to par in 2010. While both genders turned out at around 45% in 2010, almost 40,000 more women turned out than men due to the registration disparity.

Change in turnout by precinct from 2008 to 2010
Until now we've been comparing very simple counts that could have been easily been done with cross-tabs. The next chart plots turnout percentage by precinct from 2008 against turnout percentage in 2010. With 200 precincts we would be hard pressed to easily visualize these data without some sort of chart.

precinct.turnout <- ddply(vf, "precinct", summarize, turnout2010=sum(g2010.v) / length(g2010.v), turnout2008=sum(g2008.v) / length(g2010.v),reg2010=length(g2008.v) )
 
qplot(turnout2010, turnout2008, data=precinct.turnout,xlim=c(.1,1),ylim=c(.1,1),main="Turnout percentage 2008 to 2010, by Precinct") + geom_abline(intercept=0,slope=1)

Wake County Turnout Percentage 2008 and 2010 by Precinct
Click To Enlarge

This chart shows turnout percentage in 2008 along the vertical axis, turnout percentage for 2010 along the horizontal axis, and a line representing equal turnout in both. Points above the line had higher turnout in 2008, while points below the line had lower turnout. None of the points are below the line, meaning all precincts turned out lower in 2010 than 2008. We knew this, but we didn't know if the decrease was inform across precincts. Now we do know the decrease wasn't uniform, and certainly there is a relationship between turnout in 2008 and turnout in 2010.
Given this type of relationship we will fit a simple linear regression against the data and see if we can quantify change further:

# fit the turnout with a simple linear model
> lm(turnout2008~turnout2010,data=precinct.turnout)
 
Call:
lm(formula = turnout2008 ~ turnout2010, data = precinct.turnout)
 
Coefficients:
(Intercept)  turnout2010  
     0.4124       0.6216  
# summary gives us an rsqured of 0.84 
 
# plot the same graph w/ a line of best fit
qplot(turnout2010, turnout2008, data=precinct.turnout,xlim=c(.1,1),ylim=c(.1,1),main="Turnout percentage 2008 to 2010, by Precinct") + geom_abline(intercept=0,slope=1) + geom_abline(intercept=0.4124,slope=.62)

Wake County Turnout Percentage 2008 and 2010 by Precinct (2)
Click To Enlarge

In the R code above we fit a simple linear regression to the precinct turnout data, which gave us an intercept and single regression coefficient. We used these values to plot a line of best fit on the same graph. From this chart we see that the change in turnout from 2008 to 2010 by precinct was almost uniform for most data.

Democratic registration % and precinct size by 2010 turnout
This final graph is a complicated one - We'll look at Democratic voter registration and turnout in 2010 by precinct. A campaign would use this chart to find precincts with high democratic registration but low 2010 turnout. A precinct with these characteristics will be near the top on a target list for a Democratic candidate to canvas. Finding precincts like these is a valuable part of the precinct analysis process that drives a campaign plan.

# use ddply to summarize registration and turnout by precinct
dem.reg.prec <- ddply(vf, "precinct", summarize, registered=length(status),turnout2010=sum(g2010.v),dem.pct=sum(party == "DEM") / length(party) )
# now plot it
qplot(dem.pct,turnout2010/registered,data=dem.reg.prec,alpha=I(0.8), main="Democratic registration percentage and 2010 turnout\n by precinct",xlab="Democratic Registration%", ylab="Turnout 2010 (All Parties)",xlim=c(0,1), ylim=c(0,1))

Wake County 2010 Turnout by Democratic Registration Percentage
Click To Enlarge

Points in the upper left side of the graph represent precincts with high turnout and low Democratic registration - those are precincts we'll want to ignore since they most likely voted for our opponents. Precincts in the lower right side mean low turnout and high Democratic registration - those are the precincts we'd want to target most heavily for turnout efforts in 2012. Precincts with turnout below 50% with a Democratic registration percentage of 50 or greater will probably be next on the list for canvassing efforts.

Wrapping up

Thank you for reading this simple overview on performing a voter file analysis using R. The Wake County voter file is full of interesting information, we've just barely scratched the surface and I encourage the curious user to explore the file themselves. Thanks to the Wake County Board of Elections for keeping such a high quality file free and up to date. Please don't hesitate to leave any feedback in the comments below, via Follow me on twitter.

I was sure I double counted, or the candidate misfiled something, but after looking at every report I believe the number is correct. The filing’s for this candidate are a mess; several loan amounts are reported twice, and every single Form3 filing had to be amended, four times in his pre-special filing. All the fec forms for candidate Huey can be found here, and forms with loans by or contributions from the candidate can be found below.

I looked at two different types of filings to find this information: I used the Schedule C line 10 documents from Form 3 (notice of receipts and disbursements) quarterly, pre-special, and pre-runoff filings. I also used the Form 6 (48-hour notice) filings that have been piling up since 6/24.

Huey certainly isn’t the only candidate to finance his own campaign, but I think this might be a record for a runoff election in the US House. Anybody have any information on that?

Individual contributions are not required to be disclosed by a campaign unless the individual donates more than $200 during a single electoral cycle. The Smith for Congress campaign has, for their own reasons, published every individual contribution. This disclosure allows us an unprecedented look into how a modern campaign raises money. I’ve collected and scrubbed these contributions and published them for research use. In this post I will perform a detailed donor analysis on with R to better understand how the Smith for Congress campaign financed its 2010 election. Full code and graphs can be found on the simple-analysis github repository for this post:

Prepartion

We need to download the data and load it into R. The latest data can always be downloaded from: Smith for Congress Latest

# latest smith for congress data as of this writing is March 23 2011.
cd <- read.csv("smithforcongress-03232011.csv")
#subset the data to just the 2010 cycle
cd0 <- cd[cd$cycle == 2010,]
# clean up a date variable, and drop amounts < $1. 
cd$contribution_date <- as.Date(cd$contribution_date,format="%m/%d/%Y")
cd0 <- cd0[-which(cd0$amount < 1),]

Data for the 2010 electoral cycle consists of 11,721 contributions made by 6949 individuals, totaling over $770,000. Here is a sample:

Data Summary

Since the number of individual donors (6,949) is so much lower than the number of contributions (11,717) we can guess a good portion of those donors gave multiple times. The long-form contribution data is somewhat difficult to work when looking at multiple contributions from the same person. We’ll generate a summary data frame to help with our analysis. The following variables will be captured per individual donor:

• Date of first contribution
• The total value of all contributions by this individual
• The total number of contributions by this individual
• The amount of the first three contributions. Blank or NA if they have made less than 3 contributions.
• The difference in time for the first three contributions. Blank or NA if they have made less than 3 contributions.

summarize.contributions <- function(x) {
  xo <- x[order(x$contribution_date),]
  dtx <- as.integer(diff(x$contribution_date))
 
  return(data.frame(
		first.contribution=xo$contribution_date[1], 
		num.contributions = nrow(xo),
		dt1=dtx[1],
		dt2=dtx[2],
		dt3=dtx[3],
		am1=xo$amount[1],
		am2=xo$amount[2],
		am3=xo$amount[3],
		total.value=sum(x$amount)
	))
}
cd0s <- ddply(cd0, "personid", summarize.contributions)

Now the cd0s data frame holds our summary table, which looks like this:

Giving Levels

With detailed giving levels we can infer a lot of information about a campaign, and about how the fundraisers are doing their jobs. If most of the giving was in the $15-20 range we can assume they focus on small donors and maybe online contributions. If most of the giving is in the $100-250 range then maybe the campaign throws lots of medium sized dinners. If most of the donations are close to the legal maximum of $4800 then the campaign is focused on major donors, and might be ignoring smaller donors all together.

Plotting a histogram of total donation amount per individual will give us better insight into the giving levels.

> qplot(total.value,data=cd0s,geom="histogram",binwidth=50)
nrow(cd0[cd0$amount<250,]) / nrow(cd0)
summary(cd0s$total.value)

Giving Levels, Smith for Congress 2010

In 2010, 75% of contributors gave $100 or less total to the campaign. The summary table shows us the median total value donated was $50, while the overall average was $111. The maximum was $4800, which is also the maximum allowed by law for 2010. We can infer that while there was certainly some major-donor solicitation, the fundraisers were focused on much smaller donors.

Repeat donors

Now that we know more about giving levels, it would be helpful to better understand giving frequency. The amount of repeat giving may give us insight in to how involved the fundraisers are getting, and maybe even how often they are asking for money.
We’ll use a histogram and a cross-tab of the total number of contributions by individuals to help us with this analysis:

qplot(num.contributions,data=cd0s,geom="histogram",binwidth=1)
table(cd0s$num.contributions)

Giving Frequency, Smith for Congress 2010

Our plot and table shows about two thirds (61%, 4,242) of the contributors to Smith for Congress only gave one time, leaving 2,707 people who gave more than once. Most of the people who gave more than once gave twice, but there were still several hundred people who gave 3 or 4 times each.

To understand how important repeat giving might be we need more detailed information. We need to look at the total amount donated by each group of contributors; we’ll also include the cumulative total, cumulative percentage, and individual percentage of total for each group.

gft <- ddply(cd0s,"num.contributions",function(x) { data.frame(total=sum(x$total.value),n=nrow(x))})
gft$percent <- gft$total / sum(gft$total) * 100
gft$running.total <- cumsum(gft$total) 
gft$running.percent <- gft$running.total / sum(gft$total) * 100

Our gft data frame looks like this:

We see the campaign raised $284,000 (36.8% of the total raised) from the 4,242 contributors that gave only once, and $212,000 (27.5% of the total raised) from the 1,599 contributors who gave two times. We also see the campaign raised $487,378 from 2,702 repeat donors; that is almost 64% of the total value raised for the entire cycle from individuals. It is obvious the Smith for Congress campaign is good at attracting small dollar donors, one-third whom gave more man once. This is a pretty impressive repeat donor rate.

Finally I’d like to look at what kind of donations make up each level of giving. We know repeat donors gave $487,000, but we don’t know if that was mostly in $50 donations or in $250 donations. We can use a box and whisker plot to break down each giving level. I’m leaving off contribution levels 8 – 14 since giving was so sparse at those levels. We’ll be plotting this histogram with a log transform on the y axis since few very large values will skew graph and render it mostly useless. I used a trick from this stack overflow thread to get the formatting correct on the Y axis:

formatBack <- function(x) paste(round(10^x, 2), "$", sep=' ') 
qplot(factor(num.contributions),log10(total.value),data=cd0s[cd0s$num.contributions < 8,],geom="boxplot",ylab="Total Value (log)",xlab="Giving Frequency",main="Giving Levels by Giving Frequency, Smith for Congress 2010") + scale_y_continuous(formatter=formatBack)
# same data, but in table format 
ddply(cd0s,"num.contributions",function(x) { data.frame(total=sum(x$total.value),n=nrow(x), min=min(x$total.value),mean=mean(x$total.value), median=median(x$total.value),std=sd(x$total.value),max=max(x$total.value))})

Giving Levels by Giving Frequency, Smith for Congress 2010

This latest plot and table are both incredibly text heavy, but this is the critical intelligence required to start a fundraising plan.

We see the average total contribution increases with the giving frequency, this makes sense. The average increases in an approximately linear fashion which suggests the individual contribution amounts are staying constant. This may be a function of some campaign fundraising tactic, like “donate $35 now for a free tshirt.” We can also get a sense of how much success the Smith for Congress major donor program enjoys. An individual can legally donate $2,400 for both a primary and a general election per cycle. We can count how many individuals have maxed out at $4800 and measure how much impact the major donors have on the total amounts raised:

# how many individuals gave the max for one election
nrow(cd0s[cd0s$total.value == 2400,])
nrow(cd0s[cd0s$total.value == 4800,])

We see 7 individuals who gave the maximum for one election, and only 2 individuals who maxed out for the entire cycle. The maxed out donors make up only 1.2% of total giving; this is very low for the average campaign. This tells us major donors aren’t the most important segment to Smith for Congress, but it could also mean that the campaign isn’t able or isn’t willing to ask the max amount from large donors.

Take Away

We can take away the following facts from our analysis:

• 40% of individual donors gave more than once to Smith for Congress
• 80% of donors gave $100 or less to the campaign
• Repeat donors gave $487,000 total to the campaign
• Two out of 6,949 (0.028 percent) donors gave the maximum amount allowable by law for a total of 1.2% of the total amount raised

From all this we can infer that Smith for Congress is running a very strong repeat donor program, and isn’t focused on only high-dollar donors. This information could be very useful in a number of different ways. A treasurer for Smith for Congress could use this information to design a 2012 fundraising plan and campaign budget. A candidate similar to Smith, or running in a similar district, could use this same information to plan their own campaign. Or a rival campaign could use this during opposition research and financial planning. Or researchers could use this to build better generic models of US House individual fundraising. I hope this shows that detailed campaign finance analysis is pretty simple when you’ve got access to the relevant data, which unfortunately is very uncommon.

Thanks for reading, questions or comments are always appreciated: jjh@offensivepolitics.net

In the United States, a candidate for federal office can accept donations to fund their campaign from many different sources such as individuals, political action committees, and political parties. There are limits placed on how much an individual, committee or party can contribute, and there are rules for how those contributions are disclosed to the public. These rules and limits and enforced by the Federal Election Commission.

According to the 2012 FEC Contributions Brochure, an individual is allowed to contribute up to $2,500 to a given candidate in the 2012 general election cycle. If an individual gives $200 or more to a campaign in a single cycle, the campaign is required to disclose the donor’s name, address, occupation, and employer on a public donor list. If an individual gives less than $200, their personal information is not disclosed anywhere.

In FEC terminology, a donor who exceeds the $200 disclosure limit is called “itemized”, but otherwise is considered “unitemized.” A campaign discloses to the FEC, among other things, how much of their individual contributions came from unitemized donors and how much came from itemized donors.

The unitemized amount for congressional campaigns who raised at least $10,000.00 from individuals in 2010 represented on average 21% of their total individual contributions (n=1391,stddev=17%). Without the unitemized transactions we’re missing 1/5th of the total contributions of most campaigns. Without the unitemized transactions, we cannot properly build models to analyze small dollar fund raising. Tech President has an excellent writeup on small donors in 2008 and some useful links to other small donor research.

New Data

Since I spend a lot of time digging through FEC reports, I wrote some software that sort of helps. While do some unrelated research I found that one campaign was disclosing transactions that they didn’t need to disclose. Instead of only disclosing contributions above the $200-per-cycle aggregate amount, they were disclosing every single individual contribution.

As far as I can tell they are the only campaign to do so, ever, and this level of contributor data has never been made public. Big-time important congressional researchers like Fenno or Mayhew probably have access to data like this, and there may be some old stuff in ISPCR, but regular people like myself have never seen this. So I wrote some software to collect, clean, and verify the transactions and now I’m releasing it to the public.

Smith For Congress

I’m calling this data set Smith For Congress, it will live at the Smith for Congress homepage. The initial data set contains 3 election cycles (2006-2010) of individual transactional data, 49880 transactions in total. Transactions for less than $.10 were culled from the list, as were a handful of misclassified returned contributions.

The set contains five fields:

• individual identifier
• contribution amount
• contribution date
• cycle-to-date total contribution for this individual
• election cycle

The individual identifier is consistent across cycles, and used in place of a name. It is generated by running the name and address of the contributor through a standardization process, and hashing the result.

With the full transaction data provided in this data set we can perform analysis on how much, and how often donors make contributions. This can lead to better models to build better metrics about campaigns and individual giving. These metrics can help us more fully understand how congressional campaigns raise money and the role of individuals in them.

What is not included

The name of the candidate filing the reports; Smith for Congress is a code name. I’m also not releasing individual names or addresses of contributors, as they are not important for the research goals I had for putting this set together. It is all there in the FEC documents, and it shouldn’t take somebody more than 10 minutes if you feel like looking for either omission.

Finally

I believe the Smith for Congress campaign is a fair approximation for individual donations for a certain type of campaign, but this data is from one of thousands of congressional campaigns every cycle. Assumptions and models built from this data set may not hold true for other campaigns, but should help us as researchers better understand the dynamics of individual fundraising in congressional campaigns.

Over the coming weeks I’m going to be putting up short posts about different types of analysis that can be performed on these data. I will provide back links to the new posts here, but you should subscribe to the blog’s RSS feed or follow me on twitter for updates.

Data can be downloaded here: Smith For Congress

Please do not hesitate to contact me at jjh@offensivepolitics.net with comments or questions.

Researcher Drew Conway beat me to it with stalkR, but my code is sufficiently different that I think people can learn from both. I’ll walk through the code, links to the github repo are at the end of the post.

Since the location database is stored inside an iOS backup, we’ll need to understand the structure of that backup. The backup contains a bunch of files named with a long hex string, and a few files that provide a binary table of contents. There is some nice python code (iPhone Backup Decoder to open up the table of contents and locate specific files. I was going to translate this code to R, but I decided on a brute-force approach instead. The file we’re looking for is a SQLite database, and contains several unique tables. I just try to open every file in a given backup directory as a SQLite database, and look for a known table name (CellLocation). If the file isn’t a database or the table doesn’t exist then we move on.

library(RSQLite)
library(RgoogleMaps)
 
findLocationDB <- function (basePath) {
	filename <- NA
	drv <- dbDriver("SQLite")
	for(testFileName in list.files(basePath) ) {
		# brute-force the connection
		con <- dbConnect(drv,paste(basePath,testFileName,sep=''))
		# try and list the tables. 
		# this will fail if the file is not a sqlite db
		tableList <- tryCatch(dbListTables(con), error=function(e) e)	
		# if class of tableList is character then we've got a sqlite DB 
		if(any(class(tableList) == "character")) {
			# look for the CellLocation table
			if(length(grep("CellLocation", tableList))>0) {
			# we've found it. save this filename
				filename <- paste(basePath,"/",testFileName,sep='')
				dbDisconnect(con)
				break
			}
		}
		dbDisconnect(con)
	}
	dbUnloadDriver(drv)	
	return(filename)
}

Now that we’ve got a function to find a location database, we can access the database and load it into a data frame.

fetchLatLongTimestamp <- function(dbLocation,loc.table.name,accuracy=1.0) {	
	ldata <- NA
	con <- dbConnect("SQLite", dbLocation)
	ldata <- dbReadTable(con, loc.table.name)
	# drop data where lat == 0.0 && long == 0.0
	ldata <- ldata[-which(ldata$Latitude == 0.0 & ldata$Longitude == 0.0),]
	# convert the mac timestamp to unix timestamp
        ldata$datetime <- as.POSIXlt(ldata$Timestamp, origin="2001-01-01")
	# downsample the lat long by accuracy to obscure the location
	ldata$Latitude <- ldata$Latitude / accuracy
	ldata$Longitude <- ldata$Longitude / accuracy	
	ldata <- ldata[,c("Latitude","Longitude", "datetime")]
	dbDisconnect(con)
	return(ldata)
}

This fetchLatLongTimestamp function will load the entire location database into a data frame, and then clean up the timestamps and remove bad location data. I had originally seen the time stamp correction code on Prof Jackman’s blog, so thanks to him for that (and pscl!).

Now we’ve got a data frame of Latitude, Longitude, and datetime stamp that looks more or less like this:

The Lat/Lon represents downtown DC, near where I bought my iPhone last month.

Now that we’ve got a data frame full of juicy location data, we need to plot it on a map. I used the fantastic RgoogleMaps package, and ripped most of the vignette (pdf: RgoogleMaps: An R Package for plotting on Google map tiles within R) for loading a map and plotting points by latitude and longitude.

If I’ve got my location data in a data frame called ldata, I can use the following to find the correct bounds and zoom level, fetch a map, and plot my location data. Again, the drawing code is basically ripped from the RgoogleMaps vignette.

## plot a map of all the positions
	bb <- qbbox(ldata$Latitude, ldata$Longitude)
	# zoomlevel 4 works for my data (US only) 
	zoomlevel <- 4
	# grab the map
	map <- GetMap.bbox(bb$lonR, bb$latR,zoom=zoomlevel,maptype="mobile")
	# plot the points as circles 
	PlotOnStaticMap(map,lon=ldata$Longitude,lat=ldata$Latitude,col="blue",verbose=0)

Which gives us:

Obviously I spent a lot of time in Washington, DC, New York, Boston, and Las Vegas. We’re just using R, I can easily slice and dice the data. Let’s say I just wanted to see my Las Vegas data (April 1st – April 4th):

	ldata.lv <- ldata[which(ldata$datetime >= as.POSIXlt('2011-04-01 23:00:00') & ldata$datetime <= as.POSIXlt('2011-04-04 14:00:00')),]
	bb.lv <- qbbox(ldata.lv$Latitude, ldata.lv$Longitude)
	# zoom level of 12 center nicely on the las vegas strip
	zoom.lv <- 12
	map.lv <- GetMap.bbox(bb.lv$lonR, bb.lv$latR,zoom=12,maptype="mobile")
	PlotOnStaticMap(map.lv,lon=ldata.lv$Longitude,lat=ldata.lv$Latitude,col="blue",verbose=0)

Which gives us:

Yes, I spent a lot of time at the Wynn, Caesar palace, and In n Out Burger.

Here is the full driver code:

## change this to the full path of a backup of an ios 4 device
backupPath <- "C:/Documents and Settings/YourUser/Application Data/apple computer/MobileSync/Backup/a6ddb1824738f61a15b3e3c87e3e8172599b7134/"
 
dbLoc <- findLocationDB(backupPath)
 
if(!is.na(dbLoc)) {
	print(sprintf("Found location database in path: %s!",dbLoc))
 
	## for Verizon phones
	# locs <- fetchLatLongTimestamp(dbLoc, "CdmaCellLocation")
	## for AT&T phones
	ldata <- fetchLatLongTimestamp(dbLoc, "CellLocation")
 
	## plot a map of all the positions
	bb <- qbbox(ldata$Latitude, ldata$Longitude)
	# zoomlevel 4 works for my data (US only) 
	zoomlevel <- 4
	# grab the map
	map <- GetMap.bbox(bb$lonR, bb$latR,zoom=zoomlevel,maptype="mobile")
	png("all-tracks.png", width=640,height=640)
	# plot the points as circles 
	PlotOnStaticMap(map,lon=ldata$Longitude,lat=ldata$Latitude,col="blue",verbose=0)
	dev.off()
 
	## limit the data to 4/1-4/4. I was in las vegas at the time.
	ldata.lv <- ldata[which(ldata$datetime >= as.POSIXlt('2011-04-01 23:00:00') & ldata$datetime <= as.POSIXlt('2011-04-04 14:00:00')),]
	bb.lv <- qbbox(ldata.lv$Latitude, ldata.lv$Longitude)
	# zoom level of 12 center nicely on the strip
	zoom.lv <- 12
	map.lv <- GetMap.bbox(bb.lv$lonR, bb.lv$latR,zoom=12,destfile="lv.png",maptype="mobile")
	png("lv-tracks.png",width=640,height=640)
	PlotOnStaticMap(map.lv,lon=ldata.lv$Longitude,lat=ldata.lv$Latitude,col="blue",verbose=0)
	dev.off()
 
} else {
	print(sprintf("Could not find location database in path: %s",backupPath))
}

You can see this code on my iPhone location with R github repo. One big missing feature from the original application is animation, which I may add later. Patches and comments are greatly appreciated!

In a previous post on the Chicago Mayoral primary I looked at plotting returns on maps as a way to better understand the outcome. Maps help us visually determine if there is a geographic or clustering component to returns, but they aren’t the most rigorous way to compare election returns.

Another way to view the returns is a to use a Seats-Votes plot, like I did in my 2010 election returns blog post. Quickly, the Seats-Votes is a plot of a smoothed Gaussian kernel density estimator. Please refer to the Visualizing US House Election Returns post for an annotated example plot and more general information.

We’re going to create a Precincts-Votes plot of the Chicago Mayoral Democratic primary election returns. We will see a smoothed curve showing an estimation of the number of precincts with returns at a given percentage. These types of curves have been traditionally used in a two party race, meaning 50% is the cutoff for a win. But the Chicago Majoral Primary is a 6-way race, so a simple plurality is all that is required to win. Things are further complicated since the overall winner must receive 50% of the overall vote to avoid a runoff. I’m going to explore win percentages in this race from a multi-candidate perspective in a future blog post, but still keep it in mind while looking at these plots.

Updated R code and plots are available on the Chicago Mayor 2011 github page.

Data Preparation

The election returns from the previous blog post are in a wide format, one line per precinct. To effectively plot these we’re going to need to convert this to a long format. We’ll use the melt function.

# take the wide df variable and melt it to a LONG format
df.m <- melt(df,"WARD_PRECINCT", 
   c("emanuel_pct","delvalle_pct","braun_pct","chico_pct","watkins_pct","walls_pct"))
head(df.m)

Now our data looks like this:

Plots

The Winner

First we’ll make the precincts-votes plot for the winner of the election, Rham Emanuel.

qplot(emanuel_pct,data=df,geom=c("density","rug"),
      main="Precincts-Votes Curve, Chicago Mayor 2011",
      xlab="Vote %",ylab="Density")

Precincts-Votes curve, Emanuel Only, Chicago Mayoral 2011

This plot is very information-heavy, but can be decoded pretty easily. We see a large bubble of precincts where Mr. Emanuel received between 50 and 65 percent of the vote, and much fewer precincts in the 20-40 percent range, and very few at either extreme. The large spike near 60% implies Mr. Emanuel performed better in many precincts than his 55% overall vote total would lead us to believe.

All Candidates

Now we’ll create a precincts-votes plot for all candidates combined, but we’ll leave off the rug plot along the bottom. This will allow us to compare returns for all candidates at once.

qplot(x=value,group=variable,color=variable,data=df.m,geom="density",
		main="Precincts-Votes Curve, Chicago Mayor 2011",xlab="Vote %",ylab="Density") + 
		scale_color_brewer(name="Candidate") + geom_vline(x=50)

Precincts-Votes curve, All Candidates, Chicago Mayoral 2011

This combined precincts-votes chart is not as useful at comparing returns as I would like. Several of the candidates received near zero votes in many precincts, causing the scale to skew heavily towards larger values. We’ll drop the 3 worst performing candidates and build the chart again:

Top 3

qplot(x=value,group=variable,color=variable,data=subset(df.m,variable != 'walls_pct' & variable != 'braun_pct' & variable != 'watkins_pct'),
 		geom="density",main="Precincts-Votes Curve, Chicago Mayor 2011",xlab="Vote %",ylab="Density") + 
 		scale_color_brewer(name="Candidate", labels=c("Emanuel", "Del Valle", "Chico"),
		breaks=c("emanuel_pct", "delvalle_pct","chico_pct")) + geom_vline(x=50)

Precincts-Votes curve, Emanuel, Del Valle, Chico, Chicago Mayoral 2011

This chart is quite a bit better than the last. Candidates Emanuel, Chico and Del Valle received 55%, 23% and 9% of the overall vote respectively and I think this chart helps us better understand these totals. We see Del Valle underperformed in the vast majority of precincts, but was still competitive in several with returns between 20 and 40 percent. Candidate Chico has a large grouping in the 10% return range, whichis interesting given that he received 23% of the overall vote total. His ten percent wouldn’t win him a precinct, but it would keep him competitive in the overall total. This is important given that a simple plurality may win a precinct in a 6-way race. We can also see the candidates were all moderately competitive in many of the precincts around the 20-40 return range, which is what you would expect an average return to be for a contested precinct.

I hope this post has shown that precincts-votes curves can still be informative in a multicandidate race, and helped us better understand the makeup of the Chicago Mayoral Democratic primary. In my next blog post I’ll look at a way to visualize precinct returns in a multi-candidate race and how to measure the overall competitiveness of an election.

Code, data, and output are available on the Chicago Mayor 2011 github repository. Comments, questions, and pull requests are greatly appreciated.

Chicago held a Democratic primary on Feb 22nd, and I wanted to visualize the results of the 6-way Mayoral race. I’m a huge fan of mapping election results with R, but I was never able to get my favorite graphing library ggplot2 working correctly with shapefiles. After some light googling I found an excellent ggplot2 wiki page that described the whole process. Thanks to whomever put that page together, it was a really helpful resource.

In this blog post I will detail how I joined election results and maps to create a graphical summary of the mayoral primary election.

Data Sources

Precinct Boundaries

Chicago is split into 50 wards, and those wards are each split into a number of precincts. The city of Chicago helpfully provides precinct-level boundary shape files on their GIS data page.

Election Results

I pulled the precinct results from the Chicago board of elections results page. These data were provided in easily parsed HTML pages, unlike so many other states (Virginia I’m looking at you).

Data Preprocessing

Shape files

The precinct shapefiles needed no preprocessing, which is quite surprising. Usually municipality names are off, or data are missing, but not this time.

Election Results

I used Ruby to scrape each precincts results and saved them off to a CSV file. The individual files were loaded into Nokogiri and data was easily extracted with Xpath queries. The Ruby file to perform the preprocessing can be viewed here

Graphing

I made use of several external libraries when building the maps. The shapefiles were loaded with the maptools package, and then converted and plotted with ggplot2 and colored by RColorBrewer. The full R code w/ some inline comments can be found here

Results

Given the multi-party nature of this election I chose to create a turnout map for each candidate. This will allow us to more easily see the regional strengths of each candidate. Click an image below to view full-size.

Rham Emanuel with 55.2% of the overall vote	Gary Chico with 23.9% of the overall vote
Miguel Del Valle with 9.2% of the overall vote	Carol Moseley Braun with 8.9% of the overall vote
Patricia Van Pelt Watkins with 1.6% of the overall vote	William Walls III with 0.9% of the overall vote

Grab the code from Github: Chicago Mayoral 2011 code. Comments, forks, and pull requests, are greatly appreciated.

Edit: If you enjoyed this post please check out my basic precinct analysis on the Chicago Mayoral election.

This exercise will explain the different components of seats votes plot, and then look at how Democratic party support has changed from 2002 to 2010.

Seats Votes Explained.

Though it may look simple, the modified seats-votes curve is very information-heavy and can be somewhat confusing. A small change in the contours of the curve can convey a lot of information. Please refer to the annotated graph (figure 1), and the items below for instructions on how to read a seats votes-curve.

Figure 1

• 1) Uncontested Seats -The magnitude of the curve near 100 and near 0 represents the number of weakly contested or wholly uncontested seats. The curve near the left side of the graph measures how many seats were weakly contested by the Democrats, and the right side measures seats uncontested by the Republicans.
• 2 & 3) Base Seats- The lump of seats to the left of the 50% vote share line represent the seats a Republican won (2), the lump to the right belongs to Democrats (3). These lumps are the respective parties base and considered a probable win. The further from the 50% line shows how large a win it was, and how much safer a seat could be considered. A shift away from the 50% mark from one year to the next implies an electorally stronger base, and an increase in magnitude represents a larger base. A shift away and a smaller magnitude can represent an electorally weaker and smaller base, respectively.
• 4) Competitive SeatsSeats at or near the 50% mark are considered competitive seats. This portion of the curve may be a trough between the two base seat points, or sometimes a lump on its own.
• 5) Rug plotThe 1-dimensional rug plot across the bottom of the graph provides a density estimation at a given point in the curve. The more hash marks in a given location mean a higher count for that vote share, which influences the height of the density curve.

Latest Returns

Now that we can interpret the plot in terms of base, competitive, and safe seats lets look at the modified seats votes plot from the previous article but with the latest election returns.

library(ggplot2)
# load the seats-votes data from offensivepolitics.net
df <- read.csv('http://offensivepolitics.net/data/seats-votes-2010.csv')
df$year <- as.factor(df$year)
png("2010-f2.png",width=672,height=671)
qplot(voteshare,data=df, geom=c("density", "rug"),
	xlab="Democratic Vote Share (%)", ylab="Density",
	main="Democratic Vote Share US House 2002-2010") + 
	facet_wrap(~year,nrow=3) + 
	geom_vline(xintercept=50,colour="gray50")
dev.off()

Using the information from annotated reference chart (Figure 1.) to interpret to contours of the seats votes curve we can build a narrative for the mood of the electoral for any given year. Using the latest results chart (Figure 2.) we can expand that narrative across electoral cycles.

Figure 2

For elections leading up to 2010 we see a prominent bimodal curve, with a reasonably stable amount of safe seats for each party. In 2002 and 2004 we see a large number of seats in the 30-40 percent Democratic vote share, which corresponds to a strong Republican majority. In 2006 we see the number of competitive seats increase, and 2008 we see an increase in the number of safe Democratic seats. The changes in 2006 and 2008 track nicely with the Democrats narrowly control of the US House in 2006 and then strongly expanding their majority in 2008.

The 2002-2008 plots show a Republican between 15-20%, a Democratic base between 15-18%, and competitive seats between 9-15% of the total US House. In 2010 the structure of the curve changed dramatically. The Republican base seats were right around 18%, which is in line with previous years. But the Democratic base seats look nothing like previous years. There is still a contingent of the lump of safe Democratic seats, but the count dropped by about half. The rest of the seats shifted towards the 50% line where they merged with the other competitive seats. This plot of the returns lines up nicely with the pre-election narrative of an outsize number of tossup races by CQ Politics, FiveThirtyEight, Cook Political Report, and The Rothenberg Political Report.

More Precise Numbers

The smoothed density function used in the seats-votes plot is an estimation, so it is difficult to determine exactly how many seats fall within a given vote range. We could use a histogram, but I like a cummulative distribution function (CDF) plot instead. To create a CDF plot for each years results we’ll use the built-in R ECDF function and ggplot:

png("2010-f3.png",width=672,height=671)
cdf <- ddply(df,"year",function(x) data.frame(share=x$voteshare,cdf=ecdf(x$voteshare)(x$voteshare)*100))
qplot(x=share,y=cdf,data=cdf,geom="step",
	 main="Democratic Vote Share US House 2002-2010 (Cummulative)",
	 xlab="Vote Share (%)",ylab="Total %") + 
	facet_wrap(~year,nrow=3)
dev.off()

Figure 3

Using the CDF plot we can see that Democrats received 50% or less in 60% of the races in 2010, but received 50% or less in only 42% of the races in 2008. In 2010 a full 40% of the total races fell into the competitive category, defined as received between 40% and 60% of the vote. In 2008 and 2006 that number was closer to 12%, an increase of 50% in a single year.

The combination of the seats-votes plot and CDF allow us pretty powerful insights into the current electoral power of each major party in the US House. We have hard numbers and a narrative for the 2010 US House Democratic loss that goes beyond parroting the number of seats lots. We also have some historical perspective on major-party electoral returns, and it will be interesting to see if the 2010 competitive seats remain that way in 2012.