New Haven 1998 Experiment (APSR)

This site contains replication files for:
Alan S. Gerber and Donald P. Green (2004), “Correction to Gerber and Green (2000), Replication of Disputed Findings, and Reply to Imai (2005).” American Political Science Review 99 (2): 301-313.
The replication files and website were prepared by Kevin Arceneaux (kevin.arceneaux@yale.edu)
This essay responds to the following essay by Professor Imai:
Imai, Kosuke. (2005). “Go Get-Out-The-Vote Calls Reduce Turnout? The Importance of Statistical Methods for Field Experiments.” American Political Science Review 99 (2): 283-300.
Other versions of the paper submitted to the American Political Science Review

Technical Notes:

Technical Note on Monte Carlo Simulations of Randomization Checks
Do file for simulation of bias introduced by checking randomization at the individual level
Log file for simulation of bias introduced by checking randomization at the individual level
Do file for simulation of bias introduced by checking randomization by including interaction terms
Log file for simulation of bias introduced by checking randomization by including interaction terms
Technical Note on Monte Carlo Simulation of Small Sample Bias Associated with Two-Stage Least Squares
Technical Note Responding to Professor Imai’s Criticisms of Factorial Design

Data Files for Matching Analysis

2002 release of the Individual level New Haven data file analyzed by Prof. Imai.
Data for ITT matching analysis in R using the same sample restrictions as imposed by Prof. Imai.
Data for counter match analysis in R using the same sample restrictions as imposed by Prof. Imai.

Matching Analysis:

Imai’s R code* (September, 2003)
Professor Imai’s program written in R to replicate the results in Table 9 of his paper. This program contains errors discussed in Gerber and Green (2005)
*Note that this code is covered by copyright and should not be used for purposes other than replication without Professor Imai’s consent.

R code that corrects Imai’s programming errors (January, 2004)
Professor Imai’s R program modified by us to correct his programming errors.

R code that conducts matching with the actual sample (not bootstrapped samples)
Professor Imai’s program modified by us to generate matching point estimates from the actual sample rather than the bootstrapped samples.
Results based on the Actual Sample

R code that conducts matching with replacement
Professor Imai’s program modified by us to generate matching estimates with replacement. His program only generates matching estimates without replacement.

R code for Intent-to-Treat matching (using corrected and modified versions of Imai’s R program)
Intent-to-Treat Results

R code for matching untreated treatment group subjects (using corrected and modified versions of Imai’s R program)
This code estimates the effect of NOT contacting people assigned to the treatment group. If matching is unbiased, there should be no effect. We show that Professor Imai’s method produces large negative estimates.
Balance Test for Untreated Treatment Group Match Model
Untreated Treatment Group Match Results

R code for alternative propensity score models (using corrected and modified versions of Imai’s R program)
The propensity score models that Prof. Imai reports are not the only ones that satisfy the balancing criteria that he lays out in footnote 21. Here we experiment with some different specifications that also produce balance using the same set of covariates.
Balance Tests for Alternative Propensity Score Modes for New Haven Data
Results from Alternate Propensity Score Models for New Haven Data

Jasjeet Sekhon (Harvard University) has analyzed these data using matching and reports weak and insignificant treatment effects.
Results

Jake Bowers and Ben Hansen (University of Michigan) have analyzed these data using matching and report negative and insignificant ITT effects.
Results

Results from Iowa and Michigan Matching

Bootstrapped Results: Phone1-5 Matching

Running Prof. Imai’s bootstrapping program 1,000 times (using n=500 bootstrap samples as he does) shows that the estimate he reports for phone calls in Table 9 is an outlier.
Data file that contains results from 1000 replications of Imai’s bootstrapped 1-5 matching analysis for phones (bootstrap sample=500)
Do File
Log File

Analysis of Experimental Data:

Below are the programs and data files used to generate Tables 1 and A1 and the numbers reported in section 1 of Gerber and Green (2005).
2005 release of New Haven household level data file
2005 release of New Haven individual level data file
Codebook
2002 release of New Haven individual level data file analyzed by Prof. Imai (Stata)
Replication analysis do file
Replication analysis log file