2016 "cold" SMS GOTV test results

Project conception, fundraising, and execution by Debra Cleaver (VoteAmerica). Funding provided by the Y Combinator Investor Community. Technology platform by Hustle.com Experiment analysis and memo prepared by by Alex Gold, Josh Rosmarin, Lena Tom, and Miya Woolfalk, all of the Analyst Institute.

Executive Summary

A national civic organization (NCO) and Analyst Institute partnered to conduct an experiment to evaluate the GOTV impact of text  messages  sent  “cold”  to  voters  targeted  off  the  voter  file  via  Hustle.  The  test  also  examined whether plan-making messages (i.e., messages encouraging voters to make a voting plan) or polling location messages (i.e., messages helping voters find their polling place)  were more effective.

Key findings

• The polling location messages increased turnout by 0.2 percentage points (p < 0.05) relative to the no contact control group, yielding 1,473 net votes at a cost per net vote of $128 (or about 8 votes per $1,000 spent). This innovative text message included, when possible, the voter’s street address, polling location, and poll location hours.
• The  plan-making  messages  did  not  increase  turnout,  offering  suggestive  evidence that plan-making  texts  may  be less effective than polling location messages. However, the texts varied  in  personalization  and  timing  as  well,  so  we  can’t  be  certain  of  the  reason  for  the differential finding.

Key takeaways

• Polling location text messages sent cold to young voters targeted off of the voter file are   an  effective  turnout  tool,  generating  an  effect  on  par  with  that  observed  in  an academic meta-analysis of conventional nonpartisan GOTV mail programs.

Background

NCO partnered with the Analyst Institute to run a series of experiments in 2016 to evaluate the impact of text messages on civic engagement. This particular  test  examined  the  GOTV  impact of SMS  messages sent via Hustle to “cold” targets from the voter file. It also examined the relative efficacy of two different types of GOTV text messages – a plan-making or polling location message.

Research Questions

This test was designed to answer the following research questions:

• Can text messages sent cold to voters targeted off of the voter file turn them out to vote?
• What messaging is most effective: a plan-making message or a polling location message?

Experimental Design and Implementation

Experimental Universe

The experimental universe included 1,207,316 registered voters with cell phone numbers on the TargetSmart voter file. Targets were people of color under 40 and unmarried women under 35 residing in AZ, MO, NV, OH, PA, UT, and WI.

Table 1: Experimental Universe Characteristics

Experimental Conditions

Targets were randomly assigned to one of three conditions:

• Control: Targets in this condition did not receive any text messages from NCO (n = 301,920 voters).
• Plan-making: Targets in this condition received a plan-making GOTV text message with questions about their voting plan and an encouragement to make a plan online. Voters who submitted a plan online and were sent a follow-up message reminding them of their plan the day before the election. (n = 301,751 voters).
• Polling location: Targets in this condition received one polling location GOTV text message designed to help voters find their polling place. Depending on whether or not polling place information   was  readily  available  via  Google’s  Civic  API,  the  message  either  included information  on  voters’  polling  place  within  the  text  or  provided  a  link  to  look  up  this information online. (n = 603,645 voters).

Experimental Implementation

NCO obtained a list of approximately 1.2 million cell phone numbers from TargetSmart in their targeted universe. Analyst Institute then randomized the list of cell phone numbers into experimental conditions and NCO used Hustle to deliver the assigned text messages to targets in treatment conditions.

Text messages were delivered in the last two weeks leading up to the November election. The timing of the text messages varied across the experimental conditions, with plan-making texts being delivered on 10/29 through 11/4 and polling location texts being delivered on 11/1 through 11/4. Most plan-making texts were sent one week or more before Election Day while most polling location texts were sent on Thursday 11/3 and Friday 11/4 before Election Day.

A total of 714,151 texts were sent to targets, with 289,693 texts sent to those in the plan-making condition and 422,592 sent to those in the polling location condition. The overall completion rate (i.e., the share of targets receiving texts) was 96% among those in the plan-making condition and 70% among those in the polling location condition. In the plan-making condition, follow-up messages were sent to 341 voters (or 0.1% of targets), indicating that very few recipients submitted a plan to vote. In the polling location condition, 242,726 voters (or 40% of targets) were sent messages that included information on their polling place within the text and 179,866 voters (or 30% of targets) were provided a link to look up their polling place online.

Table 2: The timing of the text messages varied across experimental conditions

The response rate to text messages was low, with approximately 3.5% responding in some way to the polling location message and 5.8% responding in some way to the plan-making message. As responses to the program were not systematically captured, the content of responses is not part of this analysis. 

Implementation Challenges

Originally, this test was designed to have a total of four treatment conditions -- a control condition, a plan-making only condition, a polling location only condition and a plan-making plus polling location condition. However, due to constraints on NCO staff time, the combination plan-making plus polling location group only received polling location texts, with only about 53% of those in the combination condition receiving texts. 86% of those in the original polling-location-only condition received texts.

As the polling location texts in the combination condition were identical to those in the polling location only condition, we pooled the two conditions when analyzing the results. The low text delivery rate in the plan-making plus polling location condition did bring down the overall completion rate in the pooled polling location condition. But, the main results of this test do not change when the analysis is conducted without pooling the conditions.

Outcome Measurement

The outcome in this test was turnout in the 2016 general election. It was measured using data from the TargetSmart voter file.

Results

Main Results

Figure 1 displays the topline results from this test. It demonstrates that while the polling location messages increased voter turnout, the plan-making messages did not appear to mobilize voters. Roughly 52.4% of voters in the polling-location condition voted in the 2016 election, while 52.2% of voters  in  the  control  group  voted.   This  0.2  pp  difference  in  turnout  is  statistically significant (p < 0.05),  yielding 1,473 net votes in the polling location condition and a cost per net vote of about $128 (or 8 net votes per $1,000). In contrast, turnout in the plan-making condition was 52.1%. This 0.1 pp difference in turnout relative to the control condition is not statistically significant (p = 0.52). 

Figure 1: The polling location messages increased voter turnout by 0.2 percentage points

Variation in Treatment Effects

Effects  on  turnout  were  also  examined  by  voters’  demographic  traits,  including age, race, gender, marital status (overall and separately among women and men), and 2016 vote propensity. While effects  did  not  appear  to  vary  by  age,  race  or  vote  propensity,  there  was  some  evidence  of differences in effects by gender and marital status. In particular, unmarried women may have been more  responsive  to  the  polling  location messages compared to their married counterparts (Figure  2). But, the mechanism underlying this pattern is unclear. More research is needed to validate this result before drawing conclusions as to the relative efficacy of the messages among unmarried women. 

Figure 2: Unmarried women may have been more responsive to the polling location messages

Discussion

Although  the  plan-making  messages  did  not  appear  to  increase  turnout,  the  effect  of  the  polling location messages is notable. It confirms that GOTV text messages are an effective turnout tool even when sent “cold” to young voters targeted off the voter file in a presidential election year. That said, the  impact  of  the  polling  location  texts  was  smaller  than  effects  observed  in  tests  where  voters targeted had previously opted-in to receiving messages from the sending organization.

The  polling  location  messages  had  an  effect  on  turnout  that  is  comparable  to  that  observed  in conventional GOTV mail programs. This is based on a comparison to an academic meta-analysis of published and unpublished studies of nonpartisan GOTV mail programs, which re-analyzed the results from tests conducted in a wide variety of electoral contexts, including low salience elections, and targeting a diversity of voters. This is particularly important given the distinct electoral context and targeting deployed in this test. Specifically, this test was conducted during a presidential election year, when mobilization effects tend to be smaller, and targeted younger voters, who are generally harder to reach and thus, more di蜅cult to mobilize.

The  test  results  provide  suggestive  evidence  that  some  messages  to  “cold”  targets  may  be  more effective than others. In particular, while the polling location messages increased voter turnout, the plan-making messages did not appear to boost voting. There may be any number of explanations for  this  result.  One  possible  explanation  is  that  the  “benefits”  from  the  polling  location  message were more readily accessible. In particular, for many targets (i.e., roughly 57% of polling location text recipients), the polling location message only required individuals  to  read  messages  received  to  learn their polling location. In contrast, the plan-making texts required voters to engage in a more thoughtful process of mentally rehearsing their voting plan. This cognitively intensive and time-consuming process is key to reaping the bene fi ts of plan-making prompts. Another potential reason the polling location messages were more effective is that they were more personalized: the messages  included  targets’  first  name  (and  in  many  cases,  registration  address).  These  personal details were added to avoid misinforming recipients in the event that someone other than the intended target received the message.

More  research  is  needed  to  fully  understand  the  differential  effects  by  text message type. Future research could explore how variations in the content or format of plan-making messages or in the level of personalization in texts shape effects.