text-messages

2018 November election warm sms program evaluation

Debra Cleaver

Aug 31, 2019 — 10 min read

Which sms message copy worked best to mobilize subscribers in 2018?

Program design by Debra Cleaver (VoteAmerica) and Christopher Mann (Skidmore College). Program execution by a national civic organization (NCO) with a large list of opted-in SMS subscribers.Program evaluation by Christopher Mann. Originally published in 2019.

Executive Summary

For the 2018 general election, NCO conducted SMS voter mobilization programs covering 1,790,781 people who had used one of NCO’s tools (“subscribers”). Since these individuals had previously interacted with NCO SMS messages could be delivered via the Mobile Commons platform (rather than peer-to-peer), but this relationship was “warm” only in a very limited sense.

This memo evaluates NCO’s program to increase turnout among the full list of Vote.org subscribers. Further, the memo also assesses message test comparing four versions of the message sequence used to encourage turnout: Direct Link, Request Info, Adopt-a-voter, Recommend-a- friend.

Among the 47 states plus DC that offer in-person voting, the average effect was a 0.24 percentage point increase in turnout ($11.47/net vote; 87 votes/$1000). However, there are wide differences in the effectiveness of the treatments: The Request Information treatment was the most effective message for increasing turnout in this experiment, generating an increase in turnout of 0.49 percentage points at a cost per net vote of $5.62 (178 net votes/$1000). The Direct Link treatment generated an increase in turnout of 0.30 percentage points at a cost per net vote of $9.17 (109 net votes/$1000). The Recommend-a-friend treatment generated an increase in turnout of 0.21 percentage points at a cost per net vote of $13.11 (76 net votes/$1000). The Adopt-a-voter treatment did not generate a statistically significant effect (0.11 percentage points; $25.02/net vote; 40 votes/$1000). These results mitigate concerns about using the Request Information approach in “cold” SMS programs when it is required. The results also suggest providing information is more effective than trying to increase motivation to vote in person among subscribers.

In the three states where all registered voters were mailed a ballot in 2018, the average effect was a 0.75 percentage point increase in turnout ($2.93/net vote; 342 votes/$1000). Again, there were notable differences between the two treatments tests, although the pattern was different than the in- person voting states. The Adopt-a-voter treatment generated an increase in turnout of 1.02 percentage points at a cost per net vote of $2.15 (465 net votes/$1000). The Direct Link treatment generated an increase in turnout of 0.47 percentage points at a cost per net vote of $4.67 (214 net votes/$1000).

In future “warm” SMS mobilization programs to subscribers, Vote.org should consider the Request Information treatment to be a best practice in in-person voting states, and Adopt-a-voter message to be a best practice in postal voting states.

Objectives and Context

Despite widespread use, SMS messages have received little attention from researchers as a medium for campaign communication. In 2016, NCO established that “cold” SMS messages could increase turnout with a randomized experiment design covering 1.2 million young people of color and unmarried women in 7 states. NCO’s 2016 “cold” SMS voter mobilization program increased turnout by 0.2 percentage points. In 2017, NCO replicated and expanded testing of “cold” SMS voter mobilization with a randomized experiment covering 714k young people of color and unmarried women for the Virginia gubernatorial and legislative elections. NCO’s 2017 test of “cold” SMS voter mobilization increased turnout by 0.6 percentage points and identified Standard Practices regarding timing and message framing.

This memo evaluates an adaptation of SMS voter mobilization of NCOsubscribers. The message testing in this program takes advantage of two features of warm SMS. First, the comparison of the Direct Link and Request Info varies the inclusion of a URL to detailed voting information. Warm SMS messages can contain URLs without being blocked by mobile providers, while links cannot be included in initial contact for “cold” SMS. Therefore, this test provides insight on the impact of including a URL in the initial contact or requiring individuals to request information. Second, the comparison of Adopt-a-voter and Recommend-a-friend varies whether individuals are asked to commit to mobilizing friends and family (Adopt-a-voter) or asked to provide names so NCOcan mobilize friends and family (Recommend-a-friend).

Selected Universe

The data for the experiment was selected by NCO from the Mobile Commons communication platform used to communicate with 1,790,781 individuals who have used one (or more) of their web tools to register to vote, verify registration, obtain a mail ballot, etc. The data extraction was done on October 19, 2018.

These records were not matched to the voter file prior to treatment, so some records may contain incomplete, incorrect or be otherwise unable to match to the voter file for post-election analysis.

Treatments

The experiment compares an uncontacted control group to four different sequences of SMS messages. In states without early in person voting, each SMS message sequence consisted of two SMS messages. In states with early in person voting, each SMS message sequence consisted of four SMS messages with the first two encouraging early voting and the second two messages repeating the message content while encouraging Election Day voting. Examples of the Election Day voting treatment messages are in the Appendix.

In the three states (CO, OR, WA) where all registered voters are sent a mail ballot, variations of the Direct Link and Adopt-a-voter messages were used encouraging return of mail ballots using two SMS message sequences. The test was limited to these treatments because of weak statistical power due to small number of available records.

The Request Info treatment is based on prior tests and programs by NCO. (This treatment was intentionally very similar to and based on the “Standard Practice” in NCI’s 2018 “cold” SMS programs to individuals selected from the voter file). The Request Info treatment relies on positive descriptive norms, civic duty and information about voting to increase turnout. These tactics are very common in voter mobilization and have been successful in randomized controlled tests of mail, phone calls and canvassing (see Green and Gerber 2019 for review). The Request Info treatment offers information about polling place location when a recipient responds with a request. This procedure mimics the steps necessary to deliver URLs in “cold” SMS programs because providing links in initial contacts to a large number of individuals is very likely to be blocked by mobile providers as SMS spam.

The Direct Link treatment is identical to the Request Info treatment except that it provides a URL to the recipient (i.e. no request). The comparison between these treatments is intended to estimate the impact of the additional step of requesting information. The prior expectation was the burden of requiring respondents to request information would diminish the impact of the information.

The Adopt-a-voter treatment is based on an SMS experiment conducted in a local election in Portland, OR in 2017. This treatment generated statistically significant increases in turnout in this experiment. For the Adopt-a-voter treatment, the first message for each method of voting encouraged recipient to get friends and family members to vote. The Adopt-a-Voter treatment is based on leveraging the social rewards from voting. Delivering similar Adopt-a-Voter treatments by phone has successfully increased in-person voter turnout.iv The remaining message(s) in the treatment was identical to the Request Info treatment. The Adopt-a-voter treatment delivered to subscribers is nearly identical to the Adopt-a-voter treatments delivered in NCO’s “cold” SMS experiments for in-person voting and postal voting.

The Recommend-a-friend treatment asks the recipient to provide the phone number of a friend so NCO can mobilize this individual. It was intended as a variation of Adopt-a-voter wherein the initial recipient has Vote.org mobilize their friend rather than doing so themselves (as in Adopt-a- voter).

Intended Effects

The treatments were intended to increase turnout in the November 2018 election.
Each treatment was expected to have a different effect on turnout in the Nov. 2018 Election.

Evaluation Design

The evaluation is based on a randomized trial design (or field experiment) that is considered best practice by academic researchers and the Analyst Institute. The treatment group received SMS messages from Vote.org; the control group was sent none of the SMS messages.The randomization is conducted at the household level to reduce the risk of contaminating behavior of co-habitants. For this experiment, households were defined as people with the same mailing address. The randomization uses an automated re-randomization procedure to ensure good balance in characteristics available from the voter file prior to delivery of treatment (see Technical Appendix).

Results

Overall effect:

Among NCO subscribers in states with in-person voting, being contacted with any SMS message in 2018 generated a statistically significant increase in turnout of 0.24 percentage points (first graph below). The magnitude of the effect of encouraging in-person voting was not substantively (or statistically) different in states allowing early voting, and therefore receiving a four message sequence encouraging both methods rather than a two message sequence about Election Day voting.

Among NCO subscribers in states where all registered voters receive a mail ballot, being contacted with any SMS message in 2018 generated a marginally statistically significant increase of 0.75 percentage points.

Differences between Treatments

In the in-person voting states, the Request Information treatment increased turnout by 0.49 percentage points, the Direct Link treatment increased turnout by 0.30 percentage points, the Recommend-a-friend treatment increased turnout by 0.21 percentage points, and the Adopt-a-voter treatment appeared to increase turnout by 0.11 percentage points.

The differences across all four treatments for in person voting was statistically significant, thus suggesting the information treatments were more important among subscribers than the treatments leveraging the social aspects of voting. This pattern differs from NCO’s “cold” SMS tests where the Adopt-a-voter treatment had a larger effect than the Request Information treatment (known as Standard Practice in those experiments). Although there are many possible explanations for the difference in performance, NCO subscribers have already demonstrated motivation about voting (indicated by use of NCO tools). Therefore, fewer will be pushed across the threshold to participate by motivation than by information to help logistical overcome barriers to participation.

The difference between the Recommend-a-friend and Adopt-a-voter treatments is not statistically significant, although doubling the treatment effect suggests Recommend-a-friend is a more effective tactic. Future testing should examine whether it is the naming of an individual or the outsourcing of follow-up that make Recommend-a-friend more effective.

The difference between the Request Information and Direct Link treatments is only marginally statistically significant. However, the difference is in the opposite direction from expectations, with Request Information being more effect than providing a Direct Link. The “extra step” of requesting polling place information is certainly not a problem – and may be a benefit. This is a reassuring finding for “cold” SMS where the request process is required. Future research could examine whether the Request Information treatment is more effective because it is not perceived as spam, people feel more confident (to justify not seeking information), or delivery problems with Direct Link (possible despite being “warm” SMS).

In the postal voting states, the Adopt-a-voter treatment generated a statistically significant increase of 1.02 percentage points. The Direct Link treatment appeared to generate a 0.47 percentage point increase, although this effect is not statistically significant. The differences between the treatment effects are substantively large and approaches statistical significance. Notably, the stronger performance of the Adopt-a-voter treatment than the Direct Link treatment is the reverse of the pattern seen for in-person voting.

Net Votes

The cost per net vote (and net votes/$1000) calculation includes all costs of design, delivering, and managing the treatment delivery process.

Notes: Treatment cost reflects average cost for the message in the treatment. Net votes is the number of people who voted in response to the treatment(s), and would not have otherwise voted in the November 2018 election.

Lessons Learned

NCO’s SMS mobilization program to the subscriber list is cost effective, and should be continued in future elections.
Testing different mobilization message tactics revealed specific tactics to be more effective than others, and these differed by in-person voting vs. postal voting.
The Request Information treatment appears more effective – and certainly not less effective – than providing a URL (Direct Link treatment) with information about polling place location. Although extrapolating between the distinct populations in this test and Vote.org’s “cold” SMS mobilization targets is uncertain, this finding suggests the Request Information approach required for cold SMS messages is not problematic.

Future Steps

NCO should conduct more research on the most effective messages for mobilizing the subscriber list. The “warm” SMS population appears to respond to different treatments than the “cold” SMS mobilization targets from the voter file – at least for in-person voting states – and thus needs a separate research track.

For states with in-person voting, the Request Information treatment should be treated as “best practice” in future tests.

For states with postal voting, the Adopt-a-voter treatment should be treated as “best practice” but testing should explore whether there are types of information that can make be more effective.

Cautions

The effect of any voter mobilization communication is conditional on the execution of the program, the jurisdiction, the type of election, the level of interest in the election, and the activities of other organizations. Repeating these treatments in other election contexts or with variations of the treatments could produce different results.

Technical Appendix

Randomization Procedure:

Randomization was conducted at the household level. The random assignment was conducted in Stata using the “re-randomize” procedure developed by Kennedy and Mann (2015) to ensure balance across observable covariates.xviii

This procedure rejects any instance of randomization outside of pre-determined parameters: minimum of 10 iterations and maximum of 25 iterations. Iterations stopped between 10 and 25 when iteration had p>0.8 based on Malahanobis distance test. This procedure produced equal sized groups, and each group was designated as an experimental condition. Blocked randomization used equal probabilities of assignment in all blocks.

Blocked randomization using the following variables: State, Young (under 30 years old), Quality of cell phone match to individual (three strata based on TargetSmart cell phone match confidence code)

Balance checked using age, female, individual-level race codes (Hispanic, African American, white), past voting history (dummies for voting in the 2010, 2012, 2014, and 2016 general elections), and three-digit zip-code (geography).

Statistical Methods for Analysis:

The analysis is based on matching the pre-election experimental population to post-election vote history available on The Movement Cooperative’s instance of the Catalist voter file. Unfortunately, this dataset does not have voter registration date available to assess the rate of registering on Election Day. The matching used name, date of birth, and residential address from the pre-election TargetSmart file. Analysis was conducted using standard regression techniques for evaluating experimental results.

Hypothesis testing uses robust standard errors clustered by unique address to account for potential correlation between the behaviors of co-habitants.

All reported estimates are calculated using models that include the covariates used to check balance in the random assignment procedure. As expected from a well-balanced experiment, the estimates are essentially identical when estimated without these covariates.