text-messages

2018: We sent 25 million "cold" text messages and ran 16 different RCTs. Here's what we learned.

Debra Cleaver

Oct 1, 2019 — 13 min read

Program design by Debra Cleaver (VoteAmerica) and Professor Christopher Mann (Skidmore College). Evaluations, and memos prepared by Christopher B. Mann, Ph.D., or Christopher B. Mann, Ph.D. and Katherine Haenschen, Ph.D. This program resulted in the preparation of 17 different memos, 1 hour long video, and 1 powerpoint presentation. Individual memos are linked at the very bottom of this page.

Background

In 2016, Debra Cleaver began experimenting with peer-to-peer (P2P) SMS messages as a voter mobilization tactic via. P2P messages are considered "cold" messages in that the recipient does not have a prior relationship with the sender, and cannot be contacted via bulk messaging tactics. Hustle.com had just launched, and provided a tech platform that would allow companies and organizations to send P2P messages at scale easily. Hustle, like many San Francisco startups, had found a legal loophole that allowed them to circumvent a pesky law.

In 2016, P2P SMS was very much a novel voter mobilization tactic. Debra Cleaver tried and failed to raise money from DC-based organizations to run this program, before raising money from the YC Investor Community. Debra launched the country's first-ever P2P voter registration campaign about 78 days before election day, followed immediately by the country's first P2P GOTV campaign. Ultimately 100+ paid texters sent over 1 million SMS messages. Later analysis showed that P2P SMS was a cost effective way to register and mobilize voters.

During the 2018 general election, the national civic organization (NCO) of which Debra was CEO built upon and scaled this prior work. NCO ultimately hired 1000+ paid texters, and conducted cold SMS voter mobilization programs targeting 12,681,951 people of color and unmarried women in 33 states. None of the recipients had any prior relationship to NCO. This program resulted in the preparation of 16 evaluation memos, one video summary, and one slide deck. All memos are linked below

Memo 000: Key takeaways across all programs

In-person voting takeaways

NCO's SMS treatments increased turnout in the 2018 election, generating an average increase in turnout of 0.26 percentage points at a cost per net vote of $85.69 (11.7 net votes/$1000).
The program caused 17,586 people to vote who otherwise would have missed participating in the 2018 General Election.
If the weak-confidence cell phone matches and unsuccessful attempts to mobilize people for early in person voting only are excluded, the average effect is 0.42 percentage points at a cost per net vote of $53.14 (18.8 net votes/$1000).
NCO’s “cold” SMS voter mobilization program identified several important lessons about targeting and messages to improve future effectiveness, including two messages that were the most effective: Adopt a Voter (encouraging recipients to get their friends and family to vote) and Social Pressure (thanking recipients for being a registered voter and reminding them that their voting record is public).

Postal voting takeaways

NCO’s SMS messages increased voter turnout among people who had requested a ballot. On average, the treatments generated a 0.21 percentage point increase in turnout at a cost per net vote of $75.14 (13.3 net votes/$1000).
The Social Pressure treatment was the most effective message for increasing turnout in an experiment comparing Social Pressure vs. Political Efficacy vs. Standard Practice), generating a statistically significant increase in turnout of 0.28 percentage points at a cost per net vote of $56.36 (17.7 net votes/$1000).

Early voting and Election Day in-person voting takeaways

SMS mobilization is most cost effective for Election Day voting, generating an increase in turnout of 0.28 percentage points at a cost per net vote of $51.21 (19.5 net votes/$1000).
SMS mobilization for only Early In Person voting did not significantly increase turnout.
The impact of SMS mobilization for Early In Person voting and Election Day voting was not significantly or substantively larger than Election Day only (0.32 percentage points; cost per net vote of $113.34, 8.8 net votes/$1000).
In future “cold” SMS voter mobilization programs, organizations should focus on mobilization for Election Day voting.

In-person voting

Memo 001 - Mobilization for In-person Voting from Any Treatment

Summary

This memo evaluates the overall impact of NCO’s SMS voter mobilization program for in-person voting, including heterogeneous treatment effects across subgroups.
Embedded within this program were a series of message tests. Many tests use NCO’s Standard Practice messaging as a benchmark. The Standard Practice was developed through tests in 2016 and 2017 and is a series of three messages. The first two messages state the election date and encourage voters to “join millions of people like you voting in this important election,” while the third message provides the voter’s polling place address and an option to lookup their polling place if their address has changed.

Key Takeaways

NCO’s SMS treatments generated an average increase in turnout of 0.26 percentage points at a cost per net vote of $85.69 (11.7 net votes/$1000).
The program caused 17,586 people to vote who otherwise would have missed participating in the 2018 General Election.
If the weak confidence cell phone matches and unsuccessful attempts to mobilize people for early in-person voting only are excluded, the average effect is 0.42 percentage points at a cost per net vote of $53.14 (18.8 net votes/$1000).
NCO’s “cold” SMS voter mobilization program increased turnout in the 2018 election and identified several important lesson about targeting (discussed in this memo) and messages (discussed in other memos) to improve future effectiveness.

Memo 002 - Message Test of Adopt-a-Voter vs. Calendar Reminder vs. Standard Practice for in-person voting

Summary

The Adopt-a-voter treatment seeks to motivate turnout by reminding recipients of the social rewards of voting.
The Calendar treatment seeks to increase turnout by using smartphone technology to set a personal reminder to vote.
The test also combined these two paths to behavioral change (Adopt+Calendar).
NCO’s Standard Practice SMS treatment serves as the performance benchmark.

Key Takeaways

The Adopt-a-voter treatment was the most effective message for increasing turnout in this experiment, generating an increase in turnout of 0.22 percentage points at a cost per net vote of $101.27 (9.9 net votes/$1000).
The Standard Practice treatment generated an increase in turnout of 0.10 percentage points at a cost per net vote of $222.8 (4.4 net votes/$1000).
The Calendar and Adopt+Calendar treatments did not significantly increase turnout.
Testing the same messages for encouraging the return of mail ballots in postal voting states produced a similar pattern.

Memo 003 - Message Test of Candidate Name vs. Standard Practice

Summary

This experiment tested the effect of assignment to receive a text message including the names of gubernatorial candidates in Texas and Florida against the Standard Practice message developed by NCO.

Key Takeaways

The effects of the Standard Practice and Candidate Names treatments were statistically indistinguishable.
The Standard Practice treatment generated an increase in turnout of 0.50 percentage points at a cost per net vote of $44.56 (22.4 net votes/$1000).
The Candidate Names treatment generated an increase in turnout of 0.43 percentage points at a cost per net vote of $51.81 (19.3 net votes/$1000).
In future “cold” SMS voter mobilization programs during mid-term and Presidential elections, Vote.org should not use the Candidate Name treatment as it adds considerable complexity to implementing treatments without any gain in effectiveness over the Standard Practice treatment.

Memo 004 - Message Test of Digital Sticker vs. Standard Practice

Summary

The Digital Sticker treatment allowed an easy link from the SMS message to posting a digital “I Voted” sticker on Facebook. NCO’s Standard Practice SMS treatment serves as the performance benchmark.

Key Takeaways

Both treatments generated significant increases in turnout, and the size of the treatment effects was indistinguishable.
The Standard Practice treatment generated an increase in turnout of 0.24 percentage points at a cost per net vote of $92.83 (10.8 net votes/$1000).
The Digital Sticker treatment generated an increase in turnout of 0.25 percentage points at a cost per net vote of $89.12 (11.2 net votes/$1000).
Other message tests indicate two messages were more effective than the 2018 Standard Practice: Adopt-a-voter and Social Pressure.
In future “cold” SMS voter mobilization programs, organizations should not use the Digital Sticker treatment as it adds considerable complexity to implementing treatments without any gain in effectiveness over the Standard Practice treatment.

Summary

This experiment tested the effect of assignment to receive a text message including either positive social pressure or a political efficacy message against the "standard practice" message developed by NCO through previous testing.
Social pressure and political efficacy treatments included 3 messages, with the first two messages stating the election date and the last message providing polling place address (same format as Standard Practice). The first two messages differentiated in the following way:
- Social pressure language: Thank you for being a registered voter! This is a reminder that while your ballot is secret, whether you vote or not is public record.
- Political Efficacy language: You have the power to influence this election! Your vote matters, so make sure to cast your ballot in this important election.

Key Takeaways

The Social Pressure treatment was the most effective message for increasing turnout in this experiment, generating an increase in turnout of 0.50 percentage points at a cost per net vote of $44.56 (22.4 net votes/$1000).
The Political Efficacy treatment generated an increase in turnout of 0.26 percentage points at a cost per net vote of $85.69 (11.7 net votes/$1000).
The Standard Practice treatment appeared generated an increase in turnout of 0.11 percentage points at a cost per net vote of $202.55 (4.9 net votes/$1000), but this effect was not statistically significant.
Testing the same messages for encouraging the return of mail ballots in ballot request states produced a similar pattern (see Memo 009 - Encouraging Ballot Return in Ballot Request States Including Message Test of Social Pressure vs. Political Efficacy vs. Standard Practice.)

Memo 006 - Mobilizing Voters For Election Day With Late SMS Messages

Summary

This experiment determines whether assignment to receive one or two text messages with information about when and where to vote (the final message of the Standard Practice treatment) impacts turnout.
NCO SMS Standard Practice treatments usually include 3 text messages.

Key Takeaways

The treatment generated an increase in turnout of 0.41 percentage points at a cost per net vote of $27.17 (36.8 net votes/$1000).
The effect was indistinguishable between CA where one SMS message was sent and the other three states CT, ME, NY (excluding NYC) where 2 SMS messages were sent.

Memo 007 -Extending Mobilization to High Propensity Voters in States with In-Person Voting

Summary

This memo evaluates an extension of NCO’s original SMS voter mobilization program for in-person voting to include higher propensity voters: 70-80 on TargetSmart’s 2018 vote propensity score.
This experiment explores whether even among likely voters with turnout scores of 70 to 80 text messages are still capable of increasing participation.

Key Takeaways

Among voters with high turnout propensity scores (70 to 80), text messages did not increase turnout in the 2018 Midterm elections.

Early Voting vs Election Day In Person Voting

Memo 008- Timing of Encouraging In Person Voting for Early Voting or Election Day

Summary

This memo evaluates a test of timing of mobilization for Election Day voting, Early In Person voting, or both embedded in NCO’s SMS voter mobilization program for in-person voting in 13 states with a large share of Early In Person voting.

Key Takeaways

SMS mobilization is most cost effective for Election Day voting, generating an increase in turnout of 0.28 percentage points at a cost per net vote of $51.21 (19.5 net votes/$1000).
SMS mobilization for only Early In Person voting did not significantly increase turnout.
The impact of SMS mobilization for Early In Person voting and Election Day voting was not significantly or substantively larger than Election Day only (0.32 percentage points; cost per net vote of $113.34, 8.8 net votes/$1000).
In future “cold” SMS voter mobilization programs, NCO should focus on mobilization for Election Day voting.

Postal Voting

Memo 009 - Encouraging Ballot Return in Postal Voting States Including Message Test of Adopt-a-Voter vs. Calendar Reminder vs. Standard Practice

Summary

This memo evaluates NCO’s program to increase turnout in states that mail a ballot to every registered voter.
The memo also assesses message test comparing Adopt-a-voter vs. Calendar vs. Standard Practice messages used in these states. The same messages were tested in a program encouraging in person voting (see Memo 002: Message Test of Adopt-a-Voter vs. Calendar Reminder vs. Standard Practice).

Key Takeaways

The average effect for any treatment appears to be a 0.2 percentage point increase in turnout ($144.43/net vote; 6.9/$1000), although this effect does not reach statistical significance.
However, there are wide differences in the effectiveness of the treatments:
- The Adopt-a-voter treatment was the most effective message for increasing turnout in this experiment, generating an increase in turnout of 0.5 percentage points at a cost per net vote of $55.84 (17.9 net votes/$1000).
- The Adopt+Calendar treatment appeared to generate an increase in turnout of 0.2 percentage points at a cost per net vote of $139.60 (7.2 net votes/$1000).
- The Standard Practice treatment appeared to generate an increase in turnout of 0.1 percentage points at a cost per net vote of $279.20 (3.6 net votes/$1000).
The Calendar treatment did not significantly increase turnout overall, although it was effective in particular subgroups of the targeted population.
Testing the same messages for increasing in-person voting produced a similar pattern.
In future “cold” SMS voter mobilization programs in states where all voters receive a ballot by mail, organizations should consider the Adopt-a-voter message to be a best practice.

Summary

This experiment tested whether SMS messages could increase mail ballot returns among individuals who requested to vote absentee.

Key Takeaways

NCO’s SMS messages increased voter turnout among people who had requested a ballot. On average, the treatments generated a 0.21 percentage point increase in turnout at a cost per net vote of $75.14 (13.3 net votes/$1000).
The Social Pressure treatment was the most effective message for increasing turnout in this experiment, generating a statistically significant increase in turnout of 0.28 percentage points at a cost per net vote of $56.36 (17.7 net votes/$1000).

Memo 011 - Extending Mobilization to High Propensity Voters in States with Postal Voting

Summary

This memo evaluates an extension of NCO’s original SMS voter mobilization program for postal voting to include higher propensity voters (70-80 on TargetSmart’s 2018 vote propensity score).
This experiment explores whether even among likely voters with turnout scores of 70 to 80 text messages are still capable of increasing participation.

Key Takeaways

Among voters with high turnout propensity scores (70 to 80) in postal voting states, text messages did not increase turnout in the 2018 Midterm elections.

Memo 012 - Encouraging Ballot Return in California

Summary

This memo evaluates NCO’s program to increase turnout in California among voters who are sent mail ballots. A majority of ballots in California are cast by mail, and this share has been increasing over time.

Key Takeaways

This program encouraging Californians to return their mail ballots had no discernible effect on turnout (0.02 percentage points, cost per net vote = $791.50, 1.3 votes/$1000).
However, targeting only voters with higher cell phone match quality may produce a small (but non-significant) effect (0.14 percentage points, cost per net vote = $113.07, 8.8 votes/$1000).

2018 Runoff Elections

Memo 013 - Mobilization with 1 vs. 2 SMS Messages in the Georgia 2018 Run-Off Election

Summary

This memo evaluates an element of NCO’s SMS voter mobilization program intended to encourage participation by voters in the run-off election in Georgia. The run-off election was held on December 4, 2018.
This program delivered either one or two SMS messages to randomly assigned treatment groups. The messages were derived from NCO’s Standard Practices SMS messages, identified in tests in 2016 and 2017.

Key Takeaways

The Two Message treatment increased turnout by a statistically significant 0.57 percentage points, (cost per net vote = $30.42, 32.9 votes/$1000).
The One Message treatment increased turnout by a statistically significant 0.39 percentage points (cost per net vote = $22.82, 43.8 votes/$1000).
The difference between these treatments is statistically significant.

Summary

This memo evaluates an element of NCO’s SMS voter mobilization program intended to encourage participation by voters in the run-off election in Mississippi. The run-off election was held on November 27, 2018.
This program delivered either the Social Pressure treatment or Standard Practice treatment to randomly assigned treatment groups.

Key Takeaways

There was no statistically or substantively significant difference between these treatments in the Mississippi 2018 run-off.
Note that this result is not consistent with the stronger effect of the Social Pressure treatment in the 2018 General election message tests by NCO.

Election Day Registration

Memo 015 - Mobilization of Unregistered Movers in Election Day Registration States

Summary

This memo evaluates NCO’s program to increase turnout in five states that allow Election Day voter registration. This memo evaluates an adaptation of SMS voter mobilization to encourage people identified as having moved from their prior registration address to register and vote on Election Day in states where Election Day Registration (EDR) is allowed.

Key Takeaways

The average effect appears to be a 0.18 percentage point increase in turnout ($48.17/net vote; 20.8 votes/$1000), although this effect does not reach statistical significance. However, the positive effect is due to a large effect in one state (Wisconsin) but null effects in the remaining four states (CA, IA, IL, MN).

Hurricane Impacted Areas

Memo 016 - Extending Mobilization to High Propensity Voters in Hurricane Impacted Areas

Summary

This memo evaluates an element of NCO’s SMS voter mobilization program intended to ensure participation by voters impacted by the two major hurricanes to make landfall in the US during the 2018 election season: Hurricane Florence in North Carolina and Hurricane Michael in Florida and Georgia.
NCO was already conducting SMS voter mobilization programs to low and mid-propensity voters and to people who had requested mail ballots in these areas. Following the hurricanes, Vote.org extended the program to encourage high propensity voters in these areas to vote.
This program delivered an adapted version of NCO’s Standard Practices SMS messages, identified in tests in 2016 and 2017, to these high propensity voters.

Key Takeaways

This “cold” SMS program to high propensity voters following a hurricane had a small (and not statistically significant) effect on turnout (0.14 percentage points, cost per net vote = $126.29, 7.9 votes/$1000).
However, more judicious targeting of only voters with 70-79 vote propensity (just above NCO normal voter mobilization targeting) produced nearly all of the effect (0.51 percentage points, cost per net vote = $34.67, 28.8 votes/$1000).

2018: We sent 25 million "cold" text messages and ran 16 different RCTs. Here's what we learned.

Debra Cleaver

Background