2017 Virginia Spotify and Pandora program evaluation

Do digital GOTV ads on Pandora and Spotify increase voter turnout? A 2017 experiment

Program design by Debra Cleaver (VoteAmerica). Creative work and media placement by Solidarity Strategies, LLC. Experiment design and analysis by the Analyst Institute. The below is excerpted from a study published in 2018.

Executive Summary

In the lead up to the November 2017 gubernatorial election in Virginia, Vote.org partnered with Analyst Institute to evaluate the effect of a digital radio GOTV ad campaign on Pandora and Spotify. The campaign targeted Black, Latinx, and Asian-American listeners in the Tidewater region of Virginia with 30-second audio and clickable GOTV display ads. Ads were served to digital radio listeners in randomly selected ZIP codes. This analysis examines the effect of the program on all modeled voters of color in the experimental ZIP codes, as the test design did not allow us to identify and restrict attention to just voters who listen to digital radio. As a result, this assessment necessarily includes voters who are not digital radio listeners. Using digital radio is a novel tactic for mobilizing voters; we are unaware of previous studies that have evaluated its efficacy.

Key findings:

• Overall, Pandora and Spotify ads promoting voting do not appear to have increased turnout among voters of color in the Tidewater region. Turnout among Black, Latinx, and Asian-American voters in treatment ZIP codes was a statistically insignificant 0.2 percentage points (pp, p = 0.76) lower than in control ZIP codes.
• A separate survey of 253 voters in the same experimental ZIP codes indicates that more people in treatment ZIP codes reported hearing neutral GOTV messages (which favored no party or candidate) on digital radio than in control ZIP codes. 25 percent of voters in treatment ZIP codes, versus 10 percent in control ZIP codes (15pp difference, p < 0.01), reported hearing neutral GOTV digital radio ads.Key takeaways:
• We did not find evidence that digital radio ads increased turnout. However, we did find survey evidence that the ads were memorable among voters in the experimental ZIP codes.
• Using digital radio to mobilize voters is a novel tactic; we should continue to examine how this mode can be used effectively. In particular, we should include more ZIP codes in future tests (ideally 500 or more) in order to improve our ability to detect an effect.

Digital Radio Ads Do Not Appear to Have Increased Turnout Among Voters of Color

Voters Reported Hearing Neutral GOTV Ads at Higher Rates in Treatment ZIP Codes

Background

In the lead up to the November 2017 gubernatorial election in Virginia, Vote.org partnered with Analyst Institute to evaluate the effect of a digital radio GOTV ad campaign on Pandora and Spotify. The campaign targeted Black, Latinx, and Asian-American listeners in the Tidewater region1 of Virginia with 30-second audio and clickable GOTV display ads. Ads were served to digital radio listeners in randomly selected ZIP codes. This analysis examines the effect of the program on all modeled voters of color in the experimental ZIP codes, as the test design did not allow us to identify and restrict attention to just voters who listen to digital radio. As a result, this assessment necessarily includes voters who are not digital radio listeners.

Using digital radio is novel tactic for mobilizing voters; we are unaware of previous studies that have evaluated its efficacy. A small meta-analysis (p. 158) has identified promising, though small, effects of terrestrial radio on turnout; individual studies have found that terrestrial radio can increase turnout among specific groups, such as Latinx voters. Digital radio is the logical next step for evaluation. According to the Pew Research Center, 53% of Americans said in 2017 that they listened to some form of online radio in the past week (this category included AM/FM stations that also broadcast online in addition to streamed audio content available only on the Internet). Organizations encouraging voter turnout must continue finding new ways to contact voters who may not be reachable via other traditional GOTV modes (e.g., mail, phone, canvass); digital radio is one promising new mode of contact.

Research Questions

● Do digital GOTV ads on Pandora and Spotify increase voter turnout?

Experimental Design and Implementation

Experimental Universe

The experimental universe consisted of 318,130 modeled voters of color in 183 experimental ZIP codes in the Tidewater region of Virginia. The randomization occurred at the level of the ZIP code, but the analysis was conducted at the individual level among registered voters in the experimental ZIP codes. Table 1 shows the characteristics of the experimental universe.

Experimental Conditions

ZIP codes were randomly assigned to one of the following two conditions:

• ●  Digital Radio Ads: Pandora and Spotify listeners of color in these ZIP codes were targeted with GOTV ads from Vote.org. The ads included 30-second audio and clickable banner ads. The stations airing the ads included those modeled to have large Black, Latinx, and Asian-American audiences (N = 93 ZIPs, 155,135 voters).
• ●  No Internet Radio Ads: ZIP codes in this condition were assigned to receive no digital radio GOTV ads from Vote.org (N = 90 ZIPs, 162,995 voters).The ZIP randomization was stratified to ensure voters were comparable across conditions. Variables used in the stratification included number of voters, 2013 percent turnout, and percent white in the ZIP code.

Experimental Implementation

Between 10/17/18 and 11/7/18, Pandora and Spotify delivered the GOTV ads to listeners in the treatment ZIP codes. Black, Latinx, and Asian-American listeners were targeted based on Pandora and Spotify’s internal targeting procedures.The ads on Pandora and Spotify included audio ads and banner ads, with an option to click on most of the banner ads to go to Vote.org’s website for more information (not all ads were clickable). The audio ads provided a reminder about the upcoming election, gave polling place hours, and emphasized voting as a way to make one’s voice heard in government. The banner ads showed pictures of young people with an election reminder and polling place hours in English or Spanish. The complete text and images from the ads are available in the appendix.

Table 2 summarizes the number of impressions, unique listener accounts (only available for Pandora), and clicks for each digital radio service. Taken together, the ads had over 2.5 million impressions, and probably reached approximately 160,000 listeners in the target ZIP codes.2 While this number is slightly larger than the number of targets in our test, it is critical to note that we cannot know for certain what proportion of these listeners were, in fact, people of color or registered voters in the targeted ZIP codes based on the implementation data. We only know that people of color in the treatment ZIP codes were targeted by the ads; we do not know which individuals actually were served ads. It appears that the ads resulted in very few clicks, though given that many people listen to digital radio in the background, it may not be realistic to expect many click-throughs.

Outcome measurement

The outcome in this test was voter turnout in the 2017 Virginia general election. This outcome was measured post-election using data from TargetSmart.

Results

Main Results

Overall, Pandora and Spotify ads promoting voting do not appear to have increased turnout among voters of color in the Tidewater region. The baseline rate of voting in control was 42.4%; turnout among modeled Black, Latinx, and Asian-American voters in treatment ZIP codes was a statistically insignificant 0.2pp (p = 0.76) lower than in control ZIP codes (Figure 2).

Using data from a separate survey of 253 voters in the same experimental ZIP codes conducted by Analyst Institute in the lead up to the 2017 Virginia election, we compared the rate at which survey respondents reported hearing neutral GOTV digital radio ads (which favored no party or candidate) in the treatment and control ZIP codes. We found that 25 percent of voters in the treatment ZIP codes, versus 10 percent in control ZIP codes (15pp difference, p < 0.01), reported hearing neutral GOTV ads on digital radio (Figure 3). Among self-identified respondents of color, the difference between treatment and control ZIP codes was 21pp. There were only 58 survey respondents of color in the experimental ZIP codes so this result, while suggestive that voters of color heard the ads as intended, should be interpreted with caution.

Variation in Treatment Effects

We examined whether treatment effects varied by gender, race, age, and turnout score. We found little evidence that treatment effects varied systematically based on these characteristics.

Discussion

We did not find evidence that digital radio ads increased turnout. The results of our survey are more encouraging. The survey evidence suggests that voters in the treatment ZIP codes recalled hearing neutral GOTV ads on digital radio at a much higher rate than voters in control ZIP codes. This finding provides some evidence that the ads broke through with registered voters and were memorable.

Using digital radio to mobilize voters is a novel tactic and organizations should continue to examine how this mode can be used effectively. Given that 25% percent of voters in treatment ZIPs reported hearing neutral GOTV ads and 50% of Americans listen to digital radio in some form, we think that experiments randomized by ZIP code (or by other geographic units) are a good way to test the effect of digital radio. However, since we do not know how well digital radio vendors can target specific demographic groups, it may make sense for future tests to target individuals within treatment ZIPs more broadly. This design change relies on organizations finding enough ZIP codes where the population meets their targeting criteria (e.g., ZIP codes made up predominantly of people of color). In addition, we should include more ZIP codes in future tests (ideally 500 or more) in order to improve our ability to detect an effect. Finally, given the promising findings from our separate survey, we may want to consider using surveys in the future to measure digital radio ad recall (though measuring outcomes via voter file data would remain the priority approach).