gotv

A meta-analysis of voter mobilization tactics by electoral salience

Debra Cleaver

Debra Cleaver

— 25 min read
A meta-analysis of voter mobilization tactics by electoral salience
Photo by Elliott Stallion / Unsplash

Analysis by Christopher B. Mann PhD and Katherine Haenschen PhD.
Originally published on Science Direct at this URL: https://www.sciencedirect.com/science/article/pii/S0261379423001518. Republished with the authors' permission.

Abstract

After hundreds of field experiments assessing their effectiveness, voter mobilization tactics are often considered “settled science.” In this research note, we posit that this assumption is incorrect, due to inconsistent and insufficient attention to electoral salience in the literature. Researchers often conduct mobilization field experiments in low-salience elections due to limited resources and the need for adequate statistical power. However, practitioners often apply these findings in high-salience contexts. Theory suggests that effects of mobilization tactics will attenuate in high-salience elections due to heightened attention. We present refined meta-analytic estimates of common mobilization tactics in U.S. elections—canvassing, phone calls, direct mail, and SMS messages—based on electoral salience. Results show that effects of tactics attenuate 33%–76% from low- to high-salience contexts. We translate all findings into intent-to-treat (ITT) estimates to highlight the impact of declining contact rates. Finally, we identify significant gaps in the research and offer solutions.

1. Introduction

The experimental revolution in empirical political science has had dramatic impacts, particularly in the field of voter mobilization. Attention to causal inference about political activity is a normative good: it allows scholars to more accurately understand electoral politics and demonstrates the value of political science research to campaign practitioners. In this research note, we further urge scholars and practitioners to pay closer attention to the impact of electoral salience on effect sizes produced by mobilization efforts. In so doing, we question the conventional wisdom that the magnitude of effects of common mobilization tactics—canvassing, phone calls, direct mail, and SMS messages—are well understood. Instead, we suggest that there are critical gaps in knowledge of how tactics perform in different electoral contexts.

Scholars familiar with the voting behavior literature know that electoral salience is a key predictor of mobilization effectiveness due to its impact on overall turnout. Although a well-established and familiar concept, the mobilization literature accounts for salience only sporadically (e.g., Arceneaux and Nickerson, 2009; Bergh and Christensen, 2022; Green and Gerber, 2019; Malhotra et al., 2011). Without attention to electoral salience, scholarly understanding and empirical estimates of voter mobilization effects are unmoored from theoretically important predictors of baseline voting behavior.

Our concern is the inconsistent and therefore insufficient attention to the role of electoral salience in voter mobilization experiments. Green and Gerber’s (2019) book synthesizing this literature provides one such example of inconsistency in attending to salience. In their meta-analyses of different mobilization tactics (canvassing, phone calls, direct mail, SMS), only canvassing is examined for heterogeneity using a measure of electoral salience; this approach applied systematically is what we want to see in the literature.

Inconsistency in application of electoral salience in the mobilization literature raises the risk of misinterpretation by practitioners who draw on our work to make evidence-based decisions. Based on extensive consultation and collaboration with voter mobilization practitioners, we believe the scholarly literature is influential.1 However, there is an inherent tension between the contexts in which scholars conduct field experiments and the elections in which the findings are often applied. Researchers often conduct “get out the vote” (GOTV) experiments in low-salience elections (e.g., local elections, primaries) because mobilization tactics are more likely to have a detectable impact, necessitating smaller sample sizes to provide adequate statistical power. Conversely, U.S. practitioners may apply findings from these field experiments to high-salience elections (U.S. Midterm and Presidential elections), where mobilization effects are attenuated by heightened attention and a crowded communication landscape.

In this research note, we present a re-analysis of 240 field experiments presented in Green and Gerber (2019) and conducted in the U.S. that measure the effects of direct mail, canvassing, phone calls, and SMS messages in terms of whether they are conducted in high- or low-salience elections to further refine meta-analytic estimates of how these tactics perform. We find consistent evidence that high-salience electoral contexts attenuate treatment effects by 33%–76%, generating much lower percentage point impacts on turnout.

Using these meta-analytic results, we calculate intent-to-treat (ITT) estimates for all GOTV tactics to facilitate direct comparison of mobilization effects at the level of assignment. This is necessary because the GOTV field experiment literature traditionally reports ITT estimates for some tactics (e.g., direct mail or SMS), while others are reported using complier average causal effects (CACE; e.g., phone calls or canvassing), which makes comparisons across tactics challenging and opaque.2

Our ITT estimates demonstrate that tactics such as canvassing or phone calling--often considered the most effective--require best-case (and perhaps unrealistically high) contact rates to be as effective as SMS or social pressure mail. These key assumptions may be distorting scholars’ understanding of campaign effects and aggregate influences on voting behavior, and may lead practitioners to make flawed strategic decisions. Additionally, we identify areas where there is a paucity of available voter mobilization field experiments to support confident inferences about the effects of voter mobilization, particularly in U.S. Presidential elections and as societal change influences the impact from phone calls, at doors, and via SMS.

2. Electoral salience

Electoral salience is a measure of public interest in a given election. Scholarship offers several approaches to defining salience: an ex-ante approach based on expectations about a given type of election, and post-hoc based on overall turnout.3

The ex-ante approach categorizes uniform national elections as high-salience, and just about everything held on other dates (primaries, local elections, run-offs, stand-alone special elections) as low-salience (Gerber et al., 2017). Salience is determined by the offices on the ballot and their associated level of media and public attention. Based on this definition, high-salience elections in the United States are Presidential and Midterm cycles. The benefit to the ex-ante approach is a shared understanding among scholars, practitioners, and the public of which elections are likely to be high vs. low salience. For instance, when Gerber et al. (2017) seek to conduct an experiment in a high-salience context, they select a U.S. Midterm. When Malhotra et al. (2011, p. 669–671) similarly look for a low-salience election, they choose a local contest described as a “sleepy, off-cycle election.” Scholars use the ex-ante approach when they choose low-salience elections for mobilization field experiments because they know these elections will see lower turnout and a less-crowded communication landscape, thus powering the detection of statistically significant effects with a smaller experiment sample size.

One alternative for categorizing elections relies on a post-hoc approach that considers turnout in the control group as an indicator of salience to the experimental population. Post hoc approaches have been used in analyses looking at heterogeneity in individual-level vote propensity within experiments (e.g. Arceneaux and Nickerson, 2009). Green and Gerber (2019) use a post hoc categorization by control group turnout in their meta-analysis of canvassing experiments. Looking at control group turnout is helpful for determining what kinds of voters are mobilized by given tactics, as explained by Arceneaux and Nickerson (2009). However, the post hoc approach is less useful here, given our desire to understand meta-analytic treatment effects. Turnout in the control group of any experiment is a function of election type (the ex-ante approach) plus the types of voters selected for the experimental population. We use the ex-ante approach because we are concerned with the real-world application of meta-analysis estimates by scholars seeking to understand elections and practitioners seeking to win them.

3. The “How” vs. “Who” of mobilization

Much prior work on electoral salience focuses on which individual voters can be mobilized within a given election (e.g. Arceneaux and Nickerson, 2009; Bergh and Christensen, 2022; Malhotra et al., 2011). Essentially, high-salience elections motivate a larger share of the electorate to vote on their own via a variety of naturally occurring mechanisms. Conversely, when salience is low, only a small set of highly attentive voters are inclined to participate. Arceneaux and Nickerson (2009) introduce the theory of contingent mobilization, arguing that an individual voter’s general likelihood to vote predicts the types of elections in which they are ripe for mobilization; their focus is on who can be mobilized in a given context. Malhotra and colleagues (2011) find similar variation in the types of voters who can be mobilized in a low-salience context (habitual ones) vs. higher-salience (casual voters). Conversely, this paper considers the how of mobilization: the specific tactics and effect sizes one can reasonably expect given the election in which they are applied.

Scholars have already observed that mobilization effects are generally smaller in higher-salience compared to lower-salience contexts (Bergh and Christensen, 2022; Gerber et al., 2017; Green and Gerber, 2019; Malhotra et al., 2011). We quantify this difference using meta-analytic estimates across mobilization tactics studied extensively in the literature. We expect larger effects in low-salience contexts for two reasons. First, the pool of individuals who do not participate in high-salience elections are more difficult to mobilize due to lower interest in and information about politics, lower political efficacy, and/or weaker commitments to social norms about voting and sense of civic duty. These sources of mobilization difficulty are interrelated, and there is debate about those relationships (Blais and Daoust, 2020). However, we imagine that few will quibble with the assertion that it is harder to move people with these traits to the polls. Second, high-salience elections have more mobilization activity by a range of civic and partisan organizations. This increased activity—imagine three distinct canvassing operations all knocking on the same door—attenuates the magnitude of the effect observed in any one particular experiment.

These two phenomena are interrelated. An experiment measures the net effect of its specific assigned treatment over and above everything that happens to the control group, and the baseline propensity to vote and level of mobilization activity in the control group rises with election salience. In an extreme low-salience election, the treatment might be the only mobilization activity taking place within the electorate, so the effect is the difference between zero units of mobilization in the control group versus one unit of mobilization in the treatment group. Conversely, in a high-salience election, the entire electorate is likely to be contacted by multiple entities outside of the experiment, thus raising turnout. The net effect of being the second, 12th, or 22nd unit of mobilization in the treatment group will be smaller than the first unit, as demonstrated by the diminishing returns in experiments on multiple mobilization contacts (Gerber et al., 2003; Green and Zelizer, 2017).

We build on prior work pertaining to salience by considering the role of the mobilization tactic itself within high- and low-salience contexts. Based on theory, we expect that meta-analytic estimates of GOTV tactics will differ in high- and low-salience contexts, and that the former will be attenuated relative to the latter. Indeed, statistical tests for heterogeneity indicate that datasets containing experiments conducted in high- and low-salience elections should not be pooled. We generally find attenuation in mean treatment effect of a mobilization tactic between high and low salience elections in the range of 33–76%. Substantively, the differences between high- and low-salience estimates for each tactic make clear that our understanding of mobilization and voting behavior is distorted if we do not separate experiments by election context. Moreover, the different magnitudes of effects in high- and low-salience electoral contexts creates a different picture of the relative efficacy of these tactics—with important implications for understanding voting behavior and campaign activity.

4. Methods

4.1. Data

We analyze GOTV field experiments included in the meta-analysis appendices of Green and Gerber (2019), henceforth referred to as the GG set. Although the GG set was collected four years ago, it is the most comprehensive data available on voter mobilization field experiments. Moreover, the extensive citation of Green and Gerber’s synthesis of voter mobilization field experiments—including their own meta-analyses—makes it relevant and appropriate for our critique of the inadequate, inconsistent attention to electoral salience in the literature.4 The difficulty of identifying the gray literature—especially unpublished reports by civic and political organizations that comprise a significant portion of the GG set—highlights the value to the field of the resources Green and Gerber invest to collect the fullest possible set of experiments.

Though we reviewed each experiment, categorized the salience of the election in which it was conducted, and corrected a small number of records (see Supplement), we generally defer to the data as reported in the Green and Gerber meta-analyses (e.g., using mean effect of multiple treatment arms).5

4.2. Mobilization treatments

Our analysis examines four distinct modes of treatment: direct mail, canvassing, phone calls, and SMS messages. Based on prior work, we further disaggregate these studies as follows:

Direct mail: We distinguish between mailings using social pressure messages and non-social pressure messages since there appears to be a difference when deploying this psychological mechanism (Green and Gerber, 2019).

Phone calls: We distinguish between automated (robo) calls that deliver pre-recorded messages, live calls made by paid callers, and live calls made by volunteers, following the literature on call quality and effectiveness (Nickerson, 2006, 2007).

SMS messages: We distinguish between warm SMS, wherein the sender has a prior relationship with the recipient (Dale and Strauss, 2009), and cold SMS, wherein the sender has no relationship with the recipient, since there appears to be a meaningful difference in effectiveness based on this relationship (Malhotra et al., 2011).

Canvassing: The literature has not identified an additional dimension moderating the effectiveness of canvassing, although recent work on ‘deep canvassing’ (e.g., Broockman and Kalla, 2016) indicates that interaction quality is likely a significant factor.

4.3. Measurements

We use the ex-ante approach to election salience in American politics with a clear decision rule: even-year general elections in which seats in the U.S. House of Representatives are elected are coded as high-salience; all other elections are coded as low-salience. This definition of high-salience is broad since it includes Presidential elections in swing and non-swing states, states with and without competitive statewide races, Congressional races, etc. Unfortunately, the body of available experiments does not yet support meta-analysis of narrower definitions of salience. In the Supplement, we disaggregate estimates for Midterm vs. Presidential cycles but see no statistical signs of differences, with the large caveat that there is a severe shortage of available experiments in Presidential elections for nearly all tactics studied.

We do not examine heterogeneity based on other aspects of the treatment, such as vote propensity of the experimental population (Arceneaux and Nickerson, 2009), message content other than social pressure mail, timing of treatment delivery (Murray and Matland, 2014; Panagopoulos, 2011), type of organization (García Bedolla and Michelson, 2012) or variations in election administration procedures such as availability of early voting. Some or all of these features likely account for additional heterogeneity. We do not (yet) have sufficient data to meta-analytically investigate these dimensions.

4.4. Analytic procedure

We follow Green and Gerber (2019) in using a DerSimonian–Laird random effects model for each meta-analysis so that model specification does not contribute to differences in estimated mean effects between our meta-analyses and their meta-analyses. Each figure depicts the mean estimated treatment effect (θ). We also report Cochran’s Q test, which indicates the likelihood of further heterogeneity within each group of experiments. Further details are available in the supplement.

Each meta-analytic estimate is plotted with a 95% confidence interval; the size of each square represents the weight of an individual experiment in the meta-analysis. The diamond at the bottom of each plot shows the mean estimated effect; the width of the diamond represents the 95% confidence interval for the mean estimated effect.

4.5. Known unknowns

We note that the dataset should concern scholars based what is and what is not in it. Due to the academic “file drawer problem,” studies that fail to show clear, statistically significant effects are often unable to be published. Civic and political organizations are also often unwilling to share results of proprietary studies, whether effective or not, despite Green and Gerber’s efforts to obtain these “gray literature” studies. These entities have a strategic interest in keeping highly effective tactics secret from competitors and minimizing attention to failed experiments. These two factors produce an incomplete set of studies, without a clear sense of the direction or magnitude of the consequent bias.

5. Results

We find consistent evidence that treatments generate smaller percentage point impacts on turnout in high-salience contexts, with attenuation of 33–76%. Table 1 compares our meta-analytic estimates by salience to the estimates that commingle high- and low-salience experiments as reported in Green and Gerber’s (2019) Get Out the Vote, with 95% confidence intervals.6 Following conventions in the mobilization field experiments literature, we first report CACE estimates for canvassing and phone calls and ITT effects for robocalls, direct mail, and SMS based on the GG set, before converting all estimates to ITT in Table 2 to facilitate comparison.

Table 1. Meta-Analytic Estimates of Mobilization Tactics, Commingled vs. By Electoral Salience.

Attenuation is calculated as (low-salience mean effect – high-salience mean effect)/(low-salience mean effect) for each meta-analytic estimate; caution needed due to small cell counts.

Table 1 also reveals an alarming paucity of experiments in numerous cells. Despite the existence of 240 separate experiments in total, there are fewer than ten experiments for robo calls, social pressure mail, cold SMS and warm SMS in high-salience elections and paid calls, robo calls, cold SMS, and warm SMS in low-salience elections. Moreover, the meta-analytic weights in Fig. 1, Fig. 2, Fig. 3, Fig. 4 reveal that many of these experiments are not well-powered due to small sample sizes. Thus, meta-analytic estimates for these tactics rest on a narrow base of available evidence. Though we report estimates using the best evidence available, we caution that more experiments are needed to increase confidence about generalizability.

Fig. 1. Meta-analytic estimates of canvassing experiments by electoral salience.

Fig. 2. Meta-analytic estimates of phone experiments by electoral salience and type.

Fig. 3. Meta-analytic estimates of direct mail by electoral salience and type.

Fig. 4. Meta-analytic estimates of SMS experiments by electoral salience and type.

5.1. Canvassing

The estimated mean CACE for canvassing in low-salience elections is 4.58pp (CI: 2.97, 6.18; n = 32) and 3.05pp (CI: 1.20, 4.90; n = 24) in high-salience elections (Fig. 1 and SM Tables S1–S2). The 33.4% attenuation from low-to high-salience elections is substantial. Within both the low- and high-salience sets of canvassing experiments in Fig. 1, there is still considerable evidence of heterogeneity in the CACEs (low-salience elections: I2 = 49%, Q(23) = 45.21, p = .004; high-salience elections: I2 = 60%, Q(31) = 76.75, p < .001). Canvassing estimates also rely heavily on experiments dating back many years. Societal change has made successful contact at the door increasingly difficult, so these estimates may not reflect the current effectiveness of the tactic.

5.2. Phone calls

For volunteer calls, the estimated mean CACE is 3.62pp (CI: 1.51, 5.74; n = 12) in low-salience elections and 2.28pp (CI: 1.12, 3.43; n = 20) in high-salience elections, an attenuation of 37.0% (Fig. 2, SM Tables S3–S4). For professional calls, the estimated mean effect is 1.66pp (CI: 0.89, 2.43; n = 8) in low-salience elections and 0.74pp (CI: 0.31, 1.17; n = 14) in high-salience elections, an attenuation of 55.4% (Fig. 2 and SM Tables S5–S6).

There is still considerable indication of heterogeneity in treatment effects of volunteer calls in both contexts (high-salience: I2 = 45%, Q(19) = 34.51, p = .016; low-salience: I2 = 67%, Q(11) = 33.65, p < .001), paid calls in high-salience elections (I2 = 36%, Q(13) = 20.28, p = .088), and robocalls in low-salience elections (I2 = 62%, Q(4) = 10.55, p = .184).

Notably, our estimates suggest that robocalls have a 0.58pp impact in low-salience contexts (CI: 0.08, 1.09; n = 5), possibly because they provide a reminder about the election (Fig. 2 and SM Tables S7–S8). However, since there are only five experiments available from GG set, inferences should be tentative and replication is needed. Given the limited number (n = 2) of high-salience robocall experiments, we cannot draw meaningful conclusions about their effect.

We again note that these estimates rely heavily on experiments on phone calls dating back many years. Recent declines in response rates to phone surveys, widespread adoption of mobile phones that are more costly to dial, and regulation of robocalls present increasing challenges for the deployment of phone-based mobilization and may further weaken all treatment effects.

5.3. Direct mail

For social pressure mail, estimated mean ITT is 2.45pp (CI: 1.67, 3.23; n = 15) in low-salience elections and 1.38pp (CI: 0.308, 2.45; n = 5) in high-salience elections, an attenuation of 43.7% (Fig. 3 and SM Tables S11–S12). For non-social pressure mail, estimated mean ITT is 0.46pp (CI: 0.26, 0.66; n = 47) in low-salience elections and 0.28pp (CI: 0.13, 0.43; n = 38) in high-salience elections, an attenuation of 39.1% (Fig. 3 and SM Tables S9–S10).

Social pressure appears approximately five times more effective than non-social pressure in both low- and high-salience elections. However, additional experiments on social pressure in high-salience elections (n = 5) are needed. Moreover, political scientists and practitioners have raised concerns about social pressure that should be more fully investigated, including potential backlash against the sender, reactance against political participation, and questions about the source and force of voting norms (Bergan et al., 2022; Blais and Daoust, 2020; Fieldhouse et al., 2022; Green and Gerber, 2019; Mann, 2010; Matland and Murray, 2014).

The forest plot in Fig. 2 and treatment effect heterogeneity tests for each of the four subsets of direct mail experiments strongly suggest there are further drivers of heterogeneity even within electoral salience, which may be due to location, target universe, contents of non-social pressure mail, strength of social pressure, mail timing, or other factors (high-salience social pressure mail: I2 = 84%, Q(37) = 226.63, p < .001; high-salience non-social pressure mail: I2 = 90%, Q(4) = 38.67, p < .001; low-salience social pressure mail: I2 = 76%; Q(188.31) = 33.65, p < .001; low-salience non-social pressure mail: I2 = 95%, Q (14) = 301.89, p < .001).

5.4. SMS messages

Experiments using SMS messages in the U.S. are more limited than other tactics. Ten experiments use “cold” outreach directly from the voter file, with six in low-salience elections and four in high-salience elections. Eight experiments use “warm” outreach in which the recipient has opted to hear from the sending organization (Fig. 4 and SM Table S13). Moreover, the weights in the right column of Fig. 4 show that only 2–3 experiments of each type drive the meta-analytic estimates.

Among the warm SMS experiments, only one was conducted in a low-salience context, rendering meta-analytic estimates moot. The estimated mean ITT among warm SMS in high-salience elections is 0.68pp (CI: 0.05, 1.41, n = 8), with evidence of further heterogeneity (I2 = 55%, Q(7) = 15.67, p = .028) possibly due to variation in sender-recipient relationships. Among the cold SMS experiments, the estimated mean ITT is 0.58pp (CI: 0.37, 0.79, n = 6) in low-salience elections (I2 = 35%, Q(5) = 7.70, p = .173) and 0.22pp (CI: 0.05, 0.39, n = 4) in high-salience elections (I2 = 0%, Q(3) = 1.00, p = .801), an attenuation of 62.1% (Fig. 4 and SM Tables S14–S15).

We again note that these estimates from the GG set rely on many experiments conducted several years ago when SMS was not as central to communication as it is today (see Supplement for expanded set of studies). These societal changes may influence the effects from SMS voter mobilization in future elections.

6. Comparisons across tactics

We use these meta-analytic estimates to calculate comparable ITT rates for tactics in the GG set. Table 1 contains an inferential trap unless read carefully: The CACE and ITT effects in Table 1 cannot be directly compared. CACE accounts for whether a given tactic (canvassing, phone calls) is successfully delivered—in other words, if the voter can actually be reached, how effective is this tactic? CACE will always be larger than ITT for a given tactic, since mathematically CACE is ITT divided by the contact rate, which is unlikely to reach 100%. Conversely, ITT estimates are reported for direct mail, SMS, and robocalls because observation of treatment delivery is not possible.

In Fig. 5, below, the diagonal dashed lines represent ITT effects (Y axis) generated by canvassing, volunteer calls, and paid calls based on hypothetical contact rates (X axis). These diagonal lines can be compared to ITT estimates for social pressure mail (squares) and the cold SMS (triangles), in high- and low-salience elections, respectively (see also SM Table S16). As the plot indicates, when using realistic contact rates, canvassing and phone calls are unlikely to produce sizably larger increases in participation than social pressure mail and cold SMS. Contact rates for canvassing conventionally fall around 25–30% depending on population (Green and Gerber, 2019). Phone contact rates for professional phone centers have fallen from 36% in 1997 to just 6% in 2018 (Kennedy and Hartig, 2019). Political professionals with experience running volunteer phone banks expect contact rates around 10–12%. Moreover, attempting to increase the contact rate from phone banks is counter-productive since it often lowers treatment delivery quality (Nickerson, 2007).

Fig. 5. Comparing GOTV tactics using ITT effects.

In short, based on the available experiments, contact rates necessary for canvassing or volunteer phones to generate more votes than social pressure mail are well above real-world experience. For the ITT effect of canvassing to match the ITT effect of social pressure mail, the contact rate would have to exceed 45% in high-salience contexts and 53% in low-salience contexts. For phone calls to match the mean estimated ITT effect of social pressure mail, the contact rate would have to exceed 60% in high-salience contexts and 67% in low-salience contexts. The ITT effect of paid phones does not match social pressure mail even if 100% of intended treatment is successfully delivered.

Cold SMS mobilization treatments also compare favorably with ITT effects for phone calls, using the available experiments. In both high- and low-salience contexts, the ITT effect of cold SMS is identical to the ITT effect of volunteer phones and widely exceeds the ITT effects of paid phones. However, ITT effects for canvassing with realistic contact rates exceed the ITT for cold SMS, so, based on the available experiments, if a population can be successfully contacted at their door is it worth attempting to do so; if they cannot be contacted at their residence they should be texted.

7. Impact of mobilization efforts

A simple example illustrates the importance of accounting for election salience and treatment delivery rate when using meta-analytic estimates to understand the impact of mobilization efforts. Table 2 calculates the number of net votes that a practitioner might expect to be produced from one million mobilization targets, comparing the commingled meta-analytic estimates that combine high- and low-salience elections to those calculated using only high-salience contests.7 The first block of columns are the meta-analytic estimates that appear in Table 1. The second block of columns calculates ITT meta-analytic estimates for canvassing and phones by dividing the CACE estimates by realistic treatment delivery rates. Scanning down the columns in the second block after standardization on ITT estimates reveals how misleading it would be to compare CACE and ITT estimates directly.

Table 2. Calculating the Impact of Mobilization on One Million Registered Voters, Commingled vs. High-Salience vs. Meta-analytic Estimates.

The right side of Table 2 reports a projection of expected votes using the commingled and high-salience ITT estimates, then calculates the deficit between someone relying on the former instead of the latter. In a high-salience election, scholars or practitioners relying on the commingled estimates would find themselves 19–40% short of the votes they were mistakenly expecting. For scholars, this under-estimation leads to incorrect inferences about voting behavior and the impact of mobilization efforts. For practitioners, not turning out enough voters often leads to an undesirable outcome when ballots are counted.

8. Discussion

This research note provides updated meta-analytic estimates of mobilization tactics accounting for the salience of the election in which they are used. While we already knew generally that mobilization effects weaken between low- and high-salience elections (e.g. Bergh and Christensen, 2022; Gerber et al., 2017; Green and Gerber, 2019; Malhotra et al., 2011), our paper quantifies this degree of attenuation: commonly used mobilization tactics have 33–76% smaller effects in high-salience elections than low-salience elections.8 Furthermore, by converting CACE estimates to ITT based on hypothetical contact rates, we also facilitate the direct comparison across tactics, which shows how small expected treatment effects are at the level of assignment rather than successful delivery. Given the decline in contact rates on doors and phones and increase in the use of SMS since much of the available research was conducted, scholars must revisit these tactics to test their current impact.

We also identify substantial gaps in our understanding the impact of voter mobilization activities, suggesting that GOTV tactics are not “settled science.” Out of a corpus of 240 experiments, only 40 were conducted in U.S. Presidential cycles (traditional mail = 16; canvassing = 8; paid calls = 4; cold SMS = 4; warm SMS = 3; volunteer calls = 3; robo-calls = 1; social pressure mail = 1). Scholars often choose not to conduct GOTV experiments in such contexts because it is harder to observe an effect. Civic and political organizations may also be less cooperative with research in high-stakes elections. Based on the average effect (0.05, 95% CI -0.24-0.33) of direct mail in Presidential cycles (Supplement Materials Table S17), we surmise that there may be other null results and that all Presidential cycle estimates need to be revised downward. We urge journals and scholars to facilitate this research by accepting registered reports of GOTV experiments conducted in high-salience cycles, regardless of findings.

Statistical evidence of additional heterogeneity within subsets of experiments defined by salience and tactic suggests other directions for future meta-analyses of the voter mobilization field experiments literature. Message content, delivery timing, quality of delivery, and message source potentially matter. There is still ample room in the mobilization literature to refine our understanding of each component. Furthermore, the characteristics of the universe of voters targeted for mobilization matters (Arceneaux and Nickerson, 2009; Malhotra et al., 2011; Green and Gerber, 2019), yet many studies fail to report this in sufficient detail. Scholars should include the characteristics of their experimental populations in terms of demographics and prior voting history, ideally as part of pre-registration, and report results for theoretically relevant subgroups.

Our study has important limitations. We use data available from Green and Gerber (2019) because of its prominence in synthesizing the literature; we did not attempt to update their data collection because it was beyond our resources. We were unable to include studies from other countries but recognize the value of looking at experiments on mobilization in other systems to fully understand the impact of electoral salience. While our definition of election salience is commonly recognized and easy to operationalize within the U.S., other definitions exist, such as post-hoc and first/second order elections. Furthermore, our definition fails to account for competitiveness of jurisdictions within high-salience elections, which can explain some of the variance in turnout we see within the same election.

Despite these areas for future improvement, as well as what is included and omitted in the available set of studies, these meta-analyses are sufficient to illustrate that the effects of voter contact methods attenuate in high-salience contexts in ways that are consequential for scholars of voting behavior. We hope that future meta-analyses will account for electoral salience and other factors, and will include additional new experiments that bolster thin areas in the available evidence for understanding voter mobilization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary data to this article:

supplemental-data-for-meta-analysis
supplemental-data-for-meta-analysis.pdf
4 MB
download-circle

References

  1. Arceneaux and Nickerson, 2009Kevin Arceneaux, David W. NickersonWho is mobilized to vote? A re‐analysis of 11 field experimentsAm. J. Polit. Sci., 53 (1) (2009), pp. 1-16
  2. Bergan et al., 2022Daniel E. Bergan, et al.Promoting the youth vote: the role of informational cues and social pressurePolit. Behav., 44 (4) (2022), pp. 2027-2047
  3. Bergh and Christensen, 2022Johannes Bergh, Dag Arne ChristensenGetting out the vote in different electoral contexts: the effect of impersonal voter mobilization techniques in middle and high salience Norwegian electionsJ. Elections, Public Opin. Parties (2022), pp. 1-17, 
  4. Blais and Daoust, 2020André Blais, Jean-François DaoustThe Motivation to Vote: Explaining Electoral ParticipationUBC Press, Vancouver (2020)
  5. Broockman and Kalla, 2016David Broockman, Josh KallaDurably reducing transphobia: a field experiment on door-to-door canvassingScience, 352 (6282) (2016), pp. 220-224
  6. Dale and Strauss, 2009Allison Dale, Aaron StraussDon’t forget to vote: text message reminders as a mobilization toolAm. J. Polit. Sci., 53 (4) (2009), pp. 787-804
  7. Fieldhouse et al., 2022Edward Fieldhouse, David Cutts, Jack BaileyWho cares if You vote? Partisan pressure and social norms of votingPolit. Behav., 44 (3) (2022), pp. 1297-1316 
  8. García Bedolla and Michelson, 2012Lisa García Bedolla, Melissa R. MichelsonMobilizing InclusionYale University Press (2012)
  9. Gerber et al., 2003Alan S. Gerber, Donald P. Green, Matthew GreenPartisan mail and voter turnout: results from randomized field experiments.”Elect. Stud., 22 (4) (2003), pp. 563-579
  10. Gerber et al., 2017Alan S. Gerber, Gregory A. Huber, Albert H. Fang, Andrew GoochThe generalizability of social pressure effects on turnout across high-salience electoral contexts: field experimental evidence from 1.96 million citizens in 17 statesAm. Polit. Res., 45 (4) (2017), pp. 533-559
  11. Green and Gerber, 2019Donald P. Green, Alan S. GerberGet Out the Vote: How to Increase Voter Turnout Brookings, Washington, D.C (2019)
  12. Green and Zelizer, 2017Donald P. Green, Adam ZelizerHow much GOTV mail is too much? Results from a large-scale field experiment.”J. Exp. Polit. Sci. (2017), pp. 1-12
  13. Kennedy and Hartig, 2019Courtney Kennedy, Hannah HartigResponse Rates in Telephone Surveys Have Resumed Their DeclinePew Research Center (2019)
  14. Malhotra et al., 2011Neil Malhotra, Melissa R. Michelson, Todd Rogers, Ali A. ValenzuelaText messages as mobilization tools: the conditional effect of habitual voting and election salienceAm. Polit. Res., 39 (4) (2011), pp. 664-681
  15. Mann, 2010Christopher B. MannIs there backlash to social pressure? A large-scale field experiment on voter mobilizationPolit. Behav., 32 (3) (2010), pp. 387-407
  16. Murray and Matland, 2014Gregg R. Murray, Richard E. MatlandMobilization effects using mail: social pressure, descriptive norms, and timingPolit. Res. Q., 67 (2) (2014), pp. 304-319
  17. Nickerson, 2006David W. NickersonVolunteer phone calls can increase turnout: evidence from eight field experimentsAm. Polit. Res., 34 (3) (2006), pp. 271-292
  18. Nickerson, 2007David W. NickersonQuality is job one: professional and volunteer voter mobilization callsAm. J. Polit. Sci., 51 (2) (2007), pp. 269-282
  19. Panagopoulos, 2011Costas PanagopoulosTiming is everything? Primacy and recency effects in voter mobilization campaignsPolit. Behav., 33 (1) (2011), pp. 79-93

Read more