STUDY 3

To follow up the exploratory analyses from Study 2, we conducted a third, preregistered study. Our first aim was to re‐run bivariate correlations to confirm relationships between key variables. We expected (a) strong positive correlations between belief in unverified conspiracies, belief in verified conspiracies, and response bias, (b) moderate positive intercorrelations between our three measures of cognitive sophistication, (c) small negative associations between measures of cognitive sophistication and belief in unverified conspiracies, but non‐significant associations between these measures and belief in verified conspiracies, and (d) moderate positive associations between the cognitive sophistication measures and discrimination (AUC).

The second aim was to confirm which variables predict conspiracy discrimination by re‐running a multiple linear regression analysis. In line with Study 2, we expected general knowledge, scientific reasoning, trust in experts, conspiracy mentality, political conservatism, age and gender to uniquely predict discriminability. Additionally, for each cognitive measure, we compared two regression models—one with only response bias as a predictor, and one with both response bias and discrimination—to test whether discrimination is a stronger predictor than response bias. We expected response bias alone to negatively predict each measure, but only discrimination to be a significant (positive) predictor when both are included.

The final aim was to re‐examine our comparison of ‘Good discriminators’, ‘Indiscriminate Sceptics’ and ‘Indiscriminate Believers’ on cognitive sophistication. In line with Study 2, we expected Good Discriminators to outperform Indiscriminate Sceptics and Indiscriminate Believers on our three cognitive measures, but for Indiscriminate Sceptics and Indiscriminate Believers not to differ statistically from one another.

Methods

The methods for Study 3 were similar to Study 2, with some minor differences. Participants were recruited in the same way, but pre‐screening took place directly before the study. We also recruited a gender‐balanced sample entirely from the United States. The general knowledge items were also converted to true‐false questions, with scores computed as percentage correct (Range: 0 to 1).

We preregistered to recruit up to 350 participants. This was based on available funding, but a priori power analyses also indicated that a sample of this size ensures sufficient statistical power (>80%) to detect small‐to‐medium effect sizes (f > .20) in one‐way ANOVAs with three groups. This effect size estimate is conservative given the main effects from Study 2. A sample of this size also provides 80% power to detect effects of f > .06 in multiple regression analyses involving up to 12 predictors.

Of the 350 participants recruited, seven were excluded for failing at least one attention check, in line with the preregistration. This left a final sample of 343 participants. The mean age was 46.0 (SD = 13.4) and the sample was split evenly between genders (51.0% woman/female; 48.4% man/male; 0.6% non‐binary). Most participants (69.1%) identified as White/Caucasian (11.4% African American; 9.3% Asian; 7.6% Hispanic; 2.3% other; 0.3% did not say). Most (72.6%) also had tertiary level education or more (27.1% secondary education or less; 0.3% did not say) and the vast majority (95.6%) reported English as their first language (4.4% said it was not).

Results and discussion

Relationships between variables

Table 6 displays the zero‐order correlations for Study 3. Many of the relationships aligned with Study 2 and our preregistered predictions. There were strong positive correlations between belief in unverified conspiracies, belief in verified conspiracies and response bias. There were also moderate positive intercorrelations between our measures of cognitive sophistication (general knowledge, cognitive reflection and scientific reasoning). Additionally, we found small, negative correlations between belief in unverified conspiracies and the three cognitive sophistication measures, and non‐significant correlations between belief in verified conspiracies and both cognitive reflection and scientific reasoning. However, there was a small but significant positive correlation between belief in verified conspiracies and general knowledge. Finally, distinguishing between verified and unverified conspiracies (AUC) was moderately correlated with scores on the three cognitive sophistication measures. In addition to these preregistered predictions, AUC was positively correlated with trust in experts, age and education, and negatively correlated with conspiracy mentality and political conservatism.

 | M | SD | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14
1. Unverified consp. belief | 3.61 | 2.30 | – |  |  |  |  |  |  |  |  |  |  |  |  | 
2. Verified consp. belief | 5.21 | 2.00 | 0.51** | – |  |  |  |  |  |  |  |  |  |  |  | 
3. Response bias (c) | 0.05 | 0.84 | 0.86** | 0.80** | – |  |  |  |  |  |  |  |  |  |  | 
4. Discriminability (AUC) | 0.63 | 0.17 | −0.62** | 0.29** | −0.24** | – |  |  |  |  |  |  |  |  |  | 
5. General knowledge | 0.74 | 0.15 | −0.26** | 0.18** | −0.07 | 0.41** | – |  |  |  |  |  |  |  |  | 
6. Scientific reasoning | 6.66 | 2.86 | −0.34** | 0.06 | −0.17** | 0.41** | 0.40** | – |  |  |  |  |  |  |  | 
7. Cognitive reflection | 4.50 | 2.16 | −0.29** | 0.04 | −0.18** | 0.34** | 0.43** | 0.49** | – |  |  |  |  |  |  | 
8. Conspiracy mentality | 6.41 | 1.74 | 0.71** | 0.43** | 0.64** | −0.39** | −0.14* | −0.28** | −0.20** | – |  |  |  |  |  | 
9. Trust in experts | 4.43 | 1.50 | −0.44** | −0.16** | −0.34** | 0.34** | 0.17** | 0.15** | 0.14** | −0.37** | – |  |  |  |  | 
10. Conservatism | 4.26 | 2.01 | 0.31** | 0.15** | 0.28** | −0.24** | −0.11* | −0.14** | −0.09 | 0.30** | −0.54** | – |  |  |  | 
11. SES | 5.07 | 1.83 | −0.14** | −0.07 | −0.11* | 0.04 | 0.16** | 0.13* | 0.19** | −0.11* | 0.12* | 0.02 | – |  |  | 
12. Age | 46.03 | 13.40 | −0.14** | −0.02 | −0.10 | 0.15** | −0.06 | 0.02 | 0.00 | −0.18** | −0.18** | 0.14* | −0.02 | – |  | 
13. Education | 2.95 | 0.80 | −0.21** | −0.08 | −0.17** | 0.15** | 0.18** | 0.21** | 0.24** | −0.18** | 0.14* | 0.00 | 0.38** | 0.09 | – | 
14. Gender | – | – | 0.00 | 0.08 | 0.04 | 0.08 | 0.07 | 0.02 | 0.09 | −0.07 | 0.08 | 0.05 | −0.03 | −0.16** | −0.01 | –
15. Native English | – | – | 0.12* | 0.15** | 0.15** | −0.01 | −0.01 | −0.03 | −0.01 | 0.13* | −0.10 | 0.11* | −0.02 | 0.07 | −0.08 | −0.02

Predicting discriminability

We used a linear regression analysis to reassess the unique predictors of conspiracy discriminability (see Table 7). The predictors together explained 38.9% of the variance. Consistent with Studies 1 and 2, and with preregistered hypotheses, general knowledge and scientific reasoning scores predicted better discrimination. Being older, less conspiracy minded and more trusting of experts also predicted better discrimination, as predicted. However, political conservatism and gender were not significant independent predictors.

Predictor | β | b | SE | t‐value | p‐value
Intercept | – | 0.2 | .08 | 2.98 | .003
General knowledge | .25 | 0.29 | .06 | 4.96 | <.001
Scientific reasoning | .20 | 0.01 | .00 | 3.83 | <.001
Cognitive reflection | .08 | 0.01 | .00 | 1.61 | .108
Age | .18 | 0.00 | .00 | 3.83 | <.001
Education | .00 | 0.00 | .01 | .010 | .992
Gender | −.05 | 0.02 | .01 | −1.11 | .269
Native English | −.04 | −0.01 | .02 | −.080 | .423
Conspiracy mentality | −.17 | −0.02 | .01 | −3.39 | .001
Trust in experts | .20 | 0.02 | .01 | 3.68 | <.001
Political conservatism | −.04 | 0.00 | .00 | −0.73 | .467
SES | −.07 | −0.01 | .00 | −1.44 | .150

Predicting cognitive sophistication

As preregistered, we flipped the analytic direction by conducting linear regression analyses to assess whether AUC more strongly predicts cognitive sophistication compared to response bias. When response bias was the only predictor, it did not significantly predict general knowledge (b = −0.01, p = .197), but it was a significant negative predictor for cognitive reflection (b = −0.47, p = .001) and scientific reasoning (b = −0.59, p = .001). When AUC was added as a second predictor, however, response bias was either no longer significant or a much weaker predictor (general knowledge: b = 0.01, p = .563; cognitive reflection: b = −0.28, p = .040; scientific reasoning: b = −0.27, p = .122). AUC, by contrast, was consistently a significant, positive predictor of cognitive sophistication (general knowledge: b = 0.36, p < .001; cognitive reflection: b = 3.96, p < .001; scientific reasoning: b = 6.53, p < .001).

Comparing correlations with conspiracy mentality

Though not preregistered, we conducted a dependent samples Fisher's r‐to‐z test to compare conspiracy mentality's association with response bias and discriminability. Conspiracy mentality was more strongly associated with response bias than with discriminability, z = 21.97, p < .001. This suggests that conspiracy mentality reflects response tendency rather than accurate discrimination.

Group comparisons

For our comparison of groups, the results mostly aligned with hypotheses, but there were some deviations (see Figure 2; Table 8). We grouped participants into ‘Good Discriminators’ (AUC > .7), ‘Indiscriminate Believers’ (AUC < .7; c > 0) and ‘Indiscriminate Sceptics’ (AUC < .7, c < 0) based on principled thresholds. The ANOVAs for each cognitive sophistication measure were significant. Follow‐up analyses showed that Good Discriminators, as anticipated, outperformed Indiscriminate Sceptics and Indiscriminate Believers on general knowledge, cognitive reflection and scientific reasoning. Indiscriminate Sceptics and Indiscriminate Believers did not differ statistically for general knowledge, as expected. However, Indiscriminate Sceptics outperformed Indiscriminate Believers on cognitive reflection and scientific reasoning.

FIGURE 2: Comparison of three groups on cognitive ability measures in Study 3. In Panel a, Indiscriminate Believers (purple), Indiscriminate Sceptics (green), and Good Discriminators (dark blue) are plotted according to their mean belief in unverified and verified conspiracies. The regression line represents the correlation between the two. General knowledge (b), scientific reasoning (c), and cognitive reflection (d) scores for each group are depicted in the remaining panels. Rain clouds represent score distributions, individual points depict each participant's score and orange diamonds represent group means. Good Discriminators significantly outperformed the other two groups on all three measures. Indiscriminate Sceptics outperformed Indiscriminate Believers on scientific reasoning and cognitive reflection, but not general knowledge. When differences in discrimination were controlled for, however, the three groups did not significantly differ on any measure.

Variable | Grouping method
Thresholds | 3‐profile LPA
IB (n = 129) Mauc = .50 Mc = .82 | IS (n = 98) Mauc = .57 Mc = −.71 | GD (n = 116) Mauc = .83 Mc = −.15 | IB (n = 127) Mauc = .50 Mc = .86 | IS (n = 118) Mauc = .60 Mc = −.71 | GD (n = 98) Mauc = .84 Mc = −.06
General knowledge | Mean (SD) | 0.71 (0.13) | 0.72 (0.15) | 0.80 (0.14) | 0.71 (0.14) | 0.72 (0.15) | 0.82 (0.13)
Overall F (df) Pairwise comparisons | F(2, 340) = 15.20, p < .001 IB vs. IS: Mdiff = 0.01, p = .857 IB vs. GD: Mdiff = 0.09, p < .001 IS vs. GD: Mdiff = 0.08, p < .001 | F(2, 340) = 22.97, p < .001 IB vs. IS: Mdiff = 0.01, p = .728 IB vs. GD: Mdiff = 0.12 p < .001 IS vs. GD: Mdiff = 0.11, p < .001
Cognitive reflection | Mean (SD) | 3.75 (2.29) | 4.53 (1.98) | 5.30 (1.85) | 3.76 (2.37) | 4.52 (1.95) | 5.44 (1.72)
Overall F (df) Pairwise comparisons | F(2, 340) = 17.23, p < .001 IB vs. IS: Mdiff = 0.78, p = .014 IB vs. GD: Mdiff = 1.55, p < .001 IS vs. GD: Mdiff = 0.77, p = .019 | F(2, 340) = 18.51, p < .001 IB vs. IS: Mdiff = 0.76, p = .011 IB vs. GD: Mdiff = 1.68, p < .001 IS vs. GD: Mdiff = 0.92, p = .003
Scientific reasoning | Mean (SD) | 5.64 (2.65) | 6.50 (2.60) | 7.92 (2.83) | 5.63 (2.69) | 6.59 (2.55) | 8.06 (2.86)
Overall F (df) Pairwise comparisons | F(2, 340) = 22.17, p < .001 IB vs. IS: Mdiff = 0.86, p = .046 IB vs. GD: Mdiff = 2.29, p < .001 IS vs. GD: Mdiff = 1.42, p < .001 | F(2, 340) = 22.56, p < .001 IB vs. IS: Mdiff = 0.96, p = .015 IB vs. GD: Mdiff = 2.43, p < .001 IS vs. GD: Mdiff = 1.47, p < .001

We also grouped participants using a LPA based on the same clustering models as in Study 2, and this suggested three profiles. Results for the profile‐based group comparisons were similar to the threshold‐based comparisons. All three ANOVAs were significant. Pairwise comparisons revealed that Good Discriminators outperformed the other two groups on each measure. Indiscriminate Sceptics did not differ from the Indiscriminate Believers on general knowledge, but they outperformed Indiscriminate Believers on cognitive reflection and scientific reasoning.

That said, Indiscriminate Sceptics had a slightly higher mean AUC than Indiscriminate Believers under both grouping methods (see Table 8). Group differences in discriminability could therefore explain why Indiscriminate Sceptics scored modestly higher than Indiscriminate Believers on some cognitive measures. To test this, we followed each ANOVA above with an ANCOVA controlling for AUC. If the observed group differences are driven by discrimination rather than response bias, group differences should disappear once AUC is accounted for. Indeed, none of the groups differed significantly across all pairwise comparisons based on these ANCOVAs (_p_s > .05).

These findings, together with the linear regression analyses from earlier, demonstrate that conspiracy discrimination is a stronger predictor of cognitive sophistication than response bias. This supports the central claim from Study 2 that cognitive sophistication is more closely tied to accurately distinguishing verified from unverified conspiracies than a tendency to either believe or disbelieve conspiracies.