Abstracts

Martyna Swiatczak (Univ. of Bergen): Membership inflation in CNA. How over-and underrepresentation of factors affect CNA’s performance

Is over- and underrepresentation of factors in the data problematic for CNA's performance? And if yes: under which circumstances and to what degree? It can be shown that in ideal data, i.e. data with no noise or fragmentation, over- and underrepresentation of factors do not affect the quality of CNA’s output. However, I will present an extended series of simulation studies that reveals that, when paired with fragmentation or noise, over- and underrepresentation may substantively affect CNA's performance — a data characteristic that is defined as membership inflation. I will demarcate areas of membership inflation, discuss limitations of the presented studies, and outline countermeasures that can be taken in the face of membership inflation. 

Rafael Quintana (Univ. of Kansas): Embracing complexity in social science research

Social and behavioral phenomena are fundamentally complex in the sense that they are shaped by many interdependent causes. Researchers that adopt a complex systems perspective have argued that, rather than focusing on a single causal relationship at a time, we need to investigate how the interaction or combination of different factors generate specific outcomes. The main objective of this article is to review three methodological frameworks that have been used to investigate the interdependencies between causal factors, which is often referred to as the study of causal complexity. The three frameworks are: interaction analysis, which investigates effect heterogeneity; structural analysis, which investigates causal mechanisms; and configurational analysis, which investigates sufficient and necessary conditions. I summarize the goals and recent developments of these techniques, as well as two theoretical frameworks—intersectionality theory and the so-called “heterogeneity revolution”—that stress the importance of investigating causal complexity in social science research.

Edward Miech (Regenstrief Institute): Validating CNA models with sequestered data: A novel application of Coincidence Analysis using a split-sample design

Split-sample designs offer a novel option for assessing and validating CNA model performance using real-world data.  In a split-sample design, a dataset is randomly split into two separate sets:  a training sample and a validation sample.  The validation sample is immediately set aside and “sequestered” until the final step of the analysis.  Models are then developed using the training sample alone.  In the last phase of the analysis, the performance of these models is tested against the sequestered data in the validation sample.

This secondary analysis of a large-n observational cohort (n=3079) used a split-sample design in order to validate two CNA models.  The overall sample consisted of 3079 Veterans between the ages of 24 to 99 years (median age, 70 years; interquartile range 64-78) who presented at a US Department of Veterans Affairs hospitals with transient ischemic attack (TIA) between October 2016 and September 2017.  This dataset was randomly divided into a ~70% training sample (2192/3079) and ~30% validation sample (887/3079).

This analysis modeled two different outcomes: a patient outcome and a quality-of-care outcome.  The patient outcome was the combined endpoint of all-cause mortality or recurrent ischemic stroke within one-year post-TIA. The quality-of-care outcome was the “without-fail” rate,  the proportion of patients who received all processes for which they were eligible among seven different processes.

In this study I summarize the performance of these two different CNA models when tested against the sequestered data and discuss how split-study designs can provide strong  evidence for the validity of CNA results.

Tim Haesebrouck (Ghent University): The populist radical right and military intervention: A CNA of military deployment votes

Although populist radical right (PRR) parties have been studied intensively for decades, very few comparative studies on the parliamentary behavior of PRR parties have been conducted. This article aims to fill this gap in academic research by examining the pattern of PRR voting on military deployments. The results of our analysis indicate that PRR parties are more inclined to vote in favor of contributions to operations that are deployed to balance the threat of Jihadi terrorism. However, the majority of PRR party votes on military deployments is not determined by factors related to the operation in which forces are deployed, but is driven by the expected impact of the parliamentary vote on the PRR parties’ broader vote-, office- and policy-seeking objectives. This expected impact, in turn, is determined by the interplay between party size, government experience, the party’s level of anti-elitism, government membership and the ideological orientation of the government.

Jonathan Freitas (Univ. of Minas Gerais): A roadmap for the coincidence analysis of clustered data

The analysis of datasets consisting of multiple data points for each unit of observation opens up new opportunities for the further development of CNA. This presentation offers a roadmap that depicts seven strategies one can adopt for performing the coincidence analysis of that type of data: “multilevel modeling”, “generalizing”, “particularizing”, “clustering”, “disaggregating”, “going remote”, and “leveling up”. Each of these possible analytic moves is explained, as well as their interconnections, and pros and cons. In particular, three new procedures are introduced in the context of the roadmap: (1) the application of the alternative specification of the Totally Fuzzy and Relative (TFR) calibration approach to obliviate the need to account for inter-cluster variation during the analysis; (2) the adaptation and extension of the Cluster Diagnostic approach to assess model fit across multiple units of observation; and (3) a protocol for representing MINUS-formulas as networks to enable both the clustering and the merging of atomic solution formulas selected from analyses of different units of observation for a same outcome of interest. While initially developed as responses to specific challenges imposed by clustered data, these procedures and some modules of the proposed roadmap, taken as stand-alone methodological resources, may be applicable to many other contexts of configurational data analysis with CNA.

Reiping Huang (Northwestern): Coincidence Analysis (CNA) applications in a surgical research center

Using examples from an academic surgical research center in the U.S., the author will illustrate several challenges that health services researchers have faced in applying CNA to real-world problems. These practical challenges, partly due to data integration from mixed types and sources, call for study-specific strategies, sometimes involving combined use of other regular analytic methods. The second part of the presentation will focus on two areas that configurational methods have the greatest potential so far to enhance health services research, factor selection and contextual effect testing. Instead of elaborating on any single study, this presentation will discuss CNA applications in general, probably casting more questions than answers.   

Jiji Zhang (Hong Kong Baptist Univ.): What kind of difference-making should CNA track?

It is often taken as a truism that a cause makes a difference to its effect; theories of causation that respect this dictum sometimes include a condition of minimality or non-redundancy to disqualify factors that fail to make a difference from the list of causally relevant ones. In this talk I examine the minimality conditions required by the Coincidence Analysis (CNA) from the perspective of causal difference-making, and motivate a stronger minimality condition that is analogous to one routinely assumed in probabilistic causal modelling. Specifically, I show that a causal hypothesis that does not meet the stronger minimality requirement would feature a factor for which the coincidence data do not provide evidence for its unconfounded difference-making effect on the dependent factor. Including such hypotheses in the output of the CNA seems to be unwarranted, if the CNA aims to track only those causal relevancies that are supported by data.

Deborah Cragun  (University of South Florida): Challenges and unique insights from using CNA with real world data 

I will use real-world examples to illustrate systematic ways to combine data from multiple stakeholders across multiple analytic sites for use in CNA. I will also demonstrate the value of using FRscore software when substantial model ambiguity is identified. Unique findings reveal how causal chain models can be more robust and have higher coverage and consistency than the corresponding common cause model. Other unique findings include an example of multi-value CNA with a trichotomous outcome that uncovered one or more paths for each of the three outcome values. The practical value of these results in helping us tailor our implementation guide will be reviewed. Finally, ongoing challenges will be discussed throughout the presentation with input requested from other attendees.  

Susanne Tafvelin/Ole Henning Sørensen (Umeå Univ./NFA Denmark): Identifying conditions contributing to outcomes of a workplace intervention using Coincidence Analysis

Although organisational occupational interventions (OOHIs) currently are recommended in policies across Europe, empirical evidence for the effectiveness of OOHIs is mixed. One reason for the mixed findings of the effectiveness of OHHIs may be the way these interventions are evaluated, which typically focus on the extent to which the intervention causes a statistically significant change in a predetermined outcome. Such analyses are limited to testing only a few predicting factors and their interactions, which does not mirror the complexity of OOHIs. The purpose of the present study is to empirically test coincidence analysis (CNA) as an alternative approach to evaluate OOHIs. We apply CNA on a large, organizational-level intervention project conducted in 64 pre-schools in Denmark to examine how implementation factors contribute to intervention outcomes. Our findings suggest that one path to intervention success is through perceptions of relevance, and that another path is through a combination of either high effort among implementation team members and low time invested by intervention participants or vice versa. In addition, some issues that were identified when applying CNA on organizational intervention data will be discussed.

Veli-Pekka Parkkinen (University of Bergen): Norms of correctness for CNA models

In the context of configurational causal modeling with CNA, correctness of a model with respect to its intended search target is defined in terms of a submodel-relation between a candidate and the target. Two models related as sub- and supermodel cannot disagree on any causal relevance claim they entail, but the submodel may entail fever claims than the supermodel. Hence, a less than complete model of the target structure counts as correct as long as it abstains from ascribing false causal relevance relations. The last mentioned point is presumably uncontroversial, and any sensible notion of correctness used to benchmark causal discovery methods should incorporate it. I first argue that the submodel-definition of correctness fails to acknowledge an important subset of correct but less than complete models. In these cases, a candidate model describes some indirect causal relations as direct causal relations, without making any false causal claims. In other words, the candidate is a coarse-grained description of the target. In such a case, the target entails a difference-making dependence of the appropriate kind for each causal claim entailed by the candidate model, and the candidate does not entail any difference-making dependencies that contradict the target. Hence, there is no principled grounds for judging the candidate to be incompatible with the target. Yet, such a candidate is not a submodel of the target. I thus propose that the submodel-definition of correctness should be replaced by one that defines correctness in terms of the difference-making dependencies entailed by the candidate and the target models: A candidate model is correct if and only if ideal data returned by the target contains a difference-making pair of cases for every causal claim entailed by the candidate. This proposal has the advantage that it is directly motivated by the (M)INUS-theory of causation that is the conceptual justification of CNA. But the proposal needs refinement, as sometimes the data returned by the target are ambiguous between two or more causal interpretations, and this will affect which case comparisons count as difference-making pairs for which causal claims. Two options are open. The first one relies purely on the data returned by the target: a candidate model is correct if and only if there is any causal interpretation of the data in light of which difference-making pairs that license the candidate exist in that data. The second option involves assuming that an ambiguous target can always be disambiguated by expanding the factor set, such that a candidate model is correct if and only if the data returned by the target, in light of the single causal interpretation given by the target, contains the appropriate difference-making pairs. Both options have downsides. The latter option involves making a metaphysical assumption that is not part of the (M)INUS-theory of causation. The former option has a consequence that sometimes a candidate model that disagrees with the target on the causal ordering of some factors will count as correct. The weight of these downsides, and the practicability of applying either option in benchmarking CNA will be compared.