Objectivity in Social Research

tba

The Evaluative Aspect of the Concept of Addiction in Economics: The Case of Gary Becker.

In this talk, we explore “thick concepts,” i.e., concepts that have both an evaluative and descriptive aspect, and their relation to economics. To render this project tractable, we focus on the Theory of Rational Addiction proposed by economists Gary Becker et al. (1988, 1996) as a case study.  We argue that there is a tension between Becker’s view of economics as being a purely empirical enterprise and the concept of addiction that he ultimately proposes. While Becker acknowledges a tension between the ‘ordinary’ concept of addiction as a thick concept and economics, his attempt to remove the evaluative aspect of that concept fails. Our argument hinges on a reading of Becker as employing the strategy of explication. On this reading, Becker’s project of constructing an explicatum that discards non-epistemic values fails. Finally, some possible ramifications of this failure are discussed.

Objectivity and Experimental Practices in the Social Sciences

The experimental revolution in the Social sciences is one of the most significant methodological shifts undergone by the field  since the turn of the century, having effects on the possibility of cross-collaboration of formerly separate research areas and above all, changing the way social scientists view and deal with problems of causal identification. One of the often valued features of social science experimentation is, precisely, the fact that there are clear methodological rules regarding hypothesis testing that allow for the adjudication among contentious causal claims. The paper tries to spell out the conditions under which this kind of role is possible and underlines the often crucial though often ignored role of background knowledge in mediating between experimental results and the inferences that can be drawn from them.

For reasons that are partly contingent, as I discuss in the paper, a fundamental component of the conceptual set of tools that we use to describe social scientific experimental results and practices in includes the distinction between internal and external validity, as first conceived by Campbell (1957) and Cook and Campbell (1979). I develop the idea that one undesirable consequences of the extensive use of the distinction between internal and external validity is that the terms, and the conceptual and methodological approach in which they are embedded (the Campbellian project) tends to assume that there is a correspondence between experiments and the inferences that can be made from them.

This alleged correspondence between experiments and their inferences has, in turn, as a consequence, the underplaying of the role of background knowledge in inferring causal statements from experimental data, flattening the role of experiments to that of objective or impartial trials that can conclusively adjudicate among contentious causal claims. While this characterization may suit certain aspects of concrete experimental research programs, it does not represent the role of experimentation in much of the laboratory practices in growing subfields such as theoretical experimental sociology or behavioral economics. 

Live Empirical Issues in Debates over Objectivity

Obviously, there are many senses of objectivity detailed in the general philosophy of science literature which are relevant to the social sciences. My interest here, however, is not so much to contribute directly to that literature as it is to focus on issues related to objectivity that make a difference to how we use and practice social science. I will rely, however, on one standard distinction viz., the traditional one between objectivity as a claim about epistemology and as about ontology, though I draw on the distinction only to show how tightly claims invoking the two can be.

Part I of the paper concerns epistemological objectivity. My goal is to deflect arguments that objectivity is inherently limited in the social sciences while showing that there are a host of ongoing, quite practical issues that the social sciences can deal with successfully but nonetheless often do so badly. I look at skeptical arguments against objectivity based on underdetermination of theory by evidence, on the holism of testing, on value ladenness, and on indeterminacy of classification and kinds. In all four cases, progress first requires being explicit about what social science is at issue and exactly why that work is allegedly not objective and why. All the skeptical objections point out possibilities only, not necessities. I show how certain general epistemic virtues associated with objectivity are made concrete and can be successfully applied in the social sciences, looking in detail at experimental work on risky choice and at causal inference from observational data via regression related tools. I also suggest that actual practice often falls far short of achieving these virtues. 

Part II looks at objectivity in the sense of ontological independence. After making a few fairly obvious preliminary points, I examine claims that the social is dependent upon, constituted by, etc. individual subjectivity. There is a mass of intermixed claims here about dependency that need to be sorted. That dependency, I argue, is often not conceptual or constitutive but causal in ways often not appreciated. I also argue that some of these alleged dependencies of the social on various senses of individual subjectivity can have epistemological consequences. My main example in Part II, developed in detail, is money. The social-structural nature of the way money has developed and works is generally underappreciated by philosophers; properly understood, money provides little support for subjectivist antirealist attitudes about the social world.

Objectivity in Contexts

After Heather Douglas’s influential analysis (2004; 2009), many philosophers of science have agreed that instead of attempting to develop a coherent philosophical account of objectivity, we should either focus on distinguishing various distinct notions of scientific objectivity, or as e.g. Hacking (2015) suggests, renounce the whole notion. However, as I have recently argued (Koskinen forthcoming 2019), the diverse senses of objectivity recognised in the recent literature can be covered with a single, negative account: When we call X objective, we say that we rely on X, and that others should do so too, as important epistemic risks arising from our imperfections as epistemic agents have been effectively averted. All the positive senses of objectivity identify either some risk of this type, or some efficient strategy for averting one or more such risks. 

This "risk account" represents objectivity as a contextual matter: our diverse imperfections as epistemic agents lead to diverse epistemic risks, some of which become important in some contexts, and other ones in others. Moreover, also the risk mitigation strategies we use must vary according to context, as not all strategies are efficient everywhere. In this paper I focus on strategies developed in the humanities and qualitative social sciences, comparing them to strategies developed in the context of the natural sciences. As Daston and Galison (2007) note, objectivity has historically been strongly linked to the duty of scientists to avoid subjectivity. Strategies developed with this aim in mind typically screen out the subjective biases and idiosyncrasies of individual researchers. However, such strategies are not efficient against collective bias. I will argue that strategies developed for averting Whig historiography and ethnocentric anthropology are efficient and reflect the perceived importance of averting different types of collective bias in the social sciences and the humanities.

Statistical Objectivity and Metascience

The meaning of objectivity in any specific setting reflects historically situated understandings of both science and self. Recently, various scientific fields have confronted growing mistrust about the replicability of findings, and statistical techniques have been deployed to articulate a “crisis of false positives.” This has prompted a scientific social movement of proposed reforms, including regulating disclosure of “backstage” research details and enhancing incentives for replication. Together, these events represent the emergence of a new formulation of objectivity. Statistical objectivity assesses the integrity of research literatures in the results observed in collections of studies rather than the methodological details of individual studies and thus positions meta-analysis as the ultimate arbiter of scientific objectivity. I conclude by discussing how this movement has created a new meta-scientific field and exacerbated tensions between existing fields. 

Robust Scientific Institutions as a Solution to Fact/Value Entanglement

Freedom from the influence of non-epistemic values is one of the main senses of ‘scientific objectivity’. There are many reasons to believe, however, that scientific objectivity in this sense is an unattainable ideal. This paper takes its starting point from the reality of fact/value entanglement and addresses its consequences for the organisation of scientific inquiry. Specifically, it proposes the development of scientific institutions that function independently of the existence of consensus on visions of the good society (‘the common good’) or factual claims about effective strategies to help realise any of these visions. Its starting point is the belief that disagreements about values are here to stay because (a) individuals value many things; (b) no agreement about the relative importance of different values is forthcoming; (c) nor is agreement about what it means to realise any given value in a specific situation forthcoming. The paper offers some empirical evidence in support of these claims. It is then argued that disagreement about values feeds through to factual beliefs because of fact-value entanglement. The paper concludes by criticising some recent proposals about the management of values in science on the basis of their presumption that there is such a thing as the ‘correct set of social values’ and developing the concept of ‘Robust Relevant Research and Innovation’ (R³I) in response, the main contention of which is that institutions of relevant research and innovation undertake active steps to represent a plurality of different value profiles. 

The Problems with Context Independent Appeals to Objectivity: A Case Study of Gender Metrics

Stoet and Geary (2019) propose a new index for measuring gender inequality (the Basic Index for Gender Inequality, or BIGI). BIGI is characterised by the authors as a necessary corrective to existing ways of measuring gender inequality; where indices such as the United Nations’ Global Gender Gap Index (GGGI) focus exclusively on issues already known to disadvantage women and ignore the harms experienced disproportionately by men, BIGI only measures three ‘core’ aspects of life which are relevant to all people. Significantly, BIGI claims that, contrary to received wisdom, men are more disadvantaged than women in the majority of countries surveyed. 

There are many immediately apparent problems with the BIGI proposal, which make the metric unfit for purpose on its own terms. However, in this paper we argue that dismissing the BIGI proposal as simply bad social science will not help to explain why, despite its many flaws, the proposal has been able to gain considerable public traction in the little time since its publication (as have other recent, similarly interventionist pieces of social scientific research). This traction, we argue, derives from two related sources. The first is the work being done in the BIGI proposal by certain scientific norms, which we identify as versions of (a) objectivity, and (b) context-independence. We argue that it is by conflating an absolute understanding of objectivity (‘objective truth’) with a kind of political neutrality and presenting such neutrality as a virtue for all uses of a metric that Stoet and Geary are able to suggest BIGI is preferable to GGGI. The traction of their argument derives from the fact that both the alternative metrics they seek to criticise, and more general interpretations of what counts as a good social science metric, also rely on variations of these same norms: in this respect, BIGI is utilising a social scientific zeitgeist to its advantage. This, in turn, leads to the second source of traction for the BIGI proposal. Stoet and Geary correctly identify some challenges to indices such as the GGGI, which are used to motivate their proposed alternative: namely, an apparent lack of interest in the way that some harms impact men more than women, and a lack of an overarching concept of a good or adequate life to justify the various metrics pursued. Though these issues do not justify the move towards a BIGI-style approach to measuring gender inequality, we submit, they do reveal that indices such as GGGI must do more to contextualise their claims to objectivity, and make the theoretical postulates underpinning their approach explicit. To fail to do so is to perpetuate the same zeitgeist that leaves the door open for the BIGI proposal in the first place.

Humanistic Interpretation and Machine Learning

The objectivity of interpretive text analysis – humanistic interpretation – has been a hot potato in the social sciences since their beginning. The necessity of humanistic interpretation have been generally recognized, but many have retained their suspicions about the sources of bias that could influence the interpretive process. Thus the various attempts to formalize the interpretive process can be seen as attempts to make the interpretive choices more transparent and to control some possible biases. However, these attempts have met with opposition. Coding texts has been argued to be limited in terms of replicability and in its ability to account for nuances in textual meaning. At worst, coding procedures have been argued to impose interpretation on text data, distorting their underlying meaning structures and barring evidence important for forming a well grounded interpretation (Biernacki 2012).

The development of recent machine-learning based tools for text analysis have initiated the most recent debate about the objectivity of humanistic interpretation. While techniques based on supervised machine-learning seem to share the same problems as their coding-based ancestors, the unsupervised machine-learning techniques seem to promise something new. For example, Lee & Martin 2015 argue that what they call cultural cartography – a structural analysis of meaning will help to make interpretation more scientific. Instead of imposing interpretation on texts, unsupervised text analysis condenses information in text data into a simplified formal representation, which enables collective scrutiny of the represented textual meanings. Notice that nobody claims that computer-aided distant reading will replace humanistic close reading of documents. Rather, the claim is that incorporation of unsupervised machine-learning techniques makes the interpretation not only much more scalable, but also helps to avoid many possible biases of interpretation that derive from the interpreters’ preconceptions.

                The latter part of the paper analyzes how this might happen by close examination of the uses of unsupervised machine learning in social scientific articles. More precisely, we will investigate the prevalent ways of using topic modeling – a popular unsupervised approach among social scientists – in interpretive text analysis and analyze how it is actually used to support the interpretation. We will distinguish between the use of topic modelling for measuring theoretical constructs in text data, and for organizing textual materials in order to guide qualitative analysis. We argue that while in the latter case topic models guide interpretation through reducing the dimensionality of data into a simplified structure, in the former case unsupervised analysis works to shift the methodological role of interpretation to that of validating measured theoretical constructs. Further, both of these uses are associated with different problems pertaining to model selection and interpretation of results. 

The Objectivity of Qualitative Data Sets

The aim of the present paper is to offer a reconstructed notion of objective qualitative data sets that may serve as a useful guiding ideal in qualitative research. In this spirit, I propose that a qualitative data set is objective to the extent that it possesses various good-making features (epistemic virtues, epistemic values) in virtue of which it is suitable as evidence base for the final research report.

In the first part of the paper, I place the reconstructed notion in context. Among other things, I show how it is similar in spirit to Alison’s Wylie’s notion of objective theories. Moreover, I explicate the reconstructed notion via a discussion of the features constitutive of the objectivity of a qualitative data set. These features include, I propose, the data set containing data that are descriptively adequate, detailed, reactivity transparent, and relevant; and its being balanced and sufficiently large. In the second part of the paper, I defend the reconstructed notion against two possible lines of objections. One focuses on the specifics of the reconstructed notion arguing that it is, in one way or another, in need of amendment. The other goes further by contending that the reconstructed should be dismissed in favor of some alternative notion of objective qualitative data sets.