Centre for the Study of the Sciences and the Humanities

Numbers for policy: Practical problems in quantification

The course introduces concepts of practice and ethics of quantification, seen as an antidote to inconsiderate uses of numbers both in academia and in society. It shows the pitfalls to be avoided and offers - with examples, tools and recipes for reasonable uses of quantitative methods. The course aims at practitioners, post-docs and PhD students with an interest in the use of evidence for policy.


Main content

The course has ended. Slides for all presentations can can be found here. 

While this is not a course on impact assessment, it gives impact assessors an extra gear, plus the set of skill needed to tell apart defensible versus indefensible assessments and risk analyses, enabling participants to spot bogus, implausible or irrelevant quantifications. 
The course also invites reflection on the emergence of problems in the system of quality control of science which has in statistical and mathematical modelling a point of intense vulnerability and friction.
Examples from epidemiology to criminology, from pharmacology to psychology, from big data to unethical use of algorithms – will be discussed. Elements of sociology of quantification and ethics of algorithms will also be part of the teaching. 

The course includes an analysis of the genesis of the Cartesian Dream of prediction and control of nature and society thanks to the power of a mathematized science. We discuss the incredible success of the dream, as well as his historical, philosophical and ecological critique.

Technical material will be presented on statistical procedures and malpractices (p-hacking, p-HARKing) and how to address them.

The course takes inspiration from the works of Silvio Funtowicz and Jerome R. Ravetz, the fathers of post-normal science (PNS). Reference books of these authors for the course are about science and society, science and power, and uncertainty and quality.  
PNS-inspired practices advocated and described in the course are:

- Pedigrees for quantification such as NUSAP,

- Sensitivity auditing, quantitative story-telling and ethics of quantification

The course will also include elements of  technical sensitivity analysis.


Day one Monday, 27. August

Morning 9.00-13.00

  1. An introduction (1): The social organization of science, Matthias Kaiser
  2. An introduction (2): The case of the p-test and science’s reproducibility problems, Andrea Saltelli
  3. Prodromes of post normal science (1): Silvio Funtowicz
  4. NUSAP’s history and practice: Jeroen van der Sluijs

Afternoon 14.00-16.00

  1. Practicum with Jeroen van der Sluijs

Day two, Tuesday, 28. August

Morning 9.00-13.00

  1. On Reductionism: Ragnar Fjelland
  2. Prodromes of post normal science (2): Silvio Funtowicz
  3. Sensitivity analysis: Andrea Saltelli
  4. The licence not to quantify. Qualitative versus quantitative analyses: Bruna De Marchi

Afternoon 14.00-16.00

  1. Practicum with Bruna De Marchi 

Day three, Wednesday, 29. August

Morning 9.00-13.00

  1. When laypeople are right and experts are wrong. Lessons from Love Canal: Ragnar Fjelland
  2. Ethics of quantification: Andrea Saltelli
  3. When numbers are controversial: Jeroen van der Sluijs
  4. Gentlemen don’t publish: by Silvio Funtowicz

Afternoon 14.00-16.00

  1. Practicum with Andrea Saltelli  

Day four, Thursday 30. August

Morning 9.00-13.00

  1. Sociology of quantification: Alexandra Theben
  2. Sensitivity auditing and quantitative storytelling. The seven rules with illustrations: Andrea Saltelli
  3. Climate controversies: Jeroen van der Sluijs
  4. Issues with ethics. Hippocratic Oath and policy failures: Matthias Kaiser

Afternoon 14.00-16.00

  1. Practicum with Jeroen van der Sluijs  

Day five, Friday 31. August  

Morning 9.00-13.00

  1. The now of science: Silvio Funtowicz
  2. Ethics of algorithms: Alexandra Theben
  3. Bees and pollinators: Jeroen van der Sluijs
  4. Scenario design: conflicted facts and values, Mimi Lam 

Afternoon 14.00-16.00

  1. Practicum with Andrea Saltelli  

Day six, Saturday 1. September

Morning 10.00-12.30  

  1. General discussion and feed-back from the participants opened by two short presentation by Hans-Martin Füssel and Elodie Thirion: Was the course useful / actionable?

Duration of lessons: Morning: from 0900 to 1300, 4 times 45 minutes (total lecture time180 m per day) with three 15 m breaks. Lunch 13.00 to 14.00. Afternoon group work 14.00-16.00

Supporting material

Excerpts from Ravetz, J., 1971, Scientific Knowledge and its Social Problems, Oxford University Press.

Andrea Saltelli, Silvio Funtowicz, 2017, What is science’s crisis really about? FUTURES, Volume 91, Pages 5-11, https://doi.org/10.1016/j.futures.2017.05.010

Ragnar Fjelland, 2016, When Laypeople are Right and Experts are Wrong: Lessons from Love Canal, HYLE – International Journal for Philosophy of Chemistry, Vol. 22 (2016), 105-125.

Chapter 8 from Winner, L., 1986. The Whale and the Reactor: a Search for Limits in an Age of High Technology. The University of Chicago Press.

Excerpts from E. F. Schumacher, 1973, Small Is Beautiful. Economics as if People Mattered, Harper Perennial 2010.

Saltelli, A., Stark, P.B., Becker, W., and Stano, P. , 2015, Climate Models as Economic Guides. Scientific Challenge or Quixotic Quest? Issues in Science and Technology (IST), Volume XXXI Issue 3, Spring 2015.

Jeroen P. van der Sluijs, 2012, Uncertainty and Dissent in Climate Risk Assessment: A Post-Normal Perspective, Nature and Culture 7(2), Summer 2012: 174–195 © Berghahn Journals doi:10.3167/nc.2012.070204.

Funtowicz, S & Ravetz J, undated, NUSAP - The Management of Uncertainty and Quality in Quantitative Information.

Jeroen P. van der Sluijs, James S. Risbey and Jerry Ravetz, 2005, Uncertainty Assessment of VOC Emissions from Paint in the Netherlands Using the NUSAP System, Environmental Monitoring and Assessment (2005) 105: 229–259, DOI: 10.1007/s10661-005-3697-7.

Saltelli, A., Annoni, P., 2010, How to avoid a perfunctory sensitivity analysis, Environmental Modeling and Software, 25, 1508-1517.  

Saltelli, A., Funtowicz, S., 2014, When all models are wrong: More stringent quality criteria are needed for models used at the science-policy interface, Issues in Science and Technology, Winter 2014, 79-85.

Funtowicz, S.O. and Ravetz, J.R. (1985) Three Types of Risk Assessment: A Methodological Analysis, in: V.T. Covello, J.L. Mumpower, P.J.M. Stallen and V.R.R. Uppuluri (eds) Environmental Impact Assessment, Technology Assessment, and Risk Analysis, pp. 831–48, New York: Springer.

De Marchi, B. (2015) “Risk Governance and the integration of scientific and local knowledge”  Ch. 9, pp. 149-165 in Fra.Paleo, U. (Ed.) Risk Governance. The Articulation of Hazard, Politics and Ecology. Berlin: Springer. ISBN 978-94-017-9328-5.


Requirements for 3 ECTS version:
a) preparation: reading of course material
b) preparation: submission of 1-2 page text where they explain their "problem" or "concern" or other reflection that fit the course topic
c) attendance 
d) short "reflection memo" (1-3 page text) after the course which demonstrates having reflected on its content and assimilated some of its key messages.  

Requirements for 5 ECTS version:
a) preparation: reading of course material 
b) preparation: submission of 1-2 page text where they explain their "problem" or "concern" or other reflection that fit the course topic 
c) attendance 
d) submission of outline/topic (10-15 page text) for final essay which demonstrates having reflected on its content, assimilated some of its key messages, and put it to critical use in the context of the student's problem or concern   
e) final essay (10-15 page text)

Essays will be marked pass or fail, the course participants themselves are each responsible for course approval at home institution / department.