Understanding, Communicating, and Reducing Analytical Uncertainty: Theory, Visualization Designs, and an Augmented Presentation System to Support Validation and Interpretation of a Multiverse Analysis

Abstract

Analyzing data is a complex, multi-step process, with multiple choices available at each step, such as whether and how to exclude outliers, what approach to use to operationalize a variable, or what model and parameters to apply. While it is often possible to exclude some choices as invalid, often many alternatives remain that are equally valid, and when these alternatives lead to divergent conclusions there exists what I term analytical uncertainty. Faced with this complexity and the practical demands for professional productivity, analysts often report either a single analysis or a small number of non-divergent supporting analyses, which can result in what has been called uncertainty laundering: misrepresenting uncertainty as if it were a known quantity. Yet accurately communicating analytical uncertainty, which is often non-quantified or non-quantifiable, in a way that is both practically useful and professionally acceptable often proves to be an exceedingly difficult task.

In this dissertation, I explore ways to use data visualizations and interactive systems to support the assessment, communication, and reduction of analytical uncertainty. My aim is to pare away a slice of the ontological uncertainty present in empirical research and render it in such a way that not only can it be assessed and clearly communicated, but also so that it can be reduced.

A review of literature on topics related to uncertainty led me to reconceptualize the method of multiverse analysis as a way to assess and communicate analytical uncertainty. A systematic review and critical analysis of published multiverse analysis reports resulted in the derivation of a taxonomy of multiverse analysis tasks, as well as the identification and detailed description of multiverse visualizations archetypes. I observed that a specific set of multiverse analysis tasks---validation tasks and interpretation tasks---are critical to making a multiverse analysis useful and meaningful, yet are also the tasks that are the least supported by existing archetypes and interactive systems.

I developed a prototype system is developed named AugMeet, which uses two techniques in combination to support validation and interpretation tasks: augmented presentation; and a specially designed series of interactive visualizations, including new designs, namely: parameter-faceted outcome curves, outcome-faceted outcome curves, and twin-faceted residual plots. Based on evaluation interviews with seven experienced researchers, I offer four conclusions:

1) The augmented presentation aspects of AugMeet seemed to serve primarily as a way to make it easier for the audience to listen to and understand the information being conveyed to them about and through the visualizations themselves.

2) Comparing residuals of twin universes elicits divergent thinking, even though the design intent of twin-faceted residual plots was to avoid spurious conclusions about the superiority of one multiverse option over another.

3) Parameter-faceted outcome curves give perspective and focus first by providing an overview of all the choices that have been identified as worth considering, and then by providing visual features to identify what is important (potentially impactful) and what is not.

4) AugMeet supports the iterative group effort necessary to complete difficult validation and interpretation tasks, and participants found particular value in the demonstrated workflow that featured progressive iteration around a multiverse in a group setting.

Analytical uncertainty is far from trivial to assess, communicate, and reduce, but the techniques developed and explicated within this dissertation show how it is indeed possible and can be practicable.