Intuitive statistics

Statisticians and probability theorists have long debated about the use of various tools, assumptions, and problems relating to inductive inference in particular.[1] David Hume famously considered the problem of induction, questioning the logical foundations of how and why people can arrive at conclusions that extend beyond past experiences - both spatiotemporally and epistemologically.[9] More recently, theorists have considered the problem by emphasizing techniques for arriving from data to hypothesis using formal content-independent procedures, or in contrast, by considering informal, content-dependent tools for inductive inference.[10][11] Searches for formal procedures have led to different developments in statistical inference and probability theory with different assumptions, including Fisherian frequentist statistics, Bayesian inference, and Neyman-Pearson statistics.[1]

Gerd Gigerenzer and David Murray argue that twentieth century psychology as a discipline adopted probabilistic inference as a unified set of ideas and ignored the controversies among probability theorists. They claim that a normative but incorrect view of how humans "ought to think rationally" follows from this acceptance. They also maintain, however, that the intuitive statistician metaphor of cognition is promising, and should consider different formal tools or heuristics as specialized for different problem domains, rather than a content- or context-free toolkit. Signal detection theorists and object detection models, for example, often use a Neyman-Pearson approach, whereas Fisherian frequentist statistics might aid cause-effect inferences.[1]

Frequentist inference focuses on the relative proportions or frequencies of occurrences to draw probabilistic conclusions. It is defined by its closely related concept, frequentist probability. This entails a view that "probability" is nonsensical in the absence of pre-existing data, because it is understood as a relative frequency that long-run samples would approach given large amounts of data.[12] Leda Cosmides and John Tooby have argued that it is not possible to derive a probability without reference to some frequency of previous outcomes, and this likely has evolutionary origins: Single-event probabilities, they claim, are not observable because organisms evolved to intuitively understand and make statistical inferences from frequencies of prior events, rather than to "see" probability as an intrinsic property of an event.[13]

Bayesian inference generally emphasizes the subjective probability of a hypothesis, which is computed as a posterior probability using Bayes' Theorem. It requires a "starting point" called a prior probability, which has been contentious for some frequentists who claim that frequency data are required to develop a prior probability, in contrast to taking a probability as an a priori assumption.[1][12]

Bayesian models have been quite popular among psychologists, particularly learning theorists, because they appear to emulate the iterative, predictive process by which people learn and develop expectations from new observations, while giving appropriate weight to previous observations.[14] Andy Clark, a cognitive scientist and philosopher, recently wrote a detailed argument in support of understanding the brain as a constructive Bayesian engine that is fundamentally action-oriented and predictive, rather than passive or reactive.[15] More classic lines of evidence cited among supporters of Bayesian inference include conservatism, or the phenomenon where people modify previous beliefs toward, but not all the way to, a conclusion implied by previous observations.[6] This pattern of behavior is similar to the pattern of posterior probability distributions when a Bayesian model is conditioned on data, though critics argued that this evidence had been overstated and lacked mathematical rigor.[16]

Alison Gopnik more recently tackled the problem by advocating the use of Bayesian networks, or directed graph representations of conditional dependencies. In a Bayesian network, edge weights are conditional dependency strengths that are updated in light of new data, and nodes are observed variables. The graphical representation itself constitutes a model, or hypothesis, about the world and is subject to change, given new data.[2]

Error management theory (EMT) is an application of Neyman-Pearson statistics to cognitive and evolutionary psychology. It maintains that the possible fitness costs and benefits of type I (false positive) and type II (false negative) errors are relevant to adaptively rational inferences, toward which an organism is expected to be biased due to natural selection. EMT was originally developed by Martie Haselton and David Buss, with initial research focusing on its possible role in sexual overperception bias in men and sexual underperception bias in women.[17]

This is closely related to a concept called the "smoke detector principle" in evolutionary theory. It is defined by the tendency for immune, affective, and behavioral defenses to be hypersensitive and overreactive, rather than insensitive or weakly expressed. Randolph Nesse maintains that this is a consequence of a typical payoff structure in signal detection: In a system that is invariantly structured with a relatively low cost of false positives and high cost of false negatives, naturally selected defenses are expected to err on the side of hyperactivity in response to potential threat cues.[18] This general idea has been applied to hypotheses about the apparent tendency for humans to apply agency to non-agents based on uncertain or agent-like cues.[19] In particular, some claim that it is adaptive for potential prey to assume agency by default if it is even slightly suspected, because potential predator threats typically involve cheap false positives and lethal false negatives.[20]

Heuristics are efficient rules, or computational shortcuts, for producing a judgment or decision. The intuitive statistician metaphor of cognition[1] led to a shift in focus for many psychologists, away from emotional or motivational principles and toward computational or inferential principles.[21] Empirical studies investigating these principles have led some to conclude that human cognition, for example, has built-in and systematic errors in inference, or cognitive biases. As a result, cognitive psychologists have largely adopted the view that intuitive judgments, generalizations, and numerical or probabilistic calculations are systematically biased. The result is commonly an error in judgment, including (but not limited to) recurrent logical fallacies (e.g., the conjunction fallacy), innumeracy, and emotionally motivated shortcuts in reasoning.[22][23][24][25] Social and cognitive psychologists have thus considered it "paradoxical" that humans can outperform powerful computers at complex tasks, yet be deeply flawed and error-prone in simple, everyday judgments.[26]

Much of this research was carried out by Amos Tversky and Daniel Kahneman as an expansion of work by Herbert Simon on bounded rationality and satisficing.[27] Tversky and Kahneman argue that people are regularly biased in their judgments under uncertainty, because in a speed-accuracy tradeoff they often rely on fast and intuitive heuristics with wide margins of error rather than slow calculations from statistical principles.[28] These errors are called "cognitive illusions" because they involve systematic divergences between judgments and accepted, normative rules in statistical prediction.[29]

Gigerenzer has been critical of this view, arguing that it builds from a flawed assumption that a unified "normative theory" of statistical prediction and probability exists. His contention is that cognitive psychologists neglect the diversity of ideas and assumptions in probability theory, and in some cases, their mutual incompatibility.[30][13] Consequently, Gigerenzer argues that many cognitive illusions are not violations of probability theory per se, but involve some kind of experimenter confusion between subjective probabilities with degrees of confidence and long-run outcome frequencies.[21] Cosmides and Tooby similarly claim that different probabilistic assumptions can be more or less normative and rational in different types of situations, and that there is not general-purpose statistical toolkit for making inferences across all informational domains. In a review of several experiments they conclude, in support of Gigerenzer,[21] that previous heuristics and biases experiments did not represent problems in an ecologically valid way, and that re-representing problems in terms of frequencies rather than single-event probabilities can make cognitive illusions largely vanish.[13]

Tversky and Kahneman refuted this claim, arguing that making illusions disappear by manipulating them, whether they are cognitive or visual, does not undermine the initially discovered illusion. They also note that Gigerenzer ignores cognitive illusions resulting from frequency data, e.g., illusory correlations such as the hot hand in basketball.[25] This, they note, is an example of an illusory positive autocorrelation that cannot be corrected by converted data to natural frequencies.[31]

For adaptationists, EMT can be applied to inference under any informational domain, where risk or uncertainty are present, such as predator avoidance, agency detection, or foraging. Researchers advocating this adaptive rationality view argue that evolutionary theory casts heuristics and biases in a new light, namely, as computationally efficient and ecologically rational shortcuts, or instances of adaptive error management.[32]

People often neglect base rates, or true actuarial facts about the probability or rate of a phenomenon, and instead give inappropriate amounts of weight to specific observations.[33][34] In a Bayesian model of inference, this would amount to an underweighting of the prior probability,[6] which has been cited as evidence against the appropriateness of a normative Bayesian framework for modeling cognition.[1][21] Frequency representations can resolve base rate neglect, and some consider the phenomenon to be an experimental artifact, i.e., a result of probabilities or rates being represented as mathematical abstractions, which are difficult to intuitively think about.[13] Gigerenzer speculates an ecological reason for this, noting that individuals learn frequencies through successive trials in nature.[35] Tversky and Kahneman refute Gigerenzer's claim, pointing to experiments where subjects predicted a disease based on the presence vs. absence of pre-specified symptoms across 250 trials, with feedback after each trial.[36] They note that base rate neglect was still found, despite the frequency formulation of subject trials in the experiment.[31]

Another popular example of a supposed cognitive illusion is the conjunction fallacy, described in an experiment by Tversky and Kahneman known as the "Linda problem." In this experiment, participants are presented with a short description of a person called Linda, who is 31 years old, single, intelligent, outspoken, and went to a university where she majored in philosophy, was concerned about discrimination and social justice, and participated in anti-nuclear protests. When participants were asked if it were more probable that Linda is (1) a bank teller, or (2) a bank teller and a feminist, 85% responded with option 2, even though it option 1 cannot be less probable than option 2. They concluded that this was a product of a representativeness heuristic, or a tendency to draw probabilistic inferences based on property similarities between instances of a concept, rather than a statistically structured inference.[24]

Gigerenzer argued that the conjunction fallacy is based on a single-event probability, and would dissolve under a frequentist approach. He and other researchers demonstrate that conclusions from the conjunction fallacy result from ambiguous language, rather than robust statistical errors or cognitive illusions.[37] In an alternative version of the Linda problem, participants are told that 100 people fit Linda's description and are asked how many are (1) bank tellers and (2) bank tellers and feminists. Experimentally, this version of the task appears to eliminate or mitigate the conjunction fallacy.[21][37]

There has been some question about how concept structuring and generalization can be understood in terms of brain architecture and processes. This question is impacted by a neighboring debate among theorists about the nature of thought, specifically between connectionist and language of thought models. Concept generalization and classification have been modeled in a variety of connectionist models, or neural networks, specifically in domains like language learning and categorization.[38][39] Some emphasize the limitations of pure connectionist models when they are expected to generalize future instances after training on previous instances. Gary Marcus, for example, asserts that training data would have to be completely exhaustive for generalizations to occur in existing connectionist models, and that as a result, they do not handle novel observations well. He further advocates an integrationist perspective between a language of thought, consisting of symbol representations and operations, and connectionist models than retain the distributed processing that is likely used by neural networks in the brain.[40]

Developmental psychologists such as Jean Piaget have traditionally argued that children do not develop the general cognitive capacities for probabilistic inference and hypothesis testing until concrete operational (age 7–11 years) and formal operational (age 12 years-adulthood) stages of development, respectively.[45][46]

This is sometimes contrasted to a growing preponderance of empirical evidence suggesting that humans are capable generalizers in infancy. For example, looking-time experiments using expected outcomes of red and white ping pong ball proportions found that 8-month-old infants appear to make inferences about population characteristics from which the sample came, and vice versa when given population-level data.[47] Other experiments have similarly supported a capacity for probabilistic inference with 6- and 11-month-old infants,[48] but not in 4.5-month-olds.[49]

The colored ball paradigm in these experiments did not distinguish the possibilities of infants' inferences based on quantity vs. proportion, which was addressed in follow-up research where 12-month-old infants seemed to understand proportions, basing probabilistic judgments - motivated by preferences for the more probable outcomes - on initial evidence of the proportions in their available options.[50] Critics of the effectiveness of looking-time tasks allowed infants to search for preferred objects in single-sample probability tasks, supporting the notion that infants can infer probabilities of single events when given a small or large initial sample size.[51] The researchers involved in these findings have argued that humans possess some statistically structured, inferential system during preverbal stages of development and prior to formal education.[47][50]

It is less clear, however, how and why generalization is observed in infants: It might extend directly from detection and storage of similarities and differences in incoming data, or frequency representations. Conversely, it might be produced by something like general-purpose Bayesian inference, starting with a knowledge base that is iteratively conditioned on data to update subjective probabilities, or beliefs.[52][53] This ties together questions about the statistical toolkit(s) that might be involved in learning, and how they apply to infant and childhood learning specifically.

Gopnik advocates the hypothesis that infant and childhood learning are examples of inductive inference, a general-purpose mechanism for generalization, acting upon specialized information structures ("theories") in the brain.[54] On this view, infants and children are essentially proto-scientists because they regularly use a kind of scientific method, developing hypotheses, performing experiments via play, and updating models about the world based on their results.[55] For Gopnik, this use of scientific thinking and categorization in development and everyday life can be formalized as models of Bayesian inference.[56] An application of this view is the "sampling hypothesis," or the view that individual variation in children's causal and probabilistic inferences is an artifact of random sampling from a diverse set of hypotheses, and flexible generalizations based on sampling behavior and context.[57][58] These views, particularly those advocating general Bayesian updating from specialized theories, are considered successors to Piaget's theory rather than wholesale refutations because they maintain its domain-generality, viewing children as randomly and unsystematically considering a range of models before selecting a probable conclusion.[59]

In contrast to the general-purpose mechanistic view, some researchers advocate both domain-specific information structures and similarly specialized inferential mechanisms.[60][61] For example, while humans do not usually excel at conditional probability calculations, the use of conditional probability calculations are central to parsing speech sounds into comprehensible syllables, a relatively straightforward and intuitive skill emerging as early as 8 months.[62] Infants also appear to be good at tracking not only spatiotemporal states of objects, but at tracking properties of objects, and these cognitive systems appear to be developmentally distinct. This has been interpreted as domain specific toolkits of inference, each of which corresponds to separate types of information and has applications to concept learning.[60][63]

Infants use form similarities and differences to develop concepts relating to objects, and this relies on multiple trials with multiple patterns, exhibiting some kind of common property between trials.[64] Infants appear to become proficient at this ability in particular by 12 months,[65] but different concepts and properties employ different relevant principles of Gestalt psychology, many of which might emerge at different stages of development.[66] Specifically, infant categorization at as early as 4.5 months involves iterative and interdependent processes by which exemplars (data) and their similarities and differences are crucial for drawing boundaries around categories.[67] These abstract rules are statistical by nature, because they can entail common co-occurrences of certain perceived properties in past instances and facilitate inferences about their structure in future instances.[68][69] This idea has been extrapolated by Douglas Hofstadter and Emmanuel Sander, who argue that because analogy is a process of inference relying on similarities and differences between concept properties, analogy and categorization are fundamentally the same process used for organizing concepts from incoming data.[4]

Infants and small children are not only capable generalizers of trait quantity and proportion, but of abstract rule-based systems such as language and music.[70][71] These rules can be referred to as “algebraic rules” of abstract informational structure, and are representations of rule systems, or grammars.[72] For language, creating generalizations with Bayesian inference and similarity detection has been advocated by researchers as a special case of concept formation.[73][74] Infants appear to be proficient in inferring abstract and structural rules from streams of linguistic sounds produced in their developmental environments,[75] and to generate wider predictions based on those rules.[76]

For example, 9-month-old infants are capable of more quickly and dramatically updating their expectations when repeated syllable strings contain surprising features, such as rare phonemes.[77] In general, preverbal infants appear to be capable of discriminating between grammars with which they have been trained with experience, and novel grammars.[78][79] In 7-month-old infant looking-time tasks, infants seemed to pay more attention to unfamiliar grammatical structures than to familiar ones,[72] and in a separate study using 3-syllable strings, infants appeared to similarly have generalized expectations based on abstract syllabic structure previously presented, suggesting that they used surface occurrences, or data, in order to infer deeper abstract structure. This was taken to support the “multiple hypotheses [or models]” view by the researchers involved.[80][81]

Multiple studies by Irene Pepperberg and her colleagues suggested that Grey parrots (Psittacus erithacus) have some capacity for recognizing numbers or number-like concepts, appearing to understand ordinality and cardinality of numerals.[82][83][84] Recent experiments also indicated that, given some language training and capacity for referencing recognized objects, they also have some ability to make inferences about probabilities and hidden object type ratios.[85]

Experiments found that when reasoning about preferred vs. non-preferred food proportions, capuchin monkeys were able to make inferences about proportions inferred by sequentially sampled data.[86] Rhesus monkeys were similarly capable of using probabilistic and sequentially sampled data to make inferences about rewarding outcomes, and neural activity in the parietal cortex appeared to be involved in the decision-making process when they made inferences.[87] In a series of 7 experiments using a variety of relative frequency differences between banana pellets and carrots, orangutans, chimpanzees and gorillas also appeared to guide their decisions based on the ratios favoring the banana pellets after this was established as their preferred food item.[88]

Research on reasoning in medicine, or clinical reasoning, usually focuses on cognitive processes and/or decision-making outcomes among physicians and patients. Considerations include assessments of risk, patient preferences, and evidence-based medical knowledge.[89] On a cognitive level, clinical inference relies heavily on interplay between abstraction, abduction, deduction, and induction.[90] Intuitive "theories," or knowledge in medicine, can be understood as prototypes in concept spaces, or alternatively, as semantic networks.[91][92] Such models serve as a starting point for intuitive generalizations to be made from a small number of cues, resulting in the physician's tradeoff between the "art and science" of medical judgement.[93] This tradeoff was captured in an artificially intelligent (AI) program called MYCIN, which outperformed medical students, but not experienced physicians with extensive practice in symptom recognition.[93][94][95][89] Some researchers argue that despite this, physicians are prone to systematic biases, or cognitive illusions, in their judgment (e.g., satisficing to make premature diagnoses, confirmation bias when diagnoses are suspected a priori).[89]

Statistical literacy and risk judgments have been described as problematic for physician-patient communication.[96] For example, physicians frequently inflate the perceived risk of non-treatment,[97] alter patients' risk perceptions by positively or negatively framing single statistics (e.g., 97% survival rate vs. 3% death rate), and/or fail to sufficiently communicate "reference classes" of probability statements to patients.[98] The reference class is the object of a probability statement: If a psychiatrist says, for example, “this medication can lead to a 30-50% chance of a sexual problem,” it is ambiguous whether this means that 30-50% of patients will develop a sexual problem at some point, or if all patients will have problems in 30-50% of their sexual encounters.[99]

In studies of base rate neglect, the problems given to participants often use base rates of disease prevalence. In these experiments, physicians and non-physicians are similarly susceptible to base rate neglect, or errors in calculating conditional probability. Here is an example from an empirical survey problem given to experienced physicians: Suppose that a hypothetical cancer had a prevalence of 0.3% in the population, and the true positive rate of a screening test was 50% with a false positive rate of 3%. Given a patient with a positive test result, what is the probability that the patient has cancer? When asked this question, physicians with an average of 14 years experience in medical practice ranged in their answers from 1-99%, with most answers being 47% or 50%. (The correct answer is 5%.)[98] This observation of clinical base rate neglect and conditional probability error has been replicated in multiple empirical studies.[96][100] Physicians' judgments in similar problems, however, improved substantially when the rates were re-formulated as natural frequencies.[101]