Foundations of statistics

Fisher played a significant role in popularizing significance testing through his publications, such as "Statistical Methods for Research Workers" in 1925 and "The Design of Experiments" in 1935.[7] His aim was to achieve scientific experimental outcomes without bias from prior opinions. Significance testing is a probabilistic form of deductive inference, akin to modus tollens. A simplified statement of the test can be described as follows: "If the evidence contradicts the hypothesis to a sufficient degree, the hypothesis is rejected." In practice, a statistic is computed based on the experimental data, and the probability of obtaining a value greater than that statistic under a default or "null" model is compared to a predetermined threshold. This threshold represents the level of discord required (typically established by convention). One common application of this method is to determine whether a treatment has a noticeable effect based on a comparative experiment. In this case, the null hypothesis corresponds to the absence of a treatment effect, implying that the treated group and the control group are drawn from the same population. Statistical significance measures probability and does not address practical significance. It can be viewed as a criterion for the statistical signal-to-noise ratio. It is important to note that the test cannot prove the hypothesis (of no treatment effect), but it can provide evidence against it. The method relies on formulating an imaginary infinite population, representing the null hypothesis, within a specified statistical model.

The Fisherian significance test involves a single hypothesis, but the choice of the test statistic requires an understanding of relevant directions of deviation from the hypothesized model. In the past, the outcome of the test was simply to accept or reject the hypothesis. However, with the advent of computer technology, it is now possible to calculate the probability of observing the test results, assuming the hypothesis. This probability is known as the p-value and enables a more precise assessment of the result's significance.

Neyman and Pearson collaborated on the problem of selecting the most appropriate hypothesis based solely on experimental evidence, which differed from significance testing. Their most renowned joint paper, published in 1933,[8] introduced the Neyman-Pearson lemma, which states that a ratio of probabilities serves as an effective criterion for hypothesis selection (with the choice of the threshold being arbitrary). The paper demonstrated the optimality of the Student's t-test, one of the significance tests. Neyman believed that hypothesis testing represented a generalization and improvement of significance testing. The rationale for their methods can be found in their collaborative papers.[9]

Hypothesis testing involves considering multiple hypotheses and selecting one among them, akin to making a multiple-choice decision. The absence of evidence is not an immediate factor to be taken into account. The method is grounded in the assumption of repeated sampling from the same population (the classical frequentist assumption), although Fisher criticized this assumption (Rubin, 2020).[10]

The duration of the dispute allowed for a comprehensive discussion of various fundamental issues in the field of statistics.

During this exchange, Fisher also discussed the requirements for inductive inference, specifically criticizing cost functions that penalize erroneous judgments. Neyman countered by mentioning the use of such functions by Gauss and Laplace. These arguments occurred 15 years after textbooks began teaching a hybrid theory of statistical testing.

Fisher and Neyman held different perspectives on the foundations of statistics (though they both opposed the Bayesian viewpoint):[13]

• The interpretation of probability
  • The disagreement between Fisher's inductive reasoning and Neyman's inductive behavior reflected the Bayesian-Frequentist divide. Fisher was willing to revise his opinion (reaching a provisional conclusion) based on calculated probability, while Neyman was more inclined to adjust his observable behavior (making a decision) based on computed costs.
• The appropriate formulation of scientific questions, with a particular focus on modeling,[6][14]
• Whether it is justifiable to reject a hypothesis based on a low probability without knowing the probability of an alternative
• Whether a hypothesis could ever be accepted based solely on data
  • In mathematics, deduction proves, while counter-examples disprove.
  • In the Popperian philosophy of science, progress is made when theories are disproven.
• Subjectivity: Although Fisher and Neyman endeavored to minimize subjectivity, they both acknowledged the significance of "good judgment." Each accused the other of subjectivity.
  • Fisher subjectively selected the null hypothesis.
  • Neyman-Pearson subjectively determined the criterion for selection (which was not limited to probability).
  • Both subjectively established numeric thresholds.

Fisher and Neyman diverged in their attitudes and, perhaps, their language. Fisher was a scientist and an intuitive mathematician, and inductive reasoning came naturally to him. Neyman, on the other hand, was a rigorous mathematician who relied on deductive reasoning rather than probability calculations based on experiments.[4] Hence, there was an inherent clash between applied and theoretical approaches (between science and mathematics).

In 1938, Neyman relocated to the West Coast of the United States of America, effectively ending his collaboration with Pearson and their work on hypothesis testing.[4] Subsequent developments in the field were carried out by other researchers.

By 1940, textbooks began presenting a hybrid approach that combined elements of significance testing and hypothesis testing.[15] However, none of the main contributors were directly involved in the further development of the hybrid approach currently taught in introductory statistics.[5]

Statistics subsequently branched out into various directions, including decision theory, Bayesian statistics, exploratory data analysis, robust statistics, and non-parametric statistics. Neyman-Pearson hypothesis testing made significant contributions to decision theory, which is widely employed, particularly in statistical quality control. Hypothesis testing also extended its applicability to incorporate prior probabilities, giving it a Bayesian character. While Neyman -Pearson hypothesis testing has evolved into an abstract mathematical subject taught at the post-graduate level,[16] much of what is taught and used in undergraduate education under the umbrella of hypothesis testing can be attributed to Fisher.

There have been no major conflicts between the two classical schools of testing in recent decades, although occasional criticism and disputes persist. However, it is highly unlikely that one theory of statistical testing will completely supplant the other in the foreseeable future.

The hybrid approach, which combines elements from both competing schools of testing, can be interpreted in different ways. Some view it as an amalgamation of two mathematically complementary ideas,[13] while others see it as a flawed union of philosophically incompatible concepts.[17] Fisher's approach had certain philosophical advantages, while Neyman and Pearson emphasized rigorous mathematics. Hypothesis testing remains a subject of controversy for some users, but the most widely accepted alternative method, confidence intervals, is based on the same mathematical principles.

Due to the historical development of testing, there is no single authoritative source that fully encompasses the hybrid theory as it is commonly practised in statistics. Additionally, the terminology used in this context may lack consistency. Empirical evidence indicates that individuals, including students and instructors in introductory statistics courses, often have a limited understanding of the meaning of hypothesis testing.[18]

• The interpretation of probability remains unresolved, although fiducial probability is not widely embraced.
• Neither of the test methods has been completely abandoned, as they are extensively utilized for different objectives.
• Textbooks have integrated both test methods into the framework of hypothesis testing.
  • Some mathematicians argue, with a few exceptions, that significance tests can be considered a specific instance of hypothesis tests.
  • On the other hand, there are those who perceive these problems and methods as separate or incompatible.
• The ongoing dispute has had a negative impact on statistical education.

Fisher and Neyman were significant figures in the development of frequentist (classical) methods.[4] While Fisher had a unique interpretation of probability that differed from Bayesian principles, Neyman adhered strictly to the frequentist approach. In the realm of Bayesian statistical philosophy, mathematics, and methods, de Finetti,[20] Jeffreys,[21] and Savage[3] emerged as notable contributors during the 20th century. Savage played a crucial role in popularizing de Finetti's ideas in English-speaking regions and establishing rigorous Bayesian mathematics. In 1965, Dennis Lindley's two-volume work titled "Introduction to Probability and Statistics from a Bayesian Viewpoint" played a vital role in introducing Bayesian methods to a wide audience. Over the course of three generations, statistics have progressed significantly, and the views of early contributors are not necessarily considered authoritative in present times.

The earlier description briefly highlights frequentist inference, which encompasses Fisher's "significance testing" and Neyman-Pearson's "hypothesis testing." Frequentist inference incorporates various perspectives and allows for scientific conclusions, operational decisions, and parameter estimation with or without confidence intervals. It relies solely on a single set of evidence.

A classical frequency distribution provides information about the probability of the observed data. By applying Bayes' theorem, a more abstract concept is introduced, which involves estimating the probability of a hypothesis (associated with a theory) given the data. This concept, formerly referred to as "inverse probability," is realized through Bayesian inference. Bayesian inference involves updating the probability estimate for a hypothesis as new evidence becomes available. It explicitly considers both the evidence and prior beliefs, enabling the incorporation of multiple sets of evidence.

Frequentists and Bayesians employ distinct probability models. Frequentists typically view parameters as fixed but unknown, whereas Bayesians assign probability distributions to these parameters. As a result, Bayesians discuss probabilities that frequentists do not acknowledge. Bayesians consider the probability of a theory, whereas true frequentists can only assess the evidence's consistency with the theory. For instance, a frequentist does not claim a 95% probability that the true value of a parameter falls within a confidence interval; rather, they state that 95% of confidence intervals encompass the true value.

Both the frequentist and Bayesian schools are subject to mathematical critique, and neither readily embraces such criticism. For instance, Stein's paradox highlights the intricacy of determining a "flat" or "uninformative" prior probability distribution in high-dimensional spaces.[1] While Bayesians perceive this as tangential to their fundamental philosophy, they find frequentism plagued with inconsistencies, paradoxes, and unfavorable mathematical behavior. Frequent travelers can account for most of these issues. Certain "problematic" scenarios, like estimating the weight variability of a herd of elephants based on a single measurement ("Basu's elephants"), exemplify extreme cases that defy statistical estimation. The principle of likelihood has been a contentious arena of debate.

Both the frequentist and Bayesian schools have demonstrated notable accomplishments in addressing practical challenges. Classical statistics, with its reliance on mechanical calculators and specialized printed tables, boasts a longer history of obtaining results. Bayesian methods, on the other hand, have shown remarkable efficacy in analyzing sequentially sampled information, such as radar and sonar data. Several Bayesian techniques, as well as certain recent frequentist methods like the bootstrap, necessitate the computational capabilities that have become widely accessible in the past few decades. There is an ongoing discourse regarding the integration of Bayesian and frequentist approaches,[23] although concerns have been raised regarding the interpretation of results and the potential diminishment of methodological diversity.

Bayesians share a common stance against the limitations of frequentism, but they are divided into various philosophical camps (empirical, hierarchical, objective, personal, and subjective), each emphasizing different aspects. A philosopher of statistics from the frequentist perspective has observed a shift from the statistical domain to philosophical interpretations of probability over the past two generations.[25] Some perceive that the successes achieved with Bayesian applications do not sufficiently justify the associated philosophical framework.[26] Bayesian methods often develop practical models that deviate from traditional inference and have minimal reliance on philosophy.[27] Neither the frequentist nor the Bayesian philosophical interpretations of probability can be considered entirely robust. The frequentist view is criticized for being overly rigid and restrictive, while the Bayesian view can encompass both objective and subjective elements, among others.

• "Carefully used, the frequentist approach yields broadly applicable if sometimes clumsy answers"[28]
• "To insist on unbiased [frequentist] techniques may lead to negative (but unbiased) estimates of variance; the use of p-values in multiple tests may lead to blatant contradictions; conventional 0.95  confidence regions may consist of the whole real line. No wonder that mathematicians find it often difficult to believe that conventional statistical methods are a branch of mathematics."[29]
• "Bayesianism is a neat and fully principled philosophy, while frequentism is a grab-bag of opportunistic, individually optimal, methods."[22]
• "In multiparameter problems flat priors can yield very bad answers"[28]
• "[Bayes' rule] says there is a simple, elegant way to combine current information with prior experience to state how much is known. It implies that sufficiently good data will bring previously disparate observers to an agreement. It makes full use of available information, and it produces decisions having the least possible error rate."[30]
• "Bayesian statistics is about making probability statements, frequentist statistics is about evaluating probability statements."[31]
• "Statisticians are often put in a setting reminiscent of Arrow’s paradox, where we are asked to provide estimates that are informative and unbiased and confidence statements that are correct conditional on the data and also on the underlying true parameter."[31] (These are conflicting requirements.)
• "Formal inferential aspects are often a relatively small part of statistical analysis"[28]
• "The two philosophies, Bayesian and frequentist, are more orthogonal than antithetical."[22]
• "A hypothesis that may be true is rejected because it has failed to predict observable results that have not occurred. This seems a remarkable procedure."[21]

• Bayesian theory has a mathematical advantage.
  • Frequentist probability has existence and consistency problems.
  • But finding good priors to apply Bayesian theory remains (very?) difficult.
• Both theories have impressive records of successful application.
• Neither the philosophical interpretation of probability nor its support is robust.
• There is increasing skepticism about the connection between application and philosophy.
• Some statisticians are recommending active collaboration (beyond a cease-fire).