Maximum spacing estimation

Plots of the log value of λ for the simplistic example under both likelihood and spacing estimation. The values for which both likelihood and spacing are maximized, the maximum likelihood and maximum spacing estimates, are identified.

Suppose two values x₍₁₎ = 2, x₍₂₎ = 4 were sampled from the exponential distribution F(x;λ) = 1 − e^−xλ, x ≥ 0 with unknown parameter λ > 0. In order to construct the MSE we have to first find the spacings:

The process continues by finding the λ that maximizes the geometric mean of the “difference” column. Using the convention that ignores taking the (n+1)st root, this turns into the maximization of the following product: (1 − e^−2λ) · (e^−2λ − e^−4λ) · (e^−4λ). Letting μ = e^−2λ, the problem becomes finding the maximum of μ⁵−2μ⁴+μ³. Differentiating, the μ has to satisfy 5μ⁴−8μ³+3μ² = 0. This equation has roots 0, 0.6, and 1. As μ is actually e^−2λ, it has to be greater than zero but less than one. Therefore, the only acceptable solution is

$${\displaystyle \mu =0.6\quad \Rightarrow \quad \lambda _{\text{MSE}}={\frac {\ln 0.6}{-2}}\approx 0.255,}$$

which corresponds to an exponential distribution with a mean of 1⁄λ ≈ 3.915. For comparison, the maximum likelihood estimate of λ is the inverse of the sample mean, 3, so λ_MLE = ⅓ ≈ 0.333.

Suppose {x₍₁₎, ..., x_(n)} is the ordered sample from a uniform distribution U(a,b) with unknown endpoints a and b. The cumulative distribution function is F(x;a,b) = (x−a)/(b−a) when x∈[a,b]. Therefore, individual spacings are given by

$${\displaystyle D_{1}={\frac {x_{(1)}-a}{b-a}},\ \ D_{i}={\frac {x_{(i)}-x_{(i-1)}}{b-a}}\ {\text{for }}i=2,\ldots ,n,\ \ D_{n+1}={\frac {b-x_{(n)}}{b-a}}\ \ }$$

Calculating the geometric mean and then taking the logarithm, statistic S_n will be equal to

$${\displaystyle S_{n}(a,b)={\tfrac {\ln(x_{(1)}-a)}{n+1}}+{\tfrac {\sum _{i=2}^{n}\ln(x_{(i)}-x_{(i-1)})}{n+1}}+{\tfrac {\ln(b-x_{(n)})}{n+1}}-\ln(b-a)}$$

Here only three terms depend on the parameters a and b. Differentiating with respect to those parameters and solving the resulting linear system, the maximum spacing estimates will be

${\displaystyle {\hat {a}}={\frac {nx_{(1)}-x_{(n)}}{n-1}},\ \ {\hat {b}}={\frac {nx_{(n)}-x_{(1)}}{n-1}}.}$

These are known to be the uniformly minimum variance unbiased (UMVU) estimators for the continuous uniform distribution.[1] In comparison, the maximum likelihood estimates for this problem ${\displaystyle \scriptstyle {\hat {a}}=x_{(1)}}$ and ${\displaystyle \scriptstyle {\hat {b}}=x_{(n)}}$ are biased and have higher mean-squared error.

Density

Distribution

Plot of a “J-shaped” density function and its corresponding distribution. A shifted Weibull with a scale parameter of 15, a shape parameter of 0.5, and a location parameter of 10. The density asymptotically approaches infinity as x approaches 10, rendering the estimates of the other parameters inconsistent. Note that there is no inflection point in the graph of the distribution.

The maximum spacing estimator is a consistent estimator in that it converges in probability to the true value of the parameter, θ₀, as the sample size increases to infinity.[2] The consistency of maximum spacing estimation holds under much more general conditions than for maximum likelihood estimators. In particular, in cases where the underlying distribution is J-shaped, maximum likelihood will fail where MSE succeeds.[1] An example of a J-shaped density is the Weibull distribution, specifically a shifted Weibull, with a shape parameter less than 1. The density will tend to infinity as x approaches the location parameter rendering estimates of the other parameters inconsistent.

Maximum spacing estimators are also at least as asymptotically efficient as maximum likelihood estimators, where the latter exist. However, MSEs may exist in cases where MLEs do not.[1]

Maximum spacing estimators are sensitive to closely spaced observations, and especially ties.[8] Given

$${\displaystyle X_{i+k}=X_{i+k-1}=\cdots =X_{i},\,}$$

we get

$${\displaystyle D_{i+k}(\theta )=D_{i+k-1}(\theta )=\cdots =D_{i+1}(\theta )=0.\,}$$

When the ties are due to multiple observations, the repeated spacings (those that would otherwise be zero) should be replaced by the corresponding likelihood.[1] That is, one should substitute ${\displaystyle f_{i}(\theta )}$ for ${\displaystyle D_{i}(\theta )}$, as

$${\displaystyle \lim _{x_{i}\to x_{i-1}}{\frac {\int _{x_{i-1}}^{x_{i}}f(t;\theta )\,dt}{x_{i}-x_{i-1}}}=f(x_{i-1},\theta )=f(x_{i},\theta ),}$$

since ${\displaystyle x_{i}=x_{i-1}}$.

When ties are due to rounding error, Cheng & Stephens (1989) suggest another method to remove the effects.[note 1] Given r tied observations from x_i to x_i+r−1, let δ represent the round-off error. All of the true values should then fall in the range ${\displaystyle x\pm \delta }$. The corresponding points on the distribution should now fall between ${\displaystyle y_{L}=F(x-\delta ,{\hat {\theta }})}$ and ${\displaystyle y_{U}=F(x+\delta ,{\hat {\theta }})}$. Cheng and Stephens suggest assuming that the rounded values are uniformly spaced in this interval, by defining

$${\displaystyle D_{j}={\frac {y_{U}-y_{L}}{r-1}}\quad (j=i+1,\ldots ,i+r-1).}$$

The MSE method is also sensitive to secondary clustering.[8] One example of this phenomenon is when a set of observations is thought to come from a single normal distribution, but in fact comes from a mixture normals with different means. A second example is when the data is thought to come from an exponential distribution, but actually comes from a gamma distribution. In the latter case, smaller spacings may occur in the lower tail. A high value of M(θ) would indicate this secondary clustering effect, and suggesting a closer look at the data is required.[8]

Ranneby & Ekström (1997) generalized the MSE method to approximate other measures besides the Kullback–Leibler measure. Ekström (1997) further expanded the method to investigate properties of estimators using higher order spacings, where an m-order spacing would be defined as ${\displaystyle F(X_{j+m})-F(X_{j})}$.

Ranneby & al. (2005) discuss extended maximum spacing methods to the multivariate case. As there is no natural order for ${\displaystyle \mathbb {R} ^{k}(k>1)}$, they discuss two alternative approaches: a geometric approach based on Dirichlet cells and a probabilistic approach based on a “nearest neighbor ball” metric.