Quantile regression

Let ${\displaystyle Y}$ be a real-valued random variable with cumulative distribution function ${\displaystyle F_{Y}(y)=P(Y\leq y)}$. The ${\displaystyle \tau }$th quantile of Y is given by

${\displaystyle q_{Y}(\tau )=F_{Y}^{-1}(\tau )=\inf \left\{y:F_{Y}(y)\geq \tau \right\}}$

where ${\displaystyle \tau \in (0,1).}$

Define the loss function as ${\displaystyle \rho _{\tau }(m)=m(\tau -\mathbb {I} _{(m<0)})}$, where ${\displaystyle \mathbb {I} }$ is an indicator function. A specific quantile can be found by minimizing the expected loss of ${\displaystyle Y-u}$ with respect to ${\displaystyle u}$:[1](pp. 5–6):

${\displaystyle q_{Y}(\tau )={\underset {u}{\mbox{arg min}}}E(\rho _{\tau }(Y-u))={\underset {u}{\mbox{arg min}}}{\biggl \{}(\tau -1)\int _{-\infty }^{u}(y-u)dF_{Y}(y)+\tau \int _{u}^{\infty }(y-u)dF_{Y}(y){\biggr \}}.}$

This can be shown by computing the derivative of the expected loss with respect to ${\displaystyle u}$ via an application of the Leibniz integral rule, setting it to 0, and letting ${\displaystyle q_{\tau }}$ be the solution of

${\displaystyle 0=(1-\tau )\int _{-\infty }^{q_{\tau }}dF_{Y}(y)-\tau \int _{q_{\tau }}^{\infty }dF_{Y}(y).}$

This equation reduces to

${\displaystyle 0=F_{Y}(q_{\tau })-\tau ,}$

and then to

${\displaystyle F_{Y}(q_{\tau })=\tau .}$

If the solution ${\displaystyle q_{\tau }}$ is not unique, then we have to take the smallest such solution to obtain the ${\displaystyle \tau }$th quantile of the random variable Y.

Let ${\displaystyle Y}$ be a discrete random variable that takes values ${\displaystyle y_{i}=i}$ with ${\displaystyle i=1,2,\dots ,9}$ with equal probabilities. The task is to find the median of Y, and hence the value ${\displaystyle \tau =0.5}$ is chosen. Then the expected loss of ${\displaystyle Y-u}$ is

${\displaystyle L(u)=E(\rho _{\tau }(Y-u))={\frac {(\tau -1)}{9}}\sum _{y_{i}<u}}$${\displaystyle (y_{i}-u)}$${\displaystyle +{\frac {\tau }{9}}\sum _{y_{i}\geq u}}$${\displaystyle (y_{i}-u)}$${\displaystyle ={\frac {0.5}{9}}{\Bigl (}}$${\displaystyle -}$${\displaystyle \sum _{y_{i}<u}}$${\displaystyle (y_{i}-u)}$${\displaystyle +\sum _{y_{i}\geq u}}$${\displaystyle (y_{i}-u)}$${\displaystyle {\Bigr )}.}$

Since ${\displaystyle {0.5/9}}$ is a constant, it can be taken out of the expected loss function (this is only true if ${\displaystyle \tau =0.5}$). Then, at u=3,

${\displaystyle L(3)\propto \sum _{i=1}^{2}}$${\displaystyle -(i-3)}$${\displaystyle +\sum _{i=3}^{9}}$${\displaystyle (i-3)}$${\displaystyle =[(2+1)+(0+1+2+...+6)]=24.}$

Suppose that u is increased by 1 unit. Then the expected loss will be changed by ${\displaystyle (3)-(6)=-3}$ on changing u to 4. If, u=5, the expected loss is

${\displaystyle L(5)\propto \sum _{i=1}^{4}i+\sum _{i=0}^{4}i=20,}$

and any change in u will increase the expected loss. Thus u=5 is the median. The Table below shows the expected loss (divided by ${\displaystyle {0.5/9}}$) for different values of u.

Consider ${\displaystyle \tau =0.5}$ and let q be an initial guess for ${\displaystyle q_{\tau }}$. The expected loss evaluated at q is

${\displaystyle L(q)=-0.5\int _{-\infty }^{q}(y-q)dF_{Y}(y)+0.5\int _{q}^{\infty }(y-q)dF_{Y}(y).}$

In order to minimize the expected loss, we move the value of q a little bit to see whether the expected loss will rise or fall. Suppose we increase q by 1 unit. Then the change of expected loss would be

${\displaystyle \int _{-\infty }^{q}1dF_{Y}(y)-\int _{q}^{\infty }1dF_{Y}(y).}$

The first term of the equation is ${\displaystyle F_{Y}(q)}$ and second term of the equation is ${\displaystyle 1-F_{Y}(q)}$. Therefore, the change of expected loss function is negative if and only if ${\displaystyle F_{Y}(q)<0.5}$, that is if and only if q is smaller than the median. Similarly, if we reduce q by 1 unit, the change of expected loss function is negative if and only if q is larger than the median.

In order to minimize the expected loss function, we would increase (decrease) L(q) if q is smaller (larger) than the median, until q reaches the median. The idea behind the minimization is to count the number of points (weighted with the density) that are larger or smaller than q and then move q to a point where q is larger than ${\displaystyle 100\tau }$% of the points.

The ${\displaystyle \tau }$ sample quantile can be obtained by using an importance sampling estimate and solving the following minimization problem

${\displaystyle {\hat {q}}_{\tau }={\underset {q\in \mathbb {R} }{\mbox{arg min}}}\sum _{i=1}^{n}\rho _{\tau }(y_{i}-q),}$

${\displaystyle ={\underset {q\in \mathbb {R} }{\mbox{arg min}}}\left[(\tau -1)\sum _{y_{i}<q}(y_{i}-q)+\tau \sum _{y_{i}\geq q}(y_{i}-q)\right]}$,

where the function ${\displaystyle \rho _{\tau }}$ is the tilted absolute value function. The intuition is the same as for the population quantile.

For any ${\displaystyle a>0}$ and ${\displaystyle \tau \in [0,1]}$

${\displaystyle {\hat {\beta }}(\tau ;aY,X)=a{\hat {\beta }}(\tau ;Y,X),}$

${\displaystyle {\hat {\beta }}(\tau$ ;-aY,X)=-a{\hat {\beta }}(1-\tau ;Y,X).}

For any ${\displaystyle \gamma \in R^{k}}$ and ${\displaystyle \tau \in [0,1]}$

${\displaystyle {\hat {\beta }}(\tau ;Y+X\gamma ,X)={\hat {\beta }}(\tau ;Y,X)+\gamma .}$

Let ${\displaystyle A}$ be any ${\displaystyle p\times p}$ nonsingular matrix and ${\displaystyle \tau \in [0,1]}$

${\displaystyle {\hat {\beta }}(\tau ;Y,XA)=A^{-1}{\hat {\beta }}(\tau ;Y,X).}$

If ${\displaystyle h}$ is a nondecreasing function on ${\displaystyle \mathbb {R} }$, the following invariance property applies:

${\displaystyle h(Q_{Y|X}(\tau ))\equiv Q_{h(Y)|X}(\tau ).}$

Example (1):

If ${\displaystyle W=\exp(Y)}$ and ${\displaystyle Q_{Y|X}(\tau )=X\beta _{\tau }}$, then ${\displaystyle Q_{W|X}(\tau )=\exp(X\beta _{\tau })}$. The mean regression does not have the same property since ${\displaystyle \operatorname {E} (\ln(Y))\neq \ln(\operatorname {E} (Y)).}$