Matrix norm

If the p-norm for vectors (${\displaystyle 1\leq p\leq \infty }$) is used for both spaces ${\displaystyle K^{n}}$ and ${\displaystyle K^{m},}$ then the corresponding operator norm is:[2]

$${\displaystyle \|A\|_{p}=\sup _{x\neq 0}{\frac {\|Ax\|_{p}}{\|x\|_{p}}}.}$$

These induced norms are different from the "entry-wise" p-norms and the Schatten p-norms for matrices treated below, which are also usually denoted by ${\displaystyle \|A\|_{p}.}$

In the special cases of ${\displaystyle p=1,\infty ,}$ the induced matrix norms can be computed or estimated by

$${\displaystyle \|A\|_{1}=\max _{1\leq j\leq n}\sum _{i=1}^{m}|a_{ij}|,}$$

which is simply the maximum absolute column sum of the matrix;

$${\displaystyle \|A\|_{\infty }=\max _{1\leq i\leq m}\sum _{j=1}^{n}|a_{ij}|,}$$

which is simply the maximum absolute row sum of the matrix.

For example, for

$${\displaystyle A={\begin{bmatrix}-3&5&7\\2&6&4\\0&2&8\\\end{bmatrix}},}$$

we have that

$${\displaystyle \|A\|_{1}=\max(|{-3}|+2+0;5+6+2;7+4+8)=\max(5,13,19)=19,}$$

$${\displaystyle \|A\|_{\infty }=\max(|{-3}|+5+7;2+6+4;0+2+8)=\max(15,12,10)=15.}$$

In the special case of ${\displaystyle p=2}$ (the Euclidean norm or ${\displaystyle \ell _{2}}$-norm for vectors), the induced matrix norm is the spectral norm. (The two values do not coincide in infinite dimensions — see Spectral radius for further discussion.) The spectral norm of a matrix ${\displaystyle A}$ is the largest singular value of ${\displaystyle A}$ (i.e., the square root of the largest eigenvalue of the matrix ${\displaystyle A^{*}A,}$ where ${\displaystyle A^{*}}$ denotes the conjugate transpose of ${\displaystyle A}$):[5]

$${\displaystyle \|A\|_{2}={\sqrt {\lambda _{\max }\left(A^{*}A\right)}}=\sigma _{\max }(A).}$$

where ${\displaystyle \sigma _{\max }(A)}$ represents the largest singular value of matrix ${\displaystyle A.}$ Also,

$${\displaystyle \|A^{*}A\|_{2}=\|AA^{*}\|_{2}=\|A\|_{2}^{2}}$$

since ${\displaystyle \|A^{*}A\|_{2}=\sigma _{\max }(A^{*}A)=\sigma _{\max }(A)^{2}=\|A\|_{2}^{2}}$ and similarly ${\displaystyle \|AA^{*}\|_{2}=\|A\|_{2}^{2}}$ by singular value decomposition (SVD). There is another important inequality:

$${\displaystyle \|A\|_{2}=\sigma _{\max }(A)\leq \|A\|_{\rm {F}}=\left(\sum _{i=1}^{m}\sum _{j=1}^{n}|a_{ij}|^{2}\right)^{\frac {1}{2}}=\left(\sum _{k=1}^{\min(m,n)}\sigma _{k}^{2}\right)^{\frac {1}{2}},}$$

where ${\displaystyle \|A\|_{\textrm {F}}}$ is the Frobenius norm. Equality holds if and only if the matrix ${\displaystyle A}$ is a rank-one matrix or a zero matrix. This inequality can be derived from the fact that the trace of a matrix is equal to the sum of its eigenvalues.

When ${\displaystyle p=2}$ we have an equivalent definition for ${\displaystyle \|A\|_{2}}$ as

$${\displaystyle \sup\{x^{*}Ay:x\in K^{m},y\in K^{n}{\text{ with }}\|x\|_{2}=\|y\|_{2}=1\}.}$$

It can be shown to be equivalent to the above definitions using the Cauchy–Schwarz inequality.

Suppose vector norms ${\displaystyle \|\cdot \|_{\alpha }}$ and ${\displaystyle \|\cdot \|_{\beta }}$ are used for spaces ${\displaystyle K^{n}}$ and ${\displaystyle K^{m}}$ respectively, the corresponding operator norm is:

$${\displaystyle {\begin{aligned}\|A\|_{\alpha ,\beta }&=\sup\{\|Ax\|_{\beta }:x\in K^{n}{\text{ with }}\|x\|_{\alpha }=1\}.\end{aligned}}}$$

In the special cases of ${\displaystyle \alpha =2}$ and ${\displaystyle \beta =\infty }$, the induced matrix norms can be computed by

$${\displaystyle \|A\|_{2,\infty }=\max _{1\leq i\leq m}\|A_{i:}\|_{2},}$$

where ${\displaystyle A_{i:}}$ is the i-th row of matrix ${\displaystyle A}$. In the special cases of ${\displaystyle \alpha =1}$ and ${\displaystyle \beta =2}$, the induced matrix norms can be computed by

$${\displaystyle \|A\|_{1,2}=\max _{1\leq j\leq n}\|A_{:j}\|_{2},}$$

where ${\displaystyle A_{:j}}$ is the j-th column of matrix ${\displaystyle A}$.

Hence, ${\displaystyle \|A\|_{2,\infty }}$ and ${\displaystyle \|A\|_{1,2}}$ are the maximum row and column 2-norm of the matrix, respectively.

Any operator norm is consistent with the vector norms that induce it, giving

$${\displaystyle \|Ax\|_{\beta }\leq \|A\|_{\alpha ,\beta }\|x\|_{\alpha }.}$$

Suppose ${\displaystyle \|\cdot \|_{\alpha ,\beta }}$; ${\displaystyle \|\cdot \|_{\beta ,\gamma }}$; and ${\displaystyle \|\cdot \|_{\alpha ,\gamma }}$ are operator norms induced by the respective pairs of vector norms ${\displaystyle (\|\cdot \|_{\alpha },\|\cdot \|_{\beta })}$; ${\displaystyle (\|\cdot \|_{\beta },\|\cdot \|_{\gamma })}$; and ${\displaystyle (\|\cdot \|_{\alpha },\|\cdot \|_{\gamma })}$. Then,

${\displaystyle \|AB\|_{\alpha ,\gamma }\leq \|A\|_{\beta ,\gamma }\|B\|_{\alpha ,\beta };}$

this follows from

$${\displaystyle \|ABx\|_{\gamma }\leq \|A\|_{\beta ,\gamma }\|Bx\|_{\beta }\leq \|A\|_{\beta ,\gamma }\|B\|_{\alpha ,\beta }\|x\|_{\alpha }}$$

and

$${\displaystyle \sup _{\|x\|_{\alpha }=1}\|ABx\|_{\gamma }=\|AB\|_{\alpha ,\gamma }.}$$

Suppose ${\displaystyle \|\cdot \|_{\alpha ,\alpha }}$ is an operator norm on the space of square matrices ${\displaystyle K^{n\times n}}$ induced by vector norms ${\displaystyle \|\cdot \|_{\alpha }}$ and ${\displaystyle \|\cdot \|_{\alpha }}$. Then, the operator norm is a sub-multiplicative matrix norm:

$${\displaystyle \|AB\|_{\alpha ,\alpha }\leq \|A\|_{\alpha ,\alpha }\|B\|_{\alpha ,\alpha }.}$$

Moreover, any such norm satisfies the inequality

$${\displaystyle (\|A^{r}\|_{\alpha ,\alpha })^{1/r}\geq \rho (A)}$$

(1)

for all positive integers r, where ρ(A) is the spectral radius of A. For symmetric or hermitian A, we have equality in (1) for the 2-norm, since in this case the 2-norm is precisely the spectral radius of A. For an arbitrary matrix, we may not have equality for any norm; a counterexample would be

$${\displaystyle A={\begin{bmatrix}0&1\\0&0\end{bmatrix}},}$$

which has vanishing spectral radius. In any case, for any matrix norm, we have the spectral radius formula:

$${\displaystyle \lim _{r\to \infty }\|A^{r}\|^{1/r}=\rho (A).}$$

Let ${\displaystyle (a_{1},\ldots ,a_{n})}$ be the columns of matrix ${\displaystyle A}$. From the original definition, the matrix ${\displaystyle A}$ presents n data points in m-dimensional space. The ${\displaystyle L_{2,1}}$ norm[6] is the sum of the Euclidean norms of the columns of the matrix:

${\displaystyle \|A\|_{2,1}=\sum _{j=1}^{n}\|a_{j}\|_{2}=\sum _{j=1}^{n}\left(\sum _{i=1}^{m}|a_{ij}|^{2}\right)^{\frac {1}{2}}}$

The ${\displaystyle L_{2,1}}$ norm as an error function is more robust, since the error for each data point (a column) is not squared. It is used in robust data analysis and sparse coding.

For p, q ≥ 1, the ${\displaystyle L_{2,1}}$ norm can be generalized to the ${\displaystyle L_{p,q}}$ norm as follows:

${\displaystyle \|A\|_{p,q}=\left(\sum _{j=1}^{n}\left(\sum _{i=1}^{m}|a_{ij}|^{p}\right)^{\frac {q}{p}}\right)^{\frac {1}{q}}.}$

When p = q = 2 for the ${\displaystyle L_{p,q}}$ norm, it is called the Frobenius norm or the Hilbert–Schmidt norm, though the latter term is used more frequently in the context of operators on (possibly infinite-dimensional) Hilbert space. This norm can be defined in various ways:

${\displaystyle \|A\|_{\text{F}}={\sqrt {\sum _{i}^{m}\sum _{j}^{n}|a_{ij}|^{2}}}={\sqrt {\operatorname {trace} \left(A^{*}A\right)}}={\sqrt {\sum _{i=1}^{\min\{m,n\}}\sigma _{i}^{2}(A)}},}$

where ${\displaystyle \sigma _{i}(A)}$ are the singular values of ${\displaystyle A}$. Recall that the trace function returns the sum of diagonal entries of a square matrix.

The Frobenius norm is an extension of the Euclidean norm to ${\displaystyle K^{n\times n}}$ and comes from the Frobenius inner product on the space of all matrices.

The Frobenius norm is sub-multiplicative and is very useful for numerical linear algebra. The sub-multiplicativity of Frobenius norm can be proved using Cauchy–Schwarz inequality.

Frobenius norm is often easier to compute than induced norms, and has the useful property of being invariant under rotations (and unitary operations in general). That is, ${\displaystyle \|A\|_{\text{F}}=\|AU\|_{\text{F}}=\|UA\|_{\text{F}}}$ for any unitary matrix ${\displaystyle U}$. This property follows from the cyclic nature of the trace (${\displaystyle \operatorname {trace} (XYZ)=\operatorname {trace} (ZXY)}$):

${\displaystyle \|AU\|_{\text{F}}^{2}=\operatorname {trace} \left((AU)^{*}AU\right)=\operatorname {trace} \left(U^{*}A^{*}AU\right)=\operatorname {trace} \left(UU^{*}A^{*}A\right)=\operatorname {trace} \left(A^{*}A\right)=\|A\|_{\text{F}}^{2},}$

and analogously:

${\displaystyle \|UA\|_{\text{F}}^{2}=\operatorname {trace} \left((UA)^{*}UA\right)=\operatorname {trace} \left(A^{*}U^{*}UA\right)=\operatorname {trace} \left(A^{*}A\right)=\|A\|_{\text{F}}^{2},}$

where we have used the unitary nature of ${\displaystyle U}$ (that is, ${\displaystyle U^{*}U=UU^{*}=\mathbf {I} }$).

It also satisfies

${\displaystyle \|A^{*}A\|_{\text{F}}=\|AA^{*}\|_{\text{F}}\leq \|A\|_{\text{F}}^{2}}$

and

${\displaystyle \|A+B\|_{\text{F}}^{2}=\|A\|_{\text{F}}^{2}+\|B\|_{\text{F}}^{2}+2Re\left(\langle A,B\rangle _{\text{F}}\right),}$

where ${\displaystyle \langle A,B\rangle _{\text{F}}}$ is the Frobenius inner product, and Re is the real part of a complex number (irrelevant for real matrices)

The max norm is the elementwise norm in the limit as p = q goes to infinity:

${\displaystyle \|A\|_{\max }=\max _{ij}|a_{ij}|.}$

This norm is not sub-multiplicative.

Note that in some literature (such as Communication complexity), an alternative definition of max-norm, also called the ${\displaystyle \gamma _{2}}$-norm, refers to the factorization norm:

${\displaystyle \gamma _{2}(A)=\min _{U,V:A=UV^{T}}\|U\|_{2,\infty }\|V\|_{2,\infty }=\min _{U,V:A=UV^{T}}\max _{i,j}\|U_{i,:}\|_{2}\|V_{j,:}\|_{2}}$

Let ${\displaystyle \|A\|_{p}}$ once again refer to the norm induced by the vector p-norm (as above in the Induced Norm section).

For matrix ${\displaystyle A\in \mathbb {R} ^{m\times n}}$ of rank ${\displaystyle r}$, the following inequalities hold:[12][13]

• ${\displaystyle \|A\|_{2}\leq \|A\|_{F}\leq {\sqrt {r}}\|A\|_{2}}$
• ${\displaystyle \|A\|_{F}\leq \|A\|_{*}\leq {\sqrt {r}}\|A\|_{F}}$
• ${\displaystyle \|A\|_{\max }\leq \|A\|_{2}\leq {\sqrt {mn}}\|A\|_{\max }}$
• ${\displaystyle {\frac {1}{\sqrt {n}}}\|A\|_{\infty }\leq \|A\|_{2}\leq {\sqrt {m}}\|A\|_{\infty }}$
• ${\displaystyle {\frac {1}{\sqrt {m}}}\|A\|_{1}\leq \|A\|_{2}\leq {\sqrt {n}}\|A\|_{1}.}$