Norms
Vector Norms
- $l_p$ norm family
- quadratic norm family: for $P \in S^n_{++}$, $||x||_P = (x^T P x)^{\frac{1}{2}} = || P^{\frac{1}{2}} x ||_2$
Matrix Norms
Frobenius norm
Frobenius norm of $X \in \mathbb{R}^{m \times n} $ (Euclidean norm of vector obtained by listing all entries):
\begin{equation*}
||X||_F = (\textbf{tr}(X^TX))^{\frac{1}{2}}
\end{equation*}
Operator norm
Operator norm of $X \in \mathbb{R}^{m \times n} $, induced by norms $||\cdot||_a$ on $\mathbb{R}^{m}$ and $||\cdot||_b$ on $\mathbb{R}^{n}$:
\begin{equation*}
|| X ||_{a,b} = \sup \{ ||Xu||_a \; \bigg{|} \; \; ||u||_b \leq 1 \}
\end{equation*}
Spectral norm
When $||\cdot||_a$ and $||\cdot||_b$ are both Euclidean norms the operator norm of $X$ is its maximum singular value, also called the spectral norm or $l_2$-norm of $X$.
\begin{equation*}
||X||_2 = \sigma_{max} (X) = (\lambda_{max}(X^TX)) ^ {\frac{1}{2}}
\end{equation*}
Nuclear Norm
Dual of spectral norm is the nuclear norm, the sum of the singular values:
\begin{equation*}
||Z||_{2*} = \sup \{ \textbf{tr}(Z^TX) \; \bigg{|} \; \; ||X||_{2} \leq 1 \}
\end{equation*}
\begin{equation*}
||Z||_{2*} = \sigma_1(Z) + ... + \sigma_r(Z) = \textbf{tr}(Z^TZ)^{\frac{1}{2}}
\end{equation*}