Lecture 19 - Finishing Decomposition, Some Applications

Recall:

polar decomposition

Given $T \in L (V)$ , then there exists an isometry $S \in L (V)$ such that:
$T = S \sqrt{T^{*} T}$

In an essence, $S$ is a simple operator, so $T$ and $\sqrt{T^{*} T}$ are very similar. We want to look at the operator $\sqrt{T^{*} T}$ in more detail to work with, since we know a lot of its properties even when $T$ is a mess.

Notice that $\sqrt{T^{*} T}$ is a positive operator, so it must have non-negative eigenvalues. We used this to make:

singular values of $T$

The singular values of some $T \in L (V)$ are the eigenvalues of $\sqrt{T^{*} T}$ with each eigenvalue repeated $dim (E (λ, \sqrt{T^{*} T}))$ times.

In particular, there are exactly $dim (V)$ singular values, and they are all non-negative. Because of this, we can order them, so we typically write them in descending order:

$S_{1} \geq \dots \geq S_{dim (V)}$

or more traditionally:

$σ_{1} \geq \dots \geq σ_{dim (V)}$

An Example

Let $T \in L (F^{3})$ be defined by:

T (a, b, c) = [\begin{matrix} 1 & 0 & 1 \\ 0 & 1 & 1 \\ 0 & 0 & 0 \end{matrix}] [\begin{matrix} a \\ b \\ c \end{matrix}] = [\begin{matrix} a + c \\ b + c \\ 0 \end{matrix}]

Thus:

M (T, β) = [\begin{matrix} 1 & 0 & 1 \\ 0 & 1 & 1 \\ 0 & 0 & 0 \end{matrix}]

Further:

M (T^{*}, β) = [\begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 1 & 1 & 0 \end{matrix}]

Notice then:

M (T^{*} T, β) = [\begin{matrix} 1 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 1 & 2 \end{matrix}]

Notice that we know that $T^{*} T$ is positive, as shown since $M (T^{*} T, β)$ is symmetric. It has eigenvectors:

v_{1} = (1, 1, 2), v_{2} = (- 1, 1, 0), v_{3} = (- 1, - 1, 1)

With eigenvalues:

λ_{1} = 3, λ_{2} = 1, λ_{3} = 0

So then the matrix with respect to the eigenbasis $β^{'} = {e_{1}, e_{2}, e_{3}}$ with each $e_{i}$ is the normalized $v_{i}$ :

M (T^{*} T, β^{'}) = [\begin{matrix} 3 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{matrix}]

Thus:

M (\sqrt{T^{*} T}, β^{'}) = [\begin{matrix} \sqrt{3} & 0 & 0 \\ 0 & \sqrt{1} & 0 \\ 0 & 0 & \sqrt{0} \end{matrix}]

Notice we have the singular values on the digagonal our singular values, namely:

σ_{i} = {\sqrt{3}, 1, 0}

So the singular value decomposition is as follows, but we'll need a definition.

Singular Value Decomposition

singular value decomposition

Suppose $T \in L (V)$ , has singular values $S_{1}, . . ., S_{n}$ . Then there exist orthonormal bases $e_{1}, . . ., e_{n}$ and $f_{1}, . . ., f_{n}$ such that:

T v = T (\sum_{i = 1}^{n} ⟨ v, e_{i} ⟩ e_{i}) = \sum_{i = 1}^{n} ⟨ v, e_{i} ⟩ T (e_{i}) = \sum_{i = 1}^{n} s_{i} ⟨ v, e_{i} ⟩ f_{i}

Notice it's doing a scaling and a swap. It swaps all the $e_{i}$ 's with $f_{i}$ 's and then scales respectively by $s_{i}$ .

In general:

T e_{i} = s_{i} f_{i}

Notice that:

M (T, {e_{1}, . . ., e_{n}}, {f_{1}, . . ., f_{n}}) = [\begin{matrix} s_{1} & 0 & \dots & 0 \\ 0 & s_{2} & \dots & 0 \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 0 & 0 & \dots & s_{n} \end{matrix}]

This implies that every operator has a matrix that is diagonalizable!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Proof
The operator $\sqrt{T^{*} T}$ is:

positive
self-adjoint

So there is an ONEB $e_{1}, . . ., e_{n}$ corresponding to the eigenvalues $s_{1}, . . ., s_{n}$ for $T$ .

If our arbitrary $v = \sum_{i = 1}^{n} ⟨ v, e_{i} ⟩ e_{i}$ then:

\begin{aligned} \sqrt{T^{*} T} v & = \sum_{i = 1}^{n} s_{i} ⟨ v, e_{i} ⟩ e_{i} \end{aligned}

Apply the isometry $S$ (there may be many, but just pick one) from the polar decomposition to get:

S \sqrt{T^{*} T} v = T v = \sum_{i = 1}^{n} s_{i} ⟨ v, e_{i} ⟩ S (e_{i}) = \sum_{i = 1}^{n} s_{i} ⟨ v, e_{i} ⟩ f_{i}

where we can just define $f_{i} = S (e_{i})$ .
☐
In the matrix language, this is what Singular-Value Decomposition (SVD) looks like.

Singular-Value Decomposition (SVD)

Given an $m \times n$ complex matrix $M$ , then there exist matrices $U, S, V$ such that:

$U$ is an $m \times m$ complex unitary matrix.
$S$ is an $m \times n$ rectangular matrix with non-negative real numbers on the diagonal (namely, the singular values)
$V$ is an $n \times n$ complex unitary matrix
Such that:

M = USV^{*} $${#e2c23a}

Here unitary means:

unitary

Unitary means $U^{*} = U^{- 1}$ . The matrix representation of any isometry is always unitary.

Notice if $M$ is real, then since $V^{*} = V^{T}$ then $M = U S V^{T}$ where $U, V$ are real-orthogonal ( $U^{T} = U^{- 1}$ ) matrices.

An Example

Namely:

M = \sum_{i = 1}^{min (m, n)} s_{i} \underset{an outer product}{\underset{⏟}{\vec{u_{i}} {\vec{v_{i}}}^{T}}}

Notice if we order the $s$ 's from biggest to smallest, then for large $i \to \infty$ then our $s_{i} \to 0$ , so then we could just take a partial sum of these and get an approximation for $M$ . As such, define a matrix $M_{k}$ as this approximation:

M_{k} = \sum_{i = 1}^{k} s_{i} \vec{u_{i}} {\vec{v_{i}}}^{T}

As a result, looking at the diagram above, we would only need the first $k$ columns of $U$ and first $k$ rows of $V^{*}$ . This is called a rank $k$ approximator of $M_{k}$ .

A demo of this concept can be seen at this link.