A Course on Numerical Linear Algebra¶

Instructor: Deniz Bilman

Textbook: Numerical Linear Algebra, by Trefethen and Bau.

Lecture 24: Eigenvalue Problems¶

With this lecture we start the Part V of this course: Computing Eigenvalues.

We are seeking a subspace here.

The action of a matrix $A \in\mathbb{C}^{m\times m}$ on a subspace $S$ of $\mathbb{C}^m$ could be dilation. When this happens, the special subspace $S$ is called an eigenspace, and any nonzero $x \in S$ is an eigenvector.

This setting is phrased as follows:

Let $A \in \mathbb{C}^{m \times m}$ be a square matrix. A nonzero vector $x \in \mathbb{C}^m$ is an eigenvector of $A$, and $\lambda \in \mathbb{C}$ is its corresponding eigenvalue, if

$$ A x=\lambda x . $$

The set of all the eigenvalues of a matrix $A$ is the spectrum of $A$, denoted by $\Lambda(A) \subset \mathbb{C}$.

Eigenvalue problems make sense only when the range and the domain spaces of the linear transformation encoded by $A$ are the same.

Eigenvalue Decomposition¶

An eigenvalue decomposition of a square matrix $A$ is a factorization

$$ A=X \Lambda X^{-1} $$

where $\Lambda$ is the diagonal matrix of eigenvalues and columns of $X$ are the assoicated eigenvectors (with the same ordering as the eigenvalues).

Such factorization does not always exist. We can write the factorization above as

$$ A X = X \Lambda. $$

This encodes copies of the identities $A x_j = \lambda_j x_j$ for eigenvalues $\lambda_j$ and their associated eigenvectors $x_j\in\mathbb{C}^m$.

Recall that if $Ax = b$, we have

$$ \left(X^{-1} b\right)=\Lambda\left(X^{-1} x\right). $$

So $x\mapsto A x$ can be computed by expanding $x$ in the basis of columns of $X$, dilating, and then interpreting the outcome as a vector of linear coefficients of a linear combination of the columns of $X$.

Geometric Multiplicity¶

If $\lambda$ is an eigenvalue of $A$, denote the corresponding eigenspace by $E_\lambda$.

An eigenspace $E_\lambda$ is an example of an invariant subspace of $A$. We have $A E_\lambda \subseteq E_\lambda$.

What is the dimension of $E_\lambda$? We can interpret it as the maximum number of linearly independent eigenvectors that can be found, all assocaited with the same eigenvalue $\lambda$. This number is known as the geometric multiplicity of $\lambda$.

$$ \text{geometric multiplicity of } \lambda = \dim(E_{\lambda}) = \dim(\operatorname{null}(A-\lambda \mathbb{I})) $$

Algebraic Multiplicity¶

We know that the eigenvalues of $A$ coincide with the roots of the characteristic polynomial

$$ p_A(z)=\operatorname{det}(z \mathbb{I}-A), $$

which is monic. So, even if a matrix is real, some of its eigenvalues may be complex.

By the fundamental theorem of algebra, we can write $p_A$ in the form

$$ p_A(z)=\left(z-\lambda_1\right)\left(z-\lambda_2\right) \cdots\left(z-\lambda_m\right) $$

for some numbers $\lambda_j \in \mathbb{C}$, which are necessarily the eigenvalues of $A$. We define the algebraic multiplicity of an eigenvalue $\lambda$ of $A$ to be its multiplicity as a root of $p_A$.

Similarity Transformations¶

If $X$ is nonsingular, then $A$ and $X^{-1} A X$ have the same characteristic polynomial, eigenvalues, and algebraic and geometric multiplicities.

In this case, the matrices $X^{-1} A X$ and $A$ are called similar because they are related by a similarity transformation.

Theorem:¶

The algebraic multiplicity of an eigenvalue $\lambda$ is at least as great as its geometric multiplicity.

So the geometric multiplicity puts a lower bound on the algebraic multiplicity.

Defective Eigenvalues and Matrices¶

Generically, a matrix has eigenvalues whose algebraic and geometric multiplicities are equal (and all equal to 1). But this is by no means true for every matrix.

Example¶

Consider

$$ A=\begin{bmatrix} 2 & & \\ & 2 & \\ & & 2 \end{bmatrix}, \qquad B=\begin{bmatrix} 2 & 1 & \\ & 2 & 1 \\ & & 2\end{bmatrix}. $$

We have $p_A(z) = (z-2)^3$ and $p_B(z)=(z-2)^3$. So both matrices have eigenvalue $\lambda=2$ with algebraic multiplicity $3$.

In the case of $A$, we can choose three independent eigenvectors, for example $e_1, e_2$, and $e_3$, so the geometric multiplicity is also 3 .

For $B$, we can find only a single independent eigenvector (a scalar multiple of $e_1$ ), so the geometric multiplicity of the eigenvalue is only 1.

An eigenvalue whose algebraic multiplicity exceeds its geometric multiplicity is a defective eigenvalue. A matrix that has one or more defective eigenvalues is a defective matrix.

You may think of a defective matrix as one that had a dimension collapse in its eigenspace. $\lambda=2$ is an eigenvalue three times, but it only can generate a 1-dimensional eigenspace instead of $3$.

Diagonalizability¶

Theorem:¶

An $m \times m$ matrix $A$ is nondefective if and only if it has an eigenvalue decomposition $A=X \Lambda X^{-1}$.

In view of this result, another term for nondefective is diagonalizable.

Does a diagonalizable matrix $A$ in some sense "behave like" its diagonal equivalent $\Lambda$ ?

The answer depends on what aspect of behavior one measures and on the condition number of $X$, the matrix of eigenvectors. If $X$ is highly ill-conditioned, then a great deal of information may be discarded in passing from $A$ to $\Lambda$. See "A Note of Caution: Nonnormality" in Lecture 34 in Trefethen & Bau.

Determinant and Trace¶

The trace of $A \in \mathbb{C}^{m \times m}$ is the sum of its diagonal elements: $\operatorname{tr}(A)=\sum_{j=1}^m A_{j j}$. Both the trace and the determinant are related simply to the eigenvalues.

Theorem:¶

We have

$$ \operatorname{det}(A)=\prod_{j=1}^m \lambda_j, \quad \operatorname{tr}(A)=\sum_{j=1}^m \lambda_j. $$

Unitary Diagonalization¶

$A$ is unitarily diagonalizable if there exists a unitary matrix $Q$ such that

$$ A=Q \Lambda Q^* . $$

This factorization is both an eigenvalue decomposition and a singular value decomposition, aside from the matter of the signs (possibly complex) of the entries of $\Lambda$.

You see, the eigenvectors of a unitarily diagonalizable matrix can be chosen to be orthogonal (on top of being a set of $m$ linearly independent vector).

What are the matrices that are unitarily diagonalizable? It turns out, we can answer that.

By definition, we say that a matrix $A$ is normal if $A^* A=A A^*$.

Theorem:¶

A matrix is unitarily diagonalizable if and only if it is normal.

Schur Factorization¶

The following factorization is very useful in practice. All matrices (including the defective ones) can be factored in this way.

A Schur factorization of a matrix $A$ is a factorization

$$ A = A T Q^*, $$

where $Q$ is unitary and $T$ is upper-triangular.

Since $A$ and $T$ are similar, the eigenvalues of $A$ appear on the diagonal of $T$.

Theorem:¶

Every square matrix A has a Schur factorization.

Eigenvalue-Revealing Factorizations¶

Here we summarize such factorizations.

A diagonalization $A=X \Lambda X^{-1}$ exists if and only if $A$ is nondefective.
A unitary diagonalization $A=Q \Lambda Q^*$ exists if and only if $A$ is normal.
A unitary triangularization (Schur factorization) $A=Q T Q^*$ always exists.

Note on Future Material on Computing Eigenvalues¶

To compute eigenvalues, we rely on these factorizations. In general, we use the Schur factorization because

it applies to all matrices,
the algorithms using it tend to be numerically stable because unitary matrices are involved.

If $A$ is normal, then the Schur form comes out diagonal.

If $A$ is hermitian, then we can take advantage of this symmetry throughout the computation and reduce $A$ to diagonal form with half as much work or less than is required for general $A$.