This lecture was mostly about how to calculate the elements of $P^{(n)}$ ($=P^n$) by solving recurrence relations, or by tricks using symmetry. We ended the lecture with the definitions of communicating states, and closed and open classes. If the Markov chain consists of only one class (and so every state can be reached from every other) then the Markov chain is said to be irreducible.
The fact that we can find $p_{ij}^{(n)}$ by solving some recurrence relation follows from the fact that if $P$ is $m\times m$ and has characteristic polynomial
\[
q(x)=\det(xI−P) = c_0 +c_1 x+\cdots+c_m x^m
\]then $p_{ij}^{(n)}$ satisfies the recurrence relation
\[
c_0 + c_1 p_{ij}^{(1)}+ \cdots + c_m p_{ij}^{(m)} =0.\tag{1}
\]This follows from the Cayley-Hamilton theorem, which states that $q(P)=0$ (see IB Linear Algebra) and so in particular $(1)$. You know how to solve a recurrence relation like $(1)$ (from IA Differential Equations), and you also know that the form of the solution differs depending on whether the roots of this polynomial are real, complex or have any multiplicities greater than 1.
Notice that if $P$ is $m \times m$ and irreducible then
\[
Q=\frac{1}{m}(I+P+P^2+···+P^{m-1})
\] is a transition matrix and all its elements are positive. Can you see why? A hint is the pigeonhole principle.
Here now is a sidebar on some interesting results in matrix algebra that are related to today's topics. We said in this lecture that if $P$ is $m \times m$ and has $m$ distinct eigenvalues, $1, \mu_2,\dotsc, \mu_m$, then
\[
p_{ij}^{(n)} = a_1 + a_2 \mu_2^n + ··· + a_m\mu_m^n
\]for some constants $a_1, a_2,\dotsc,a_m$.
We would like to know more about the eigenvalues $\mu_2,\dotsc,\mu_m$. In particular, let $|\mu_j|$ denotes the modulus of $\mu_j$. If $|\mu_j| < 1$ for all $j > 1$ then $p_{ij}^{(n)}\to a_1$ as $n\to\infty$ (as we see happening in Example 2.1)
In fact, the following are true.
- $|\mu_j| \leq 1$ for all $j > 1$.
- Suppose there exists $n$ such that $P^n$ is strictly positive. Then $|\mu_j| < 1$ for all $j > 1$.
Suppose $A$ is square matrix, which is non-negative and irreducible (in the sense that for all $i,j$, we have $(A^n)_{ij}>0$ for some $n$). Then there exists a positive real number, say $\lambda$, such that
(i) $\lambda$ is an eigenvalue of $A$,
(ii) $\lambda$ has multiplicity $1$,
(iii) both the left and right-hand eigenvectors corresponding to $\lambda$ are strictly positive,
(iv) no other eigenvector of $A$ is strictly positive,
(v) all eigenvalues of $A$ are in modulus no greater than $\lambda$,
(vi) if, moreover, $A$ is strictly positive then all other eigenvalues of $A$ are in modulus strictly less than $\lambda$, and
(vii) $\min_i \sum_j a_{ij} \leq \lambda \leq \max_i \sum_j a_{ij}$, i.e. $\lambda$ lies between the minimum and maximum of row sums of $A$.
So in the case that $A =P$, the transition matrix of an irreducible Markov chain, (vii) implies that $\lambda=1$ (and $1=(1,1,\dotsc,1)$ is the corresponding right-hand eigenvector).
Of the claims made earlier, 1 follows from (v) and (vii). To see 2, we have from (vi) that if $P^n$ is strictly positive then all its eigenvalues different to $1$ have modulus less strictly than $1$. But if $\mu$ is an eigenvalue of $P$ then $\mu^n$ is an eigenvalue of $P^n$. Hence we must have $|\mu| < 1$.
