Lecture 1 was on on Thursday October 4 at 10am. It was attended by almost all of the IB students.
Complete course notes are available. This is the second year I have lectured this course, so I hope these notes are mostly free of typos. However, I may make small changes to the notes during this term. I will be copying into the blog that I write this year some things I wrote in last year's blog, so not all of the following is original.
The example of a frog who jumping amongst lily pads comes from Ronald A. Howard's classic book, Dynamic Programming and Markov Processes (1960). I read this book long ago and the image has stuck with me and been helpful. It gets you thinking in pictures, which is good.
Markov chains are a type of mathematics that can be highly visual, by which I mean that the problems, and even the proofs (I'll give examples later), can be run through in the mind in a very graphic way --- almost like watching a movie play out.
I mentioned that our course only deals with Markov chains in which the state space is countable. Also, we consider only a discrete-time process, $X_0, X_1, X_2,\dots$ . It is not much different to address Markov processes whose state space is uncountable (such as the non-negative real numbers) and which evolve in continuous time. The mathematics becomes only mildly more tricky. An important and very interesting continuous-time Markov process is Brownian motion. This is a process $(X_t)_{t ≥ 0}$ which is continuous and generalizes the idea of random walk. It is very useful in financial mathematics.
Section 1.6 is about the calculation of $P^{(n)}$ ($=P^n$) for the 2-state case of
\[
P=\begin{pmatrix}
1-\alpha & \alpha\\
\beta & 1-\beta
\end{pmatrix}.
\]We found $P^n$ by writing $P=UDU^{−1}$, where $D$ is the diagonal matrix
\[
D=\begin{pmatrix}
1 & 0\\
0 & 1-\alpha-\beta
\end{pmatrix}.
\]So $P^n=UD^nU^{-1}$. We did not need to know $U$. But, as an exercise, we can easily find it. Notice that for any stochastic matrix $P$ there is always a right-hand eigenvector of $1=(1,1,\dotsc,1)^\top$ (a column vector of $1$s). This is because every row of $P$ sums to $1$. So there is always an eigenvalue of $1$. The other right-hand eigenvector of $P$ is $(\alpha,−\beta)^\top$, with eigenvalue $1−\alpha−\beta$. So we may take
\[
U=\begin{pmatrix}
1 & \alpha\\
1 & -\beta
\end{pmatrix},\quad\quad
U^{-1}=\frac{1}{\alpha+\beta}\begin{pmatrix}
\beta & \alpha\\
1 & -1
\end{pmatrix}.
\]The left-hand eigenvectors of $P$ are of course the rows of $U^{-1}$.
If $P$ has repeated eigenvalues we cannot diagonalize it as above. But we can write
\[
P= U J U^{-1}
\]where $J$ is a Jordan matrix. If $\lambda$ is an eigenvalue of $P$ with multiplicity $k$ then $p_{ij}^{(n)}$ will be of a form looking like
\[
p_{ij}^{(n)} = \cdots +\bigl(a_0+a_1 n+\cdots+a_{k-1}n^{k-1}\bigr)\lambda^n.
\] I will say more about this in Lecture 2.
In Section 1.6 I gave two methods of finding $p_{11}^{(n)}$. The first method obviously generalizes to chains with more than 2 states. The second method is specific to this 2-state example, but it is attractive because it gets to the answer so quickly.
If you find the first lecture is too easy-going, then there is something fun for you to think about in Appendix C.
