Lecture 1

Lecture 1 was on on Thursday October 4 at 10am. It was attended by almost all of the IB students.

Complete course notes are available. This is the second year I have lectured this course, so I hope these notes are mostly free of typos. However, I may make small changes to the notes during this term. I will be copying into the blog that I write this year some things I wrote in last year's blog, so not all of the following is original.

The example of a frog who jumping amongst lily pads comes from Ronald A. Howard's classic book, Dynamic Programming and Markov Processes (1960). I read this book long ago and the image has stuck with me and been helpful. It gets you thinking in pictures, which is good.

Markov chains are a type of mathematics that can be highly visual, by which I mean that the problems, and even the proofs (I'll give examples later), can be run through in the mind in a very graphic way --- almost like watching a movie play out.

I mentioned that our course only deals with Markov chains in which the state space is countable. Also, we consider only a discrete-time process, $X_0, X_1, X_2,\dots$ . It is not much different to address Markov processes whose state space is uncountable (such as the non-negative real numbers) and which evolve in continuous time. The mathematics becomes only mildly more tricky. An important and very interesting continuous-time Markov process is Brownian motion. This is a process $(X_t)_{t ≥ 0}$ which is continuous and generalizes the idea of random walk. It is very useful in financial mathematics.

Section 1.6 is about the calculation of $P^{(n)}$ ($=P^n$) for the 2-state case of
$$P=\begin{pmatrix} 1-\alpha & \alpha\\ \beta & 1-\beta \end{pmatrix}.$$ We found $P^n$ by writing $P=UDU^{−1}$, where $D$ is the diagonal matrix
$$D=\begin{pmatrix} 1 & 0\\ 0 & 1-\alpha-\beta \end{pmatrix}.$$ So $P^n=UD^nU^{-1}$. We did not need to know $U$. But, as an exercise, we can easily find it. Notice that for any stochastic matrix $P$ there is always a right-hand eigenvector of $1=(1,1,\dotsc,1)^\top$ (a column vector of $1$s). This is because every row of $P$ sums to $1$. So there is always an eigenvalue of $1$. The other right-hand eigenvector of $P$ is $(\alpha,−\beta)^\top$, with eigenvalue $1−\alpha−\beta$. So we may take
$$U=\begin{pmatrix} 1 & \alpha\\ 1 & -\beta \end{pmatrix},\quad\quad U^{-1}=\frac{1}{\alpha+\beta}\begin{pmatrix} \beta & \alpha\\ 1 & -1 \end{pmatrix}.$$ The left-hand eigenvectors of $P$ are of course the rows of $U^{-1}$.

If $P$ has repeated eigenvalues we cannot diagonalize it as above. But we can write
$$P= U J U^{-1}$$ where $J$ is a Jordan matrix. If $\lambda$ is an eigenvalue of $P$ with multiplicity $k$ then $p_{ij}^{(n)}$ will be of a form looking like
$$p_{ij}^{(n)} = \cdots +\bigl(a_0+a_1 n+\cdots+a_{k-1}n^{k-1}\bigr)\lambda^n.$$ I will say more about this in Lecture 2.

In Section 1.6 I gave two methods of finding $p_{11}^{(n)}$. The first method obviously generalizes to chains with more than 2 states. The second method is specific to this 2-state example, but it is attractive because it gets to the answer so quickly.

If you find the first lecture is too easy-going, then there is something fun for you to think about in Appendix C.