## Thursday, October 13, 2011

### Lecture 3

You should now be able to do all of Example Sheet 1 (excepting that #12 will be easier after seeing Section 4.1 in the next lecture).

I mentioned that Theorem 3.4 in this lecture is similar to the result that you were taught in Probability IA, concerning the probability of ultimate extinction of a branching process. Remember that in a branching process each individual independently produces offspring for the next generation, according to a distribution in which there are k offspring with probability p_k (k=0,1,...). Given that we start with one individual, the probability of ultimate extinction, say u, is the minimal solution to
u = G(u) = sum_k p_k uk
where G is the probability generating function for the number of offspring.

Do you remember the proof that u is the minimal solution to u=G(u) , and do you see how similar it is to the proof in Theorem 3.4?

I have called the equations of the form x=Px (i.e. x_i = sum_j p_{ij}x_j) "right-hand equations", because x appears on the right-hand side of P. Later in the course we will find a use for "left-hand equations", of the form x=xP. So far as I can tell, this terminology does not appear in any modern books on Markov chains. However, it was language used by David Kendall in the course I took from him in 1972 and I have always found it to be helpful terminology.

I hope you enjoy question #13 on Example Sheet 1 (which is a new question this year). It contains a result that many people find surprising. In a Red-Black game in which p<q the strategy of bold play is optimal (but not necessarily uniquely so). This fact is proved in the Part II course Optimization and Control (see Section 4.3 "Optimal gambling" in the Optimization and Control course notes.) Part of that course is about Markov Decision Processes, which are Markov chains in which we have some control over the transitions that occur, and we try to minimize (or maximize) costs (or rewards) that accrue as we move through states.

Question #14 on the examples sheet extends the idea of a gambling game between 2 players to that of a game amongst 3 players. It is remarkable that there is such a nice formula for the expected number of games until one of the first player becomes bankrupt.