1. We make the definition that state i is recurrent if P_i(V_i = \infty)=1. It is defined to be transient otherwise, i.e if P_i(V_i = \infty) < 1. Later (in Theorem 5.3) we show that if i is transient then actually P_i(V_i = \infty)=0 (but this is a consequence, not part of our starting definition of transience).
2. In the proof of Theorem 5.4 we use the fact that p_{ii}^{(n+m+r)} \geq p_{ij}^{(n)} p_{jj}^{(r)} p_{ji}^{(m)}. Please don't think that we are using any summation notation! (We never use summation convention in this course.) This inequality is a simply product of three terms on the right hand side and is a simple consequence of the fact that one way to go i\to i in n+m+r steps is to first take n steps to go i\to j, then r steps to go j\to j, and finally m steps to go j\to i. There is a \geq because there are other ways to go i\to i in n+m+r steps.
3. The proof of Theorem 5.3 can be presented in one line:
\begin{align*} \sum_{n=0}^\infty p_{ii}^{(n)}&= \sum_{n=0}^\infty E_i\left(1_{\{X_n=i\}}\right) =E_i\left(\sum_{n=0}^\infty 1_{\{X_n=i\}}\right)=E_i(V_i)\\[12pt] &=\sum_{r=0}^\infty P_i(V_i> r) =\sum_{r=0}^\infty f_i^r= \left\{\begin{array}{cc} \infty, & f_i=1\\[6pt] \frac{1}{1-f_i}, & f_i<1 \end{array}\right. \end{align*}
\begin{align*} \sum_{n=0}^\infty p_{ii}^{(n)}&= \sum_{n=0}^\infty E_i\left(1_{\{X_n=i\}}\right) =E_i\left(\sum_{n=0}^\infty 1_{\{X_n=i\}}\right)=E_i(V_i)\\[12pt] &=\sum_{r=0}^\infty P_i(V_i> r) =\sum_{r=0}^\infty f_i^r= \left\{\begin{array}{cc} \infty, & f_i=1\\[6pt] \frac{1}{1-f_i}, & f_i<1 \end{array}\right. \end{align*}
Remember that if A is an event then P(A)=E(1_{\{A\}}), where 1_{\{A\}} is the indicator random variable that =1 or =0 as A does or does not occur.
4. In Theorem 5.5 we gave an important way to check if a state is recurrent or transient, in terms of the summability of the p_{ii}^{(n)}. This criterion will be used in Lecture 6. There are other ways to check for transience. One other way is explained in Theorem 5.9. This is to solve the RHE for the minimal solution to
y_j = \sum_k p_{jk} y_k,\quad j \neq i,\quad \text{and } y_i=1.
y_j = \sum_k p_{jk} y_k,\quad j \neq i,\quad \text{and } y_i=1.
So y_j =P_j(\text{return to }i). Now check the value of \sum_k p_{ik} y_k. If it is < 1 then i is transient. This is essentially the content of Theorem 5.9, which I have put in my published notes (in blue type) but am not going to discuss in lectures. However, you may find it helpful to read the Theorem. It's proof is simple.
