# Properties of Markov chains

We covered Markov chains in class and after going through the details, I still have a few questions. (I encourage you to give short answers to the question, as this may become very cumbersome otherwise)

1.) There are many theorems like: If $(X_n)$ is an irreducible( with a state space $S$), positive recurrent markov chain, then...(for example there is a unique stationary distribution $\pi>0$). Now, assuming that $(X_n)$ is not irreducible, but contains a positive recurrent class $A \subset S$. Does this mean that we can say that there is a unique stationary distribution $\pi|_A>0$ and $\pi=0$ elsewhere? So my question is: If we want to talk about one positive recurrent class instead of one irreducible chain, does this method of transferring the results ( by restricting the result to the subset $A$) always work? When I thought about it, I considered this to be a trivial conclusion, but most books don't even talk about this simple generalisation, so I wanted to ask this here.

2.) When you want to identify communication classes of a Markov chain on a finite set you can do this by drawing a picture and look whether there is a pathway between two states. But is there also a way of calculating the classes directly from the transition matrix? If my guess in question (1) holds, then there should be a one-one correspondence between eigenvectors $\pi^T = \pi^T P$ with $\pi_A >0$ (and $\pi_{A^C}=0$) and positive recurrent classes $A$. In that case the only question would be how to find the remaining classes. But probably, the remaining transient states are just the states that always have a zero component in every stationary distirbution we get from the eigenvalue problem $P^T \pi = \pi$.

3.) The same as question 2) for the period. Is there a faster method of calculating the period of a state than calculating the powers of the transition matrix? Or at least a fast way to determine whether a state is periodic or aperiodic?

4.) I am struggeling with closed states. My intuition tells me that: Every finite closed class is recurrent and every recurrent class is closed. So for finite chains the terms closed and (positive) recurrent should be $1:1$. Is this true? It obviously diverges over infinite sets as the randomn walk in $\mathbb{Z}^3$ is transient, but the whole state space is closed.

5.) In case that we regard the continuous case of a homogenous markov chain and $P(X_{n+1} \in A |X_n =x) = \int_A q(x,y) dy$. How do we get $P(X_{n+k} \in A |X_n = x)$?

6.) For a transient states $i,j$ in one class we have: $\sum_{n = 0}^{\infty} p^{(n)}_{ij} < \infty$. For a (positive) recurrent states $i,j$ in one class we have: $\sum_{n = 0}^{\infty} p^{(n)}_{ij} = \infty$. This is in my opinion a consequence of the fact that transience/recurrence is a class property, but I just wanted to be sure about this.

Unfortunately, this questions seems to be very unpopular. Any suggestion how I could improve my question is highly appreciated.

• I think one reason people might shy away from answering this is that, though none of your concerns seem that difficult, typing up an answer to all this stuff in one shot would involve a lot of $\LaTeX$. Good question(s) though, +1! – Robert Lewis Jul 1 '14 at 21:43
• @RobertLewis thank you for this hint. In that case: in 6 a short yes or no with a correction should do it. The same for 1 and 4. in 5, one equation should do it. So, the only thing where a few lines would be probably needed are 2 and 3. Hope, this makes my questions a little bit more comfortable. – user159356 Jul 1 '14 at 22:09
• In general, it is better to ask one question per post. – Nate Eldredge Jul 2 '14 at 0:30
• @NateEldredge sorry, I am not permitted to do that (by the forum laws). Will do it that way in the future. – user159356 Jul 2 '14 at 16:54

A) You did not consider the case say (12) and (34) are in two seperate communicating class such that

$p_{12}=p_{21}=p_{34}=p_{43}=1/2$, then there is no unique stationary distribution. If there is an only 1 closed communicating class, $A$, and all other states are transient, what you said is true.

If we want to talk about one positive recurrent class instead of one irreducible chain, does this method of transferring the results ( by restricting the result to the subset A) always work?

Yes.

B) I do not really understand you are trying to pull here? Take an example of a chain with two communicating classes. A possible solution to $\pi = \pi^T P$ is setting 0 for all states in one class and find the unique distribution for the other class.

Then the set of equilibrium measures are linear combinations of these vectors. You cannot use $\pi = \pi^T P$ unless you know that there is only $1$ closed communicating class (by same reason as question 1)

A finite state Markov Chain has a unique stationary distribution iff it contains only one closed communicating class. (so what you said is true, if you assume there is just 1 closed communicating class, everything else is open)

C) Not as far as I know. To see a state is periodic, find two path of different length on which it returns itself such that the highest greatest common factor. This is normally quite easy for a simple Markov Chain.

D) consider something like

1 -> 2 <-> 3

then 2 and 3 are closed. 1 is open. You know that open and closed are class properties, so are transient and recurrence. In a finite chain, recurrence and closed are the same thing.

The only difference occurs in infinite state Chains.

E) For a uncountable states, you can just integrate $k$ times

For example

$$P(X_2\in A|X_0=x) = \int_{y\in\Omega}\int_{z\in A} q(x,y)q(y,z)dydz$$

where $\Omega$ is the entire state space. For $X_3$, you just need to integrate 3 times etc. Notice that summation (used for countable state) is just integral with respect to counting measure.

F) you stated the sufficient and necessary condition for recurrence. There is a standard textbook proof, and yes verifying this for 1 pair of $i, j$ is enough because recurrence and transience are class properties. In fact often it is verified for $i=j$.

• I replaced numbering 1,2,3... by A-F because the automatic numbering mechanism is weird it changed all my number to 1... – Lost1 Jul 2 '14 at 18:18
• there are two things I noticed: in E) I am not sure whether the order of the integrals is correct there? Don't you want to integrate $y$ over $\Omega$? ---- in B) why do you say that I cannot use $\pi^T = \pi^T P$? I mean, if I calculate all solutions of this equation, I will certainly also get the solutions that don't vanish on one recurrent class but do vanish everywhere else. By looking at these solutions, I should be immediately able to identify all recurrence classes. – user159356 Jul 3 '14 at 0:13
• @user159356 you are right about E, take a markov chain which looks like 1<->2 3<->4, with $p12=p21=p34=p43=1/2$, you notice that $(0.25,0.25,0.25,0.25)$ is a possible staionary distribution, so is $(0.3,0.3,0.2,0.2)$ but so is $(0,0,1,1)$ and $(1,1,0,0)$. Since the equations have (infinitely) many solutions, the method does not work UNLESS you know there is only 1 CLOSED class. In the example I just gave you, there are two closed classes, so $(0,0,1,1)$ and $(1,1,0,0)$ (any possible stationary distributions are linear combinations of these), your method fails. – Lost1 Jul 3 '14 at 10:59