Markov chain basic positive recurrency question If a discrete markov chain is stationary (as far as I know: doesn't modify itself with time), irreducible (doesn't have transient states) and aperiodic (no periodic states), is it positive recurrent?
This answer might be answered or not, the problem is: I don't know whether it can.
A chain is positive recurrent if mean recurrence time is finite, but it seems to me that I don't know how many states are there in a generic discrete markov chain (could it be infinite states?).
 A: If by "stationary" you mean that the transition probabilities $P(X_{n+1} = y \mid X_n = x)$ are independent of $n$ (also called "time homogeneous"), then the answer is certainly no, if the chain can have an infinite number of states.  Consider a chain on the integers where at each step you move one unit to the right with probability 1 (i.e. $P(X_{n+1} = x+1 \mid X_n = x) =1$); then every state is transient.
If by "stationary" you mean the chain admits a stationary distribution, i.e. there is a probability measure $\pi$ such that $P_\pi(X_n = x) = \pi(x)$ for every $n,x$, then the answer is yes, every state is positive recurrent, and indeed the expected return time $E_x T_x$ is given by $1/\pi(x)$.  You do not need aperiodicity.  This is a standard fact and should be in most textbooks.
A chain with a finite number of states always admits a stationary distribution, so in the first example, having an infinite number of states was necessary.
A: Yes a Markov chain could have an infinite number of states and so I think the best you can say is that the chain is null recurrent meaning every state with be returned to but the waiting time to return has no finite bound.
