Can I define recurrent/transient states through number of visits in a Markov chain I'm trying to appropriate myself some notions related to Markov chains (for which I'm not at all a specialist) and I have this question about the notion of "number of visits" in a certain state.
Let $(X_t)_{t \in \mathbb{N}}$ be a homogeneous Markov chain with transition matrix $(P_{i,j})$ and states ${1,2,…,n}$.
I define the number of visits to state $j$ by $$N_j = \sum_{t=0}^\infty \mathbb{1}_{X_t = j}$$
Does it make sense to define the mean number of visits to state $j$ starting from state $i$ by $v_{j,i} = \mathbb{E} (N_j \, | \, X_0=i)$ ?
In that case, is it right to write the following:
$$ v_{j,i} = \mathbb{E} \left((\sum_{t=0}^\infty \mathbb{1}_{X_t = j}) \, | \, X_0=i \right) = \mathbb{E}  \left( \sum_{t=0}^\infty (\mathbb{1}_{X_t = j} \, | \, X_0=i) \right) = \sum_{t=0}^\infty \mathbb{E}(\mathbb{1}_{X_t = j} \, | \, X_0=i) =  \sum_{t=0}^\infty P^t_{i,j} $$
If it is, I then say that a state $j$ in recurrent if and only if $v_{j,j} = \infty$ and transient if and only if $v_{j,j} < \infty$. Am I right in using these definitions?
Thank you for any comments that would help me better understand if something is wrong...
 A: The standard definition of recurrence for a state $j$ in a time-homogenous Markov chain, is that starting from $j$, the chain eventually returns to $j$ with probability 1. This is equivalent to $v_{jj}=\infty$. Indeed, if $x$ is recurrent, then after the first return, there is another one with probability  1, etc. Iterating and intersecting countably many events of probability 1, we see that assuming $j$ is recurrent, the number of returns is infinite almost surely. Conversely, if the probability of return is $q<1$, then the number of returns is a geometric variable with parameter $p=1-q$ and mean $v_{jj}=1/p$. See any standard book on Markov chains, e.g. [1] or Section 21.1 page 291 in [2]. The graduate probability textbooks by Durrett and by Billingsley also discuss this.
[1] James Robert Norris. Markov chains.  Cambridge university press, 1998.
[2] Levin, David A., and Yuval Peres. Markov chains and mixing times. Vol. 107, American Mathematical Soc., 2017.  PDF available at https://yuvalperes.com/markov-chains-and-mixing-times-2/
A: $v_{j,j}=\infty$ is not a priori sufficient for recurrence. In principle, you could have something like $P(N_j=n \mid X_0=j) \propto 1/n^2$. In this case $j$ would not be recurrent but the mean number of visits is infinite anyway.
For a time-homogeneous chain what you want is actually true but this should be viewed as a theorem, not an immediate consequence of the definitions. One way to prove it is to prove that $P(N_j \geq n)=P(N_j \geq 1)^n$ using the strong Markov property and the time homogeneity and then use $E[N_j]=\sum_{n=1}^\infty P(N_j \geq n)=\sum_{n=1}^\infty P(N_j \geq 1)^n$, which is $\infty$ if and only if $P(N_j \geq 1)=1$.
Note that in the above I am redefining $N_j=\sum_{t=1}^\infty 1_{X_t=j}$ (ignoring time $0$). You can count the initial state as a "visit" if you want, that's just not what I wrote above.
Everything else you said makes sense/is correct.
