Markov Chain? (definition and/or material recommendation needed)

Question

I am trying to figure out whether there are a specific definition or materials I can study for analyzing the following system.

From Markov Chain definition, I think we need to have fixed probability transition matrix. However, I am interested in calculating the hitting time when we have dynamic probability transition matrix.

For example, let's imagine a simple discrete random walk with states $i \in (-\infty, 0].$ Hence, 0 is the absorbing state. However, $p^t_{i,j}$ is monotone increasing with $t$. Then we are sure about the convergence. When we want to know $\mathbb{E}[T_{-1,0}]$, which is the expected hitting time, what we can do?

I will be really appreciated for any recommendation to the materials I can study. Thank you!

Answer 1

On the state space $(-\infty,0] \times [0,\infty)$, consider a new Markov chain (knows as the space-time chain) with transition matrix $\widetilde{P}$ defined by $\widetilde{P}\Bigl((i,t),(j,t+1)\Bigr)=p_{ij}^t $ and $\widetilde{P}\Bigl((i,t),(j,s)\Bigr)=0$ for $s \ne t+1$. The space-time chain has fixed transition matrix, and you are asking for $h(-1,0)$, where $h(v)$ is the mean hitting time of the line $\{0\} \times [0,\infty)$ from the initial state $v$. Now the setting is more standard, and methods you know to find hitting times apply. For instance, the $h(v)$ satisy linear equations.