I am trying to figure out whether there are a specific definition or materials I can study for analyzing the following system.

From Markov Chain definition, I think we need to have fixed probability transition matrix. However, I am interested in calculating the hitting time when we have dynamic probability transition matrix.

For example, let's imagine a simple discrete random walk with states $i \in (-\infty, 0].$ Hence, 0 is the absorbing state. However, $p^t_{i,j}$ is monotone increasing with $t$. Then we are sure about the convergence. When we want to know $\mathbb{E}[T_{-1,0}]$, which is the expected hitting time, what we can do?

I will be really appreciated for any recommendation to the materials I can study. Thank you!


1 Answer 1


On the state space $(-\infty,0] \times [0,\infty)$, consider a new Markov chain (knows as the space-time chain) with transition matrix $\widetilde{P}$ defined by $\widetilde{P}\Bigl((i,t),(j,t+1)\Bigr)=p_{ij}^t $ and $\widetilde{P}\Bigl((i,t),(j,s)\Bigr)=0$ for $s \ne t+1$. The space-time chain has fixed transition matrix, and you are asking for $h(-1,0)$, where $h(v)$ is the mean hitting time of the line $\{0\} \times [0,\infty)$ from the initial state $v$. Now the setting is more standard, and methods you know to find hitting times apply. For instance, the $h(v)$ satisy linear equations.

  • $\begingroup$ Thank you so much! $\endgroup$
    – Hariboo
    Apr 21 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.