Let's say there are 2 states (A & B), where the probability of going from state A to B in interval i is $P_{ab}$. State B always lasts for n intervals, then always goes to back to state A.

I want to know what is the probability of being in state B at any given interval. Is it correct to calculate that probability like this:

$P(B)= (P_{ab}*n)/(1+P_{ab}*n)$ ?

I see Markov chains being applied to problems similar to this, but I just can't seem to find examples of this exact type of problem.

I don't have enough reputation to embed a picture, but the link shows a diagram of the situation I'm describing.



One option is to make a larger Markov chain that embeds the history information in the state space. Rather than having 1 State that corresponds to B, you would have $n$ states that correspond to how many intervals you've been in State B (aka "dummy states" that each record how long you've been in state B). Whenever you're in state B, each time interval you would transition to the next dummy state B up until the n-th one. Then transition back to A.

I.e., you would have the following states:

  • State $A$

  • State $B_1$

  • ...

  • State $B_n$

Transition probabilities:

  • State $A$ to State $B_1$ $:P_{ab}$

  • State $B_j$ to $B_{j+1}$ for all $\{j\in 1..n-1\}$ $:1 $

  • State $B_n$ to $A:1$

This should get you your Markov chain. From here, I'll let you think about how to calculate the probability of being in State A and any of the dummy State B's.

  • $\begingroup$ Good news - I worked through the math of my solution, and the answer is equivalent to the formula you gave. Apologies for initially indicating your response was wrong. Generally, it's a bad idea to consider durations within a Markov chain, but in this case, it worked out fine. $\endgroup$ – E. Tucker Apr 15 at 17:06
  • $\begingroup$ Thank you for the response Mr./Mrs Tucker, That is wonderful :D Got that equation from a completely different approach, but I wasn't 100% confident in it being right.. If its not too much to ask, could I see the work to get there? A bit rusty I must admit, spent some hours trying to relearn steady state Markov chains, but didn't quite get there myself :p I intend to use this "equation" in my masters, so thank you so much for your thourough response. $\endgroup$ – Tage Wærdahl Apr 15 at 18:37
  • $\begingroup$ After a good rest I got to the formula myself. Again, thank you for explaining the procedure. I would like to acknowledge your help in the acknowledgement section of my report. Please hit me back, to let me know if that is okay, and how you would like to be titled. $\endgroup$ – Tage Wærdahl Apr 16 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.