Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
17 views

Proof of mean recurrence time theorem in a Markov chain?

How can this formula been proven? $$\lim_{n\to \infty} p_{i,i}^{[n]} = {1\over \mu_{i,i}}$$ where $p_{i,i}^{[n]}$ is the probability that we've returned to state $j$ after $n$ steps in the Markov ...
2
votes
1answer
30 views

Markov chains by hand

If I have a starting point: $A_T=[0,1]$ at $T=1$ and a one step transition matrix of: $B=\left[ \begin{align} &\frac34 & \frac14& \\& \frac1{20}& \frac {19}{20} &\end{align} ...
0
votes
1answer
32 views

A discrete time Markov chain with such a transient state that $\mathbb P(T_i<\infty \ | \ X_0=i) \neq 0$

All examples of discrete time Markov chains my text provides are where $S$ is finite, and as far as I can tell, it makes all transient states have $$\mathbb P(T_i<\infty \ | \ X_0=i) = 0.$$ Are ...
0
votes
1answer
30 views

Why does this hold for the mean hitting time?

Let $X$ be a Markov chain and $T_A$ the hitting time. My text uses this in a proof: $$\mathbb E[T_A \ | \ X_0=k ] = \sum_{l\in S} \mathbb P(X_1=l \ | X_0=k\ )(1+\mathbb E[T_A \ | \ X_0=l ])$$ and I ...
0
votes
1answer
33 views

Integration with respect to conditional measure?

Let $(X_n)$ be a Markov chain. For $i\in S$ my text defines $$N_i:=\sum_{n=0}^\infty \mathbf 1_{\{ X_n=i \}}$$ and then, as a part of a larger proof, claims that $$\mathbb E_i(N_i)=\sum_{n=0}^\infty ...
0
votes
1answer
32 views

Inferring transition rates from continuous markov chain question

A house has 2 rooms of similar sizes with identical air conditioners equipped with thermostats which turn on and off as needed to maintain the temperature in each room to a desired level of 22 ...
1
vote
1answer
62 views

random walk with sticky barriers

Consider a random walk on the line 1,...,d. You start at point 1. At each step you flip a coin: heads means go left, tails means go right. If you're at 1 and get a heads, just stay where you are ...
0
votes
0answers
28 views

Transition matrix in left-right hidden semi-Markov model

I'm developing a hidden semi-Markov model left-right . In a left-right model a sequence of $M$ states starts in state $1$ and ends in state $M$, with no repetition of states. Since the model is ...
0
votes
2answers
39 views

four state Markov chain

If there are four states:A,B,C,D. Probability of moving to the left is b and prob of moving to the right is a. If starting at state B, what is probability of arriving at state D? The hit says to ...
0
votes
0answers
26 views

Does Markov property imply $\mathbb P (X_n=i \ | \ X_0=j)= \mathbb P (X_{n+1}=i \ | \ X_1=j)$?

If the future depends only on the present and not on the past (aka Markov property), one could expect $$\mathbb P (X_n=i \ | \ X_0=j)= \mathbb P (X_{n+1}=i \ | \ X_1=j)$$ to hold. Is that true? I've ...
1
vote
0answers
33 views

Finding a probability measure

Could someone helpme with this problem? First, consider the transition kernel in $\mathbb{R^2}\times B(\mathbb{R})$ given by $K(x,A)=U_{S^1}(A-x)$. We can than define an other kernel in ...
1
vote
2answers
26 views

Can the ergodic theorem for Markov chains be proved with linear algebra?

This theorem is in my book, let me just say that it is for discrete-time Markov chains, that are time-homogeneous. Ergodic is defined in the book as being positive recurrent and aperiodic. The ...
0
votes
0answers
32 views

Probability of not reaching completion in Markov process

This question is supposed to be easy but is very hard for me. The Norwegian Skating Association has mass produced certain "collectors' cards" with all $N$ speedskaters (Norwegian as well as ...
0
votes
1answer
43 views

Markov chain exercise

Hello i have this Markov chain exercise: Basically we can always move up 1 step, but there is always a possibility that we will go down to the first state 0, the Markov chain consists of N states. ...
0
votes
1answer
72 views

The expected number of visits before hitting zero in simple random walk

I am learning Markov chains and encounter the following problem: Suppose in simple random walk, we start from state k. What's the expected number of visits to k before we hit 0? The book does not ...
0
votes
0answers
15 views

action of transition operator on function

Let $P$ be the transition operator of a markov chain with discrete time and discrete state space $X$. The action of the transition operator on a function $X \to \mathbb{R}$ is defined by $Pf(x) = ...
0
votes
1answer
36 views

Markov’s inequality

The annual return, R, of a certain stock is a random variable with mean 10. Use Markov’s inequality to obtain a bound for the probability of the stock return being at least 20. Assuming now that R ...
1
vote
1answer
40 views

Visualizing second-order Markov chain

You can visualize a first-order Markov chain as a graph with nodes corresponding to states and edges corresponding to transitions. Are there any known strategies to visualize a second-order Markov ...
0
votes
1answer
42 views

Markov Chains and Return Times

Let $(X_n)_{n≥0}$ be a Markov chain with transition kernel $p$ on a countable state space $S$, starting at $x∈S$ $T^{(1)}=\inf\{n≥1:X_n=x\} \quad \quad$ first return time to $x$ ...
1
vote
0answers
27 views

Markov decision processes with action space only revealed at point of decision.

I have a problem which looks like a finite horizon Markov decision process, except the actions space at each time is revealed at the decision making point. There is no way to know before hand the ...
0
votes
1answer
58 views

Understanding detailed balance equations

I'm trying to understand how the equilibrium distribution satisfy the detailed balance equation. To my understanding, I only understand that a detailed balance equation would only be satisfied if ...
4
votes
1answer
53 views

Finding Hitting probability from Markov Chain

I have a Markov chain with states {1,2,3,4,5} which has the following transition matrix: $$P= \begin{bmatrix} 0.3 & 0 & 0.7 & 0 & 0\\ 0 & 1 & 0 & 0 & 0\\ 0.5 & 0 ...
5
votes
1answer
97 views

Compute a probability in Random Walk by Martingales

Let $X_n$ be the state at time $n$ of a Markov chain with these transition probabilities : $$p_{i,i+1}=p_i\qquad,\qquad p_{i,i-1}=q_i=1-p_i$$ $(a)$ Show that $Z_n=g(X_n)\,;\,n\geq0$, is a ...
1
vote
1answer
26 views

Characterization of conditional independence

Definition: Let $\mathcal{G},\mathcal{K},\mathcal{H}$ be $\sigma $-subalgebras of $\mathcal{F}$, where $\left( {\Omega ,\mathcal{F},\mathbb{P}} \right)$ is a given probability space. We say that ...
0
votes
2answers
46 views

Proof that steady state is not affected by initial distribution in Markov chain.

I was following a proof provided in Gilbert Strang's book "Introduction to Linear Algebra". And I am confused by one step of the proof. Suppose we have a $n$ by $n$ stochastic matrix $A$, where all ...
1
vote
0answers
18 views

Markov chains mixing time

Informally, the mixing time of a Markov chain is the time it takes to reach “nearly uniform” distribution from any arbitrary starting distribution. What does it mean by nearly uniform? I hope some one ...
0
votes
0answers
34 views

Perron Frobenius Theorem and Markov chains and more

I came across few ways of calculating convergence rates of Markov chains but I am a bit confused as to how these differ from each other and what may be the best way to calculate. The second ...
2
votes
1answer
66 views

Monotonicity and Convexity of Stochastic Matrices

The definition of stochastic monotonicity and convexity is given by "Stochastic Orders and Their Applications" by Moshe Shaked and George Shanthikumar (1994) as: Let $P = \{p_{i,j} \}$ be a ...
1
vote
2answers
54 views

Convergence time of a Markov chain

We know that a regular Markov chains converges to a unique matrix. The convergence time maybe finite or infinite. My interest is in the case where the convergence time is finite. How can we accurately ...
0
votes
3answers
91 views

How fast does this Markov chain converge?

Observe the above a Markov chain and limiting matrix of it. Finding the limiting matrix if it exists is easy but I am curious as to how fast this given matrix converges to its limiting matrix. Is ...
1
vote
0answers
32 views

A Central Limit Theorem to Markov Chains

I am looking for some textbook or paper that treats this question: Let be $X_{1}, X_{2}, \ldots$ the random variables from a Markov Chain (MC). Is there any Central Limit Theorem (CLT) envolving ...
0
votes
0answers
98 views

Application of Markov Chain to Game of Life Board Game

I need to calculate the expected outcomes for the Game of Life. I believe that if I multiply the probability of landing on a particular square with the payoff of said square and add up all these ...
1
vote
1answer
28 views

Equality involving a sequence of independent exponentially distributed variables

I'm trying to prove the following statement: Let $\left( {{T_n}:n \geqslant 1} \right)$ be a sequence of independent, exponentially distributed random variables with ${T_n} \sim Exp\left( {{q_n}} ...
1
vote
0answers
12 views

Birkhoff-Neumann like result for stochastic matrices?

during my research I came along a nice lemma which looks like a Birkhoff-Neumann-theorem result, but in a version for stochastic matrices. Namely, I have: Lemma. Let $M$ be a stochastic matrix, then ...
2
votes
1answer
43 views

probability that a game finishes at $n$th step

A coin is flipped sequentially. The game finishes when the sequence TTH is formed(player X wins) or the sequence HTT is formed(player Y wins). I can find the expected time until absorption by X or Y ...
1
vote
1answer
33 views

Markov: Expected time of first visit to a state starting from that state.

Question: Calculate the expected time of first visit to state 2 given we start in state 2. Is the answer to this the mean recurrence time of 2 or simply zero? I at first thought that the answer ...
0
votes
0answers
20 views

The second eigenvalue of a reducible stochastic matrix

The magnitude of the second dominant eigenvalue of a reducible matrix, as I know, is supposed to be 1, why it's not the case for this matrix : $$ \begin{matrix} 0 & 1 & 0 ...
1
vote
1answer
26 views

Markov processes limiting probability questions

I am going over previous mock exams in preparation for an upcoming exam and am having problems with parts (ii) and (iv) and was looking for some guidance. For part (ii), my thinking was that the ...
2
votes
1answer
64 views

Safe small wins vs. risky large wins at roulette

Short statement of problem : Two players play roulette at a casino. They both start with the same initial amount. Each player always plays his favorite bet each time, and stops playing as soon as he ...
5
votes
2answers
202 views

Generalization of the Jordan form for infinite matrices

Under what conditions is it the case that for a matrix $M$ whose rows and columns are indexed by a countably infinite set $S$ one has a Hamel basis consisting of generalized eigenvectors (i.e. $v \in ...
0
votes
0answers
19 views

How to use symmetry of transition rate matrix in a continuous-time Markov chain?

This is part of a bigger question, so I have to change the question a bit to focus on the point. We have a continuous- time Markov chain with the following transition rate matrix: $$Q= \begin{pmatrix} ...
0
votes
0answers
31 views

G/G/1 Queues - Book with Discrete Time Markov Chain examples

Need some book recommendation or links which have examples how to solve G/G/1 queues with detailed Discrete Time Markov Chain drawn and how to get the steady state distribution, the average number of ...
1
vote
1answer
132 views

Probability of a trajectory in Markov processes

I need help with a simple formula! (My question is taken from here, pag 26 eq 1.112. ) Consider a Markov Process with associated Master Equation: \begin{equation*} ...
0
votes
1answer
28 views

Calculating probability from Markov Chain

I have a Markov Chain with states {1,2,3,4,5} which has the following transition matrix below: $$P= \begin{bmatrix} 0.3 & 0 & 0.7 & 0 & 0\\ 0 & 1 & 0 & 0 & 0\\ 0.5 ...
1
vote
1answer
25 views

Finding $P_{11}(n)$ in Markov Chains

Calculate: $P_{11}(n)=P(X_n=1|X_0=1)$ where the transition matrix is of the form: $$\left[\begin{matrix}0 & 1 &0 \\ 0 & \dfrac{1}{2} & \dfrac{1}{2} \\ \dfrac{1}{2} & 0 & ...
2
votes
1answer
33 views

Proving that a HMC state is recurrent or transient?

Looking at the HMC $$\begin{bmatrix} 1-\alpha & \alpha \\ 0 & 1 \end{bmatrix} $$ How do I prove that the state 2 is recurrent and that state 1 is transient? What does it actually mean by ...
0
votes
1answer
20 views

Continuous time Markov chain. proportion of time spent in state i

If a question asks for the proportion of time spent in a specific state is this the same as the stationary distribution or something else? For continuous time Markov chain with finite state space.
0
votes
2answers
32 views

Proof of Markov Property

I'm trying to understand a simple proof for the markov property which states that: "$A_1$, $A_3$ are conditionally independent given $A_2$ iff $P(A_3 | A_1 \cap A_2)=P(A_3|A_2)$" The Proof begins as ...
0
votes
2answers
52 views

Continuous Markov chains, arriving pairs

I have been trying to sort out this exercise but really stuck on this. Preparing myself for exams and found many exercise on continuous Markov chains but I am always stuck when it comes to transition ...
1
vote
0answers
21 views

Changes in the transition matrix of a Markov chain

In most or all Markov chain theories that I know of assumes that the transition matrix does not change over time. But what if certain changes are expected to occur at certain times in the transition ...