Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
1answer
33 views

A question about markov chain's transition matrix

Suppose $P$ is the transition matrix of some homogeneous finite Markov chain with state space $Ω=\{1,2,…,n\}$, then $P(x,y)$ is the probability that the next state is $y$ if the current state is $x$. ...
0
votes
1answer
36 views

Calculate probability of two different outcomes where history is governed by markov chain

Let the state space, $s_t$, be $\{0,1\}$ and be governed by a Markov chain with probability $\pi(s_0=1) =1$ for the initial state and time-varying transition probabilities $\pi_1(s_1=1|s_0=1)=1$, ...
0
votes
1answer
21 views

Is a Markovian chain irreducible when one state does not have a recursive path?

Let be the following homogeneous Markovian chain with three state: \begin{pmatrix} 1/2 & 1/4 & 1/4 \\ 2/3 & 0 & 1/3\\ 3/5 & 1/5 & 1/5 \end{pmatrix} Is this ...
1
vote
1answer
33 views

Recurrence of a Markov chain (lemma of Pakes)

For my course on Markov chains, we have to think about the following problem: Consider the irreducible Markov chain with $P$ on the state space $S={0,1,2,...}$, with $p_{0,1}=1$, ...
2
votes
1answer
29 views

How to find Kolmogorov Forward Equations, given generator matrix Q?

I am having difficulty in forming Kolmogorov Forward Equations. I understand how the KFE is derived and that $$\frac {d}{ds} p_{ij} (s) = \sum_{k \neq j} p_{ik} (s) \lambda_{k} r_{kj} - p_{ij} ...
1
vote
1answer
37 views

Modelling probability problems by Markov chains

For one of my courses, we have to think about how we could model certain problems with the help of Markov chains. Most are straight-forward, but I find it difficult to choose the right states and ...
0
votes
0answers
19 views

Reference request for name of matrix similar to transition matrix of Markov chain

If $P$ is the transition matrix of a Markov chain, and $\pi$ is its stationary distribution, define matrices $Q,$ $R$ by the formulae: $$Q_{ij} = \pi_i P_{ij}~.$$ $$R_{ij} = ...
3
votes
1answer
60 views

Computational methods for the limiting distribution of a finite ergodic Markov chain

We wish to show what can be discovered about the limit of a finite, homogeneous, ergodic Markov Chain $X_1, X_2, \dots,$ using simple methods of computation and simulation. Specifically, consider the ...
0
votes
0answers
30 views

Markov chain: Find expected value to get back to starting state

I wonder why they complicate this solution? Call the mean time to get from i to j $M_{i,j}$ and set up three simple equations starting with $$M_{0,0} = 1 + (1/3)M_{1,0} + (1/3)M_{2,0}$$ and you ...
1
vote
0answers
54 views

Does a Markov process have memory zero?

I have the following question: The Markov process with two states and a transition matrix $$P =\begin{pmatrix} 0.3 & 0.7 \\ 0.3 & 0.7 \end{pmatrix}$$ has memory zero. Is it true? My ...
3
votes
1answer
188 views

Show $S_N = \sum\limits_{n=1}^{N} \text{sign}(Y-X_n)$ is Markov, $(X_n),Y $ iid Uniform(0,1)

Let $(X_n)$ and Y be i.i.d. Uniform$(0,1)$ random variables and let $$S_N = \sum\limits_{n=1}^{N} \text{sign}(Y-X_n)$$ Show that $S_n$ is a Markov Chain and find its transition probabilities. Any ...
1
vote
0answers
28 views

Simple random walk on the $N$-cycle

I am considering the following example: In my lecture notes we noted that "the functions $(\phi_j)_j$ form a basis". I think they refer to the space $\mathbb{C}^G$ where $G$ is the above ...
2
votes
0answers
49 views

distribution of times that a traveller passes by vertex

a traveller is travelling on a map. arriving every vertex of the map, the traveller could choose to go to next vertex according to a constant probability. The probabilities are represented in a matrix ...
0
votes
1answer
423 views

Linear Algebra Stochastic Matrix and Markov Chains

I have a few true and false questions I need help with. Can someone please check my work? The product of two stochastic matrices is a stochastic matrix. This is false I found a counterexample. 2 ...
1
vote
1answer
56 views

How to simulate visits to a transient state of a Markov chain.

Consider a discrete-parameter Markov chain $\{X_n, n ≥ 0\}$ with state space $E$, transition probability matrix $P$ and initial-state probabilities $p(0)$ given by $E = \{0, 1, 2, 3\}$, P = ...
1
vote
0answers
24 views

Quasistationary distribution for the Moran model.

The Moran model is a model for genetic drift. Basically, it is a finite Markov chain (more precise: a birth-death chain) with state space $S:=\{0,...,N\}$ and the following transition probabilites: ...
1
vote
0answers
67 views

Show crazy identity with too many sums in it (for numbers $\tau_1,…, \tau_{N-1}$). [closed]

Let $N \geq 2$ be a natural number and $ i \in \{1,\ldots,N-1 \}$. Then we define the number $\tau_i$ via $$ \tau_1 := \frac{1}{N} \sum_{k=1}^{N-1} \sum_{l=1}^k \frac{N^2}{l(N-l)}, $$ $$ \tau_i := ...
3
votes
1answer
34 views

Markov Chain with two components

I am trying to understand a question with the following Markov Chain: As can be seen, the chain consists of two components. If I start at state 1, I understand that the steady-state probability of ...
15
votes
2answers
469 views

Knight returning to corner on chessboard — average number of steps

Context: My friend gave me a problem at breakfast some time ago. It is supposed to have an easy, trick-involving solution. I can't figure it out. Problem: Let there be a knight (horse) at a ...
2
votes
0answers
37 views

Markov chain limit problem

Let $X_n$ be a Markov chain on a countable state space, $\mathbb{S}$. Let $N_n(x) = \sum_{k=1}^n\mathbb{1}_{\{X_k=x\}}$ denote the number of times the chain visits state $x\in \mathbb{S}$. Let ...
0
votes
0answers
13 views

Log moment generating function from two state transition matrix of markov process

How to find the log moment generating function of two state Markov process where the distribution is gamma distribution. The transition matrix is $$ P=\begin{pmatrix}1-\sigma & \sigma\\ \tau ...
1
vote
0answers
40 views

Gambler's ruin problem - expected time

I have troubles seeing the following. Consider the classical gambler's ruin problem, betting 1 at each time $t\in \mathbb{N}$, and losing or winning -1 respectively +1 at each time till the fortune of ...
1
vote
1answer
17 views

Finite, irreducible Markov chains - Is the mean arrival time at $j$ always finite?

We consider an irreducible Markov chain $(X_0,X_1,...)$ with finite state space $S$ and transition probabilities $p_{ij}$. Then, for $j \in S$, we can define the random variable $$ T_j :=\min{\{ n \in ...
0
votes
0answers
21 views

Problem in an example of Introduction to stochastic processes by Lawler page 25

Example. page 25: Consider the two-state Markov chain with $S=\{0,1\}$ and P= $\begin{pmatrix} 1-p & p \\ q & 1-q \\ \end{pmatrix}$ where $0< p,q< 1 $ Asuume the chain starts in ...
0
votes
0answers
23 views

Confusion in the defition of 'first passage time' (Markov Chains)

Consider a state $i$ from some state space $A$. First passage time to state $i$ is the random variable $T_i$ defined by $T_i(\omega) = inf$ { $n \geq 1: X_n(\omega) = i$ }. Does this means that ...
0
votes
0answers
10 views

Example of the strong Markov property

Can someone give me an example of strong Markov property? I have been looking Markov chains by J.R. Norris for this and the example given in that book is confusing. If anyone has read that example ...
0
votes
0answers
27 views

Convergence to equilibrium

Hi I have a question about the following proof. By definition then $\mathbb{P}$ should refer to the distribution of $X_n$, so something like $P_\lambda=\mathbb{P}$. What it confuse me a bit is the ...
0
votes
0answers
27 views

Markov chains: absorption time for upper-triangular state transition matrix

I have a (time-homogeneous, discrete-time) Markov chain with $K+1$ states $\{0,1,\ldots,K\}$. The last state $K$ is an absorbing state, all other states are transient states. Furthermore, from each ...
4
votes
1answer
41 views

Stopping time in Markov chains

A random variable $T : \Omega \rightarrow ${$1,2,3...$} $\cup$ {$ \infty$} is called a stopping time if the event {$T=n$} depends only on $X_0 , X_1 ,X_2 ,..., X_n$ for $n = 0,1,2,...$ I have trouble ...
1
vote
1answer
71 views

Understanding the random variable definition of Markov chains

Update This question is answered in section 3.2 of these notes. As a probability novice, I'm struggling to completely understand the definition of a Markov chain as a sequence of random variables. ...
0
votes
1answer
40 views

Has a Markov chain in compact metric space a stationary distribution (possibly non-unique)?

Let $Y_n$ be a Markov chain in a compact metric space. Is it true that it has a stationary distribution?
0
votes
1answer
23 views

What is does a steady state vector tell us if the matrix is irregular?

When a Markov chain is regular, the finding the steady state vector (i.e. the eigenvector corresponding to the eigenvalue $1$) will tell us the long term probability of ending up in any of the states, ...
0
votes
2answers
92 views

Random Walk on a Cube

A particle performs a randowm walk on the vertices of a cube. At each step it remains where it is with probability 1/4, or moves to one of its neighbouring vertices each having probability 1/4. Let ...
1
vote
0answers
36 views

Markov chain of transition probabilities

Let $P$ be a transition matrix on a discrete state space with $N$ elements. $P_{i,j}$ is the probability of going from state $i$ to state $j$. Let $\pi$ be the stationary distribution. Let $\{X_n\}$ ...
0
votes
0answers
19 views

steady state distribution of the following Markov jump process?

Consider a queueing process with the following rate transition matrix: $\mathbf{P}=\left( \begin{smallmatrix} -\lambda & \lambda & & & & & & &\\ \mu & ...
1
vote
1answer
26 views

Markov chain with dynamic higher orders

Let $X_i$ be the node visited by a random walk at step $i$, and the following equations be the transition probabilities. $Pr(X_n = x_n | X_{n-1} = x_{n-1}, \cdots, X_1 = x_1) = Pr(X_n = x_n | X_{n-1} ...
1
vote
1answer
27 views

optimal utility calculation for a simple discrete Markov chain

I am trying to calculate analytically the optimal decision rule for a simple discrete markov chain, following standard decision theory framework (slide 17 in ...
1
vote
1answer
27 views

Success-run on $\mathbb{Z}^+$

Let the walk on the positive integer axis $\{0,1,2,...\}$ with the following step probabilities. $p_i:=p_{i,i+1}=1-(1/2)^{i+1}$, $q_i:=p_{i,0}=(1/2)^{i+1}$. I know that the chain is transient. I want ...
0
votes
0answers
30 views

Birth-Death process with shifted exponential distribution

In the general framework of $M/M/1$ queue we have rate $\lambda$ and an exponential service time $\mu$, we can set up the transition rate matrix intuitively. However, if the service times satisfy ...
1
vote
1answer
77 views

Show that the probability of never hitting 0 on a birth-death chain is $6/\pi^2$.

In the question we have a birth-death chain on $\{0,1,2,...\}$ whose only non-zero transitions from $i$ are to $i+1$ and $i-1$, with probabilities $p_i$ and $q_i$, respectively. I have that $p_i$ and ...
1
vote
1answer
78 views

A property of Poisson process

Let $Y_t$ be a centered Poisson process, why \begin{equation} \lim_{n \to \infty} \sup_{s<t} |n^{-1}Y(ns)| = 0 \qquad a.s. \qquad \forall t\ge 0 \end{equation} This is a fundamental step in the ...
1
vote
2answers
16 views

A property of irreducible and aperiodic Markov chains

Let $P$ denote the $s\times s$ Markov transition matrix. We know that irreducibility and aperiodicity implies the following: There exists an integer $N\geq 1$, such that $[P^n]_{ij}>0$ for all ...
2
votes
1answer
45 views

Markov Process: predict the weather using a stochastic matrix

I have the following stochastic matrix $$ P = \begin{pmatrix} P(S \mid S) = 0.5 & P(F \mid S) = 0.2 & P(R \mid S) = 0.3 \\ P(S \mid F) = 0.2 & P(F \mid F) = 0.7 & P(R \mid F) = 0.1 ...
0
votes
0answers
11 views

steady state distribution of the following infinite-state Markov chain

Given the following state transition equation: $P_0(n+1)=P_0(n)(1-\lambda \Delta t)+ P_1(n)\mu \Delta t$ $P_j(n+1)=P_{j}(n)(1-\lambda \Delta t-\mu \Delta t)+\lambda \Delta t P_{j-1}(n)+ \mu \Delta ...
1
vote
1answer
53 views

Matrix Form for Hitting Probabilities in a Markov Chain

I've just read that the vector of hitting probabilities $h^A=\left(h^A_i=P(H^A<\infty|X_0=i):i \in S\right)$ is the minimal solution to the linear system: $h^A_i=1, i \in A$ and $h^A_i=\sum_{j \in ...
1
vote
1answer
76 views

Chuck Norris' Coupling of Markov Chains: An Invariant Distribution

I'm having some difficulty understanding a proof in James('Chuck') Norris book on markov chains. Let $P$ be irreducible and aperiodic, with an invariant distribution $\pi$. Let ...
2
votes
1answer
62 views

Find expected time to reach a state in a Markov chain

Consider a Markov chain $ (X_n)_{n\geq 0} $ with state space $E$, initial distribution $p(0)$ and transition probability matrix $P$ given by $E = \{0, 1, 2\}, p(0) = [1\;\; 0\;\; 0]$ and $$ P= ...
2
votes
3answers
120 views

Find expected value of time to reach a state in Markov chain, by simulation

Consider a time homogeneous Markov chain $ (X_n)_{n=0} $ with state space $E$, initial distribution $p(0)$ and transition probability matrix $P$ given by $E = \{0, 1, 2\}, p(0) = [1\;\; 0\;\; 0]$ and ...
1
vote
0answers
18 views

Understanding a Markov decision process

We have an insect that is resting on a vertex of a square at each point of time $t=0,1,2..$. The vertices are labelled from 1 to 4. 1 is given to the lower left vertex, 3 to the upper left vertex, ...
3
votes
1answer
64 views

Using Strong Markov Property

Let $X_n$ be a DTMC, with transition matrix P and state-space I. Let $Y_m=X_{T_m}$ for $m \in \mathbb{N}$. Define $T_0=\inf\{n\geq0:X_n\in J\subset I\}$ and $T_{m+1}=\inf\{n> T_{m}:X_n\in J\subset ...