Questions tagged [markov-chains]

Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Balance equation of stationary distribution (Markov chains)

Let $P$ be a transition matrix on the finite state space $S$. Show that every stationary distribution $\pi$ satisfies the following equation for all $A \subset S$: $\sum_{i \in A} \sum_{j \in S \...
0
votes
0answers
15 views

Stationary distribution in a Markov process.

Consider the homogeneous Markov process with matrix $W$ which describes the "probability of transition" to pass from a state $a$ to $b$. So in the time $t+1$ the probability to be in the state $a$, $...
0
votes
0answers
20 views

Stationary distribution of a Markov chain on the nonnegative integers

Let $\lambda_k,\mu_k\in\mathbb R_{\ge0}$ $(k\ge1)$ be nonnegative real numbers such that $\sum_{k=1}^\infty k\lambda_k<\infty,$ let $S=\mathbb Z_{\ge0}$ be the nonnegative integers, let $T=\mathbb ...
0
votes
1answer
14 views

How do we show $Z=g(Y) \implies X \Rightarrow Y \Rightarrow Z$ i.e. $X,Y,Z$ form a Markov chain?

I am reading Elements of Information Theory by Cover and Thomas, and I have come across the above corollary to Markov chains. In their notation, this would amount to showing that $Z=g(Y) \implies p(x,...
0
votes
0answers
12 views

Using Chapman-Kolmogorov Property to prove v=Qv

How would you use the Chapman-Kolmogorov property ($Q_{t+s}=Q_tQ_s$) to prove that v (a column vector distribution over the sample space) is a stationary distribution of Markov Chain $X_t$ with ...
1
vote
0answers
17 views

Stationary Distribution of Ehrenfest Markov Chain

The example in my book for an Ehrenfest Markov chain is: A system of of two urns, A & B where there are 2n balls total in both urns. We are assuming that there are $i$ balls in urn A and $2n - i$ ...
0
votes
1answer
10 views

Expected hitting times for simple random walk on a hypercube

Setup In an $n$-dimensional hypercube $C_n = \{0,1\}^n$, we define the Hamming distance of two vertices $d(A,B)$ to be the number of coordinates in which they differ. (e.g. $d((0,0,1), (1,0,1)) = 2$.)...
0
votes
1answer
11 views

Markov chain covariance calculation

Suppose $Y_t=1$ with probability $\pi_s$ and $0$ with probability $1-\pi_s$. Moreover, we know that $Y_t$ is a Markov chain with transition probabilities $Pr(Y_t=1|Y_{t-1}=1)=(p^3+(1-p)^3)/p_s$ and $...
0
votes
2answers
24 views

Reference request: convergence for periodic Markov chains

Consider an irreducible but possibly periodic Markov chain on a finite state space with transition matrix $P$. We know there exists a unique stationary distribution $\pi$. If the Markov chain were ...
2
votes
1answer
25 views

For ergodic Markov chains, when does $\lim_{N\to\infty} \mathbb{E}[\sum_{n=1}^{N}f(X_n)] - N\mu(f)$ exist

For an ergodic Markov chain (process would be even better) $X_{n}$ with stationary distribution $\mu$, under which conditions does $$ L:=\lim_{N\to\infty} \mathbb{E}[\sum_{n=1}^{N}f(X_n)] - N\mu(f) $$ ...
0
votes
2answers
38 views

Conditional expectation of markov chain

Let$\{X_n:n\geq0\}$ be a Markov chain with $X_{n+1}$, conditionally on $X_n=x$, a Poisson r.v. with parameter $\lambda x$ where $\lambda$ is a positive parameter. $X_n$ can be thought of as the number ...
0
votes
0answers
18 views
+50

Question on the use of the Markov Kernel for conditional probability

We define a Markov kernel Let $(\Omega_{1},\mathcal{A}_{1})$ and $(\Omega_{2},\mathcal{A}_{2})$ be some measurable spaces. A map $K$ where $K : \Omega_{1}\times \mathcal{A}_{2}\to [0,\...
1
vote
1answer
32 views

Extracting a smaller Markov chain from a larger Markov chain

I am not very familiar with Markov chains, hence the probably ill titled questions. If we have 5 random variables $X, Y, Z, W$ and they form a Markov chain such that $$X \rightarrow Y \rightarrow ...
3
votes
0answers
35 views

is there any relation between the right eigenvectors of $P^\top$ and $\begin{bmatrix}0 &P^\top \\P &0\end{bmatrix}$?

I'm trying to analyze the stationary distribution of a Markov Chain parameterized by some n by n matrix $P$ (top right eigenvector of $P^\top$). While there is a relation between the singular vectors ...
-1
votes
1answer
28 views

About measures acting on measurable functions.

If $\phi : X \rightarrow \mathbb{R}$ is a measurable function and $\mu$ is a measure on $X$ then what is $``\mu(\phi)"$? Is this a notation for some function? I came across this notation for the ...
0
votes
0answers
55 views

Need a help with Markov Chain

We have Markov Chain with continuous time, three conditiions and generator: \begin{equation*} Q = \begin{bmatrix} -3 & 1 & 2\\ 1 & -1 & 0\\ 1 & 0 & -1 \end{bmatrix} \end{...
1
vote
1answer
37 views

Dyck path - Probability of stopping time $2n$

Can someone explain to me, why the probability of returning to the origin of a Dyck path with length $2n$ is: $\mathbb{P}(\tau=2n) = 2C_{n-1}4^{-n}$? $C_n$ stands for the Catalan number and $\tau :=...
0
votes
2answers
42 views

How to prove that Markov chain with specific transition probabilities has independent increments?

I have Markov chain $N=\{N(t) \mid t\geq 0 \}$ with the state space $\{0,1,2,\dots\}$. I know that it is homogeneous and transition probabilities are: $$ p_{ij}(s,t)=P(N(t)=j\mid N(s)=i) = p_{i,j}(t-...
1
vote
0answers
35 views

Consider a branching process $\{X_n , n \geq 0 \}$ in which the offspring distribution is binomial $(k,p)$

Consider a branching process $\{ X_n , n \geq 0 \}$ in which the offspring distribution is binomial $(k,p)$. Find probability of ultimate extinction when $k = 3$. So I've tried this: $P_k = P(\...
0
votes
0answers
16 views

Markov chain convergence in probability

The Markov chain $X_{n}$ takes values 1, 2, 3. The matrix of probability transitions is: $$ \begin{bmatrix} 0.6 & 0.4 & 0\\ 0.2 & 0.5 & 0.3\\ 0 & 0.1 & 0.9 \end{bmatrix} $$ ...
-1
votes
1answer
22 views

How to derive first and second moments? [closed]

given the equations: $\frac{d}{dt} p_{j} = \gamma (j-1) p_{j-1} +\gamma (j+1) p_{j+1} -2\gamma j p_{j}$ for $j=1,2,..$ $\frac{d}{dt} p_{0} = \gamma p_{1} (t)$ and $p_{j} (t) =0$ if $j<0$ how to ...
0
votes
0answers
18 views

Proving distribution is reversible condition

I'm working on the following question: Consider an irreducible homogeneous Markov chain over a discrete state space $S$, all of whose states are positive recurrent. Prove that its stationary ...
0
votes
0answers
14 views

Transition probability matrix for n x n matrix

I'm given the following problem: Consider a Markov chain on $S = \{0,1,\ldots,n\}$ with transition probabilities $p(j,j+1) = \frac{n-j}{n}$ and $p(j,j-1) = \frac{j}{n}.$ I'm having enormous ...
0
votes
0answers
18 views

How do you find the probability of a continuous-time markov chain, where having started in state $i$, will be in $j$ state at time $t$?

This is the three-state continuous-time Markov chain in which the transition rates are given by: $$Q = \left[ \begin{matrix} 0 & 2\lambda & 0 \\ \lambda & 0 & \lambda \\ ...
1
vote
0answers
25 views

Markov chain on $\mathcal{Z}$, hitting probability after having done negative jumps of total magnitude $k$

I'm considering a random walk Y on $\mathbb{Z}$ with jump sizes in $[-E, E]$, defined as follow: $Y_0 = 0$ and for $n \ge 1 $: $$Y_n = \sum_{i = 1}^n X_i,$$ with $(X_i)$ a family of i.i.d random ...
0
votes
0answers
13 views

ergodicity of non-linear stochastic dynamical system

Is there any ergodicity result( existance and uniqueness of invariant probability measure) for the Random dynamical system which form a discrete-time markov chain by its realization equation as: $y_{t+...
0
votes
0answers
20 views

Definitions of recurrence / transience (Discrete-time Markov Chains)

We have that the definition of state $i$ being transient is $\mathbb{P}(\text{# times we are at $i$ is finite})=1$. Is this equivalent to saying $\mathbb{P}(\text{the first time step we return to $i$ ...
0
votes
0answers
11 views

Every finite closed class is recurrent proof clarification

In Norris: Markov Chains the closed class C is defined as one for which $i\in C$ and $P_i(X_n=j \text{ for some }n\ge0)>0$ implies that $j\in C$. Here's theorem 1.5.6 from the book with proof ...
0
votes
0answers
8 views

Measurability of the exterior boundary of a set for a transition kernel

I am learning the potential theory for the Markov chains. And I've encountered a problem: Let $\pi$ be a transition kernel on a Polish space $(S,\mathcal{B})$, and let $D \in \mathcal{B}$. The ...
1
vote
0answers
29 views

Are limiting and stationary distributions of a Markov chain always unique?

Firstly am I correct in saying that that for an irreducible, aperiodic, positive recurrent Markov chain, a limiting distribution exists, and this distribution is the same as the chain's stationary ...
1
vote
1answer
12 views

Why is M/G/1 queue viewed only at departure times?

It is said in the book that for M/G/l queue, viewed only at departure times, leads to an imbedded discrete-time Markov chain. Viewing the queue only at arrival times does not yield a Markov chain. ...
0
votes
0answers
26 views

Markov Chain Transition matrix Question involving steady state vector, transition matrixes etc?

In an office complex of 1170 employees, on any given day some are at work and the rest are absent. It is known that if an employee is at work today, there is a 76% chance that she will be at work ...
0
votes
1answer
11 views

Proving doubly stochastic matrix limit

Let n > 0 and Xn be an irreducible aperiodic Markov chain having a doubly stochastic transition matrix. By definition, $\sum_{y∈S} P(x,y) = 1$ and $\sum_{x∈S} P(x,y) = 1$ for all x y ∈ S. I want ...
1
vote
2answers
53 views

A vowel is followed by a consonant 80% of the time and a consonant is followed by a vowel 50% of the time. Proportion between cs and vs?

A text is such that the probability that a vowel is followed by a consonant is 80% and a consonant is followed by a vowel 50% of the time. What is the proportion of vowels in the text? I don't ...
0
votes
1answer
157 views

Write down the backward equations for $P_{12}$ and $P_{21}$ and use the symmetry of Q to solve these equations.

Hint: Whenever confronted with an ordinary differential equation of the form x′(t) = ax(t)+b(t), it might be beneficial to consider the function y(t) = $e^{−at}x(t)$. $$Q = \left[ \begin{matrix} ...
1
vote
1answer
33 views

What is meant with persistence time in stochastic models?

I've been reading about epidemiological modeling using stochastic models (discrete/continuous Markov chains and stochastic differential equations). I've come across the term persistence time multiple ...
0
votes
1answer
20 views

How to understand the first formula in Case 2 of Example 4.26 in ross's probability model(11th)

Ross's Probablity Models(11th) Example 4.26 (Mean Pattern Times in Markov Chain Generated Data) Consider an irreducible Markov chain $\{X_n, n\geqslant0\}$ with transition probabilities $P(i,j)$ and ...
1
vote
2answers
93 views

Proving that in an homogeneous DTMC $i\leftrightarrow j$ and $j$ is recurrent $\implies f_{ij}=1$

I'm struggling to understand the proof of the following theorem: Theorem: Let $\big\{\mathsf X\big\}_{n\in\mathbb N}$ be an homogeneous DTMC, P its transition matrix and $\mathsf S$ its states space; ...
2
votes
0answers
16 views

Prove $Y_k$ is a homogeneous markov chain

Let $(X_n)_{n\in\mathbb{N}_0}$ be a homogeneous markov chain with starting distribution $\mu$, transition matrix $P$ and $P(x,x)<1$ for all $x\in S$, and $\tau_0:=0$ and $\tau_{k+1}:=\inf\{n\geq \...
0
votes
0answers
16 views

Recurrent Harris Chain

A Harris chain, $X_n$, as defined in Durrett, Probability, sec. 5.8 of the V ed., depends on the definition of two measurable sets A and B and some probability measure $\rho$ (details are given on ...
0
votes
0answers
31 views

Could indicator function be introduced into probability?

Suppose there are 3 random variables: $A_t\in\{0,1\},B_t,\text{and}\; B_{t-1}$. I would like to write a joint probability distribution to express the idea: At time step $t$, $B_{t-1}=b$ has been ...
0
votes
1answer
15 views

Clarification of finding this transition probability matrix

Let $X_n$ denote the two-state Markov chain with transition probability matrix P= $ \begin{bmatrix} \alpha & 1-\alpha\\ 1-\beta & \beta \end{bmatrix} $ given states 0 and 1. Let $Z_n=(X_{n-1},...
-1
votes
0answers
19 views

Markov chains, probability

We have two coins, one regular and the other has both sides of the head. We give these coins to a friend, who chooses one of them with a probability of 1/2. He then uses the currency he has chosen. a) ...
1
vote
2answers
18 views

Expected number of throws in a custom dice game, difference manual and simulation

There is a difference in my computed expected number of throws by hand and by simulation. The difference is about 1/4, but my question is, which one is wrong? Clarification of the rules of this game: ...
1
vote
1answer
26 views

Markov chain with external input

Could anyone explain to me this Markov chain model? $$S_{k+1}= P(S_k+S_k^0).$$ Please allow me to give a link from the paper I was read this equation $(6)$ here https://drive.google.com/file/d/...
4
votes
1answer
98 views

Proof using strong Markov Property

Let $X = (X_n)_{n\in\mathbb{N}_0}$ be a homogenous Markov Chain with starting distribution $\mu$ and transition matrix $P$, where $P(x,x)<1$ for all $x\in S$ and $\tau_0:=0$ and $\tau_{k+1}:=$ inf$...
0
votes
0answers
13 views

Non-linear Markov chain model [duplicate]

Could anyone explain to me this Markov chain model? $$S_{k+1}= P(S_k+S_k^0).$$ Please allow me to give a link from the paper I was read this equation $(6)$ here https://drive.google.com/file/d/...
0
votes
1answer
12 views

Markov chain connected with recurrent events

I am reading Feller Volume 1, and this example is in page 382. I understand that $f_1= q_1$ and $f_2 = p_1 q_2$, but I don't understand how to derive $p_k$ in general (which I highlight with the ...
0
votes
0answers
35 views

Consider a two-state continuous-time Markov chain with rates λ and μ. Write and solve the Kolmogorov backward equations.

This is What I have so far and I am not sure if I am on the right track: Consider the matrix Q = $ \left[ \begin{matrix} λ & 0 \\ 0 & μ\\ \end{matrix} \right] $ So then the ...
1
vote
1answer
58 views

methods for finding the limit of $n$-step probability

I have the following transition matrix of a markov chain: $$\mathbb{P}= \begin{bmatrix} 1 & 0 & 0 & 0 & 0\\ q & \delta & p & 0 & 0\\ 0 & q & \delta & p &...

1
2 3 4 5
86