Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.
1
vote
1answer
42 views
Finding Markov chain transition matrix using mathematical induction
Let the transition matrix of a two-state Markov chain be
$$P = \begin{bmatrix}p& 1-p\\
1-p& p\end{bmatrix}$$
Questions:
a. Use mathematical induction to find $P^n$.
b. When n goes to ...
1
vote
1answer
46 views
A simple case of random walk
$\forall n \in \mathbb{N}$ we can either move from state $S_n$ to state $S_{n+1}$ with probability $p$ or to state $S_{n-1}$ with probability $q=1-p$. Also we move from state $S_0$ to state $S_{1}$ ...
1
vote
2answers
55 views
How can I calculate the expected number of changes of state of a discrete-time Markov chain?
Assume we have a 2 state Markov chain with the transition matrix:
$$
\left[
\begin{array}
(p & 1-p\\
1-q & q
\end{array}
\right]
$$
and we assume that the first state is the starting state. ...
1
vote
1answer
32 views
Finding the limiting distribution of a $3 \times 3$ Markov chain
This is a question from a book.
Find $\lim_{n\rightarrow \theta}P^n$ where $$P=\begin{pmatrix}0 & 1 & 0\\
\frac{1}{6} & \frac{1}{2} & \frac{1}{3}\\
0 & \frac{2}{3} & ...
1
vote
1answer
74 views
A few questions about Markov chains
Let $\{X_n\}$, $n \geq 0$ be a Markov chain with the transition matrix $P$ such that
$$
\begin{array}{c|ccc}
&A &B &C \\
\hline
A &0.2 & 0.2 &0.6\\
B &0 & 0.25 ...
1
vote
1answer
35 views
Markov chain - recurrence and transience
Does a Markov chain with infinite recurrence states and infinite transience exists?
I believe it doesn't exists but I'm not sure how can I prove it.
Thanks guys! :D
1
vote
2answers
59 views
Simple Symmetric Random Walk : $P_{00}^{2n}=\binom{2n}{n}\left(\dfrac{1}{2}\right)^{2n}$
I was studying Simple Symmetric Random Walks and my notes state (without proof) that $$P_{00}^{2n}=\binom{2n}{n}\left(\dfrac{1}{2}\right)^{2n}$$
That is the probability of going from $0$ to $0$ in ...
1
vote
2answers
86 views
Discrete-time Markov chain
Consider the following simplistic model of transitions between social classes as defined by sociologists. Only males are considered and by assumption every male has exactly 1 son. Let $X_n$ denote the ...
1
vote
1answer
47 views
HMM as special case of MRF
I have learned that any Hidden Markov Model (HMM) can be described as a special case of a Markov Random Field (MRF) model.
However, AFAIK, the dependencies in a HMM are directed, while the ...
1
vote
1answer
45 views
How the equation is derived using geometric sequence
I need to know how equation (3.7) is derived from equation (3.6). I contacted the authors and they said "expression inside parenthesis is the summation geometric sequence with ratio of (p1/p2)." In my ...
1
vote
1answer
122 views
Why we can calculate the limiting probability of $\begin{pmatrix}0.6&0.2&0.1&0.1\\0.6&0&0.3&0.1\\0&0.6&0&0.4\\0&0&0.6&0.4\end{pmatrix}$ in this way?
Why we can calculate the limiting probability of $\begin{pmatrix}0.6&0.2&0.1&0.1\\0.6&0&0.3&0.1\\0&0.6&0&0.4\\0&0&0.6&0.4\end{pmatrix}$ in this way?
...
1
vote
2answers
49 views
How to get the proportion of time of a state of this markov chains process
Suppose each day there is a $0.2$ probability will rain in the morning. $P(\text{rains afternoon}|\text{rain morning})=0.6$ and $P(\text{rains afternoon}|\text{not rain morning})=0.3$. Suppose John ...
1
vote
1answer
97 views
Expected number of visits to state $j$ between successive visits to a state $i$ in a Markov chain given conditional information
Say I have a Markov chain $\{X_n: n \geq 1\}$ with state space $E = \{1,2,3,4,5\}$ and transition matrix,
$$ P = \begin{bmatrix} 0 & 1/2 & 1/2 & 0 & 0 \\\ 1/2 & 0 & 1/2 & ...
1
vote
1answer
83 views
Chernoff bound for Geometric RVs compared to exact tail bound
I keep getting a result I can't interpret.
X is a Geometric RV with distribution ($0<\rho<1$)
$$
\pi_k = \rho^k(1- \rho)
$$
so directly applying Geometric series the tail bound is
$$
B_1 = ...
1
vote
2answers
184 views
Markov chain with infinitely many states
I understand that a Markov chain involves a system which can be in one of a finite number of discrete states, with a probability of going from each state to another, and for emitting a signal.
...
1
vote
1answer
75 views
Why is the Markov property implied by the existence of a transition matrix?
If $\left(X_n\right)_{n\in\mathbb{N}_0}$ is an $E$-valued stochastic process with distributions $\left(P_x\space:\space x\in E\right)$ satisfying
$$\mathrm{P}_x\left(X_0=x\right)=1$$
and stochastic ...
1
vote
1answer
124 views
Transforming an inhomogeneous Markov chain to a homogeneous one
I fail to understand Cinlar's transformation of an inhomogeneous Markov chain to a homogeneous one. It appears to me that $\hat{P}$ is not fully specified. Generally speaking, given a $\sigma$-algebra ...
1
vote
1answer
69 views
Is this a valid proof for recurrence time?
The following is a well known result of Markov chain:
Given a Markov chain $(X_t)_{t \ge 0}$, if $T_{ii}$ denote the time of
the first return to state $i$ when starting at state $i$, then we ...
1
vote
1answer
94 views
Eigenvalues of a quasi-stochastic matrix
Quasi-stochastic
In order not to make the title too long I used the term Quasi-Stochastic with this meaning: a quasi-stochastic matrix $Q$ is a square matrix $Q = ...
1
vote
2answers
413 views
Can Markov Chain state space be continuous?
I looked for a formal definition of Markov chain and was confused that all definitions I found restrict chain's state space to be countable. I don't understand purpose of such a restriction and I have ...
1
vote
2answers
95 views
Finding again the stationary distribution of a markov chain
I am asked to compute the stationary distribution of the markov chain with state space $E=\{0\dots,n\}$ and transition matrix below:
\begin{bmatrix}
0 & 1 \\
\frac{1}{n} ...
1
vote
1answer
188 views
Why does this example of Hardy-Weinberg equilibrium not work?
Background Info
Hardy-Weinberg equilibrium is a mathematical model of the frequencies of alleles (i.e., versions of a gene) in a population. The model states that the frequency of the 2 alleles in a ...
1
vote
1answer
175 views
Finite State Markov Chain Stationary Distribution
How does one show that any finite-state time homogenous Markov Chain has at least one stationary distribution in the sense of $\pi = \pi Q$ where $Q$ is the transition matrix and $\pi$ is the ...
1
vote
1answer
502 views
Proving a process is Markov chain
Could anyone give me an example of a problem where it is requested to prove rather than assume that a stochastic process forms a Markov chain. I can think of something like this: if $X_{n+1} = X_{n} + ...
1
vote
2answers
167 views
Calculating probabilities (Markov Chain)
Let $\mathcal{X}=(X_n:n\in\mathbb{N}_0)$ denote a Markov chain with state space $E=\{1,\dots,5\}$ and transition matrix
...
1
vote
1answer
22 views
Mean number of particle present in the system: birth-death process, $E(X_t|X_0=i)$, $b_i=\frac{b}{i+1}$, $d_i=d$
Let $\{X_t\}$ be a birth–and–death process with birth rate
$$
b_i = \frac{b}{i+1},
$$
when $i$ particle are in the system, and a constant death rate
$$
d_i=d.
$$
Find the expected number of particle ...
1
vote
1answer
55 views
Limit theorem of Markov chains applied to higher order Markov chains
I have a second order Markov chain with 4 states {A,T,C,G} (the 4 DNA nucleotides).
the transition matrix looks like this:
...
1
vote
1answer
39 views
Hitting times of Markov chain/process have always finite moments?
Consider an irreducible ergodic Markov chain on a finite state space $\Omega$. Then any state is positive recurrent and this should suffice to conclude that the mean hitting time of state $s \in ...
1
vote
1answer
38 views
Asymmetric random walk with unequal step size other than 1.
Say, an asymmetric random walk, at each step it goes left by 1 step with chance $p$, and goes right by $a$ steps with chance $1-p$. (where $a$ is positive constant).
The chain stops whenever it ...
1
vote
1answer
56 views
Markov chain property
Suppose $\{Y_{n}, n \ge 0\}$ is a Markov chain consisting of $N$ states. Suppose that $i$ and $j$ are states of this Markov chain and that $i \hookrightarrow j$, i.e state $j$ can be reached from ...
1
vote
1answer
261 views
Example of a Markov chain transition matrix that is not diagonalizable?
It is well-known that every detailed-balance Markov chain has a diagonalizable transition matrix. I am looking for an example of a Markov chain whose transition matrix is not diagonalizable. That is:
...
1
vote
2answers
41 views
How to prove the existence of the limit of Markov transition matrix?
Does the limit of a Markov transition matrix $M$:
$$\lim_{n\to\infty}M^n$$
always exist? And if yes, how to prove it?
1
vote
2answers
138 views
How to create a transition matrix that will guarantee an outcome after infinite transitions
Let's assume we have the a transition matrix like:
0 0 0
1 2 0
2 4 0
3 6 0
4 7 2
5 9 3
6 6 6
7 7 7
8 8 8
9 9 9
First ...
1
vote
2answers
43 views
Period of Markov Chain when no chance of return
Everywhere I look I see that the period of a state $i$ in a Markov chain is given by
$$
\gcd\{n>0 : P_{ii}^n>0 \}
$$
but what do we mean if the set $\{n>0 : P_{ii}^n>0 \} = \emptyset$? For ...
1
vote
1answer
140 views
Markov chain with finite positive recurrent states
If I have a Markov chain with finite positive recurrent states $\in S$, then that means starting from a given state $y$, the expected number of steps to return to state $y$ is finite.
Now, if I start ...
1
vote
1answer
51 views
Symmetry of hitting times in a Markov chain
Consider an irreducible, aperiodic Markov chain with stationary distribution $\pi$. We will use $E_{\pi} T_j$ to be the hitting time of node $j$ when the initial distribution is $\pi$, and $E_i T_j$ ...
1
vote
1answer
220 views
Irreducible, finite Markov chains are positive recurrent
I am under the impression that an irreducible, finite Markov chain is necessarily positive recurrent. How might I show this?
Regards,
Jon
1
vote
1answer
79 views
Classify the states of a markov chain
a)
P =$\begin{bmatrix}
{1-2p} & 2p & {0} \cr
{p} & {1-2p} & {p} \cr
{0} & 2p & {1-2p} \cr
\end{bmatrix}$
b)
P = $\begin{bmatrix}
0 & p & 0 & 1-p \cr
1-p & 0 ...
1
vote
2answers
88 views
Game with losing and winning a dollar
I found an interesting problem in my book:
There is a game where player starts with $k\$$. In each step he wins or loses $1\$$ (both with probability $p=\frac{1}{2}$). The game ends when player ...
1
vote
1answer
40 views
Why do all $X_0 … X_n$ have the stationary distribution?
Say I have an irreducible Markov chain with state space $\{1, 2, 3 ... m\}$, where $m > 2$ and stationary distribution $s = \{s_1, s_2, ... s_m\}$. The initial state is given by the stationary ...
1
vote
1answer
64 views
Explanation of Parsimony
Can someone explain what Parsimony is in the context of probability, more specifically in Parsimonious Markov models?
I have been trying to search around a simple explanation of this but I only seem ...
1
vote
1answer
30 views
is this the correct way to impose an additional condition on a hidden markov model?
Suppose I have an observation $Y_t$ that is conditionally dependent on $X_t$. (More specifically, Y is a series of observations emitted by underlying hidden markov state sequence X.)
I can describe ...
1
vote
1answer
78 views
Markov Property
this relates to an unanswered question I posted a few days ago:
Let $\{ X_t : t = 1, 2, 3 \dots \}$ follow a 2-state Markov chain with transition matrix P. Does the Markov property mean I can break ...
1
vote
1answer
448 views
When is a Markov chain null recurrent?
Are there any necessary and sufficient conditions for a Markov chain to be null recurrent? What about sufficient conditions?
Naturally, I am not looking for tautological statements, e.g., a Markov ...
1
vote
1answer
57 views
How does this Markov process involving balls and bins behave?
I have some set $S_1,\ldots,S_k$ ($k \geq 3$) of bins, each initially with $N_0(S_i)$ balls ($N_t(S_i)$ denotes the number of balls in $S_i$ at time $t$). A bin can contain a negative number of balls. ...
1
vote
1answer
92 views
Expected number of jumps in a regular pure-birth process with Malthusian parameter.
Consider a pure-birth process $X(t)$ with rates $\lambda_i$ that satisfies
$$\sum_{i=0}^\infty \frac{1}{\lambda_i} = \infty.$$ By Reuter's criterion this is sufficient for $X(t)$ to be regular, ie ...
1
vote
1answer
175 views
A Markov chain probability calculation.
I'm taking a course about Markov chain, and here's a snippet from the lecture notes:
Let $(X_i, i \ge 0)$ be a time homogeneous Markov chain, let $V$ be
the state space, let $\lambda$ be the ...
1
vote
1answer
945 views
Expected number of steps/probability in a Markov Chain?
Can anyone give an example of a Markov Chain and how to calculate the expected number of steps to reach a particular state? Or the probability of reaching a particular state after T transitions?
I ...
1
vote
1answer
825 views
Show irreducibility of markov chain
I need to show that the markov chain that has transition matrix written below is irreducible.
\begin{bmatrix}
0.2 & 0.5 & 0.1 & 0.1 & 0.1 \\
0.2 & 0.5 ...
1
vote
1answer
149 views
Finding the distribution function of a markov process
Given a markov process $\mathcal{Y} = (Y_t : t \ge 0)$ with state space $E=\{1,2,3\}$ and with generator matrix
$G = \left[ \begin{array}{ccc} -3 & 1 & 2 \\ 1 & -2 & 1 \\ 0 & 0 ...
