Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
9 views
1
vote
0answers
22 views

Example of Markov property

I am reading Durrett's Probability: Theory and Examples, and trying to understand its context about Markov Chains. It is not hard to understand the theorem and proofs but when it comes to concrete ...
0
votes
1answer
34 views

proof of Markov chain Monte Carlo

This is the first step of proof of MCMC in my notes I have a question, how come $\pi(x)\pi(x_p\mid x)=\pi(x_p)\pi(x\mid x_p)$? Is it true for any markov chains which are ergodic and aperiodic? The ...
0
votes
1answer
26 views

confused about the proof of Markov chain Monte Carlo

This is the proof from notes I'm confused about the $\pi(x_p|x)$ and $\pi(x|x_p)$ Let's say $X\sim $Bin$(10,0.3)$, so $\pi(x)=\binom{10}{x}0.3^x0.7^{(10-x)}$, so what does $\pi(x_p|x)$ or $\pi(x|...
1
vote
0answers
16 views

Asymptotic Growth of Markov Chain

I'm interested in the following problem: We have got a time-discrete Markov chain $(X_n)$ with state space $S=\mathbb{R}_+^d$. The transition kernel is discrete in the sense, that for each $s \in S$ ...
0
votes
1answer
50 views

confused about proposal distribution in MCMC

This is a question from notes I have some questions regarding the proposal distribution which is $N(x,1)$ Is the proposal distribution symmetric i.e. $g(x_p|x)=g(x|x_p)$? I'm not sure whether it ...
1
vote
1answer
29 views

Markov chain average waiting time

Given the following $E=\{1,2,3\}$ and matrix $P=\begin{bmatrix} 1/2 & 1/3 & 1/6\\ 1/4 & 3/4 & 0\\ 1/2& 0 & 1/2 \end{bmatrix}$, assuming that chain starts from point 1 find ...
0
votes
0answers
9 views

Is Markov Chain property true for correlated inputs?

I have a finite state machine (FSM). At time $k$, state is $\theta^k$ and input is $x^k$. The next state $\theta^{k+1}$ and output $y^k$ are completely determined by \begin{align} \theta^{k+1} &=...
1
vote
2answers
58 views

When a Markov chain converges to a steady state, what kind of convergence is it?

Let $A$ be a transition matrix, the steady state distribution $x$ satisfies the distribution $Ax = x$. One can prove that under certain circumstances, $$\lim_{n\rightarrow\infty}A^n q=x$$ where $q$ is ...
1
vote
1answer
56 views

1D random walk probability distribution

I am way more physicist than mathematician and this question arises from experimental physics/engineering. The motivation is dealing with small amount of random discrete shifts between measured ...
1
vote
0answers
45 views

How can I generate random samples from following probability density function?

Let $\mathbf{\alpha}=(\alpha_1, \ldots, \alpha_m)$. The posterior density function of $\mathbf{\alpha}$ is given by $$h_0(\mathbf{\alpha}|\mathbf{x})=‎\frac{\prod_{i=1}^{m}\alpha_i^{a_i}}{\left(1+\...
1
vote
0answers
33 views

The average number of transitions to a particular state from only two particular states.

I know that in a Markov chain, $$\mathbf{N} = (\mathbf{I} - \mathbf{Q})^{-1}$$ gives a matrix to calculate the expected number of times before absorption that a particular [transient] state is visited,...
-1
votes
1answer
11 views

Markov Equations Balance equations and Normalising equations

I am looking at a question involving three equations: $A=0.6667A+0.2222B+0.1667C$ $B= 0.2A+0.3333B+0.5C$ $C=0.1333A+0.4444P+0.3333C$ The solution then goes on to say, that these equations can be ...
0
votes
0answers
14 views

expected number of steps to absorption when there are two absorbing states

The expected number of steps to absorption for an absorbing state markov chain can be obtained from the fundamental matrix F. By adding up the first row of F we get the expected number of steps to ...
0
votes
0answers
108 views

Calculating probabilities in a Markov chain process [on hold]

I have 3 variables A, B and C with each variable having a probability of 0.6 and 0.4 i.e. A can have states (ON) with probability of 0.6 as well as can remain in certain states (OFF) with probability ...
2
votes
1answer
43 views

Does absorbing Markov chain have steady state distributions?

If I am not mistaken, the steady state distribution is independent of initial state distribution, and regular Markov chains satisfies this definition. On the other hand, since the row of each ...
1
vote
0answers
10 views

How to solve stationary disribution with higher dimension?

I have a problem with stationary disribution for Markov chain. Let say we have $\pi P=\pi$ where P is $n\times n$ transition matrix and $\pi$ is $n\times 1$ vector. Then, we have $\pi (P-I)=0$. ...
0
votes
0answers
15 views

Absorbing State vs Closed Communicating Class

According to Wikipedia, A set of states C is a communicating class if every pair of states in C communicates with each other. A communicating class is closed if the probability of leaving the class is ...
1
vote
0answers
37 views

Is it an absorbing state if it does not communicate with other states?

$$\begin{matrix} 1 & 0 & 0\\ 0 & 0.5 & 0.5\\ 0 & 0.5 & 0.5 \end{matrix}$$ Given that this is a right transition matrix, would you call the state in the first row, say A, an ...
7
votes
1answer
105 views

How long until I get out of bed?

Suppose I have two independent alarm clocks which I set right before I go to bed. Their ring times are exponentially distributed with rates $\lambda_1$ and $\lambda_2$. Whenever alarm 1 goes off I ...
0
votes
1answer
45 views

Finding the stationary distribution

For a Markov process with state space $S= \{ 0,1,2,\dots\}$ The one step probabilities are: $p_{0,0}=q$, $ p_{0,2}=p$ and $p_{i,i-1}=q$, $ p_{i,i+1}=p$ for $i \geq1$ where $p+q=1$. The one step ...
0
votes
2answers
44 views

transition matrix for urn model

There are slides regrading to urn model I have two questions 1.if a Species A dies and a Species A is born, the original text says the probability is 0.4*0.4, but since a Species A has died , ...
0
votes
0answers
14 views

What is the definition of perturbed Markov chain?

Please correct me if I'm wrong: A perturbed version of a Markov chain with transition matrix $P$ is exactly the same Markov chain where its transition matrix is slightly perturbed, i.e. $P_\mu$. Here,...
-1
votes
0answers
48 views

Birth-death process and transience.

I am unable to tackle part c) and d) can anyone help/ sugesstions? A Markov chain with state space ${0,1,2,...}$ is called a “birth-and-death chain” if the only non-zero transitions from state $i$ ...
1
vote
1answer
33 views

periodic period for Markov Chain

I don't understand why the only state with period > 1 is 1 Let's take state 2 for example, what's the period for state 2? Another question is, does an absorbing state(state 4 in this example) only ...
0
votes
0answers
18 views

showing that a matrix has repetitive values?

Here my primary aim is to calculate the stationary distribution of a DTMC using left-eigen values i.e, $ \pi = \pi*P$. But for some matrices, I observe that some states a same stationary probability. ...
0
votes
0answers
42 views

$(X_n)$ transient Markov chain with transition probability matrix $P = \|P_{ij}\|$. Define $u(i) = \sum P_{i0}^{(n)}$. Then $u(X_n)$ is submartingale

Let $(X_n), n \geq 0$ be a transient Markov chain on the non-negative integers with transition probability matrix $P = \|P_{ij}\|$. Define $u(i) = \sum_{n=0}^{+\infty} P_{i0}^{(n)}$. Then $u(X_k)$ is ...
0
votes
1answer
15 views

Is this a Markov chain when dealing with minimum?

So in my probability studies I just encountered this: For a Markov chain $ X_n $ we have a finite state space $ \{1,2,...,k\} $ such that we can transit from one index to the next and from k only ...
1
vote
0answers
18 views

Lipschitz constant/derivative of the stationary distribution of a Markov chain under perturbations in the transition kernel

I'm interested in the following question: Given a parameter $t\in \mathbb{R}$ and a column stochastic matrix $P(t)$ (i.e., $e^T P(t)=e^T$ and $P(t)_{ij}\ge 0$), calculate the Lipschitz constant of ...
2
votes
1answer
66 views

Can you suggest a method to generate random sample from following PDF?

‎Let‎ ${‎‎\bf{\alpha}}=(\alpha_1, \alpha_2, \ldots, \alpha_m)$ ‎and ‎‎$‎‎\textbf{b}=(b_1, b_2, \ldots, b_m, b_{m+1}).$ I intend ‎to ‎generate ‎sample ‎from PDF $$ g(\alpha_1, \alpha_2, \ldots, \...
2
votes
1answer
38 views

Conditional expected number of visits in symmetric random walk with two absorbing barriers

Consider a symmetric random walk on vertices $\{0,1,2,\ldots,n\}$. Suppose that we are at vertex $1$ initially. At each step, we move left with probability $1/2$ and right with probability $1/2$. We ...
1
vote
0answers
20 views

Conditional probability of a random walk hits position $b$ in $n$ steps

This question comes from my question Modified gambler's ruin problem: quit when going bankruptcy or losing $k$ dollars in all Generally, I know the probability that a random walk hits position $b&...
5
votes
1answer
82 views

Modified gambler's ruin problem: quit when going bankruptcy or losing $k$ dollars in all

In each round, the gambler either wins and earns 1 dollar, or loses 1 dollar. The winning probability in each round is $p<1/2$. The gambler initially has $a$ dollars. He quits the game when he has ...
2
votes
0answers
35 views

Distribution of Markov Chain at a Stopping Time

Suppose $(X_t)_{t \geq 0}$ is a Markov chain on the state space $S$ with transition probability $p$, and that $\pi$ is a stationary distribution for $p$. If $X_0 \sim \pi$, then we know $X_t \sim \pi$ ...
1
vote
0answers
29 views

Differences of Markov chain is Markov

In my studies of Markov chains, I was tackled with this tough problem: Let $ \{ X_n \}_{n=0}^{\infty} $ be a homogeneous Markov chain with transition probabilities satisfying $ | i-j | > 1 \to ...
0
votes
0answers
12 views

Drift analysis of an absorbing Markov chain

Consider a set $S$, and suppose we have a sequence of random subsets $$ \zeta_t = \{x_1, \dots, x_n\} $$ for $x_1, \dots, x_n \in S$. We do not know with which probability density the points of each $\...
1
vote
0answers
26 views

Basic Limit Theorem for Markov Chain (Knowing the odds)

In the book "Knowing the Odds", Basic Limit Theorem for Markov Chain is stated as follows. Theorem 7.41 (Basic Limit Theorem). Suppose $j$ is a recurrent aperiodic state in an irreducible Markov ...
0
votes
1answer
30 views

What is the difference between a reversible markov chain and a reversible in equilibrium markov chain?

In the text I'm using it says: Let X = {$X_n : 0 \leq n \leq N$} be an irreducible Markov chain such that $X_n$ has the stationary distribution $\pi$ for all $n$. The chain is called reversible if ...
0
votes
0answers
18 views

stopping criteria for power-iteration to find rank-1 matrix

I start with B=I, A positive matrix, and compute B=(BA)/norm(B) by iterating until B is sufficiently close to rank-1 matrix. What is a good stopping criterion for this algorithm? There's Birkhoff ...
-1
votes
0answers
57 views

Simple Markov property on stopping times [on hold]

Suppose $(S_n)_{n\geq1}$ is a Markov chain on the two dimensional lattice of the integers. Then define the stopping time $\tau_A'=\inf\{n\geq1:S_n\notin A'\}$ and consider the following for $A\subset ...
1
vote
1answer
26 views

Show that it is a Markov chain, determine the transition-probability matrix and reversibility

Certain machine has three possible states: $0=working,\,1=broken\,and\, awaiting\,repair,\,2=broken\,and\,being\,repaired$. The permanence times (in minutes) in each state have independent geometric ...
4
votes
2answers
34 views

Distribution of throws of die rigged to never produce twice in a row the same result

A die is “fixed” so that each time it is rolled the score cannot be the same as the preceding score, all other scores having probablity 1/5. If the first score is 6, what is the probability that the ...
0
votes
1answer
19 views

Random walk mean number of visits to state before absorption

This is from Stirzaker's book Random Processes. Suppose we have a simple random walk with probability going "up" p, "down" q. At time 0 it stats at 0, so $$S_0 = 0$$ Now let $u_b $ be the mean ...
1
vote
1answer
29 views

Extension of ergodic theorem with WLLN

Suppose you have a ergodic (or irreducible) Markov chain $(A_t)_{t\geq0}$ in continuous time. Denote by $\pi$ the invariant distribution of $A$. If $f$ is a function of $A_s$ which is integrable w.r.t....
1
vote
2answers
76 views

Expected steps to eliminate a character? [closed]

This came up in a game theory crafting exercise. Imagine a character has 195 hit points. You can shoot at them and there are three results: Critical - 100 damage - 40% of shots Body - 50 damage -...
1
vote
0answers
17 views

First time of passage, discrete random walk with disjoint absorbing regions

I have a sum $T^i$ of zero/one Bern$(p)$ random variables $T_i$ and multiple disjoint absorbing regions, i.e. the absorbing region is a union of disjoint, closed sets: $$T^i \in \bigcup_{u \in \...
1
vote
0answers
12 views

Filtering/MCMC methods for this HMM

I have a Discrete HMM with hidden Markovian signals of the form $\{X_t\}_{t \in [0, \infty)} \in \{ 1,2,3\}$ and observed outputs of the form $\{Y_n\}_{n \in \mathbb{N}} \in \{ 1,2\}$. Each ...
0
votes
0answers
31 views

Application Strong Markov Property

I am considering a random walk $S_n$ on a state space $\mathbb{Z}^d$. I want to show that $E_x\left[\sum_{n=0}^{\tau_A-1}{1_{\{S_n=y\}}}\right]=\frac{1}{P_x[\tau_A<\tau_y]}$, where $\tau_A=\inf\{n\...
1
vote
1answer
30 views

Steady state distribution needed

I have a chain $C_t$. At every instant $t$ an exponential random variable $X_t$ with parameter $\lambda$ is added to the chain or if the chain has a value greater than $Q$ then a value $Q$ is ...
1
vote
0answers
40 views

Better to go first in Snakes and Ladders?

We consider the game as described in http://www.datagenetics.com/blog/november12011/ . Each person rolls a dice and the person who gets 6 on the face can start and the other keeps waiting. If the ...