Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

2
votes
2answers
290 views

What does it mean for MCMC to converge?

I know that a Markov Chain is a discrete random process where the current state decides the next and in a random walk, the probability that we move from node u to v is 1/N(u). An MCMC sample will ...
2
votes
2answers
291 views

Calculating probabilities (Markov Chain)

Let $\mathcal{X}=(X_n:n\in\mathbb{N}_0)$ denote a Markov chain with state space $E=\{1,\dots,5\}$ and transition matrix ...
2
votes
1answer
81 views

Irreducible MCs

Why is it that theorems for (discrete) Markov chains always require that the MC concerned is irreducible? Can problems with reducible MCs can be simplified to considering the irreducible components? ...
2
votes
3answers
679 views

How to show Martingale property for sum of $S_k-E(S_k)$-summands where $S_k$ is a function of two RV's

EDIT: new formulation of the question (old version below). In a paper I found the statement that a certain sum $M_n =Y_1+\dotsb Y_n$ is a martingale, $Y_i=f (X_k, Z_k) - E ( f(X_k, Z_k) | X_k)$. (The ...
2
votes
2answers
25 views

Probability of returning to a given state after n transitions-Markov chains

Let us denote $f_j^{(n)}$ denote the probability of the first return to state $j $after n transitions. Let $p_{jj}^{(n)}$ be the probability of returning to the state $j$ after $n$ transitions when ...
2
votes
1answer
24 views

Applying transition matrix to a probability vector seems to ruin its normalization

I had a little bit about stochastic processes during my "Statistical Physics" course and on my exam I got a problem with a Markov chain. My solution seems to be without computational mistakes (checked ...
2
votes
1answer
30 views

Markov chains by hand

If I have a starting point: $A_T=[0,1]$ at $T=1$ and a one step transition matrix of: $B=\left[ \begin{align} &\frac34 & \frac14& \\& \frac1{20}& \frac {19}{20} &\end{align} ...
2
votes
1answer
95 views

Inverse of a regular stochastic matrix

Is it true that the inverse of a regular stochastic matrix is also regular? Are there any other interesting features that the inverse may have of a regular stochastic matrix? Hope someone could answer ...
2
votes
1answer
69 views

time-homogeneous continuous time Markov chain

I have a question about the continuous time Markov chain. In the Poisson process we have independent and stationary increments. Do we have this in a continuous time Markov chain that is ...
2
votes
1answer
39 views

Is the following Markov Chain a martingale?

Say I have a finite, ergodic Markov chain with states ${0,1,2,3}$ and with the following transition matrix: $$\begin{bmatrix} \frac{7}{10} & \frac{3}{10} & 0 &0\\ \frac{1}{10} & ...
2
votes
1answer
57 views

Probability distribution of Poisson process

Let $X_t$ and $Y_t$ be two independent Poisson process with rate parameter $\lambda_1$ and $\lambda_2$, respectively, measuring the number of customers arriving in stores $1$ and $2$, respectively. ...
2
votes
2answers
48 views

Generate random sample with three-state Markov chain

I have a Markov chain with the transition matrix $$\pmatrix{0 & 0.7 & 0.3 \\ 0.8 & 0 & 0.2 \\ 0.6 & 0.4 & 0}$$ and I would like to generate a random sequence between the three ...
2
votes
1answer
203 views

Markov chains: is “aperiodic + irreducible” equivalent to “regular”?

I have two books on stochastic processes. In one book, it says that the limiting matrix is possible to find if the matrix is regular, that is, if for some $n$ $P^n$ has only positive values. The ...
2
votes
1answer
84 views

Probability problem with markov property

Problem: In a test paper, the questions are arranged so that 3/4's of the time a True is followed by a True and 2/3's of the time a False is followed by a False. You are confronted with a 100 ...
2
votes
1answer
136 views

Proof that Markov Chains converges to the stationary distribution

Let $P$ is a transition matrix of a Markov Chain, which is irreducible, aperiodic and lets assume $\pi$ is its stationary distribution: $\pi = \pi P$. Does anyone knows the proof for the following ...
2
votes
1answer
396 views

Expected time for winning in biased Gambler's Ruin

Consider the random walk $X_0, X_1, X_2, \ldots$ on state space $S=\{0,1,\ldots,n\}$ with absorbing states $A=\{0,n\}$, and with $P(i,i+1)=p$ and $P(i,i-1)=q$ for all $i \in S \setminus A$, where ...
2
votes
2answers
99 views

Continuous-time finite-state Markov chain as a subordinated Brownian motion

I think I read somewhere that every semimartingale is representable as a time changed Brownian motion (sorry, I don't have a reference). This suggests that in particular a continuous-time finite-state ...
2
votes
3answers
108 views

why is this Markov Chain aperiodic

I have this Matrix: $$P=\begin{pmatrix} 0 & 1 \\ 0.3 & 0.7 \end{pmatrix}$$ this markov chain is said to be aperiodic, I dont understand how it comes to it. Period $\delta$ is the gcd of ...
2
votes
1answer
155 views

Construction of positive recurrent Markov chain

Let $\{X_i\}_{i\geq 1}$ be i.i.d. with values in $\mathbb N_0$. Define a Markov chain via the following transition matrix: $$p(0,n) = \mathbb P(X_1 = n-1) \qquad p(m,n) = \mathbb P\left(\sum_{k=1}^m ...
2
votes
2answers
313 views

Finding the exact stationary distribution for a biased random walk on a bounded interval

Imagine we have a biased random walk on an interval $[0, L]$, where the probability of taking a $+1$ step is $p$ and the probability of taking a $-1$ step is $(1-p)$. At the reflecting boundary $0$, ...
2
votes
1answer
182 views

Random walk with 3 possible steps

I have i.i.d. random variables with following distribution: $$ P(\xi_i =1) = p_1, \ P(\xi_i = 0) = p_0, \ P(\xi_i = -1) = p_{-1}; \quad S_n = \sum^n_{i=1}\xi_i.$$ I am interested in probability of ...
2
votes
1answer
134 views

Markov Property

this relates to an unanswered question I posted a few days ago: Let $\{ X_t : t = 1, 2, 3 \dots \}$ follow a 2-state Markov chain with transition matrix P. Does the Markov property mean I can break ...
2
votes
1answer
82 views

How does this Markov process involving balls and bins behave?

I have some set $S_1,\ldots,S_k$ ($k \geq 3$) of bins, each initially with $N_0(S_i)$ balls ($N_t(S_i)$ denotes the number of balls in $S_i$ at time $t$). A bin can contain a negative number of balls. ...
2
votes
1answer
102 views

Expected number of jumps in regular jump HMC

Consider a homogeneous Markov Chain $X$ on a countable state space, ie a jump process. It is said to be regular (does not explode) if there are only a finite number of jumps in every finite interval. ...
2
votes
1answer
469 views

Understanding a Markov Chain

I am using a Markov Chain to get the 10 best search results from the union of 3 different search engines. The top 10 results are taken from each engine to form a set of 30 results. The chain starts ...
2
votes
2answers
188 views

Markov chain basic positive recurrency question

If a discrete markov chain is stationary (as far as I know: doesn't modify itself with time), irreducible (doesn't have transient states) and aperiodic (no periodic states), is it positive recurrent? ...
2
votes
1answer
438 views

Expected time of mouse's survival (stochastic matrix)

In the following wikipedia page explaining stochastic matrices, there is an example with 5 boxes and a cat and a mouse where they jump to a left or right box at every turn and it explains how to ...
2
votes
2answers
619 views

Proof of Markov property for Ehrenfest urn

[the question got downvoted on MO with the recommendation to ask here] In many books Ehrenfest Urn is used as an example of a homogeneous Markov chain, where entries in transition probabilities ...
2
votes
2answers
409 views

Markov Chain: Pensioner Problem

A pensioner receives 2000 dollars at the beginning of each month. The amount of money he needs to spend during a month is independent of the amount he has and is equal to i (i.e. i thousand dollars) ...
2
votes
1answer
78 views

Invariance on the product space

Let us consider two spaces $\mathbb{X},\mathbb{Y}$. For simplicity we put $\mathbb{X} = \mathbb{Y} = \mathbb{R}$. On the product space $\mathbb{S} = \mathbb{X}\times \mathbb{Y}$ we consider a Markov ...
2
votes
1answer
44 views

Why left multiplication when it comes to Markov chains?

When working with Markov chains and transition matrices $P$ we multiply from the left, meaning that for example $\mu^{(n)} = \mu^{(0)}P^n$ or that the stationary distribution satisfies $\pi = \pi P$. ...
2
votes
1answer
43 views

probability that a game finishes at $n$th step

A coin is flipped sequentially. The game finishes when the sequence TTH is formed(player X wins) or the sequence HTT is formed(player Y wins). I can find the expected time until absorption by X or Y ...
2
votes
1answer
64 views

Safe small wins vs. risky large wins at roulette

Short statement of problem : Two players play roulette at a casino. They both start with the same initial amount. Each player always plays his favorite bet each time, and stops playing as soon as he ...
2
votes
1answer
82 views

What does a customer see when it begins to be served in $M/M/1$ queue?

In queueing theory, the PASTA (Poisson Arrivals See Time Averages) principle [wiki] justifies $a_n = P_n$ where $$a_n = \text{proportion of customers that find } n \text{ customers in the system when ...
2
votes
1answer
30 views

How to get transition matrix of markov process?

I am monitoring a Markov process with ~21 states. I know all the states, initial state and what states transitions can/cannot be, so that zero elements of transition matrix are known. I know the ...
2
votes
1answer
64 views

continuous time Markov chain, something the book does not explain

I have a problem with something in my book, under the chapter of continuous time Markov chains. I will post a link to what the book does. They do something which they seem to take for granted, but I ...
2
votes
2answers
205 views

Mean recurrence time and stationary distribution of a Markov chain?

In a Markov chain is there a theorem relating the existence of the stationary distribution and the mean recurrence time? E.g. impossible for stationary distribution to exist therefore mean recurrence ...
2
votes
1answer
50 views

Coarser cyclic decomposition of Markov chain

For a irreducible Markov chain with period $d$ there is a standard construction which shows that the state space can be partitioned into $d$ sets $C_1, \ldots, C_d$ such that $P(x,y)>0$ only if $x ...
2
votes
1answer
64 views

Can I invert a single row of a very large sparse matrix?

Problem I research electron behavior in organic solar cells and have found a way to recast this problem in terms of a large (n=~60 million) Absorbing Markov Chain that I represent as a sparse matrix. ...
2
votes
2answers
84 views

Expected value of a series of random variables in a markov chain

I have a Markov Chain such that $X_n = \max(X_{n-1}+\xi _n,0)$ where the $\xi_n$ series is independent and identically distributed. I want to show that if $\mathbb E(\xi_n) > 0$ (where $\mathbb ...
2
votes
1answer
93 views

What is the difference between positive presistent and null persistent state in a Markov Chain?

I'm not looking for the difference in the mathematical definition, but rather for an intuitive explanation of their differences and possible examples, so that I can have them in my head when ...
2
votes
1answer
54 views

Elementary probability question (Random walks)

Given a random walk $X_{t \ge 0}$ on $\mathbb{Z}$ starting at $0$ with probabilities $P(n, n + 1) = p$ and $P(n, n - 1) = 1 - p$, let $Y = \min\{X_0, X_1 \dots \}$. What is the probability that $Y = ...
2
votes
1answer
126 views

If $P$ is an invertible transition probability matrix, does $P^{-1}[i,j]$ have any interesting meaning?

Suppose we have a Markov chain transition probability matrix $P$ that is invertible, i.e., $P^{-1}$ exists. Question: Does there exist a meaningful interpretation of the $(i,j)$ entry in $P^{-1}$? ...
2
votes
1answer
183 views

Markov Chain Ergodic Theorem

Consider a discrete time Markov Chain on countable state space $X_{0},X_{1},\ldots$. Assume that the chain satisfies the Foster Lyapunov criteria, and since it is countable state space chain, we ...
2
votes
1answer
102 views

Confidence intervals on maximum likelihoods of observed data

I observed 400 episodes of nursing care in a hospital. I tracked the movement of the nurses between 5 rooms $A-E$. The maximum likelihood of them moving from room $i\rightarrow j$ is given by: ...
2
votes
1answer
225 views

Two different ways of constructing a continuous time Markov chain from discrete time one

Consider a homogeneous continuous time Markov chain (CTMC) with Markov transition function $p(t)$ and infintesimal generator $G$. Its embeded discrete time Markov chain (DTMC) has its transition ...
2
votes
2answers
199 views

$(X_n)$ an irreducible transient Markov chain. Is $f(x) = \mathbb{P}(X_n = x_0 \text{ for some } n > 0 | X_0=x)$ constant?

Let $(X_n)_{n=0}^{\infty}$ be an irreducible transient Markov chain with countably infinite state space $E$. Let $T_x = \inf\{n > 0 : X_n = x\}$. Let $\mathbb{P}_x$ be probability conditioned on ...
2
votes
3answers
423 views

DTMC : Example of Irreducible Aperiodic Null Recurrent Chain

Can someone give me an example of a Discrete Time Markov Chain (DTMC) which is Irreducible Aperiodic Null Recurrent I know that a Simple Symmetric Random Walk on Integers is Irreducible ...
2
votes
2answers
61 views

Convergence of an Ergodic process

I'm having trouble working through the math of the problem below. I believe the problem as described is an ergodic process. I've written a simple simulation of the problem, that converges to 66.6...6% ...
2
votes
1answer
269 views

Markov Chains - How to calculate prob. a state is visited at least N times? what about Expectation?

In Markov chains, if I was given a transition probability matrix with each of the probabilities specified, then how do I determine the following: 1- Probability that state y is visited at least n ...