Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
84 views

Markov Chains Proof using Statistics

Source: This came from "Introduction to probability" by Charles Miller Grinstead, and James Aurie Snell. It was located on page 407 and is Theorem 11.1 in the section 11.1 Introduction. Theorem: The ...
2
votes
0answers
72 views

simple proof of the $L^2$ weak law for discrete-time ergodic Markov processes

Let $\{X_t\}_{t\in\mathbb{Z}}$ be a stationary and ergodic stochastic process with finite second moment. Von Neuman's ergodic theorem implies that the time average $(1/N)\sum_{j=0}^{N-1} X_j$ ...
0
votes
2answers
87 views

Discrete Markov Transition Matrix

Well, I've been reading over the internet but I've been unable to find a straight answer. I've got a transition matrix for a Markov Discrete Chain. I've made the graph and according to my knowledge, ...
2
votes
2answers
74 views

Random walk on one-dimensional lattice - understanding the expression $pe^{i\theta} + qe^{-i\theta}$

I've started reading the book - First Steps in Random Walks and in the very first example in Chapter 1 they talk about a random walk on a one-dimensional lattice. If we consider a particle starting ...
1
vote
1answer
94 views

Why Gibbs sampling needn't “remixing”

I am generating $\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \dots, \mathbf{x}^{(n)}$ using Gibbs sampling methods. So I want $\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \dots, \mathbf{x}^{(n)} \sim$ some ...
2
votes
1answer
54 views

Elementary probability question (Random walks)

Given a random walk $X_{t \ge 0}$ on $\mathbb{Z}$ starting at $0$ with probabilities $P(n, n + 1) = p$ and $P(n, n - 1) = 1 - p$, let $Y = \min\{X_0, X_1 \dots \}$. What is the probability that $Y = ...
0
votes
1answer
218 views

What values of $p$ make this a transient chain?

Suppose we have a Markov chain with state space $S = \{0, 1, 2, \dots \}$ and probabilities $p(x, x + 2) = p$, $p(x, x - 1) = 1-p$ for $x > 0$ and $p(0, 2) = p$ and $p(0, 0) = 1 - p$ I would ...
1
vote
1answer
183 views

How to show that this process is a Markov chain?

This question is from DEGROOT's "Probability and Statistics". Question: Suppose that a coin is tossed repeatedly in such a way that heads and tails are equally likely to appear on any given ...
1
vote
1answer
47 views

makov chains and property

let $X_t$ for $t \in \{1,2,...,10\}$ be a sequence of independent tosses of a fair coin. We denote heads with 1 and tails with 0. Define the random variable $S_n=\sum_{t=1}^n X_t$ for $n \in ...
1
vote
1answer
199 views

Steady State Markov Chain

I was reading http://www.ams.org/bookstore/pspdf/mbk-58-prev.pdf and going through the first example for the frog jumping between the lily pads. I'm interested in find the steady-state probability for ...
0
votes
1answer
93 views

Markov time $ T= \min\{n : X[n] = 1\}$

Let $T$ is a Markov time such that $T= \min \{ n : X[n] = 1\}$ , $X[n]$ is the number of $h$ (heads) in coin tossing for $n$ times. Let's say I will toss the coin 3 times, so the event collection is ...
0
votes
1answer
118 views

Is aperiodicity necessary for a irreducible Markov chain with finite state space to exclude positive probability of infinite hitting time?

I encountered a Lemma: For any irreducible aperiodic Markov chain $(X_0, X_1, \ldots)$with state space $S =\{s_1,\ldots, s_k \}$ and transition matrix $P$, we have for any two states $s_i,s_j \in ...
0
votes
1answer
2k views

Stochastic process that is Martingale but not Markov? [duplicate]

Can you please help me by giving an example of a stochastic process that is Martingale but not Markov process for discrete case?
0
votes
1answer
43 views

Can a stationary distribution be zero vector

Suppose I have probabilities matrix between 3 states, for exampel we can take $P=\left(\begin{array}{ccc} \frac{1}{9} & \frac{8}{9} & 0\\ 0 & 0.3 & 0.7\\ 0 & 0 & 1 ...
1
vote
1answer
91 views

Waiting time in an immigration-birth process

Could someone please verify that none of the four given choices are correct? Isn't the correct answer $$\frac 1{(\lambda + 4\beta)^2} + \frac 1{(\lambda + 5\beta)^2} + ... +\frac 1{(\lambda + ...
1
vote
1answer
83 views

How to Prove by definition, the given process is a Markov Process?

Define the process Xt by X0 = 1, and for t = 1, 2, . . . Xt = { uXt-1, with probability p, { vXt-1, with probability 1-p where 0 < v < 1 ...
1
vote
1answer
133 views

Exponentially-distributed lifetimes (death process)

In a pure death process where the individual death rate is fixed at v, because the process is a time-homogeneous Markov process, the wating time till the next "event" (i.e. the wating time till the ...
1
vote
0answers
83 views

Stationary distribution behavior - Markov chain

I have modeled a process with a Markov chain with K+1 states which is irreducible and apperiodic. The transition matrix is a centrosymmetric matrix where all it's entries has a positive probability. ...
1
vote
1answer
97 views

How is a simple birth process is time-homogeneous?

Why is it that a simple birth process is time-homogeneous? The incidence of a birth in a small time interval depends on the population size at the time of start of the interval. Doesn't this ...
2
votes
1answer
357 views

Is a Bernoulli process a Markov chain?

For a Bernoulli process, the outcome of a future trial is independent of the outcome of past trials. I.e., the future behaviour of a Bernoulli process is independent of its past, i.e. a Bernoulli ...
1
vote
1answer
363 views

Why is this infinite-state-space Markov chain positive recurrent?

Given the following transition matrix for a Markov chain, how can I see that the chain is positive recurrent? I want to convince myself that the chain has a limiting distribution, and the chain is ...
1
vote
4answers
183 views

Why do these expressions tend to zero?

In a chapter on Markov chains, it is claimed that $\binom {2n}{n} p^n(1-p)^n$ (where $p$ is a probability) tends to $0$ as $n$ tends to $\infty$. But why is this so? It is also claimed that for an ...
0
votes
1answer
226 views

Expectation problem in Absorbing Markov Chain(exercise on Grinstead and Snell 11.2 18 )

Hi I encountered this problem. It took me quite long but I could not solve it. The problem is as follows: Assume that a student going to a recently established school in a university has, each year, ...
2
votes
2answers
115 views

How to understand Markov property?

I'm learning stochastic process in college. How to understand Markov property?I'm curious about what is the power and validity of Markov property ? A stochastic process has the Markov property if ...
4
votes
0answers
78 views

A Continuous-Time Markov Process Taking All Possible Values

Let $\mathbb{N}$ be the set of positive integers. For each $n \in \mathbb{N}$, let $X^{(n)}=\{ X^{(n)}(t): t \geq 0 \}$ be a Markov chain with state-space the two point set $\{0,1\}$ and $Q$-matrix ...
1
vote
0answers
77 views

Expected distance traversed between 2 vertices on probabilistic graph

Let $V = \{1,2,3,...,N\}$ be the vertex set of a graph. Let $d(i,j)>=0$ represent the $(i,j)$ vertex distance between vertices $i$ and $j$, $i \in V, j \in V$. Now, define a non-negative number ...
2
votes
1answer
134 views

If $P$ is an invertible transition probability matrix, does $P^{-1}[i,j]$ have any interesting meaning?

Suppose we have a Markov chain transition probability matrix $P$ that is invertible, i.e., $P^{-1}$ exists. Question: Does there exist a meaningful interpretation of the $(i,j)$ entry in $P^{-1}$? ...
1
vote
1answer
108 views

Markov Property as given in Norris' book on Markov chains

In the book, Markov Chains, the following theorem is mentioned: Let $(X_n)_{n≥0}$ be Markov$(λ,P)$. Then, conditional on $X_m = i, (X_{m+n})_{n≥0}$ is Markov$( δ_i,P)$ and is independent of the ...
0
votes
1answer
141 views

Markov Chain Monte Carlo in plain English

I barely know what a markov chain is (I had a terrible teacher) and I probably have an idea of what a stationary distribution is... but I don't know how a Monte Carlo method works and I don't know how ...
2
votes
1answer
451 views

Expected time for winning in biased Gambler's Ruin

Consider the random walk $X_0, X_1, X_2, \ldots$ on state space $S=\{0,1,\ldots,n\}$ with absorbing states $A=\{0,n\}$, and with $P(i,i+1)=p$ and $P(i,i-1)=q$ for all $i \in S \setminus A$, where ...
1
vote
1answer
87 views

What is strictly periodic cycle

In AI book by Norvig and Russell ergodic Markov Chains are defined as follows: Every state is reachable from every other. There are no strictly periodic cycles in it. Can someone explain what is ...
0
votes
1answer
104 views

Markov Chain discarding balls from urn

The following question has me stumped. Any ideas on how to get started? An urn contains $n$ green balls and $n+2$ red balls. A ball is picked at random: if it is green then a red balls is also ...
0
votes
1answer
63 views

Markov chain - clique

Is there a special name (or case) for a finite Markov chain which all states are reachable from any state with positive probability? Does anyone familiar with a problem modeled by this kind of chain?
1
vote
2answers
117 views

expected life absorbing Markov Chain

No idea on how to start this question. Any help would be much appreciated. A flea lives on a polyhedron with N vertices, labelled $1, . . . , N$. It hops from vertex to vertex in the following manner: ...
1
vote
0answers
75 views

Intuitive meaning of spectral radius of a Markov chain transition matrix?

What is the intuitive meaning of the eigenvalues and in particular of the spectral radius of the transition matrix corresponding to a Markov chain?
0
votes
1answer
33 views

Stoppingtimes: Why demand $\mathbb{E}[\tau]<\infty$?

I'm working with a discrete-time Markov Chain $\{Y_j, j \geq 0 \}$ that evolves untill a stoppingtime $\tau$ is reached. $X$ is een stochastic variable which depends on the state of the Markov Chain. ...
1
vote
1answer
111 views

Markov chain transition matrix

if $P$ and $Q$ are $n \times n$ transition matrices for two Markov chain, then product $R=PQ$ is also a transition matrix. is this true ? why is it ? looks like product of transition matrix means ...
1
vote
0answers
53 views

Conditions for Markov chain

Let $\{X_n\}$ be a Markov chain with transition matrix $P$, and $Y_n := X_{m-n}$, $m\ge n$. Under what conditions is $\{Y_n\}_{n\ge 0}$ Markov chain? I stared by proving that conditional probability ...
3
votes
0answers
101 views

SLLN of Markov chains .

Let $X_1$, $X_2$,... be a finite state, irreducible and aperiodic Markov chain with initial state $X_0=i$. It is known that \begin{equation} \mathrm{P}\Big\{\lim_{n\rightarrow\infty} ...
0
votes
1answer
65 views

Markov Model transition probability

Hy, i have a little doubt about a Markov model problem. The problem requests to find a transition probability matrix for a situation with two statistically independent person that can be in 4 ...
1
vote
0answers
79 views

Markov kernels and update functions

I would like to prove, that for any Markov kernel $K$ on a Polish space $(F,\mathcal{F})$ (with a $\sigma$-field) you can find a measurable space $(S,\mathcal{S})$, a random element $Z$ on $S$ and an ...
0
votes
1answer
63 views

Stationary probabilities of markov chain

I am confused in which conditions the stationary probabilities of both discrete and continuous Markov chain donot exist. If it is due to periodic chain then is it for both discrete and continuous. ...
1
vote
0answers
51 views

Gibbs / MCMC sampling for sum of parameters - how to improve slow mixing?

Suppose I have a hierarchical Bayesian model, where my observational prediction, $y'$, is calculated as the sum of other parameters, ${\alpha_i}$. My observation equation (the likelihood) is: $P(y | ...
1
vote
0answers
78 views

Markov chain weak convergence

consider a sequence of Markov chains $\Phi^{(n)}$whose transition kernel $P^{(n)}$ converges to $P$. Now let $\Phi$ be the Markov chain with the limiting kernel $P$. How do I show that $\Phi^{(n)}\to ...
2
votes
1answer
313 views

If $P$ is a regular transition probability matrix then $P^{n^2}$ has no zero element

A transition probability matrix $P\in M_{n\times n}$ is regular if for some $k$ the matrix $P^k$ has all of its elements strictly positive. I read that this can be tested by using the following ...
1
vote
1answer
36 views

Prove that if $a_k=x_{k1}a_{k-1}+x_{k2}a_{k-2}+\cdots+x_{kt}a_{k-t},$ ($x_{ki}$~U(0,b)) then $\dfrac{\log{a_k}}{k}\to^{p} c$

Let $a_1=a_2=\cdots=a_t= 1,a_k=x_{k1}a_{k-1}+x_{k2}a_{k-2}+\cdots+x_{kt}a_{k-t},$ where $x_{ki}$~U(0,b), and $x_{ki}(k>t,i=1,2,\cdots,t)$ are independent each other. Prove that $\exists c\in ...
2
votes
2answers
458 views

Expected state of a Markov chain

Let's start with a slightly trivial Markov chain defined as follows: the beginning state is called $1$ and the set of states is $\mathbb{N}$. At each step, when the current state is $n$, the ...
1
vote
1answer
75 views

Upper Bound of Markov Chain Convergence?

Reading about Markov Chain Monte Carlo in this book on Probability (DeGroot), it says In general, the distribution will get pretty close to the stationary distribution in finite time, but how ...
1
vote
1answer
197 views

Probability of being at a certain point after $N$ steps in Random Walk with a single absorbing barrier

A random walker in $1$ dimension starts walking from a point $k>0$ with an absorbing barrier at point $0$. What is the probability that he will reach a point $m>0$ in $N$ steps? How should I ...
2
votes
1answer
207 views

Markov Chain Ergodic Theorem

Consider a discrete time Markov Chain on countable state space $X_{0},X_{1},\ldots$. Assume that the chain satisfies the Foster Lyapunov criteria, and since it is countable state space chain, we ...