Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
1answer
280 views

Most likely path through a continuous time Markov chain

If I have a discrete Markov chain, it's easy to find the most likely path through it: just look at the probabilities of following each possible path independently, and take the largest one. In a ...
0
votes
1answer
108 views

Decision theory question about selling a house

I have a real world problem and I was wondering if you guys have any nice insight on the best way to solve it mathematically. I'm not sure there is a decisive solution, but it would be nice to have a ...
0
votes
1answer
135 views

Need help in understanding state transition diagram of a convolutional coder. How are the output bits calculated?

Have a look at the above figure. I am confused in how the output bits are calculated. e.g. according to my understanding a state transition from 00 to 10 (with input bit 1) should produce output 10 ...
0
votes
0answers
159 views

Markov chains, limiting distribution and periodicity

My textbook on Markov chains has theorems on when a chain has a unique limiting distribution, but not the other way around, i.e. when a chain does not have a limiting distribution. My question is the ...
0
votes
1answer
66 views

Stochastic matrix with structure

Let $P \in [0,1]^{(n \times n)}$ be a stochastic matrix i.e $P_{ij} > 0 ~ \forall i,j$ and $\sum_{j}P_{ij} = 1~ \forall i$. Now let us impose additional structure on $P$ by saying that $P_{ij} + ...
2
votes
1answer
122 views

Markov Chains Proof using Statistics

Source: This came from "Introduction to probability" by Charles Miller Grinstead, and James Aurie Snell. It was located on page 407 and is Theorem 11.1 in the section 11.1 Introduction. Theorem: The ...
3
votes
0answers
77 views

simple proof of the $L^2$ weak law for discrete-time ergodic Markov processes

Let $\{X_t\}_{t\in\mathbb{Z}}$ be a stationary and ergodic stochastic process with finite second moment. Von Neuman's ergodic theorem implies that the time average $(1/N)\sum_{j=0}^{N-1} X_j$ ...
0
votes
2answers
103 views

Discrete Markov Transition Matrix

Well, I've been reading over the internet but I've been unable to find a straight answer. I've got a transition matrix for a Markov Discrete Chain. I've made the graph and according to my knowledge, ...
3
votes
2answers
86 views

Random walk on one-dimensional lattice - understanding the expression $pe^{i\theta} + qe^{-i\theta}$

I've started reading the book - First Steps in Random Walks and in the very first example in Chapter 1 they talk about a random walk on a one-dimensional lattice. If we consider a particle starting ...
1
vote
1answer
108 views

Why Gibbs sampling needn't “remixing”

I am generating $\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \dots, \mathbf{x}^{(n)}$ using Gibbs sampling methods. So I want $\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \dots, \mathbf{x}^{(n)} \sim$ some ...
2
votes
1answer
59 views

Elementary probability question (Random walks)

Given a random walk $X_{t \ge 0}$ on $\mathbb{Z}$ starting at $0$ with probabilities $P(n, n + 1) = p$ and $P(n, n - 1) = 1 - p$, let $Y = \min\{X_0, X_1 \dots \}$. What is the probability that $Y = ...
1
vote
1answer
238 views

What values of $p$ make this a transient chain?

Suppose we have a Markov chain with state space $S = \{0, 1, 2, \dots \}$ and probabilities $p(x, x + 2) = p$, $p(x, x - 1) = 1-p$ for $x > 0$ and $p(0, 2) = p$ and $p(0, 0) = 1 - p$ I would ...
1
vote
1answer
268 views

How to show that this process is a Markov chain?

This question is from DEGROOT's "Probability and Statistics". Question: Suppose that a coin is tossed repeatedly in such a way that heads and tails are equally likely to appear on any given ...
1
vote
1answer
53 views

makov chains and property

let $X_t$ for $t \in \{1,2,...,10\}$ be a sequence of independent tosses of a fair coin. We denote heads with 1 and tails with 0. Define the random variable $S_n=\sum_{t=1}^n X_t$ for $n \in ...
1
vote
1answer
368 views

Steady State Markov Chain

I was reading http://www.ams.org/bookstore/pspdf/mbk-58-prev.pdf and going through the first example for the frog jumping between the lily pads. I'm interested in find the steady-state probability for ...
0
votes
1answer
112 views

Markov time $ T= \min\{n : X[n] = 1\}$

Let $T$ is a Markov time such that $T= \min \{ n : X[n] = 1\}$ , $X[n]$ is the number of $h$ (heads) in coin tossing for $n$ times. Let's say I will toss the coin 3 times, so the event collection is ...
0
votes
1answer
141 views

Is aperiodicity necessary for a irreducible Markov chain with finite state space to exclude positive probability of infinite hitting time?

I encountered a Lemma: For any irreducible aperiodic Markov chain $(X_0, X_1, \ldots)$with state space $S =\{s_1,\ldots, s_k \}$ and transition matrix $P$, we have for any two states $s_i,s_j \in ...
1
vote
1answer
4k views

Stochastic process that is Martingale but not Markov? [duplicate]

Can you please help me by giving an example of a stochastic process that is Martingale but not Markov process for discrete case?
0
votes
1answer
49 views

Can a stationary distribution be zero vector

Suppose I have probabilities matrix between 3 states, for exampel we can take $P=\left(\begin{array}{ccc} \frac{1}{9} & \frac{8}{9} & 0\\ 0 & 0.3 & 0.7\\ 0 & 0 & 1 ...
1
vote
1answer
140 views

Waiting time in an immigration-birth process

Could someone please verify that none of the four given choices are correct? Isn't the correct answer $$\frac 1{(\lambda + 4\beta)^2} + \frac 1{(\lambda + 5\beta)^2} + ... +\frac 1{(\lambda + ...
1
vote
1answer
91 views

How to Prove by definition, the given process is a Markov Process?

Define the process Xt by X0 = 1, and for t = 1, 2, . . . Xt = { uXt-1, with probability p, { vXt-1, with probability 1-p where 0 < v < 1 ...
1
vote
1answer
190 views

Exponentially-distributed lifetimes (death process)

In a pure death process where the individual death rate is fixed at v, because the process is a time-homogeneous Markov process, the wating time till the next "event" (i.e. the wating time till the ...
1
vote
0answers
94 views

Stationary distribution behavior - Markov chain

I have modeled a process with a Markov chain with K+1 states which is irreducible and apperiodic. The transition matrix is a centrosymmetric matrix where all it's entries has a positive probability. ...
1
vote
1answer
122 views

How is a simple birth process is time-homogeneous?

Why is it that a simple birth process is time-homogeneous? The incidence of a birth in a small time interval depends on the population size at the time of start of the interval. Doesn't this ...
2
votes
1answer
547 views

Is a Bernoulli process a Markov chain?

For a Bernoulli process, the outcome of a future trial is independent of the outcome of past trials. I.e., the future behaviour of a Bernoulli process is independent of its past, i.e. a Bernoulli ...
1
vote
1answer
547 views

Why is this infinite-state-space Markov chain positive recurrent?

Given the following transition matrix for a Markov chain, how can I see that the chain is positive recurrent? I want to convince myself that the chain has a limiting distribution, and the chain is ...
1
vote
4answers
190 views

Why do these expressions tend to zero?

In a chapter on Markov chains, it is claimed that $\binom {2n}{n} p^n(1-p)^n$ (where $p$ is a probability) tends to $0$ as $n$ tends to $\infty$. But why is this so? It is also claimed that for an ...
0
votes
1answer
286 views

Expectation problem in Absorbing Markov Chain(exercise on Grinstead and Snell 11.2 18 )

Hi I encountered this problem. It took me quite long but I could not solve it. The problem is as follows: Assume that a student going to a recently established school in a university has, each year, ...
2
votes
2answers
140 views

How to understand Markov property?

I'm learning stochastic process in college. How to understand Markov property?I'm curious about what is the power and validity of Markov property ? A stochastic process has the Markov property if ...
4
votes
0answers
95 views

A Continuous-Time Markov Process Taking All Possible Values

Let $\mathbb{N}$ be the set of positive integers. For each $n \in \mathbb{N}$, let $X^{(n)}=\{ X^{(n)}(t): t \geq 0 \}$ be a Markov chain with state-space the two point set $\{0,1\}$ and $Q$-matrix ...
1
vote
0answers
85 views

Expected distance traversed between 2 vertices on probabilistic graph

Let $V = \{1,2,3,...,N\}$ be the vertex set of a graph. Let $d(i,j)>=0$ represent the $(i,j)$ vertex distance between vertices $i$ and $j$, $i \in V, j \in V$. Now, define a non-negative number ...
2
votes
1answer
177 views

If $P$ is an invertible transition probability matrix, does $P^{-1}[i,j]$ have any interesting meaning?

Suppose we have a Markov chain transition probability matrix $P$ that is invertible, i.e., $P^{-1}$ exists. Question: Does there exist a meaningful interpretation of the $(i,j)$ entry in $P^{-1}$? ...
1
vote
1answer
119 views

Markov Property as given in Norris' book on Markov chains

In the book, Markov Chains, the following theorem is mentioned: Let $(X_n)_{n≥0}$ be Markov$(λ,P)$. Then, conditional on $X_m = i, (X_{m+n})_{n≥0}$ is Markov$( δ_i,P)$ and is independent of the ...
0
votes
2answers
225 views

Markov Chain Monte Carlo in plain English

I barely know what a markov chain is (I had a terrible teacher) and I probably have an idea of what a stationary distribution is... but I don't know how a Monte Carlo method works and I don't know how ...
2
votes
1answer
745 views

Expected time for winning in biased Gambler's Ruin

Consider the random walk $X_0, X_1, X_2, \ldots$ on state space $S=\{0,1,\ldots,n\}$ with absorbing states $A=\{0,n\}$, and with $P(i,i+1)=p$ and $P(i,i-1)=q$ for all $i \in S \setminus A$, where ...
1
vote
1answer
100 views

What is strictly periodic cycle

In AI book by Norvig and Russell ergodic Markov Chains are defined as follows: Every state is reachable from every other. There are no strictly periodic cycles in it. Can someone explain what is ...
0
votes
1answer
114 views

Markov Chain discarding balls from urn

The following question has me stumped. Any ideas on how to get started? An urn contains $n$ green balls and $n+2$ red balls. A ball is picked at random: if it is green then a red balls is also ...
0
votes
1answer
79 views

Markov chain - clique

Is there a special name (or case) for a finite Markov chain which all states are reachable from any state with positive probability? Does anyone familiar with a problem modeled by this kind of chain?
1
vote
2answers
128 views

expected life absorbing Markov Chain

No idea on how to start this question. Any help would be much appreciated. A flea lives on a polyhedron with N vertices, labelled $1, . . . , N$. It hops from vertex to vertex in the following manner: ...
1
vote
0answers
112 views

Intuitive meaning of spectral radius of a Markov chain transition matrix?

What is the intuitive meaning of the eigenvalues and in particular of the spectral radius of the transition matrix corresponding to a Markov chain?
0
votes
1answer
34 views

Stoppingtimes: Why demand $\mathbb{E}[\tau]<\infty$?

I'm working with a discrete-time Markov Chain $\{Y_j, j \geq 0 \}$ that evolves untill a stoppingtime $\tau$ is reached. $X$ is een stochastic variable which depends on the state of the Markov Chain. ...
2
votes
1answer
161 views

Markov chain transition matrix

if $P$ and $Q$ are $n \times n$ transition matrices for two Markov chain, then product $R=PQ$ is also a transition matrix. is this true ? why is it ? looks like product of transition matrix means ...
1
vote
0answers
64 views

Conditions for Markov chain

Let $\{X_n\}$ be a Markov chain with transition matrix $P$, and $Y_n := X_{m-n}$, $m\ge n$. Under what conditions is $\{Y_n\}_{n\ge 0}$ Markov chain? I stared by proving that conditional probability ...
3
votes
0answers
111 views

SLLN of Markov chains .

Let $X_1$, $X_2$,... be a finite state, irreducible and aperiodic Markov chain with initial state $X_0=i$. It is known that \begin{equation} \mathrm{P}\Big\{\lim_{n\rightarrow\infty} ...
0
votes
1answer
81 views

Markov Model transition probability

Hy, i have a little doubt about a Markov model problem. The problem requests to find a transition probability matrix for a situation with two statistically independent person that can be in 4 ...
1
vote
0answers
83 views

Markov kernels and update functions

I would like to prove, that for any Markov kernel $K$ on a Polish space $(F,\mathcal{F})$ (with a $\sigma$-field) you can find a measurable space $(S,\mathcal{S})$, a random element $Z$ on $S$ and an ...
0
votes
1answer
67 views

Stationary probabilities of markov chain

I am confused in which conditions the stationary probabilities of both discrete and continuous Markov chain donot exist. If it is due to periodic chain then is it for both discrete and continuous. ...
1
vote
0answers
64 views

Gibbs / MCMC sampling for sum of parameters - how to improve slow mixing?

Suppose I have a hierarchical Bayesian model, where my observational prediction, $y'$, is calculated as the sum of other parameters, ${\alpha_i}$. My observation equation (the likelihood) is: $P(y | ...
2
votes
0answers
107 views

Markov chain weak convergence

consider a sequence of Markov chains $\Phi^{(n)}$whose transition kernel $P^{(n)}$ converges to $P$. Now let $\Phi$ be the Markov chain with the limiting kernel $P$. How do I show that $\Phi^{(n)}\to ...
3
votes
1answer
485 views

If $P$ is a regular transition probability matrix then $P^{n^2}$ has no zero element

A transition probability matrix $P\in M_{n\times n}$ is regular if for some $k$ the matrix $P^k$ has all of its elements strictly positive. I read that this can be tested by using the following ...