Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

2
votes
1answer
513 views

Is a Bernoulli process a Markov chain?

For a Bernoulli process, the outcome of a future trial is independent of the outcome of past trials. I.e., the future behaviour of a Bernoulli process is independent of its past, i.e. a Bernoulli ...
1
vote
1answer
546 views

Why is this infinite-state-space Markov chain positive recurrent?

Given the following transition matrix for a Markov chain, how can I see that the chain is positive recurrent? I want to convince myself that the chain has a limiting distribution, and the chain is ...
1
vote
4answers
189 views

Why do these expressions tend to zero?

In a chapter on Markov chains, it is claimed that $\binom {2n}{n} p^n(1-p)^n$ (where $p$ is a probability) tends to $0$ as $n$ tends to $\infty$. But why is this so? It is also claimed that for an ...
0
votes
1answer
281 views

Expectation problem in Absorbing Markov Chain(exercise on Grinstead and Snell 11.2 18 )

Hi I encountered this problem. It took me quite long but I could not solve it. The problem is as follows: Assume that a student going to a recently established school in a university has, each year, ...
2
votes
2answers
135 views

How to understand Markov property?

I'm learning stochastic process in college. How to understand Markov property?I'm curious about what is the power and validity of Markov property ? A stochastic process has the Markov property if ...
4
votes
0answers
93 views

A Continuous-Time Markov Process Taking All Possible Values

Let $\mathbb{N}$ be the set of positive integers. For each $n \in \mathbb{N}$, let $X^{(n)}=\{ X^{(n)}(t): t \geq 0 \}$ be a Markov chain with state-space the two point set $\{0,1\}$ and $Q$-matrix ...
1
vote
0answers
85 views

Expected distance traversed between 2 vertices on probabilistic graph

Let $V = \{1,2,3,...,N\}$ be the vertex set of a graph. Let $d(i,j)>=0$ represent the $(i,j)$ vertex distance between vertices $i$ and $j$, $i \in V, j \in V$. Now, define a non-negative number ...
2
votes
1answer
176 views

If $P$ is an invertible transition probability matrix, does $P^{-1}[i,j]$ have any interesting meaning?

Suppose we have a Markov chain transition probability matrix $P$ that is invertible, i.e., $P^{-1}$ exists. Question: Does there exist a meaningful interpretation of the $(i,j)$ entry in $P^{-1}$? ...
1
vote
1answer
119 views

Markov Property as given in Norris' book on Markov chains

In the book, Markov Chains, the following theorem is mentioned: Let $(X_n)_{n≥0}$ be Markov$(λ,P)$. Then, conditional on $X_m = i, (X_{m+n})_{n≥0}$ is Markov$( δ_i,P)$ and is independent of the ...
0
votes
2answers
216 views

Markov Chain Monte Carlo in plain English

I barely know what a markov chain is (I had a terrible teacher) and I probably have an idea of what a stationary distribution is... but I don't know how a Monte Carlo method works and I don't know how ...
2
votes
1answer
728 views

Expected time for winning in biased Gambler's Ruin

Consider the random walk $X_0, X_1, X_2, \ldots$ on state space $S=\{0,1,\ldots,n\}$ with absorbing states $A=\{0,n\}$, and with $P(i,i+1)=p$ and $P(i,i-1)=q$ for all $i \in S \setminus A$, where ...
1
vote
1answer
98 views

What is strictly periodic cycle

In AI book by Norvig and Russell ergodic Markov Chains are defined as follows: Every state is reachable from every other. There are no strictly periodic cycles in it. Can someone explain what is ...
0
votes
1answer
114 views

Markov Chain discarding balls from urn

The following question has me stumped. Any ideas on how to get started? An urn contains $n$ green balls and $n+2$ red balls. A ball is picked at random: if it is green then a red balls is also ...
0
votes
1answer
79 views

Markov chain - clique

Is there a special name (or case) for a finite Markov chain which all states are reachable from any state with positive probability? Does anyone familiar with a problem modeled by this kind of chain?
1
vote
2answers
128 views

expected life absorbing Markov Chain

No idea on how to start this question. Any help would be much appreciated. A flea lives on a polyhedron with N vertices, labelled $1, . . . , N$. It hops from vertex to vertex in the following manner: ...
1
vote
0answers
106 views

Intuitive meaning of spectral radius of a Markov chain transition matrix?

What is the intuitive meaning of the eigenvalues and in particular of the spectral radius of the transition matrix corresponding to a Markov chain?
0
votes
1answer
34 views

Stoppingtimes: Why demand $\mathbb{E}[\tau]<\infty$?

I'm working with a discrete-time Markov Chain $\{Y_j, j \geq 0 \}$ that evolves untill a stoppingtime $\tau$ is reached. $X$ is een stochastic variable which depends on the state of the Markov Chain. ...
2
votes
1answer
156 views

Markov chain transition matrix

if $P$ and $Q$ are $n \times n$ transition matrices for two Markov chain, then product $R=PQ$ is also a transition matrix. is this true ? why is it ? looks like product of transition matrix means ...
1
vote
0answers
64 views

Conditions for Markov chain

Let $\{X_n\}$ be a Markov chain with transition matrix $P$, and $Y_n := X_{m-n}$, $m\ge n$. Under what conditions is $\{Y_n\}_{n\ge 0}$ Markov chain? I stared by proving that conditional probability ...
3
votes
0answers
111 views

SLLN of Markov chains .

Let $X_1$, $X_2$,... be a finite state, irreducible and aperiodic Markov chain with initial state $X_0=i$. It is known that \begin{equation} \mathrm{P}\Big\{\lim_{n\rightarrow\infty} ...
0
votes
1answer
81 views

Markov Model transition probability

Hy, i have a little doubt about a Markov model problem. The problem requests to find a transition probability matrix for a situation with two statistically independent person that can be in 4 ...
1
vote
0answers
82 views

Markov kernels and update functions

I would like to prove, that for any Markov kernel $K$ on a Polish space $(F,\mathcal{F})$ (with a $\sigma$-field) you can find a measurable space $(S,\mathcal{S})$, a random element $Z$ on $S$ and an ...
0
votes
1answer
67 views

Stationary probabilities of markov chain

I am confused in which conditions the stationary probabilities of both discrete and continuous Markov chain donot exist. If it is due to periodic chain then is it for both discrete and continuous. ...
1
vote
0answers
62 views

Gibbs / MCMC sampling for sum of parameters - how to improve slow mixing?

Suppose I have a hierarchical Bayesian model, where my observational prediction, $y'$, is calculated as the sum of other parameters, ${\alpha_i}$. My observation equation (the likelihood) is: $P(y | ...
1
vote
0answers
100 views

Markov chain weak convergence

consider a sequence of Markov chains $\Phi^{(n)}$whose transition kernel $P^{(n)}$ converges to $P$. Now let $\Phi$ be the Markov chain with the limiting kernel $P$. How do I show that $\Phi^{(n)}\to ...
2
votes
1answer
472 views

If $P$ is a regular transition probability matrix then $P^{n^2}$ has no zero element

A transition probability matrix $P\in M_{n\times n}$ is regular if for some $k$ the matrix $P^k$ has all of its elements strictly positive. I read that this can be tested by using the following ...
1
vote
1answer
39 views

Prove that if $a_k=x_{k1}a_{k-1}+x_{k2}a_{k-2}+\cdots+x_{kt}a_{k-t},$ ($x_{ki}$~U(0,b)) then $\dfrac{\log{a_k}}{k}\to^{p} c$

Let $a_1=a_2=\cdots=a_t= 1,a_k=x_{k1}a_{k-1}+x_{k2}a_{k-2}+\cdots+x_{kt}a_{k-t},$ where $x_{ki}$~U(0,b), and $x_{ki}(k>t,i=1,2,\cdots,t)$ are independent each other. Prove that $\exists c\in ...
0
votes
1answer
598 views

Expected state of a Markov chain

Let's start with a slightly trivial Markov chain defined as follows: the beginning state is called $1$ and the set of states is $\mathbb{N}$. At each step, when the current state is $n$, the ...
1
vote
1answer
88 views

Upper Bound of Markov Chain Convergence?

Reading about Markov Chain Monte Carlo in this book on Probability (DeGroot), it says In general, the distribution will get pretty close to the stationary distribution in finite time, but how ...
3
votes
1answer
309 views

Probability of being at a certain point after $N$ steps in Random Walk with a single absorbing barrier

A random walker in $1$ dimension starts walking from a point $k>0$ with an absorbing barrier at point $0$. What is the probability that he will reach a point $m>0$ in $N$ steps? How should I ...
2
votes
1answer
268 views

Markov Chain Ergodic Theorem

Consider a discrete time Markov Chain on countable state space $X_{0},X_{1},\ldots$. Assume that the chain satisfies the Foster Lyapunov criteria, and since it is countable state space chain, we ...
1
vote
0answers
44 views

how to describe this case with markov-chain

I want to describe this case in markov chain: The case: Mr. Meier reads NYTimes everyday and puts the newspaper on news rack. His wife sometimes cleans the house(with prob $1/3$ each day) and throws ...
0
votes
1answer
164 views

Inner product space and Reversible Markov chain

I am trying to clarify the following: Suppose P is the $N\times N$ transition matrix of a finite-dimensional Markov Chain, with invariant distribution given by N-vector $\mu$ (i.e. $\mu^T=\mu^T P$). ...
0
votes
1answer
221 views

Markov Chain with two states

A Markov Chain has two states, $A$ and $B$, and the following probabilities: If it starts at $A$, it stays at $A$ with probability $\frac13$ and moves to $B$ with probability $\frac23$; if it starts ...
0
votes
1answer
72 views

question about the transformation of a Markov process

I have a question about Markov Process: Let $X_t=(X_t^1, X_t^2,..., X_t^n)$ be a Markov process with regard to the filtration $\mathcal{F}_t$, let $Y_t:=\max_{1\leq k\leq n}X_t^k$, then is $Y_t$ a ...
-1
votes
1answer
547 views

What is the Expectations of all 3 ants meeting at same point?

Say we have 3 ants in three corner's of triangle. What is the expectations that all 3 ants meeting together given that the ant moves in any direction. So by just seeing it I figured out that in 2 ...
1
vote
0answers
49 views

Poisson distributed variable after iterative process

The value of $x$ is changed in a stochastic iterative process. Changes of $\pm1$ are possible. I am searching transition probabilities $p(x=n \rightarrow x=n+1)$ and $p(x=n \rightarrow x=n-1)$ that ...
0
votes
1answer
76 views

Expected number of shots for a game to end

A basketball player plays a shooting game. He gets +1 point if he scores a basket and -2 points if he misses. He starts with 0 points. The game ends when the player reaches +10 or -10. What is the ...
1
vote
3answers
643 views

Estimating the transition matrix given the stationary distribution

Let's say we are given a Markov chain for variable $X = [x_1, ..., x_n]$; also we are given a desired stationary distribution for this graph $P_\infty = [p_1, ..., p_n]^\top$. How can we design an ...
1
vote
0answers
652 views

Compute stationary distribution of a general markov chain

I have a stochastic matrix P that represents a markov chain. I know that the markov chain is irreducible and aperiodic and therefore, I know the existence of a unique normalised left eigenvector to ...
2
votes
2answers
134 views

Continuous-time finite-state Markov chain as a subordinated Brownian motion

I think I read somewhere that every semimartingale is representable as a time changed Brownian motion (sorry, I don't have a reference). This suggests that in particular a continuous-time finite-state ...
1
vote
0answers
72 views

Iterative process that leads to Poisson distribution

I want $x$ to be Poisson distributed. I will call occupation probability $p(x=n) =: p(n)$ and the transition probability $p(x=n \rightarrow x=n+1) =: p(n \rightarrow n+1)$ The value of $x$ is ...
1
vote
1answer
383 views

Eigenvector of transition matrix for Markov chain

Why is the only eigenvector of the transition matrix for an irreducible Markov chain with eigenvalue $= 1$ the eigenvector with all ones?
0
votes
0answers
45 views

When random walk Markov matrix of the graph is normal?

I'm considering random walk on undirected graph $G$. At every time step, walker moves to a random neighbour, with all neighbours being equally likely. With adjacency matrix $A$ the random walk Markov ...
1
vote
1answer
324 views

Reverse engineer transition matrix from steady state?

I had this open ended question. Is it possible to reverse engineer a transition matrix from the steady state. Is there a unique solution or could there be many? Basically given $P^{\infty}$ is there ...
0
votes
0answers
81 views

Markov Chain : Montonicity of Sample Mean

Let $\{X_n\}_{n\geq1}$ be an irreducible, ergodic Markov chain with discrete state-space $S$, transition probability matrix $P$ and steady state distribution $\pi = \{\pi_j\}_{j\in S}$. Let $f$ be a ...
1
vote
1answer
81 views

Absorbing state for a collection of random walks

Further to this question; having learned some stuff since I posed it. Consider a collection of random walks $X_i$ which take finite integer values. These evolve as time-inhomogeneous Markov Chains. ...
2
votes
1answer
111 views

Confidence intervals on maximum likelihoods of observed data

I observed 400 episodes of nursing care in a hospital. I tracked the movement of the nurses between 5 rooms $A-E$. The maximum likelihood of them moving from room $i\rightarrow j$ is given by: ...
3
votes
1answer
182 views

recurrence criterion for random-walk like (simple) inhomogeneous Markov chain

This question is to some degree a follow-up of this question. Suppose we have a random-walk like Markov chain, i.e. state space is the set of all integers from $-\infty$ to $\infty$, the transition ...
1
vote
0answers
55 views

$\psi$ irreducibility and ergodicity of Markov Processes

How is Markov chain splitting technique useful for inferring ergodicity of a Markov Chain?Assume that I am working with general state space (uncountable say $R^{N}$ but time is discrete. I want to ...