Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
207 views

Markov chains - classify states and find stationary distribution

Consider the Markov Chain with the matrix $$ \begin{vmatrix} 0 & 0 & 1 & 0 & 0 \\ 0 & {\frac2 3} & 0 & {\frac 1 3 } & 0 \\ 1 & 0 & 0& 0& 0 \\ 0 & ...
1
vote
0answers
61 views

Recurrence criterion for a specific Markov chain

Let $(X_n)$ be a Markov chain on $\mathbb N_0$ defined by $(\alpha \geq 0)$ $$ p(0,1) = 1 \\ p(x,x+1) = 1-\frac{1}{(1+x)^\alpha} \\ p(x,0)= \frac{1}{(1+x)^\alpha}$$ Define the shifted moments for ...
1
vote
0answers
85 views

Finding probability from a markov chain

If I have a markov chain transition matrix for 2 states. Specifically in my case, it is a transition matrix for a bacterial genome with 4 random variables being A,C,G and T. (The bases) If I want to ...
1
vote
0answers
80 views

How to prove ergodic property from aperiodicity and positive recurrence

How to prove that in case of an irreducible, aperiodic and positive recurrent Markov Chain time average along sample paths is equal to the ensemble average ? i.e. $$\lim_{n\to \infty ...
1
vote
0answers
111 views

A question about discrete and continuous-time Markov Chains

I have a test tomorrow about Stochastics Process and I couldn't solve the following questions: A gambler starts with 500\$ and plays till he runs out of money. In each round the probability to win ...
1
vote
0answers
51 views

Efficient random number generation for sojourn times in semi-Markov processes

I'm doing a self-study of semi-Markov processes and was wondering if there are efficient methods for generating random numbers for sojourn times. For example, generating a bunch of random numbers from ...
1
vote
0answers
28 views

Multi-variate monotonic function

This is a question continued from here... Proof about Steady-State distribution of a Markov chain I have a stochastic matrix $P_\delta$ of dimensions $n\times n$. I look at the matrix as a transition ...
1
vote
0answers
127 views

how to calculate limit of P(Xn = j | X0 = i) in markov chain?

1. in http://robotics.eecs.berkeley.edu/~wlr/126/w12.htm lim N ® ¥ [1{X1 = j} + 1{X1 = j} + … + 1{XN = j}]/N = 0 What is N? how it limit to zero? and what do 1 in 1{X1 = j} represent? /N must be ...
1
vote
0answers
133 views

a problem on DTMC

For a Markov chain $\{X_n, n\ge0\}$ with transition probabilities $P_{i,j}$, consider the conditional probability that $X_n = m$ given that the chain started at time $0$ in state $i$ and has not ...
1
vote
0answers
71 views

A question on irreducible markov chains

I have a question on irreducible Markov Chains that has been bugging me for a few hours now I have the markov chain defined by: $P(i, i-1) = 1 - P(i,i+1) = \frac{1}{2(i+1)}$ for $i>=1$, and $p(0,1) ...
1
vote
0answers
28 views

immigration from one to other group

I'm studying Morkov chains and I have a question about immigration process. Let's say I have two groups $X$ and $Y$ each individual of these groups give birth with the same rate $b$ and members of $X$ ...
1
vote
0answers
66 views

Solve Variable Order Markov Chain(VOMC)

I am using VOMC in order to implement a real-time system. The Loop of the system is the following: LOOP{ 1)Get Input and Train VOMC according to it 2)Get output from Markov Chain } With the above ...
1
vote
0answers
34 views

Books about Markov Models

I am looking about books on Markov chains, with recent findings such as autoregressive HMM, HMM with inputs, multiple HMM connected together. Is there anything I can look at?
1
vote
0answers
60 views

An Iterated function system with probabilities and overlapping supports of its invariant measures

Let $(X, \rho)$ be a Polish space. Consider an Iterated Function System $(S_i,p_i)_{i=1,...,N}$, where $S_i:X\rightarrow X$, $p_i: X\rightarrow \left[0,1\right]$ are continuous functions and ...
1
vote
0answers
112 views

Random Walk on $N\times N$ grid

I would appreciate any help (answers, pointers to the literature etc.) on the following problem. Consider a (discrete time) random walk on an N-by-N grid which has two absorbing nodes, namely $(1,1)$ ...
1
vote
0answers
47 views

Markov chain supplementary litterature

I'm studying markov chains through Durrett and I'm finding it quite hard to read. Does anyone have a good idea to supplementary book, preferably one on his level of generality and which studies some ...
1
vote
0answers
45 views

How to sample the walk which visits each vertex of a graph specific number of times?

Is there any MCMC mathod that allow me to uniformly sample from all feasible walks where the following restrictions apply: ...
1
vote
0answers
67 views

Positive recurrence of a continuous-time jump process, from its jump chain

I am looking at an irreducible, continuous-time jump process $(X_t)_{t\geq0}$ with the following jump times. Let a Poisson process $(T_i)_{i=1}^\infty$ determine the event times. With probability ...
1
vote
0answers
153 views

Continuous-time Markov process - need help with flux and discrete-time equivalent

I have a continuous-time markov process and I need to calculate the following transition frequencies matrix (aka intensity matrix) transition probabilities all parameters which define permanence ...
1
vote
0answers
88 views

Quasi-stationary distribution of a state in a birth-and-death MC

I need to find an expression for the first state in an MC with transition matrix $P$ with tridiagonal entries. The state space is $U={1,2,..n}$ with the last state being absorbing. Expressions for ...
1
vote
0answers
52 views

Green kernel of Hunt processes

Let $X$ be a regular Hunt process on $R^+$ with starting at $x$ and $T_y :=\inf\{t>0: X_t=y\}, y\in R^+$. $G_q(\cdot,\cdot), q >0$ denotes Green kernel of the process $X$. We have the following ...
1
vote
0answers
161 views

Markov Chains - Using Gibbs & Metropolis algorithm.

Suppose $f_x,_y$ is bivariate normal distribution. I was given the parameters $(μ_1, μ_2, σ_1^2, σ_2^2)$ and $ρ=0.95$ the correlation coefficient. I want to generate $(x_1,y_1), ...
1
vote
0answers
26 views

How many observations is the minimum?

I want to estimate model transition matrix for a process (Markov chain). How much observiations of state do I need? I would prefer this as a function dependent on $n$, where $n$ is number of possible ...
1
vote
0answers
38 views

Integrating with respect to an MCMC Kernel

I'm currently trying to evaluate an integral with respect to an MCMC kernel, and I wanted someone else to read what I `proved' to see if it's really correct. It just doesn't feel right. Suppose I ...
1
vote
0answers
130 views

Cesaro mixing time: Show $t_m(2^{-k}) \le k t_m(1/4), k \ge 1$

Let $(X_t)_{t \ge 0}$ be a finite Markov chain with state space $\Omega$, transition matrix $P$ and stationary distribution $\pi$. Let $\| \cdot \|$ denote the total variation distance and define  ...
1
vote
0answers
107 views

Irreducible Markov: harmonic function based on stationary distribution

Let $P$ be the transition matrix of an irreducible Markov chain on a finite state space $\Omega$. Let $\pi_1$ and $\pi_2$ be two stationary distributions for $P$. Is the function $$h(x)={\pi_1(x) ...
1
vote
0answers
119 views

Expectation of an event

Let $S[4]$ be a binary array with elements of $S$ are taken uniformly and independently from $\{0,1\}$. Also take $k$ uniformly from $\{0,1\}$. Take $i=1$. Now run the following process: Take $a,b$ ...
0
votes
0answers
14 views

Proof of Hammersley and Clifford theorem in Besag's paper

I am reading Besag's paper on Spatial Interaction and the Statistical Analysis of Lattice Systems, see http://www.cise.ufl.edu/~anand/fa11/Besag_Spatial_interaction.pdf. In section 3, it introduces ...
0
votes
0answers
22 views

Continuous time markov chains, is this step by step example correct

I have some questions regarding CTMC... and most importantly whether the step-by-step example I provide below is correct. My main sources about CTMC are: ([1], and [2]). Let's assume 3 possible ...
0
votes
0answers
25 views

Can the transition probabilities of an inhomogeneous Markov chain be written as an exponential?

If $Z_t$ is a homogeneous continuous-time Markov chain with finite state space $E=\{1,\ldots,p\}$, transition matrices $(P(t))$ and intensity matrix $Q$, it holds that $$ P(t) = \exp(tQ), $$ see for ...
0
votes
0answers
18 views

Initializing MCMC walkers with ambiguous direction (-/+)

I'm running a sampler program where there are observations given as sample data which are derived from an equal sized population of parameters that are converted to the observations using a known ...
0
votes
0answers
14 views

Simultaneous multiple perturbations in Markov chain Monte Carlo

I'm coding a McMC algorithm for geophysical applications. Using the Metropolis-Hastings scheme to accept/reject the proposed models is smth that i thought i completely understood, but i don't. To be ...
0
votes
0answers
39 views

Can ergodic Markov chains be periodic?

I found a statement in one of my notes which said If a state is persistent, aperiodic and not null the it is said to be ergodic Is it necessary that it should be aperiodic? This statement ...
0
votes
0answers
16 views

Solution to linear system around the topic of Markov-chains

Let $(X_n)_{n\geq 0}$ be a Markov-chain with the state space $S$ and transition matrix $P=(p_{xy})_{x, y \in S}$. For $A\subset S$ be $H^A:=\inf\{n = 0, 1, \dots | X_n \in A\}$ the first visit time ...
0
votes
0answers
13 views

Skew and Kurtosis of Absorbing Markov Chains

An absorbing Markov chain $P$ can be put in canonical form: $$ P = \left( \begin{array}{cc} Q & R\\ \mathbf{0} & I_r \end{array} \right), $$ where $Q$ is a t-by-t matrix, $R$ is a nonzero ...
0
votes
0answers
13 views

Proof of mean recurrence time theorem in a Markov chain?

How can this formula been proven? $$\lim_{n\to \infty} p_{i,i}^{[n]} = {1\over \mu_{i,i}}$$ where $p_{i,i}^{[n]}$ is the probability that we've returned to state $j$ after $n$ steps in the Markov ...
0
votes
0answers
19 views

Transition matrix in left-right hidden semi-Markov model

I'm developing a hidden semi-Markov model left-right . In a left-right model a sequence of $M$ states starts in state $1$ and ends in state $M$, with no repetition of states. Since the model is ...
0
votes
0answers
23 views

Does Markov property imply $\mathbb P (X_n=i \ | \ X_0=j)= \mathbb P (X_{n+1}=i \ | \ X_1=j)$?

If the future depends only on the present and not on the past (aka Markov property), one could expect $$\mathbb P (X_n=i \ | \ X_0=j)= \mathbb P (X_{n+1}=i \ | \ X_1=j)$$ to hold. Is that true? I've ...
0
votes
0answers
29 views

Probability of not reaching completion in Markov process

This question is supposed to be easy but is very hard for me. The Norwegian Skating Association has mass produced certain "collectors' cards" with all $N$ speedskaters (Norwegian as well as ...
0
votes
0answers
14 views

action of transition operator on function

Let $P$ be the transition operator of a markov chain with discrete time and discrete state space $X$. The action of the transition operator on a function $X \to \mathbb{R}$ is defined by $Pf(x) = ...
0
votes
0answers
28 views

Perron Frobenius Theorem and Markov chains and more

I came across few ways of calculating convergence rates of Markov chains but I am a bit confused as to how these differ from each other and what may be the best way to calculate. The second ...
0
votes
0answers
59 views

Application of Markov Chain to Game of Life Board Game

I need to calculate the expected outcomes for the Game of Life. I believe that if I multiply the probability of landing on a particular square with the payoff of said square and add up all these ...
0
votes
0answers
16 views

The second eigenvalue of a reducible stochastic matrix

The magnitude of the second dominant eigenvalue of a reducible matrix, as I know, is supposed to be 1, why it's not the case for this matrix : $$ \begin{matrix} 0 & 1 & 0 ...
0
votes
0answers
14 views

How to use symmetry of transition rate matrix in a continuous-time Markov chain?

This is part of a bigger question, so I have to change the question a bit to focus on the point. We have a continuous- time Markov chain with the following transition rate matrix: $$Q= \begin{pmatrix} ...
0
votes
0answers
23 views

G/G/1 Queues - Book with Discrete Time Markov Chain examples

Need some book recommendation or links which have examples how to solve G/G/1 queues with detailed Discrete Time Markov Chain drawn and how to get the steady state distribution, the average number of ...
0
votes
0answers
16 views

Markov chains, Essential, Inessential, Transient and Recurrent states - redundant definitions (what is the difference)

I've searched my books and gone through a tonne of lecture notes, I am now very sure that we have some redundant definitions. this question is about the difference between essential and recurrent, I ...
0
votes
0answers
16 views

discrete time Markov chain, difference between absorbing and recurrent classes.

In a discrete time Markov chain, are there any differences between an absorbing and a recurrent class? Recurrence is that we with probability 1 will reenter a state that we are in, this is a class ...
0
votes
0answers
14 views

Differentiating Dynkin's Formula

Let us assume that we are given a right-continuous, non-explosive continuous time Markov chain $(X_t)_{t\geq0}$ with infinite state space in $\mathbb{N}_0^d$. (Think, for instance, about a Markov ...
0
votes
0answers
12 views

Struggling to prove Markov chain property / State machine property - I can see what I need to do just can't write it

First notation, I have created the shorthand: $$P_x(A)=\mathbb{P}(A|X_0=x)$$ just to save time. I wish to prove "Lemma 1.3" which states the following: If $0<\alpha\le P_x(T_y\le k) \forall x\in ...
0
votes
0answers
18 views

Improve stationary distribution of one vertex

Suppose you have a Markov chain graph G and want to improve the stationary distribution of a node A the most by adding a single edge. The question asks to prove the best edge is one from some other ...