Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
21 views

Some questions about a “relaxed” invariant probability problem $|\mu(P-I)|\leq \epsilon$

Let's consider the set $\mathcal{M}=\{\mu:|\mu(P-I)|_1\leq \epsilon\}$ where $\mu$ is a probability vector, $P$ is the transition matrix of a discrete homogeneous Markov chain, $I$ is the identity ...
1
vote
0answers
97 views

Kolmogorov backward and forward equations for a discrete-time Markov chain?

I found Kolmogorov backward equations and forward equations for diffusion processes, and for continuous time Markov chains in Wikipdia. I was wondering what Kolmogorov backward and forward equations ...
1
vote
0answers
38 views

What is the matrix norm in defining the generator of a continuous time Markov chain?

For a continuous time Markov chain with finite state space and Markov transition function $p(t)$, its generator $G$ can be defined entry-wise as $$ G_{i,j}:=\lim_{t\to 0^+} \frac{p_{i,j}(t) - ...
1
vote
0answers
256 views

Random walk, Cat and mouse

Here is the problem. In graph G, on different vertices there is cat and mouse. Cat and mouse do independent random walk, but time is synchronous, in one unit of time both cat and mouse do one step. ...
1
vote
0answers
84 views

Projected Markov chain evaluated at hitting times again Markov chain?

Consider the 2D Markov chain $X_n = (Y_n,Z_n)$ with a simple symmetric random walk along $(y,0),y\in \mathbb Z$ and simple symmetric random walks along the vertical direction for every $y \neq 0$. ...
1
vote
0answers
209 views

Markov chains - classify states and find stationary distribution

Consider the Markov Chain with the matrix $$ \begin{vmatrix} 0 & 0 & 1 & 0 & 0 \\ 0 & {\frac2 3} & 0 & {\frac 1 3 } & 0 \\ 1 & 0 & 0& 0& 0 \\ 0 & ...
1
vote
0answers
61 views

Recurrence criterion for a specific Markov chain

Let $(X_n)$ be a Markov chain on $\mathbb N_0$ defined by $(\alpha \geq 0)$ $$ p(0,1) = 1 \\ p(x,x+1) = 1-\frac{1}{(1+x)^\alpha} \\ p(x,0)= \frac{1}{(1+x)^\alpha}$$ Define the shifted moments for ...
1
vote
0answers
85 views

Finding probability from a markov chain

If I have a markov chain transition matrix for 2 states. Specifically in my case, it is a transition matrix for a bacterial genome with 4 random variables being A,C,G and T. (The bases) If I want to ...
1
vote
0answers
80 views

How to prove ergodic property from aperiodicity and positive recurrence

How to prove that in case of an irreducible, aperiodic and positive recurrent Markov Chain time average along sample paths is equal to the ensemble average ? i.e. $$\lim_{n\to \infty ...
1
vote
0answers
112 views

A question about discrete and continuous-time Markov Chains

I have a test tomorrow about Stochastics Process and I couldn't solve the following questions: A gambler starts with 500\$ and plays till he runs out of money. In each round the probability to win ...
1
vote
0answers
51 views

Efficient random number generation for sojourn times in semi-Markov processes

I'm doing a self-study of semi-Markov processes and was wondering if there are efficient methods for generating random numbers for sojourn times. For example, generating a bunch of random numbers from ...
1
vote
0answers
28 views

Multi-variate monotonic function

This is a question continued from here... Proof about Steady-State distribution of a Markov chain I have a stochastic matrix $P_\delta$ of dimensions $n\times n$. I look at the matrix as a transition ...
1
vote
0answers
128 views

how to calculate limit of P(Xn = j | X0 = i) in markov chain?

1. in http://robotics.eecs.berkeley.edu/~wlr/126/w12.htm lim N ® ¥ [1{X1 = j} + 1{X1 = j} + … + 1{XN = j}]/N = 0 What is N? how it limit to zero? and what do 1 in 1{X1 = j} represent? /N must be ...
1
vote
0answers
133 views

a problem on DTMC

For a Markov chain $\{X_n, n\ge0\}$ with transition probabilities $P_{i,j}$, consider the conditional probability that $X_n = m$ given that the chain started at time $0$ in state $i$ and has not ...
1
vote
0answers
71 views

A question on irreducible markov chains

I have a question on irreducible Markov Chains that has been bugging me for a few hours now I have the markov chain defined by: $P(i, i-1) = 1 - P(i,i+1) = \frac{1}{2(i+1)}$ for $i>=1$, and $p(0,1) ...
1
vote
0answers
28 views

immigration from one to other group

I'm studying Morkov chains and I have a question about immigration process. Let's say I have two groups $X$ and $Y$ each individual of these groups give birth with the same rate $b$ and members of $X$ ...
1
vote
0answers
66 views

Solve Variable Order Markov Chain(VOMC)

I am using VOMC in order to implement a real-time system. The Loop of the system is the following: LOOP{ 1)Get Input and Train VOMC according to it 2)Get output from Markov Chain } With the above ...
1
vote
0answers
35 views

Books about Markov Models

I am looking about books on Markov chains, with recent findings such as autoregressive HMM, HMM with inputs, multiple HMM connected together. Is there anything I can look at?
1
vote
0answers
60 views

An Iterated function system with probabilities and overlapping supports of its invariant measures

Let $(X, \rho)$ be a Polish space. Consider an Iterated Function System $(S_i,p_i)_{i=1,...,N}$, where $S_i:X\rightarrow X$, $p_i: X\rightarrow \left[0,1\right]$ are continuous functions and ...
1
vote
0answers
113 views

Random Walk on $N\times N$ grid

I would appreciate any help (answers, pointers to the literature etc.) on the following problem. Consider a (discrete time) random walk on an N-by-N grid which has two absorbing nodes, namely $(1,1)$ ...
1
vote
0answers
47 views

Markov chain supplementary litterature

I'm studying markov chains through Durrett and I'm finding it quite hard to read. Does anyone have a good idea to supplementary book, preferably one on his level of generality and which studies some ...
1
vote
0answers
45 views

How to sample the walk which visits each vertex of a graph specific number of times?

Is there any MCMC mathod that allow me to uniformly sample from all feasible walks where the following restrictions apply: ...
1
vote
0answers
68 views

Positive recurrence of a continuous-time jump process, from its jump chain

I am looking at an irreducible, continuous-time jump process $(X_t)_{t\geq0}$ with the following jump times. Let a Poisson process $(T_i)_{i=1}^\infty$ determine the event times. With probability ...
1
vote
0answers
155 views

Continuous-time Markov process - need help with flux and discrete-time equivalent

I have a continuous-time markov process and I need to calculate the following transition frequencies matrix (aka intensity matrix) transition probabilities all parameters which define permanence ...
1
vote
0answers
90 views

Quasi-stationary distribution of a state in a birth-and-death MC

I need to find an expression for the first state in an MC with transition matrix $P$ with tridiagonal entries. The state space is $U={1,2,..n}$ with the last state being absorbing. Expressions for ...
1
vote
0answers
52 views

Green kernel of Hunt processes

Let $X$ be a regular Hunt process on $R^+$ with starting at $x$ and $T_y :=\inf\{t>0: X_t=y\}, y\in R^+$. $G_q(\cdot,\cdot), q >0$ denotes Green kernel of the process $X$. We have the following ...
1
vote
0answers
162 views

Markov Chains - Using Gibbs & Metropolis algorithm.

Suppose $f_x,_y$ is bivariate normal distribution. I was given the parameters $(μ_1, μ_2, σ_1^2, σ_2^2)$ and $ρ=0.95$ the correlation coefficient. I want to generate $(x_1,y_1), ...
1
vote
0answers
26 views

How many observations is the minimum?

I want to estimate model transition matrix for a process (Markov chain). How much observiations of state do I need? I would prefer this as a function dependent on $n$, where $n$ is number of possible ...
1
vote
0answers
38 views

Integrating with respect to an MCMC Kernel

I'm currently trying to evaluate an integral with respect to an MCMC kernel, and I wanted someone else to read what I `proved' to see if it's really correct. It just doesn't feel right. Suppose I ...
1
vote
0answers
130 views

Cesaro mixing time: Show $t_m(2^{-k}) \le k t_m(1/4), k \ge 1$

Let $(X_t)_{t \ge 0}$ be a finite Markov chain with state space $\Omega$, transition matrix $P$ and stationary distribution $\pi$. Let $\| \cdot \|$ denote the total variation distance and define  ...
1
vote
0answers
107 views

Irreducible Markov: harmonic function based on stationary distribution

Let $P$ be the transition matrix of an irreducible Markov chain on a finite state space $\Omega$. Let $\pi_1$ and $\pi_2$ be two stationary distributions for $P$. Is the function $$h(x)={\pi_1(x) ...
1
vote
0answers
119 views

Expectation of an event

Let $S[4]$ be a binary array with elements of $S$ are taken uniformly and independently from $\{0,1\}$. Also take $k$ uniformly from $\{0,1\}$. Take $i=1$. Now run the following process: Take $a,b$ ...
0
votes
0answers
42 views

Follow-up on solution to markov process equation

I asked a question here about solving a system related to an absorbing markov chain. I now have a variation where there are $m$ types (of student, job seeker, etc) each of which applies to ...
0
votes
0answers
13 views

Distributive Law on Sum Product

I am reading a tutorial on Conditional Randome Fileds, Here is the link: http://people.cs.umass.edu/~mccallum/papers/crf-tutorial.pdf in the equation 1.24 it defines: $p(x,y) = \prod_{t=1}^{T} ...
0
votes
0answers
13 views

Markov Models and Applications

I am looking for resources in Markov models and its applications. I'm looking for tutorials, videos, books etc which provide the following Explain Markov chains in layperson terms and provide ...
0
votes
0answers
11 views

Definition of Perron-Frobenius eigenvalue

Consider a Markov chain with state space $X$ and transition prob. matrix $P=(p_{ij})$. Then a paper claims the following : Let $\theta \in X$ denote some fixed state. The Perron-Frobenius eigenvalue ...
0
votes
0answers
15 views

Probabilities in a Markov Model

I am reading a paper on Markov Models and I am trying to figure out how to compute the probabilities for the $\alpha$-pass. I am given an $N\times N$ matrix $A$, that has the probabilities of ...
0
votes
0answers
28 views

Can ergodic theorem be used here

Suppose I have an ergodic Markov Chain $\{X_n\}$ where $X_n$ are bounded. Now, Can I say anything on the limit $$ \lim_{n\to\infty} \frac{1}{n}\ln E\left[e^{\sum_{i=0}^{n} X_i}\right]$$ I don't ...
0
votes
0answers
21 views

Treatment of Markov process with absolute states

In the standard treatment of a markov process, the state vector is a probability vector, whose elements can be between zero and one. But I have a need to constrain the state vector to zeros or ones. ...
0
votes
0answers
12 views

For a general absorbing markov chain, if we have that $I-Q$ can be inverted, is it possible to prove the chain covers all stationary distributions?

If I have a general absorbing markov chain, there are nice properties when $I-Q$ is invertible. In my book, it claims it can be shown that a vector: $(0,0,0,...,0,v_1,...,v_{N-r+1} \in ...
0
votes
0answers
26 views

Exact probability distribution for hitting time of simple random walk

Consider simple random walk on the line starting from the site $y \in \mathbb{N}$. With probability $p$ the walker moves to the right and with probability $1-p$ to the left. Call $\tau$ the first time ...
0
votes
0answers
27 views

Proof of Hammersley and Clifford theorem in Besag's paper

I am reading Besag's paper on Spatial Interaction and the Statistical Analysis of Lattice Systems, see http://www.cise.ufl.edu/~anand/fa11/Besag_Spatial_interaction.pdf. In section 3, it introduces ...
0
votes
0answers
19 views

Initializing MCMC walkers with ambiguous direction (-/+)

I'm running a sampler program where there are observations given as sample data which are derived from an equal sized population of parameters that are converted to the observations using a known ...
0
votes
0answers
16 views

Simultaneous multiple perturbations in Markov chain Monte Carlo

I'm coding a McMC algorithm for geophysical applications. Using the Metropolis-Hastings scheme to accept/reject the proposed models is smth that i thought i completely understood, but i don't. To be ...
0
votes
0answers
40 views

Can ergodic Markov chains be periodic?

I found a statement in one of my notes which said If a state is persistent, aperiodic and not null the it is said to be ergodic Is it necessary that it should be aperiodic? This statement ...
0
votes
0answers
16 views

Solution to linear system around the topic of Markov-chains

Let $(X_n)_{n\geq 0}$ be a Markov-chain with the state space $S$ and transition matrix $P=(p_{xy})_{x, y \in S}$. For $A\subset S$ be $H^A:=\inf\{n = 0, 1, \dots | X_n \in A\}$ the first visit time ...
0
votes
0answers
15 views

Skew and Kurtosis of Absorbing Markov Chains

An absorbing Markov chain $P$ can be put in canonical form: $$ P = \left( \begin{array}{cc} Q & R\\ \mathbf{0} & I_r \end{array} \right), $$ where $Q$ is a t-by-t matrix, $R$ is a nonzero ...
0
votes
0answers
14 views

Proof of mean recurrence time theorem in a Markov chain?

How can this formula been proven? $$\lim_{n\to \infty} p_{i,i}^{[n]} = {1\over \mu_{i,i}}$$ where $p_{i,i}^{[n]}$ is the probability that we've returned to state $j$ after $n$ steps in the Markov ...
0
votes
0answers
21 views

Transition matrix in left-right hidden semi-Markov model

I'm developing a hidden semi-Markov model left-right . In a left-right model a sequence of $M$ states starts in state $1$ and ends in state $M$, with no repetition of states. Since the model is ...
0
votes
0answers
23 views

Does Markov property imply $\mathbb P (X_n=i \ | \ X_0=j)= \mathbb P (X_{n+1}=i \ | \ X_1=j)$?

If the future depends only on the present and not on the past (aka Markov property), one could expect $$\mathbb P (X_n=i \ | \ X_0=j)= \mathbb P (X_{n+1}=i \ | \ X_1=j)$$ to hold. Is that true? I've ...