Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
21 views

Some questions about a “relaxed” invariant probability problem $|\mu(P-I)|\leq \epsilon$

Let's consider the set $\mathcal{M}=\{\mu:|\mu(P-I)|_1\leq \epsilon\}$ where $\mu$ is a probability vector, $P$ is the transition matrix of a discrete homogeneous Markov chain, $I$ is the identity ...
1
vote
0answers
101 views

Kolmogorov backward and forward equations for a discrete-time Markov chain?

I found Kolmogorov backward equations and forward equations for diffusion processes, and for continuous time Markov chains in Wikipdia. I was wondering what Kolmogorov backward and forward equations ...
1
vote
0answers
39 views

What is the matrix norm in defining the generator of a continuous time Markov chain?

For a continuous time Markov chain with finite state space and Markov transition function $p(t)$, its generator $G$ can be defined entry-wise as $$ G_{i,j}:=\lim_{t\to 0^+} \frac{p_{i,j}(t) - ...
1
vote
0answers
264 views

Random walk, Cat and mouse

Here is the problem. In graph G, on different vertices there is cat and mouse. Cat and mouse do independent random walk, but time is synchronous, in one unit of time both cat and mouse do one step. ...
1
vote
0answers
84 views

Projected Markov chain evaluated at hitting times again Markov chain?

Consider the 2D Markov chain $X_n = (Y_n,Z_n)$ with a simple symmetric random walk along $(y,0),y\in \mathbb Z$ and simple symmetric random walks along the vertical direction for every $y \neq 0$. ...
1
vote
0answers
214 views

Markov chains - classify states and find stationary distribution

Consider the Markov Chain with the matrix $$ \begin{vmatrix} 0 & 0 & 1 & 0 & 0 \\ 0 & {\frac2 3} & 0 & {\frac 1 3 } & 0 \\ 1 & 0 & 0& 0& 0 \\ 0 & ...
1
vote
0answers
61 views

Recurrence criterion for a specific Markov chain

Let $(X_n)$ be a Markov chain on $\mathbb N_0$ defined by $(\alpha \geq 0)$ $$ p(0,1) = 1 \\ p(x,x+1) = 1-\frac{1}{(1+x)^\alpha} \\ p(x,0)= \frac{1}{(1+x)^\alpha}$$ Define the shifted moments for ...
1
vote
0answers
85 views

Finding probability from a markov chain

If I have a markov chain transition matrix for 2 states. Specifically in my case, it is a transition matrix for a bacterial genome with 4 random variables being A,C,G and T. (The bases) If I want to ...
1
vote
0answers
80 views

How to prove ergodic property from aperiodicity and positive recurrence

How to prove that in case of an irreducible, aperiodic and positive recurrent Markov Chain time average along sample paths is equal to the ensemble average ? i.e. $$\lim_{n\to \infty ...
1
vote
0answers
115 views

A question about discrete and continuous-time Markov Chains

I have a test tomorrow about Stochastics Process and I couldn't solve the following questions: A gambler starts with 500\$ and plays till he runs out of money. In each round the probability to win ...
1
vote
0answers
51 views

Efficient random number generation for sojourn times in semi-Markov processes

I'm doing a self-study of semi-Markov processes and was wondering if there are efficient methods for generating random numbers for sojourn times. For example, generating a bunch of random numbers from ...
1
vote
0answers
29 views

Multi-variate monotonic function

This is a question continued from here... Proof about Steady-State distribution of a Markov chain I have a stochastic matrix $P_\delta$ of dimensions $n\times n$. I look at the matrix as a transition ...
1
vote
0answers
129 views

how to calculate limit of P(Xn = j | X0 = i) in markov chain?

1. in http://robotics.eecs.berkeley.edu/~wlr/126/w12.htm lim N ® ¥ [1{X1 = j} + 1{X1 = j} + … + 1{XN = j}]/N = 0 What is N? how it limit to zero? and what do 1 in 1{X1 = j} represent? /N must be ...
1
vote
0answers
133 views

a problem on DTMC

For a Markov chain $\{X_n, n\ge0\}$ with transition probabilities $P_{i,j}$, consider the conditional probability that $X_n = m$ given that the chain started at time $0$ in state $i$ and has not ...
1
vote
0answers
71 views

A question on irreducible markov chains

I have a question on irreducible Markov Chains that has been bugging me for a few hours now I have the markov chain defined by: $P(i, i-1) = 1 - P(i,i+1) = \frac{1}{2(i+1)}$ for $i>=1$, and $p(0,1) ...
1
vote
0answers
29 views

immigration from one to other group

I'm studying Morkov chains and I have a question about immigration process. Let's say I have two groups $X$ and $Y$ each individual of these groups give birth with the same rate $b$ and members of $X$ ...
1
vote
0answers
67 views

Solve Variable Order Markov Chain(VOMC)

I am using VOMC in order to implement a real-time system. The Loop of the system is the following: LOOP{ 1)Get Input and Train VOMC according to it 2)Get output from Markov Chain } With the above ...
1
vote
0answers
35 views

Books about Markov Models

I am looking about books on Markov chains, with recent findings such as autoregressive HMM, HMM with inputs, multiple HMM connected together. Is there anything I can look at?
1
vote
0answers
60 views

An Iterated function system with probabilities and overlapping supports of its invariant measures

Let $(X, \rho)$ be a Polish space. Consider an Iterated Function System $(S_i,p_i)_{i=1,...,N}$, where $S_i:X\rightarrow X$, $p_i: X\rightarrow \left[0,1\right]$ are continuous functions and ...
1
vote
0answers
116 views

Random Walk on $N\times N$ grid

I would appreciate any help (answers, pointers to the literature etc.) on the following problem. Consider a (discrete time) random walk on an N-by-N grid which has two absorbing nodes, namely $(1,1)$ ...
1
vote
0answers
48 views

Markov chain supplementary litterature

I'm studying markov chains through Durrett and I'm finding it quite hard to read. Does anyone have a good idea to supplementary book, preferably one on his level of generality and which studies some ...
1
vote
0answers
45 views

How to sample the walk which visits each vertex of a graph specific number of times?

Is there any MCMC mathod that allow me to uniformly sample from all feasible walks where the following restrictions apply: ...
1
vote
0answers
68 views

Positive recurrence of a continuous-time jump process, from its jump chain

I am looking at an irreducible, continuous-time jump process $(X_t)_{t\geq0}$ with the following jump times. Let a Poisson process $(T_i)_{i=1}^\infty$ determine the event times. With probability ...
1
vote
0answers
157 views

Continuous-time Markov process - need help with flux and discrete-time equivalent

I have a continuous-time markov process and I need to calculate the following transition frequencies matrix (aka intensity matrix) transition probabilities all parameters which define permanence ...
1
vote
0answers
90 views

Quasi-stationary distribution of a state in a birth-and-death MC

I need to find an expression for the first state in an MC with transition matrix $P$ with tridiagonal entries. The state space is $U={1,2,..n}$ with the last state being absorbing. Expressions for ...
1
vote
0answers
54 views

Green kernel of Hunt processes

Let $X$ be a regular Hunt process on $R^+$ with starting at $x$ and $T_y :=\inf\{t>0: X_t=y\}, y\in R^+$. $G_q(\cdot,\cdot), q >0$ denotes Green kernel of the process $X$. We have the following ...
1
vote
0answers
162 views

Markov Chains - Using Gibbs & Metropolis algorithm.

Suppose $f_x,_y$ is bivariate normal distribution. I was given the parameters $(μ_1, μ_2, σ_1^2, σ_2^2)$ and $ρ=0.95$ the correlation coefficient. I want to generate $(x_1,y_1), ...
1
vote
0answers
26 views

How many observations is the minimum?

I want to estimate model transition matrix for a process (Markov chain). How much observiations of state do I need? I would prefer this as a function dependent on $n$, where $n$ is number of possible ...
1
vote
0answers
39 views

Integrating with respect to an MCMC Kernel

I'm currently trying to evaluate an integral with respect to an MCMC kernel, and I wanted someone else to read what I `proved' to see if it's really correct. It just doesn't feel right. Suppose I ...
1
vote
0answers
130 views

Cesaro mixing time: Show $t_m(2^{-k}) \le k t_m(1/4), k \ge 1$

Let $(X_t)_{t \ge 0}$ be a finite Markov chain with state space $\Omega$, transition matrix $P$ and stationary distribution $\pi$. Let $\| \cdot \|$ denote the total variation distance and define  ...
1
vote
0answers
108 views

Irreducible Markov: harmonic function based on stationary distribution

Let $P$ be the transition matrix of an irreducible Markov chain on a finite state space $\Omega$. Let $\pi_1$ and $\pi_2$ be two stationary distributions for $P$. Is the function $$h(x)={\pi_1(x) ...
1
vote
0answers
119 views

Expectation of an event

Let $S[4]$ be a binary array with elements of $S$ are taken uniformly and independently from $\{0,1\}$. Also take $k$ uniformly from $\{0,1\}$. Take $i=1$. Now run the following process: Take $a,b$ ...
0
votes
0answers
10 views

Probability transition matrix for maximum of iid random variables

I have a homework problem that goes as follows: Let $\xi_i, \ i=0,1,2,\ldots$ be i.i.d. random variables of discrete type. The distribution of $\xi_0$ is given by: $$\mathbb{P}\{\xi_0=1\} = a_i, \ ...
0
votes
0answers
22 views

“Taking expectation” to yield conditional probability

This argument is taken from Resnicks Adventures in stochastic processes and let $T _{\infty } < \infty $ denote that an infinite number of transitions in a continuous time markov chain has occurd ...
0
votes
0answers
7 views

Equivalent Formulation of Markov Property for Homogeneous Chains

In Shiryaev's Probability (just above the strong Markov property, p.568), the author says that an equivalent formulation of the usual Markov property for homogeneous chains is $$P[\theta_nX\in B\mid ...
0
votes
0answers
20 views

Irreducible Markov chain being recurrent

I've come across the following theorem in Sheldon Ross's book whose converse part I am unable to prove. Theorem: An irrreducible Markov chain with state space 0,1,2,... is recurrent if and only $\ ...
0
votes
0answers
28 views

Conditional expectation of a Markov-chain - can the conditioning Sigma-algebra be changed?

Let $(X_n)_{n \in \mathbb{N}_0}$ be a Markov-chain and $(\mathfrak{F}_n)_{n \in \mathbb{N}_0}$ the induced filtration $\mathfrak{F}_n := \sigma(X_0, \dots, X_n)$. Is then ...
0
votes
0answers
15 views

Markov vs reinforcement learning

What's the different between markov chain ,markov decision process and reinforcement learning? when we can apply these theories?
0
votes
0answers
15 views

markov process with extra boundary

In a markov process a random walker has to reach N (absorbing boundary) from $x_o$ on a $[0,N]$ lattice, where $0$ is the reflecting boundary. To find the first exit time of the random walker via N, i ...
0
votes
0answers
16 views

A basic problem on Markov chain

Consider an irreducible finite state Markov chain. Can we say that the following quantity is independent of $s$. $$\frac{E[\sum_{i=1}^{T} X_i]}{E[T]}$$ where $T$= time between two successive visit to ...
0
votes
0answers
8 views

minimum condition under which time avg. and ensemble avg. are equal for a markov chain

What is the minimum condition under which time avg. and ensemble avg. are equal for a markov chain. Is it ergodicity ?
0
votes
0answers
48 views

Dining philosophers using markov chains

We have 5 philosophers sitting at a table, where between each pair of philosophers is a single chopstick. They alternatively think and eat. When they want to eat, they pick up the chopsticks either ...
0
votes
0answers
25 views

A linear-algebraic property of stochastic matrices.

All matrices are real, $n \times n$. By a stochastic matrix, I mean any non-negative real matrix with rows summing to one. Denote the set of all stochastic matrices by $\mathcal{S}$. By $I_k$ I mean ...
0
votes
0answers
15 views

Markov Models and Applications

I am looking for resources in Markov models and its applications. I'm looking for tutorials, videos, books etc which provide the following Explain Markov chains in layperson terms and provide ...
0
votes
0answers
11 views

Definition of Perron-Frobenius eigenvalue

Consider a Markov chain with state space $X$ and transition prob. matrix $P=(p_{ij})$. Then a paper claims the following : Let $\theta \in X$ denote some fixed state. The Perron-Frobenius eigenvalue ...
0
votes
0answers
44 views

A question about a Markov Chain

I encountered a question about Markov Chains which looks interesting. Given a homogeneous, irreducible, non cyclic Markov Chain with $K$ possible states and a transition matrix $Q$. We define $T_i$ ...
0
votes
0answers
16 views

Probabilities in a Markov Model

I am reading a paper on Markov Models and I am trying to figure out how to compute the probabilities for the $\alpha$-pass. I am given an $N\times N$ matrix $A$, that has the probabilities of ...
0
votes
0answers
28 views

Can ergodic theorem be used here

Suppose I have an ergodic Markov Chain $\{X_n\}$ where $X_n$ are bounded. Now, Can I say anything on the limit $$ \lim_{n\to\infty} \frac{1}{n}\ln E\left[e^{\sum_{i=0}^{n} X_i}\right]$$ I don't ...
0
votes
0answers
29 views

Treatment of Markov process with absolute states

In the standard treatment of a markov process, the state vector is a probability vector, whose elements can be between zero and one. But I have a need to constrain the state vector to zeros or ones. ...
0
votes
0answers
12 views

For a general absorbing markov chain, if we have that $I-Q$ can be inverted, is it possible to prove the chain covers all stationary distributions?

If I have a general absorbing markov chain, there are nice properties when $I-Q$ is invertible. In my book, it claims it can be shown that a vector: $(0,0,0,...,0,v_1,...,v_{N-r+1} \in ...