Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
55 views

Two independent renewal processes

We have two urns (blue and red) that are connected, and two particles, $p_1$ and $p_2$, are traveling between these urns independently. The amount of time $Z_1$ that $p_1$ spends in blue urn is iid ...
1
vote
0answers
40 views

probability with exponential distribution

there is a barber shop with two waiting chair and one barber.suggest that,when a customer comes in,and two waiting chair are full,he goes to another barber shop. The time between entering of two ...
1
vote
0answers
21 views

Continuous time markov chain: $p_{ij}(t + h) - p_{ij}(t)$

Given a continuous time markov chain: According to Ross: $P_{ij}(t+h) - P_{ij}(t) = \sum_{k \ne i} P_{ik}(h)P_{kj}(t) - (1 - P_{ii}(h)P_{ij}(t)$ What does $(1-P_{ii}(h))$ mean here? I am assuming ...
1
vote
0answers
59 views

A probability problem on Markov Chain

I am copying one example problem from Sheldon Ross which I am not understanding : Suppose we are given a set of $n$ elements, numbered $1$ through $n$, which are to be arranged in some ordered list. ...
1
vote
0answers
101 views

An urn probability problem

Suppose that $M$ balls are distributed among two urns and at each time point one of the balls is chosen at random, removed from its urn and placed in the other one. Now, in the long run the ...
1
vote
0answers
50 views

Is a deterministic path a Markov process

I have a question about what classifies as a Markov chain and what does not. Consider a system with state space $\left\{ 1,\ldots,n \right\}$ and a trajectory for the system defined by the following ...
1
vote
0answers
77 views

Rate of convergence of $n$-fold convolution over $\mathbb{Z}_p$.

Suppose that $\mu$ is a probabilist measure on $\mathbb{Z}_p$ such that $d_{TV}(\mu,\mu_U)\leqslant \delta < 1$. What are the best upper bound on $$ d_{TV}(\underbrace{\mu*\mu*\ldots*\mu}_{n ...
1
vote
0answers
101 views

First and second moments of recurrence time in a finite two-dimensional Markov chain

I have a two dimensional finite Markov chain with $(m+1)^2$ states, and with transition rates: $q_x((x,y)\to (x+1,y))=(m-x)\lambda,\quad 0\leq x< m, 0\leq y \leq m$, $q_x((x,y)\to ...
1
vote
0answers
45 views

Finding Steady state using markov chains. Am I right?

Suppose that there are two doctors in a country town, Dr Black and Dr White. Each year, 13% of patients move from Dr Black to Dr White, while 19% of patients move from Dr White to Dr Black. Suppose ...
1
vote
0answers
75 views

A matrix-multiplication random walk

Let $x \in \mathbb{R}^n$. Consider an $n\times n$ matrix $A$. Suppose we're interested in how $||A^nx||$ grows with $n$, the answer (excluding pathological cases) is that it scales exponentially with ...
1
vote
0answers
83 views

Stationary distribution behavior - Markov chain

I have modeled a process with a Markov chain with K+1 states which is irreducible and apperiodic. The transition matrix is a centrosymmetric matrix where all it's entries has a positive probability. ...
1
vote
0answers
77 views

Expected distance traversed between 2 vertices on probabilistic graph

Let $V = \{1,2,3,...,N\}$ be the vertex set of a graph. Let $d(i,j)>=0$ represent the $(i,j)$ vertex distance between vertices $i$ and $j$, $i \in V, j \in V$. Now, define a non-negative number ...
1
vote
0answers
75 views

Intuitive meaning of spectral radius of a Markov chain transition matrix?

What is the intuitive meaning of the eigenvalues and in particular of the spectral radius of the transition matrix corresponding to a Markov chain?
1
vote
0answers
53 views

Conditions for Markov chain

Let $\{X_n\}$ be a Markov chain with transition matrix $P$, and $Y_n := X_{m-n}$, $m\ge n$. Under what conditions is $\{Y_n\}_{n\ge 0}$ Markov chain? I stared by proving that conditional probability ...
1
vote
0answers
79 views

Markov kernels and update functions

I would like to prove, that for any Markov kernel $K$ on a Polish space $(F,\mathcal{F})$ (with a $\sigma$-field) you can find a measurable space $(S,\mathcal{S})$, a random element $Z$ on $S$ and an ...
1
vote
0answers
51 views

Gibbs / MCMC sampling for sum of parameters - how to improve slow mixing?

Suppose I have a hierarchical Bayesian model, where my observational prediction, $y'$, is calculated as the sum of other parameters, ${\alpha_i}$. My observation equation (the likelihood) is: $P(y | ...
1
vote
0answers
78 views

Markov chain weak convergence

consider a sequence of Markov chains $\Phi^{(n)}$whose transition kernel $P^{(n)}$ converges to $P$. Now let $\Phi$ be the Markov chain with the limiting kernel $P$. How do I show that $\Phi^{(n)}\to ...
1
vote
0answers
33 views

how to describe this case with markov-chain

I want to describe this case in markov chain: The case: Mr. Meier reads NYTimes everyday and puts the newspaper on news rack. His wife sometimes cleans the house(with prob $1/3$ each day) and throws ...
1
vote
0answers
39 views

Poisson distributed variable after iterative process

The value of $x$ is changed in a stochastic iterative process. Changes of $\pm1$ are possible. I am searching transition probabilities $p(x=n \rightarrow x=n+1)$ and $p(x=n \rightarrow x=n-1)$ that ...
1
vote
0answers
514 views

Compute stationary distribution of a general markov chain

I have a stochastic matrix P that represents a markov chain. I know that the markov chain is irreducible and aperiodic and therefore, I know the existence of a unique normalised left eigenvector to ...
1
vote
0answers
63 views

Iterative process that leads to Poisson distribution

I want $x$ to be Poisson distributed. I will call occupation probability $p(x=n) =: p(n)$ and the transition probability $p(x=n \rightarrow x=n+1) =: p(n \rightarrow n+1)$ The value of $x$ is ...
1
vote
0answers
42 views

$\psi$ irreducibility and ergodicity of Markov Processes

How is Markov chain splitting technique useful for inferring ergodicity of a Markov Chain?Assume that I am working with general state space (uncountable say $R^{N}$ but time is discrete. I want to ...
1
vote
0answers
81 views

From Q matrix to Markov Chain

We are in the setting of a continuous time MC, as defined by Liggett in his book on continuous time markov processes, on a countable state space $S$. All of his MCs are defined on the space of right ...
1
vote
0answers
34 views

literature to learn more on ergodic harris recurrent chains with an atom

I'm trying to learn more on the topic mentioned in the title. Namely I'd like to get more information on the behavior of the boundary terms. ie if I decompose sum of my chain (suppose it's ...
1
vote
0answers
63 views

A probability problem on periodicity of Markov Chain

Assume for a Markov Chain with period $d$, $\{C_0, C_1, \dots, C_{d−1}\}$ be the equivalence classes induced by $∼$ $(i$~$j$ means all the paths from $i$ to $j$ is of length $0$ mod $d$ )and numbered ...
1
vote
0answers
51 views

Meaning of $\pi$ in case of irreducible positive recurrent DTMC which is not aperiodic

In case of a irreducible positive recurrent DTMC which is not aperiodic, we know that there exist a positive unique probability mass function $\pi$ satisfying $\pi=\pi p$. The meaning of this can be : ...
1
vote
0answers
109 views

Log Moment Generating function of a two-state Markov source

Let's say you have a two-state markovian source whose transition matrix is $P=\begin{pmatrix}1-\sigma & \sigma\\ \tau & 1-\tau\end{pmatrix}$, for the state 0 the data rate is 0 and for the ...
1
vote
0answers
40 views

Estimate on Galton-Watson process distribution

Let $(Zn)_{n\in \mathbb N_0}$ be a Galton-Watson process, i.e. $$ Z_{n+1} = \sum_{k=1}^{Z_n}\xi_{n,k},\qquad (\xi_{n,k})_{n\in \mathbb N_0,k \in \mathbb N} \quad \text{i.i.d } \mathbb N_0 \text{ ...
1
vote
0answers
250 views

Modified M/M/1/2 with 2 possible arrival rates and M/M/1/5 queue

I've been stuck on this question for hours, and could use some help :) "An M/M/1/2 queue has service rate $\mu$ and arrival rate of either $\lambda_1$ or $\lambda_2$. The rate can change only when ...
1
vote
0answers
21 views

Some questions about a “relaxed” invariant probability problem $|\mu(P-I)|\leq \epsilon$

Let's consider the set $\mathcal{M}=\{\mu:|\mu(P-I)|_1\leq \epsilon\}$ where $\mu$ is a probability vector, $P$ is the transition matrix of a discrete homogeneous Markov chain, $I$ is the identity ...
1
vote
0answers
104 views

Kolmogorov backward and forward equations for a discrete-time Markov chain?

I found Kolmogorov backward equations and forward equations for diffusion processes, and for continuous time Markov chains in Wikipdia. I was wondering what Kolmogorov backward and forward equations ...
1
vote
0answers
43 views

What is the matrix norm in defining the generator of a continuous time Markov chain?

For a continuous time Markov chain with finite state space and Markov transition function $p(t)$, its generator $G$ can be defined entry-wise as $$ G_{i,j}:=\lim_{t\to 0^+} \frac{p_{i,j}(t) - ...
1
vote
0answers
85 views

Projected Markov chain evaluated at hitting times again Markov chain?

Consider the 2D Markov chain $X_n = (Y_n,Z_n)$ with a simple symmetric random walk along $(y,0),y\in \mathbb Z$ and simple symmetric random walks along the vertical direction for every $y \neq 0$. ...
1
vote
0answers
269 views

Markov chains - classify states and find stationary distribution

Consider the Markov Chain with the matrix $$ \begin{vmatrix} 0 & 0 & 1 & 0 & 0 \\ 0 & {\frac2 3} & 0 & {\frac 1 3 } & 0 \\ 1 & 0 & 0& 0& 0 \\ 0 & ...
1
vote
0answers
65 views

Recurrence criterion for a specific Markov chain

Let $(X_n)$ be a Markov chain on $\mathbb N_0$ defined by $(\alpha \geq 0)$ $$ p(0,1) = 1 \\ p(x,x+1) = 1-\frac{1}{(1+x)^\alpha} \\ p(x,0)= \frac{1}{(1+x)^\alpha}$$ Define the shifted moments for ...
1
vote
0answers
86 views

Finding probability from a markov chain

If I have a markov chain transition matrix for 2 states. Specifically in my case, it is a transition matrix for a bacterial genome with 4 random variables being A,C,G and T. (The bases) If I want to ...
1
vote
0answers
81 views

How to prove ergodic property from aperiodicity and positive recurrence

How to prove that in case of an irreducible, aperiodic and positive recurrent Markov Chain time average along sample paths is equal to the ensemble average ? i.e. $$\lim_{n\to \infty ...
1
vote
0answers
120 views

A question about discrete and continuous-time Markov Chains

I have a test tomorrow about Stochastics Process and I couldn't solve the following questions: A gambler starts with 500\$ and plays till he runs out of money. In each round the probability to win ...
1
vote
0answers
51 views

Efficient random number generation for sojourn times in semi-Markov processes

I'm doing a self-study of semi-Markov processes and was wondering if there are efficient methods for generating random numbers for sojourn times. For example, generating a bunch of random numbers from ...
1
vote
0answers
31 views

Multi-variate monotonic function

This is a question continued from here... Proof about Steady-State distribution of a Markov chain I have a stochastic matrix $P_\delta$ of dimensions $n\times n$. I look at the matrix as a transition ...
1
vote
0answers
145 views

how to calculate limit of P(Xn = j | X0 = i) in markov chain?

1. in http://robotics.eecs.berkeley.edu/~wlr/126/w12.htm lim N ® ¥ [1{X1 = j} + 1{X1 = j} + … + 1{XN = j}]/N = 0 What is N? how it limit to zero? and what do 1 in 1{X1 = j} represent? /N must be ...
1
vote
0answers
141 views

a problem on DTMC

For a Markov chain $\{X_n, n\ge0\}$ with transition probabilities $P_{i,j}$, consider the conditional probability that $X_n = m$ given that the chain started at time $0$ in state $i$ and has not ...
1
vote
0answers
79 views

A question on irreducible markov chains

I have a question on irreducible Markov Chains that has been bugging me for a few hours now I have the markov chain defined by: $P(i, i-1) = 1 - P(i,i+1) = \frac{1}{2(i+1)}$ for $i>=1$, and $p(0,1) ...
1
vote
0answers
29 views

immigration from one to other group

I'm studying Morkov chains and I have a question about immigration process. Let's say I have two groups $X$ and $Y$ each individual of these groups give birth with the same rate $b$ and members of $X$ ...
1
vote
0answers
69 views

Solve Variable Order Markov Chain(VOMC)

I am using VOMC in order to implement a real-time system. The Loop of the system is the following: LOOP{ 1)Get Input and Train VOMC according to it 2)Get output from Markov Chain } With the above ...
1
vote
0answers
38 views

Books about Markov Models

I am looking about books on Markov chains, with recent findings such as autoregressive HMM, HMM with inputs, multiple HMM connected together. Is there anything I can look at?
1
vote
0answers
60 views

An Iterated function system with probabilities and overlapping supports of its invariant measures

Let $(X, \rho)$ be a Polish space. Consider an Iterated Function System $(S_i,p_i)_{i=1,...,N}$, where $S_i:X\rightarrow X$, $p_i: X\rightarrow \left[0,1\right]$ are continuous functions and ...
1
vote
0answers
119 views

Random Walk on $N\times N$ grid

I would appreciate any help (answers, pointers to the literature etc.) on the following problem. Consider a (discrete time) random walk on an N-by-N grid which has two absorbing nodes, namely $(1,1)$ ...
1
vote
0answers
50 views

Markov chain supplementary litterature

I'm studying markov chains through Durrett and I'm finding it quite hard to read. Does anyone have a good idea to supplementary book, preferably one on his level of generality and which studies some ...
1
vote
0answers
46 views

How to sample the walk which visits each vertex of a graph specific number of times?

Is there any MCMC mathod that allow me to uniformly sample from all feasible walks where the following restrictions apply: ...