Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

2
votes
0answers
125 views

Markov chain from Poisson

Let $K_t$ be a Poisson process with rate $1$ and $X_n=K_n-n$ $, \ \ \ n\in \mathbb{N}$ am asked to determine whether it is null or positive recurrent, we already know it is recurrent. I ...
2
votes
0answers
29 views

Broken disk head

Broken disk head: We would like to read 1 byte = sequence of 8 bits from a disk, starting from bit 0. Our disk head reads 1 bit at a time. Disk head can only move forward, but after reaching bit 7 ...
2
votes
0answers
84 views

Is the following interpretation for the stationary distribution of a Markov process correct?

Imagine I have some Markov process with stationary distribution $\pi$ and a mixing time of $\tau$ after which $|Prob[x=s_i] - \pi(s_i)| \leq \epsilon$. Can I assume the following: A state $(x=s_i)$ ...
2
votes
0answers
21 views

Problem with the uniform transience

Let $X$ be a Borel space and let us consider a Markov Chain $(\Phi_n)_{n\geq 0}$ on this space given by the stochastic kernel $$ P(x,\mathrm dy) = p(x,y)\mu(\mathrm dy) $$ where the density $p$ is ...
2
votes
0answers
333 views

In finite-state Markov chain state $i$ is transient

Can you help me please with proof of this question: Prove, that in finite-state Markov chain state $i$ is transient if and only if is exist state $k$ such that $i\rightarrow k$ but k $\nrightarrow i$....
2
votes
0answers
240 views

Computing the stationary distribution of a markov chain

I have a markov chain with transition matrix below, $$\begin{bmatrix} 1-q & q & & & \\ 1-q & 0 & q & & \\ & 1-q & ...
2
votes
0answers
57 views

Green kernel of Hunt processes

Let $X$ be a regular Hunt process on $R^+$ with starting at $x$ and $T_y :=\inf\{t>0: X_t=y\}, y\in R^+$. $G_q(\cdot,\cdot), q >0$ denotes Green kernel of the process $X$. We have the following ...
2
votes
0answers
283 views

Probability question about change in vending machines; maybe markov chain?

Suppose there are vending machine that sells its goods for $3$. It's known that a third of the buyers use three coins of $1$, a third of the buyers use $2$ and $1$, and the last third use $5$. The ...
1
vote
0answers
19 views

Extension of erdodic theorem with WLLN

Suppose you have a ergodic (or irreducible) Markov chain $(A_t)_{t\geq0}$ in continuous time. denote by $\pi$ the invariant distribution of $A$. If $f$ is a function of $A_s$ which is integrable w.r.t....
1
vote
0answers
17 views

First time of passage, discrete random walk with disjoint absorbing regions

I have a sum $T^i$ of zero/one Bern$(p)$ random variables $T_i$ and multiple disjoint absorbing regions, i.e. the absorbing region is a union of disjoint, closed sets: $$T^i \in \bigcup_{u \in \...
1
vote
0answers
10 views

Filtering/MCMC methods for this HMM

I have a Discrete HMM with hidden Markovian signals of the form $\{X_t\}_{t \in [0, \infty)} \in \{ 1,2,3\}$ and observed outputs of the form $\{Y_n\}_{n \in \mathbb{N}} \in \{ 1,2\}$. Each ...
1
vote
0answers
33 views

Better to go first in Snakes and Ladders?

We consider the game as described in http://www.datagenetics.com/blog/november12011/ . Each person rolls a dice and the person who gets 6 on the face can start and the other keeps waiting. If the ...
1
vote
0answers
92 views

Probability of losing lotteries needed

A person earns $x_i$ amount of money every month where $x_i$ is an exponential random variable with parameter $\lambda_1$. The amount $x_i(1-p)y$, here $0 \leq p\leq 1$ and $y$ is exponentially ...
1
vote
0answers
30 views

Statement of the strong Markov property in Norris' book

In J.R.Norris' Markov chains book, the strong Markov property for discrete-time, Markov chains is stated and proved as follows: Let $(X_n)_{n \geqslant 0}$ be a Markov chain with transition ...
1
vote
0answers
22 views

Perron-Frobenius Theorem: Markov Chain -> Matrices

I am interested in finding out a way how to transform the stochastic results of perron-frobenius for markov chains to any matrix. I am aware that perron-frobenius was originally proofed with linear ...
1
vote
0answers
29 views

Series with Markov Chains Probabilities

Notation For each $t \in \mathbb{N}$, let $h_t \in H$ be a random variable that follows a Markov chain, and $h^t \equiv \{h_0,h_1,\dots,h_t\} \in H^t$. Let $\Pi(h^{t})$ be the probability that a ...
1
vote
0answers
29 views

Mean value function of a discrete-time Markov-modulated Poisson process

By "discrete-time Markov-modulated Poisson process" I mean a semi-Markov process $\{(X_n,T_n):n=0,1,\ldots\} $ which satisfies $$T_{n+1}-T_n\mid X_n\sim\operatorname{Exp}(\lambda_{X_n}), $$ with $\...
1
vote
0answers
21 views

Reducing sequential correlations in Metropolis Algorithm

In our last lab, we use MCMC method to simulate a walker walking in the phase space. Using the Metropolis method, a walker at its currect position will sample another point inside a cube (centered at ...
1
vote
0answers
20 views

Pure Birth Question. Find the probability that the population at time $t$ is an odd # given it starts at $0$.

Here is the question. Consider a pure birth process $\{X(t) : t ≥ 0\}$ with birth parameters $\lambda_{2n} = α>0$ and $\lambda_{2n+1} =β>0$ for $n∈N$. Compute $Pr\{X(t) \text{ is odd } \mid X(0)...
1
vote
0answers
44 views

For a finite state irreducible aperiodic MC, show that $P^{d^2}$ has all coordinates positive.

Suppose $X_n$ is an irreducible aperiodic finite state MC, with $P$ being the transition matrix. Then we know that $P^n$ has all positive entries for some $n\in\mathbb N$. If the state space $S$ of ...
1
vote
0answers
25 views

How to test quality of probability estimates?

I have a Markov chain model which produces a probability distribution for absorption in 4 possible absorbing states. I.e. the model estimates the probability distribution for a discrete random ...
1
vote
0answers
58 views

Prediction Interval from Markov Chains

Thank you for taking the time to look at my question. Short, less involved question: How do you find Prediction Intervals with non-Gaussian residuals? Here is the situation: I have made a model that ...
1
vote
0answers
24 views

When can an embedded Markov chain X for a Markov process Y be reducible?

It's pretty widely documented that a Markov process Y is reducible/irreducible if and only if the embedded Markov chain X is reducible/irreducible. However I'm not sure this works in reverse. I'm ...
1
vote
0answers
20 views

Harris Chain and Stationary Distribution

Consider a Harris chain given by $\{X_n\}$ with the following transition function, $X_{n+1}=\max \{0,X_n-b\} $ with probability $p$ and $X_{n+1}=\max \{0,a-\tau\} $ with probability $1-p$, where $\tau ...
1
vote
0answers
14 views

How to determine the transition probability in Sequential Importance Sampling (SIS) for Particle Filter

Given a state-space model \begin{align} x_k &= f_k(x_{k-1}, v_{k-1}),\\ z_k &= h_k(x_k, w_k), \end{align} where $x_k \in {\mathbb R}^{n}$ and $y_k \in {\mathbb R}^{m}$ are the system state ...
1
vote
0answers
23 views

How is the following attitude true?(Markov chains)

Let us have an $X_n$ Markov-chain with finite $S$ set as domain. $A \subset S$ is given, so that $P_x(T_A < \infty) > 0$ for all $x \in S$. $T_A=\inf \{ n \geq 1: X_n \in A\}$. Then we take a $...
1
vote
0answers
30 views

Transform Markov chain that doesn't have stationary transition probabilities to one that does?

This question concerns Exercise 7.3 in Walsh's Knowing the Odds. A Markov chain is defined as having stationary transition probabilities if for all $i, j, n$ we have $P(X_{n+1} = j \mid X_n=i) = P(...
1
vote
0answers
20 views

Two Markov chains $X_t$, $Y_t$ have the same transition matrix $P$, show $\Bbb P(\tau_c\le t_0) = \Bbb P(\tau_c\le 2t_0|\tau_c> t_0)$

Given two Markov chains $X_t$, $Y_t$ characterized by the same transition matrix $P$, let $\tau_c$ be the first time the two chains have the same state, i.e. $\tau_c = \min\{t:X_t=Y_t\}$. The ...
1
vote
0answers
19 views

Is this the same thing as some simple discrete probability distribution?

I want to count the number of trials until one success in a sequence where the success probability is increased with each failure. For each trial, the success probability is $1-\binom{k+1+i}{k-1}^{-1}$...
1
vote
0answers
30 views

Probability that sample path ends at a high.

I am trying to get the probability that a sample path ends at a high. To formulate the problem, let sequence $\{S_n\}$ be a random walk, with $S_0 = 0$, defined by $$ S_n = \sum_{k=1}^n X_k$$ Where $...
1
vote
0answers
18 views

marginalised markov chain

If there is a Markov chain for the joint variable $z=(x,y)$, the marginal process $x$ is not, in general, Markovian itself. However, if we consider the probability of a two time step process $$p(z_0,...
1
vote
0answers
42 views

What is the invariance principle of Random Walks?

Several papers I have read allude to the fact that a random walk is invariant; however, I have been unable to find any reference to support this fact. Could anyone explain why random walks are ...
1
vote
0answers
52 views

Finite irreducible Markov chain

The question I have is stated as follows: Show that for any finite-state irreducible Markov chain $$\max_{i,j}\mathbb E_iT_j\le C$$where the constant $C$ only depends on the number of states and $\...
1
vote
0answers
80 views

Deduce the definition of a harmonic function in the context of a Markov Chain

We know from PDE that a harmonic function $f$ satisfies the mean value property, namely, $f(x)$ = $\frac{1}{\vert{B_r(x)}\vert}\int_{B_r(x)}f(y)dy$ where $B_r(x)$ is the ball about $x$ with radius $r$....
1
vote
0answers
32 views

Irreducible and positive recurrent CTMC — first passage times are finite?

Consider a continuous-time Markov chain (CTMC) $X$ on a countably infinite state space $S$. The CTMC is irreducible and all the states are positive recurrent. Let $T(i,j)$ be the first passage time to ...
1
vote
0answers
15 views

Rigorous Derivation of Metropolis-Hastings Transition Density

The Metropolis-Hastings MCMC algorithm is as follows. Set $X_0$ to some initial value in the support of the target density $f$ and choose a proposal density $q(y \mid x)$; a density in $y$ for each $...
1
vote
0answers
31 views

The “how many pieces do you have buy on average” problem, a markov problem?

I recently discovered a problem similar to this one in a book about Markov chains: Assume you can buy $n-$ different set of cards in a store, but you do not know which one you'll buy: What is the ...
1
vote
0answers
47 views

Identity for return times in continuous Markov chain

I need help with this problem about return times in continuous time Markov chains: We are given a continuous time Markov chain $(X_t)$. Define $T_i=\inf{\{t>0; X_t=i, X_{t-} \neq i\}}$, which ...
1
vote
0answers
34 views

An example of Markov chain with no closed class?

What is an example of Markov chain with no closed communicating class? Closed class means that once we are in that class, there would be no escape from it. I am thinking that an example would be ...
1
vote
0answers
20 views

recurrence time for transient state

I have the following transition matrix for a MC with state space $S = \{ 1,2,3,4,5,6,7,8 \}$ \begin{bmatrix} 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0.4 &...
1
vote
0answers
51 views

Eigenvectors of Approximations to Infinite Stochastic Matrices

Given a function $[0,1]\to[0,1]\times[0,1]$ on the reals, such that the function is "stochastic" (probably an abuse of vocabulary: defined such that integrating along any vertical line gives $1$), ...
1
vote
0answers
64 views

Does a Markov process have memory zero?

I have the following question: The Markov process with two states and a transition matrix $$P =\begin{pmatrix} 0.3 & 0.7 \\ 0.3 & 0.7 \end{pmatrix}$$ has memory zero. Is it true? My ...
1
vote
0answers
34 views

Simple random walk on the $N$-cycle

I am considering the following example: In my lecture notes we noted that "the functions $(\phi_j)_j$ form a basis". I think they refer to the space $\mathbb{C}^G$ where $G$ is the above ...
1
vote
0answers
44 views

Markov chain of transition probabilities

Let $P$ be a transition matrix on a discrete state space with $N$ elements. $P_{i,j}$ is the probability of going from state $i$ to state $j$. Let $\pi$ be the stationary distribution. Let $\{X_n\}$ ...
1
vote
0answers
45 views

Birth-Death process with shifted exponential distribution

In the general framework of $M/M/1$ queue we have rate $\lambda$ and an exponential service time $\mu$, we can set up the transition rate matrix intuitively. However, if the service times satisfy ...
1
vote
0answers
25 views

Understanding a Markov decision process

We have an insect that is resting on a vertex of a square at each point of time $t=0,1,2..$. The vertices are labelled from 1 to 4. 1 is given to the lower left vertex, 3 to the upper left vertex, 2 ...
1
vote
0answers
54 views

Finding a stationary distribution for a Markov chain with a binomially distributed transition matrix

The distribution isn't exactly binomial.. I'll explain what I mean: I am considering a Markov chain with N+1 states (denoted by $0,1,...,N$), the probability of going from state $i$ to state $j$ is ...
1
vote
0answers
19 views

convergence of markov chain, find distribution of limit

let $T=\mathbb{N}$ and $(X_t)_{t \in T}$ be a markov chain with state space $E=\{a,b\}$. the one step transition matrices are given by \begin{equation*} P(t,t+1)= \begin{pmatrix} 0,2+0,8f(t) &...
1
vote
0answers
16 views

Alternate Markvov Chain Model.

In class we are working on DTMCs for past couple of week, and in our last lecture we did an example. Question was: In a city of nowhere it only rains if there are clouds for successive m-days. Any ...
1
vote
0answers
31 views

Irreducible finite-state Markov Chain

Could anyone help me with this? Let $X_n$ be an irreducible Markov chain on the state space $S=\{1,\ldots,N\}$. Show that there exist $C<\infty$ and $\rho <1$ such that for any states $i,j$, $...