0
votes
0answers
20 views

Markov Chain Geometric Convergence

Perhaps due to my non-mathematician background (lack of Measure Theory knowledge), I have some difficulties about the Markov Chain Theory. Given an ergodic (irreducible and aperiodic) Markov Chain ...
0
votes
0answers
24 views

Markov chain definition: should the conditional probability also hold if $P(X_{n} = i_{n}, \ldots, X_0 = i_0) = 0$? Is $S$ a set of real numbers?

Definition of Markov Chain (as it is stated in my textbook): Let $S$ be a set of states and $\mathbb P = \{p_{i,j}\}$, $i,j \in S$ a transistion matrix . Then the sequence of RV's $(X_{n})_{n \ge 0}$ ...
0
votes
0answers
19 views

Discrete Markov Chain: probabilities

I'm confused about these: steady-state transition probabilities limiting probabilities stationary probabilities how are they different? I know the question is pretty vague, but I feel like I'm ...
3
votes
0answers
34 views

Markov chains and natural filtration

I have the following problem Consider a homogeneous Markov chain $(X_n)$ with countable state set $E$. Suppose that $A$ is a proper subset of $E$ and consider the stopping times $\tau^0=0 $ and ...
2
votes
1answer
50 views

continuous time Markov chain, something the book does not explain

I have a problem with something in my book, under the chapter of continuous time Markov chains. I will post a link to what the book does. They do something which they seem to take for granted, but I ...
1
vote
0answers
45 views

Markov chain question

Consider an irreducible, recurrent Markov Chain ($X_n$) on a countable state space $S$ with transition probability $p(x,y).$ Pick a sigma-algebra $A \subset S$ and let $T_k=\inf\{n>T_{k-1}:X_n \in ...
0
votes
1answer
46 views

Proving a chain is aperiodic, and finding a stationary distribution.

We have an irreducible Markov chain with a not necessarily finite state space. It has a transition matrix $P$ such that $P^2=P$. Prove (1) the chain is aperiodic, and (2) prove $p_{ij}=p_{jj}$ ...
0
votes
1answer
17 views

Question involving an invariant measure on a Markov chain

Suppose $\mu$ is an invariant measure for a Markov chain with state space $S$ with $\mu(i)p_{ij}=\mu(j)p_{ji}$ $\forall i,j \in S$. Describe a Markov chain with this property. Also, show that $\mu$ is ...
0
votes
0answers
26 views

Continuous time Markov chains, how would this definition be expanded from time-homogeneous to time-inhomogeneous.

Below I have a picture of how we can view a continuous time Markov chain that is time-homogeneous. Now, I am wondering what happens when we have a inhomogeneous continuous Markov chain. I have ...
0
votes
3answers
65 views

Markov Chain: classify states of finite Markov chain

I can easily see the states of this MC, recurrent and transient if I graph them, but how do I prove that a state is recurrent or transient. My book refers to probability to ever returning to state $j$ ...
2
votes
1answer
27 views

On track Prerequisite for Statistics and Probability

I do not really have a solid mathematical background because of the range of courses i had back in high school/university that wasn't really scientific oriented. Presently i am doing an MSc in ...
1
vote
2answers
64 views

Expected number of steps between states in a Markov Chain

Suppose I am given a state space $S=\{0,1,2,3\}$ with transition probability matrix $\mathbf{P}= \begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 \\[0.3em] \frac{2}{3} ...
1
vote
2answers
38 views

Counterintuitive Markov chain problem

In class, my professor said that given a Markov chain $\{X_k\}$ it intuitively should be true that $P(X_{k+1} = x_{k+1} \, \mid \, X_0 = a_0, \dots, X_{k-1}= a_{k-1}) = P(X_{k+1} = x_{k+1}\, \mid ...
0
votes
1answer
62 views

Exercise on Markov chain

Prove, or give an explicit counterexample to refute, the following assertion: if $\{X_n\}$ is a Markov chain, then $\{X_n^2\}$ is also a Markov chain. It's easy to show that ...
3
votes
1answer
50 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is ...
0
votes
0answers
18 views

Computing standard errors using EM algorithm

I'm applying the EM algorithm to a hidden markov chain (the $\mathbf{Z}=\{Z_1,...,Z_n\}$ variable), with observations(the $\mathbf{Y}=\{Y_0,...,Y_n\}$ variable) dependent not only on the hidden markov ...
2
votes
1answer
92 views

Markov chain problem with finite states

Consider any Markov chain on a state space with exactly $r$ states. We want to find the largest $N>0$ such that there exists states $i,j$ for every $n < N$ we have $p_{ij}^{(N)} > 0$ and ...
1
vote
1answer
45 views

The “maximum” of a simple random walk

Suppose $S_n$ is a simple random walk started from $S_0=0$. Denote $M_n$ to be the maximum of the walk in the first $n$ steps, i.e. $M_n=\max_{k\leq n}S_k$. Show that $M_n$ is not a Markov chain, but ...
1
vote
0answers
32 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
1
vote
1answer
57 views

Library chain stationary distribution

This is an exercise 1.47 from Richard Durrett's Essentials of Stochastic Processes p.85 (doi: 10.1007/978-1-4614-3615-7_1 or Google Books). On each request the ith of the $n$ possible books is the ...
1
vote
1answer
39 views

$Q$-matrices and Markov Chain properties

I would like to start with the definition of a $Q$-matrix on a countable set $I$. A $Q$-matrix is a matrix $Q=(q_{ij}:i,j\in I)$ satisfying the following conditions: (i) $0\le-q_{ii}<\infty\forall ...
1
vote
1answer
77 views

First step analysis on random walk

Let us consider random walk on integers {0,1,...,N} where $P(N,N)=1$,$P(0,1)=1$, $P(N,N-1)=0$ and all other connections have probability $\frac{1}{2}$. Using first step analysis, compute $p_{00}$ for ...
4
votes
1answer
76 views

random walk in a certain environment

Consider the following random walk in one dimension, starting from $r(0)=0$. $$ r(i+1) = r(i) + \xi, $$ where $\xi(i, r(i))$ is an increment with distribution $P(\xi=1) = \frac{c^{r(i)}}{i-r(i)+1}$ ...
2
votes
2answers
88 views

Wald's equation example controversy

I'm trying to get a grip of Wald's equation, applying it to the following example. Suppose, we have a simple sequence of fair coin flips, where heads wins us a dollar, while tails means loss of a ...
5
votes
1answer
111 views

Log-likelihood function

I'm not sure if this could be asked here, or in math overflow... In the following paper Cho, Jin Seo, and Halbert White. "Testing for regime switching." Econometrica 75.6 (2007): 1671-1720. doi: ...
0
votes
1answer
38 views

how to reformulate general markov property in discrete case

I read the wiki article on the markov property http://en.wikipedia.org/wiki/Markov_property#Definition and wondered how to work out this reformulation. It seems intuitively but I can not work it out. ...
0
votes
1answer
41 views

Stationary distribution for a Markov chain which is not irreducible

I have a Markov chain with $K$ states $S$: {$s_1,s_2,...,s_K$}. $s_1$ is reachable from any state in $S$; however not all the states can be reached from $s_1$. What does the stationary distribution ...
0
votes
1answer
52 views

Load balance equation of birth-death process

I am not able to understand the reasoning behind the load-balance equation of the birth-death processes (Markov chain) . The equation is $\pi_ib_i = \pi_{i+1}d_{i+1}$ , where the symbol have their ...
0
votes
0answers
44 views

Continuous-Time Markov Chain conditioned on not visiting a part of the state space

Let $(X_t)_{t\geq 0}$ be a homogeneous continuous-time Markov chain with state space $\Psi =\{1,\dots,N\}$. Consider $S=\{1,\dots,M\} \subset \Psi$. Define $T = \inf \{ t \geq 0 : (X_t \not \in S ...
1
vote
1answer
81 views

Stopping times of Markov chains

I have the following problem: Consider a state space $E$ and a Markov chain $X$ on $E$ with transition matrix $Q$ such that for every $x \in E$, $Q(x,x)<1$. Define: $\tau:=\inf\{n\geq 1:X_n\neq ...
1
vote
1answer
39 views

Finding the general pattern form

Suppose a Markov chain given as follows. $p_{ii}=1-3a$ And $p_{ij}=a\ \forall i\ne j$ where $P=(p_{ij}), 1\le i,j\le 4$. Find $P_{1,1}^n$. Attempts: I have tried to compute for the case $n=1, 2, 3$. ...
3
votes
1answer
90 views

Markov Chain Initial Distribution

Suppose $\{X_0,X_1,X_2,\dots\}$ is a discrete-time Markov chain taking values in a finite set $\{1,\dots,N\}$ with initial distribution $p_i(0) = P(X_0 = i)$ for $i\in\{1,\dots,N\}$ and transition ...
2
votes
1answer
66 views

Markov chain and hitting times

I have a Problem about hitting times. That's the following: Let $A\subset E$ and the first passage time $T_A$ and the hitting time $H_A$. Define: $T_A =\inf\{n\geq 0;X_n \in A\}$ and $H_A ...
1
vote
1answer
107 views

Markov Chain, closed, recurrent states

There are two urns, with the first one containing three white balls and the second one containing three black balls. At each step, we draw a ball from each urn, and then put the ball drawn from the ...
1
vote
2answers
61 views

Expected value of a series of random variables in a markov chain

I have a Markov Chain such that $X_n = \max(X_{n-1}+\xi _n,0)$ where the $\xi_n$ series is independent and identically distributed. I want to show that if $\mathbb E(\xi_n) > 0$ (where $\mathbb ...
0
votes
0answers
63 views

Markov Chains Proof using Statistics

Source: This came from "Introduction to probability" by Charles Miller Grinstead, and James Aurie Snell. It was located on page 407 and is Theorem 11.1 in the section 11.1 Introduction. Theorem: The ...
0
votes
1answer
37 views

Can a stationary distribution be zero vector

Suppose I have probabilities matrix between 3 states, for exampel we can take $P=\left(\begin{array}{ccc} \frac{1}{9} & \frac{8}{9} & 0\\ 0 & 0.3 & 0.7\\ 0 & 0 & 1 ...
1
vote
1answer
82 views

How is a simple birth process is time-homogeneous?

Why is it that a simple birth process is time-homogeneous? The incidence of a birth in a small time interval depends on the population size at the time of start of the interval. Doesn't this ...
1
vote
4answers
168 views

Why do these expressions tend to zero?

In a chapter on Markov chains, it is claimed that $\binom {2n}{n} p^n(1-p)^n$ (where $p$ is a probability) tends to $0$ as $n$ tends to $\infty$. But why is this so? It is also claimed that for an ...
0
votes
0answers
37 views

Two Markov chains: Optimal choice of initial distributions

Consider two Markov chains on the state space $X = [1;n] = \{1,2,\dots,n\}$ given by stochastic matrices $P$ and $Q$. Let $\alpha$ be the initial distribution for the first Markov Chain and $\beta$ is ...
1
vote
0answers
53 views

Intuitive meaning of spectral radius of a Markov chain transition matrix?

What is the intuitive meaning of the eigenvalues and in particular of the spectral radius of the transition matrix corresponding to a Markov chain?
1
vote
0answers
71 views

Markov kernels and update functions

I would like to prove, that for any Markov kernel $K$ on a Polish space $(F,\mathcal{F})$ (with a $\sigma$-field) you can find a measurable space $(S,\mathcal{S})$, a random element $Z$ on $S$ and an ...
0
votes
1answer
57 views

Stationary probabilities of markov chain

I am confused in which conditions the stationary probabilities of both discrete and continuous Markov chain donot exist. If it is due to periodic chain then is it for both discrete and continuous. ...
1
vote
0answers
56 views

Markov chain weak convergence

consider a sequence of Markov chains $\Phi^{(n)}$whose transition kernel $P^{(n)}$ converges to $P$. Now let $\Phi$ be the Markov chain with the limiting kernel $P$. How do I show that $\Phi^{(n)}\to ...
1
vote
1answer
62 views

Upper Bound of Markov Chain Convergence?

Reading about Markov Chain Monte Carlo in this book on Probability (DeGroot), it says In general, the distribution will get pretty close to the stationary distribution in finite time, but how ...
0
votes
0answers
62 views

Formal Theory Regarding M/M/s Queue

I have some difficulties with formally deducing the Q-matrix or infinitesimal generator for M/M/s Queues. Although I undestrand the intuitive idea I would like to know the real formal definition of ...
-1
votes
1answer
258 views

What is the Expectations of all 3 ants meeting at same point?

Say we have 3 ants in three corner's of triangle. What is the expectations that all 3 ants meeting together given that the ant moves in any direction. So by just seeing it I figured out that in 2 ...
1
vote
0answers
28 views

Poisson distributed variable after iterative process

The value of $x$ is changed in a stochastic iterative process. Changes of $\pm1$ are possible. I am searching transition probabilities $p(x=n \rightarrow x=n+1)$ and $p(x=n \rightarrow x=n-1)$ that ...
1
vote
0answers
56 views

Iterative process that leads to Poisson distribution

I want $x$ to be Poisson distributed. I will call occupation probability $p(x=n) =: p(n)$ and the transition probability $p(x=n \rightarrow x=n+1) =: p(n \rightarrow n+1)$ The value of $x$ is ...
1
vote
1answer
180 views

Eigenvector of transition matrix for Markov chain

Why is the only eigenvector of the transition matrix for an irreducible Markov chain with eigenvalue $= 1$ the eigenvector with all ones?