0
votes
0answers
25 views

Can the transition probabilities of an inhomogeneous Markov chain be written as an exponential?

If $Z_t$ is a homogeneous continuous-time Markov chain with finite state space $E=\{1,\ldots,p\}$, transition matrices $(P(t))$ and intensity matrix $Q$, it holds that $$ P(t) = \exp(tQ), $$ see for ...
2
votes
1answer
28 views

Finding the probability of ever visiting a transient state for a zero-seeking device for a Markov Chain?

A zero-seeking device operates as follows: if it is in state $j$ at time $n$, then at time $n+1$, its position is $0$ with probability $\frac{1}{j}$ or $k$ with probability $\frac{2k}{j^2}$, where $k$ ...
0
votes
1answer
25 views

Q-matrix vs. P-matrix description of a Markov chain

Consider a continuous time Markov chain $(X_t)_{t \geq 0}$ on some state space $S$ with transition matrix (P-matrix) $p_t(x,y)$, the probability density of jumping from $x$ to $y$ in time $t$. The ...
0
votes
1answer
71 views

How to make sense out of this: Ergodic theorem for Markov chains

We had the ergodic theorem for Markov chains, stating that: For a state space $S \subset \mathbb{N}$ and all functions $f \in L^1$ (meaning that $\sum_{s \in S} |f(s)|\pi(s) < \infty$) and an ...
1
vote
1answer
56 views

Need help with a basic exercise about Markov chains

Suppose $\left\{ X_{n}\right\} _{n=1}^{\infty}$ is a Markov Chain taking real values. Are the following Markov Chains? $$Y_{n}=\sum_{i=1}^{n}X_{i} , Z_{n}=\left(X_{n},X_{n-1}\right)$$ Edit1 I ...
0
votes
1answer
17 views

Is there a relationship between the mean matrix and the transition matrix of a multi type branching process?

Let $\mathbf{M}$ be the mean matrix of a multi type branching process $(\mathbf{Z}^{(n)})_{n\geq1}=((Z^{(n)}_1,\ldots,Z^{(n)}_k))_{n\geq1}$. This matrix is defined as follows $$M_{i,j}=\mathbb ...
1
vote
1answer
46 views

First hitting time expectation and Markov property

Let $H_A$ be the first hitting time, such that $H_A\geqslant1$, so we have $X_0=i\notin A$. All texts I looked at, state without any further justification that $$ \mathbb E(H_A\mid X_1=j, ...
2
votes
0answers
27 views

Geometric ergodicity and mixing - stationary case

I have this theorem: The Markov Chain {$X_n$} is stationary and geometrically ergodic if and only if {$X_n$} is stationary and absolutely regular with $\beta_n=O(\gamma^n)$ for some $\gamma\in ...
0
votes
1answer
28 views

A discrete time Markov chain with such a transient state that $\mathbb P(T_i<\infty \ | \ X_0=i) \neq 0$

All examples of discrete time Markov chains my text provides are where $S$ is finite, and as far as I can tell, it makes all transient states have $$\mathbb P(T_i<\infty \ | \ X_0=i) = 0.$$ Are ...
1
vote
1answer
56 views

random walk with sticky barriers

Consider a random walk on the line 1,...,d. You start at point 1. At each step you flip a coin: heads means go left, tails means go right. If you're at 1 and get a heads, just stay where you are ...
0
votes
0answers
23 views

Does Markov property imply $\mathbb P (X_n=i \ | \ X_0=j)= \mathbb P (X_{n+1}=i \ | \ X_1=j)$?

If the future depends only on the present and not on the past (aka Markov property), one could expect $$\mathbb P (X_n=i \ | \ X_0=j)= \mathbb P (X_{n+1}=i \ | \ X_1=j)$$ to hold. Is that true? I've ...
0
votes
1answer
60 views

The expected number of visits before hitting zero in simple random walk

I am learning Markov chains and encounter the following problem: Suppose in simple random walk, we start from state k. What's the expected number of visits to k before we hit 0? The book does not ...
0
votes
0answers
14 views

action of transition operator on function

Let $P$ be the transition operator of a markov chain with discrete time and discrete state space $X$. The action of the transition operator on a function $X \to \mathbb{R}$ is defined by $Pf(x) = ...
0
votes
1answer
36 views

Markov Chains and Return Times

Let $(X_n)_{n≥0}$ be a Markov chain with transition kernel $p$ on a countable state space $S$, starting at $x∈S$ $T^{(1)}=\inf\{n≥1:X_n=x\} \quad \quad$ first return time to $x$ ...
1
vote
1answer
23 views

Characterization of conditional independence

Definition: Let $\mathcal{G},\mathcal{K},\mathcal{H}$ be $\sigma $-subalgebras of $\mathcal{F}$, where $\left( {\Omega ,\mathcal{F},\mathbb{P}} \right)$ is a given probability space. We say that ...
1
vote
1answer
25 views

Equality involving a sequence of independent exponentially distributed variables

I'm trying to prove the following statement: Let $\left( {{T_n}:n \geqslant 1} \right)$ be a sequence of independent, exponentially distributed random variables with ${T_n} \sim Exp\left( {{q_n}} ...
0
votes
0answers
14 views

How to use symmetry of transition rate matrix in a continuous-time Markov chain?

This is part of a bigger question, so I have to change the question a bit to focus on the point. We have a continuous- time Markov chain with the following transition rate matrix: $$Q= \begin{pmatrix} ...
0
votes
1answer
18 views

A problem on Markov process

Suppose, $\Pi_{\theta}$ be the transition probability function of a Markov chain. For any function $f$ define $$\Pi_{\theta}f_{\theta}(x) = \int f(y,\theta)\Pi_{\theta}(x,dy).$$ Is there any ...
1
vote
1answer
39 views

PRobability Markov chain, system of equations

I'm looking for techniques or tricks to solve a system of linear equations you get where you want to find the limiting probabilities. The system is this: $\pi_0 = 0.7\pi_0 + 0.2\pi_1 + 0.1\pi_2$ ( ...
0
votes
0answers
16 views

discrete time Markov chain, difference between absorbing and recurrent classes.

In a discrete time Markov chain, are there any differences between an absorbing and a recurrent class? Recurrence is that we with probability 1 will reenter a state that we are in, this is a class ...
0
votes
1answer
39 views

Markov chain notation

In a book of stochastic approximation, in the convergence of the ODE method chapter I see the following notation : the state vector of a system $X_n$ has a dynamic representation controlled by ...
2
votes
0answers
56 views

What does a customer see when it begins to be served in $M/M/1$ queue?

In queueing theory, the PASTA (Poisson Arrivals See Time Averages) principle [wiki] justifies $a_n = P_n$ where $$a_n = \text{proportion of customers that find } n \text{ in the system when they ...
0
votes
2answers
48 views

Prove that something is a Markov chain

Let $\xi_0, \xi_1, \xi_2, ...$be independent, identically distributed, integer valued random variables. Define $Y_n$ = max{$\xi_i: 0 \leq i \leq n$}. Show that $(Y_{n)n\geq0}$ is a Markov chain and ...
0
votes
0answers
28 views

Derive a simple relation between two probabilistic quantities of a two-dimensional continuous-time Markov chain

We have two random variables $X(t)$ and $Y(t)$ represented as a continuous-time Markov chain with equilibrium probabilities \begin{align} p(i,j) := \lim_{t \to \infty} \mathbb{P}(X(t) = i,Y(t) = j), ...
2
votes
1answer
31 views

Is the following Markov Chain a martingale?

Say I have a finite, ergodic Markov chain with states ${0,1,2,3}$ and with the following transition matrix: $$\begin{bmatrix} \frac{7}{10} & \frac{3}{10} & 0 &0\\ \frac{1}{10} & ...
1
vote
0answers
43 views

Continuous transition kernel for Markov Chains

I am trying to show that the stationary distribution for a Markov Chain on a continuous state space can be obtained by building a transition density kernel, which obeys the detailed balance rule where ...
1
vote
0answers
22 views

Analysis of Steady State Probability for Markov Process

I have a balance equation, representing a Markov Chain, which yields $$ (K - z) \pi(Z_c = z) = (\lambda_c + (z+1)x) \pi(Z_c = z+1) $$ where K is the maximum state of the server. The term $\lambda_c$ ...
0
votes
0answers
23 views

Restarting a Markov chain

I'm reading an article and having difficulty understanding some basic stochastic processes (I'm new to the subject so please pardon my wording of the question). Let $S$ be a set of states and $Q$ be a ...
1
vote
1answer
44 views

Markov Chain Ergodic Theorem (Proof references)

Where can I find a proof of the erogidc theorem for Markov chains that doesn't use Birkhoff? The theorem states the following : Let $(X_n)_{n\in \mathbb{N}}$ be an irreducible and positively ...
1
vote
1answer
29 views

$X_n, n> 0$ is a Markov Chain, how to interpret $Z_n = (X_n,X_{n+1}), n > 0$?

Am a newbie to Markov Chain. So, this might be incredibly naive/stupid question. If $X_n, \, n > 0$ is MC, am having difficulty imagining/interpreting process $Z_n = (X_n,X_{n+1}), n > 0$. I ...
0
votes
0answers
36 views

Markov Chain Geometric Convergence

Perhaps due to my non-mathematician background (lack of Measure Theory knowledge), I have some difficulties about the Markov Chain Theory. Given an ergodic (irreducible and aperiodic) Markov Chain ...
0
votes
0answers
26 views

Markov chain definition: should the conditional probability also hold if $P(X_{n} = i_{n}, \ldots, X_0 = i_0) = 0$? Is $S$ a set of real numbers?

Definition of Markov Chain (as it is stated in my textbook): Let $S$ be a set of states and $\mathbb P = \{p_{i,j}\}$, $i,j \in S$ a transistion matrix . Then the sequence of RV's $(X_{n})_{n \ge 0}$ ...
3
votes
0answers
41 views

Markov chains and natural filtration

I have the following problem Consider a homogeneous Markov chain $(X_n)$ with countable state set $E$. Suppose that $A$ is a proper subset of $E$ and consider the stopping times $\tau^0=0 $ and ...
2
votes
1answer
63 views

continuous time Markov chain, something the book does not explain

I have a problem with something in my book, under the chapter of continuous time Markov chains. I will post a link to what the book does. They do something which they seem to take for granted, but I ...
1
vote
0answers
51 views

Markov chain question

Consider an irreducible, recurrent Markov Chain ($X_n$) on a countable state space $S$ with transition probability $p(x,y).$ Pick a sigma-algebra $A \subset S$ and let $T_k=\inf\{n>T_{k-1}:X_n \in ...
0
votes
1answer
52 views

Proving a chain is aperiodic, and finding a stationary distribution.

We have an irreducible Markov chain with a not necessarily finite state space. It has a transition matrix $P$ such that $P^2=P$. Prove (1) the chain is aperiodic, and (2) prove $p_{ij}=p_{jj}$ ...
0
votes
1answer
24 views

Question involving an invariant measure on a Markov chain

Suppose $\mu$ is an invariant measure for a Markov chain with state space $S$ with $\mu(i)p_{ij}=\mu(j)p_{ji}$ $\forall i,j \in S$. Describe a Markov chain with this property. Also, show that $\mu$ is ...
0
votes
0answers
50 views

Continuous time Markov chains, how would this definition be expanded from time-homogeneous to time-inhomogeneous.

Below I have a picture of how we can view a continuous time Markov chain that is time-homogeneous. Now, I am wondering what happens when we have a inhomogeneous continuous Markov chain. I have ...
0
votes
3answers
93 views

Markov Chain: classify states of finite Markov chain

I can easily see the states of this MC, recurrent and transient if I graph them, but how do I prove that a state is recurrent or transient. My book refers to probability to ever returning to state $j$ ...
2
votes
1answer
35 views

On track Prerequisite for Statistics and Probability

I do not really have a solid mathematical background because of the range of courses i had back in high school/university that wasn't really scientific oriented. Presently i am doing an MSc in ...
1
vote
2answers
95 views

Expected number of steps between states in a Markov Chain

Suppose I am given a state space $S=\{0,1,2,3\}$ with transition probability matrix $\mathbf{P}= \begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 \\[0.3em] \frac{2}{3} ...
1
vote
2answers
40 views

Counterintuitive Markov chain problem

In class, my professor said that given a Markov chain $\{X_k\}$ it intuitively should be true that $P(X_{k+1} = x_{k+1} \, \mid \, X_0 = a_0, \dots, X_{k-1}= a_{k-1}) = P(X_{k+1} = x_{k+1}\, \mid ...
0
votes
1answer
73 views

Exercise on Markov chain

Prove, or give an explicit counterexample to refute, the following assertion: if $\{X_n\}$ is a Markov chain, then $\{X_n^2\}$ is also a Markov chain. It's easy to show that ...
3
votes
1answer
67 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is ...
0
votes
0answers
24 views

Computing standard errors using EM algorithm

I'm applying the EM algorithm to a hidden markov chain (the $\mathbf{Z}=\{Z_1,...,Z_n\}$ variable), with observations(the $\mathbf{Y}=\{Y_0,...,Y_n\}$ variable) dependent not only on the hidden markov ...
2
votes
1answer
103 views

Markov chain problem with finite states

Consider any Markov chain on a state space with exactly $r$ states. We want to find the largest $N>0$ such that there exists states $i,j$ for every $n < N$ we have $p_{ij}^{(N)} > 0$ and ...
1
vote
1answer
81 views

The “maximum” of a simple random walk

Suppose $S_n$ is a simple random walk started from $S_0=0$. Denote $M_n$ to be the maximum of the walk in the first $n$ steps, i.e. $M_n=\max_{k\leq n}S_k$. Show that $M_n$ is not a Markov chain, but ...
1
vote
0answers
37 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
1
vote
1answer
64 views

Library chain stationary distribution

This is an exercise 1.47 from Richard Durrett's Essentials of Stochastic Processes p.85 (doi: 10.1007/978-1-4614-3615-7_1 or Google Books). On each request the ith of the $n$ possible books is the ...
1
vote
1answer
41 views

$Q$-matrices and Markov Chain properties

I would like to start with the definition of a $Q$-matrix on a countable set $I$. A $Q$-matrix is a matrix $Q=(q_{ij}:i,j\in I)$ satisfying the following conditions: (i) $0\le-q_{ii}<\infty\forall ...