-1
votes
0answers
22 views

Stationary distribution of a birth-death model where a parameter follows a uniform distribution.

I asked this question about some type a markov process I was interested in. @Did offers an answer but I fail to understand how to apply his answer to a concrete example. I am therefore seeking for an ...
1
vote
0answers
48 views

Can we find a correlation between states of a Markov chain?

I have a fair bit of knowledge on Markov chains but I recently wondered if there is a way to find out a correlation between the states of a finite Markov chain. I could not find any material on this. ...
0
votes
0answers
25 views

Covariance of states of a finite Markov chain

I know it is possible to construct a covariance matrix for states of a Markov chain but I cannot seem to find a proper way to compute it. I will attach some theories I found from Kemeny and Snell's ...
1
vote
1answer
29 views

Literature request on Markov Chains which state transition probability matrix evolves over time

I want to know is there any literature on markov chains who's state transition probability matrix evolves over time? For instance, I have 2 states, 1 and 2. With $$P = \begin{bmatrix} p_{11} & ...
0
votes
1answer
15 views

Recurrence of states in a function of a Markov chain

Suppose $X$ is a Markov chain (or process, for that matter) and suppose further $f(X)$ is also a Markov chain. Let $s$ be a recurrent state in $X$. Is there a general way to determine the recurrence ...
0
votes
0answers
9 views

Equivalent Formulation of Markov Property for Homogeneous Chains

In Shiryaev's Probability (just above the strong Markov property, p.568), the author says that an equivalent formulation of the usual Markov property for homogeneous chains is $$P[\theta_nX\in B\mid ...
0
votes
1answer
17 views

$n$-step transition probability of a Markov chain

Let $(X_t)_{t\in\mathbb{N}_0}$ be a time-homogenous Markov chain over a finite state space $\left\{1,\dots,m\right\}$, so that we've got $$\Pr(X_{t+1}=j\mid X_t=i_t,\dots,X_0=i_0)=\Pr(X_{t+1}=j\mid ...
0
votes
0answers
20 views

Irreducible Markov chain being recurrent

I've come across the following theorem in Sheldon Ross's book whose converse part I am unable to prove. Theorem: An irrreducible Markov chain with state space 0,1,2,... is recurrent if and only $\ ...
2
votes
0answers
13 views

Proving that an inductively defined function is a Markov chain

Let $X_0$ be a random variable with values in a countable set $I$. Let $Y_1,Y_2,\ldots$ be a sequence of independent random variables, uniformly distributed on $[0,1]$. Suppose we are given a function ...
1
vote
1answer
47 views

Proof of extinction probability in Galton-Watson-process using a Martingale

this problem is somewhat similar to the thread The extinction probability of Galton-Watson process from a Martingale perspective. I want to show, that for a Galton-Watson-process $Z_0,Z_1,\ldots$ with ...
1
vote
1answer
54 views

Markov Chains : Can anything be said about what happens in between two transition?

In time homogeneous discrete Markov chains we take a set period for a single transition. In examples we see sometimes depending on the examples the transition period being a a month a week etc. I'm ...
1
vote
1answer
27 views

In an irreducible, aperiodic, null-recurrent Markov chain holds $\sum_n p_{ij}^{(n)} = \infty$

My lecture notes state the following theorem: Theorem 2. Let $(X_n)$ be an irreducible, aperiodic, null-recurrent Markov chain. Then $$\forall i,j \in S : p_{ij}^{(n)}\to 0 \text{ and } \sum_n ...
0
votes
1answer
41 views

How to apply the strong Markov property in this case?

I'm trying to understand the following proof: Theorem: Let $(X_n)$ be an irreducible $(\alpha, \mathbf p)$-Markov chain with a finite state space $S$. Then $(X_n)$ is positive recurrent. ...
2
votes
1answer
36 views

Green's function of a Markov Chain, and maybe of a Feller Process?

How are the Green's functions of a Markov chain related to the notion from PDE theory? For instance, if the Markov chain (i.e. discrete state space) is continuous time, then the Green's function I'm ...
0
votes
1answer
52 views

Can the transition probabilities of an inhomogeneous Markov chain be written as an exponential?

If $Z_t$ is a homogeneous continuous-time Markov chain with finite state space $E=\{1,\ldots,p\}$, transition matrices $(P(t))$ and intensity matrix $Q$, it holds that $$ P(t) = \exp(tQ), $$ see for ...
2
votes
1answer
47 views

Finding the probability of ever visiting a transient state for a zero-seeking device for a Markov Chain?

A zero-seeking device operates as follows: if it is in state $j$ at time $n$, then at time $n+1$, its position is $0$ with probability $\frac{1}{j}$ or $k$ with probability $\frac{2k}{j^2}$, where $k$ ...
1
vote
1answer
39 views

Q-matrix vs. P-matrix description of a Markov chain

Consider a continuous time Markov chain $(X_t)_{t \geq 0}$ on some state space $S$ with transition matrix (P-matrix) $p_t(x,y)$, the probability density of jumping from $x$ to $y$ in time $t$. The ...
0
votes
1answer
76 views

How to make sense out of this: Ergodic theorem for Markov chains

We had the ergodic theorem for Markov chains, stating that: For a state space $S \subset \mathbb{N}$ and all functions $f \in L^1$ (meaning that $\sum_{s \in S} |f(s)|\pi(s) < \infty$) and an ...
1
vote
1answer
66 views

Need help with a basic exercise about Markov chains

Suppose $\left\{ X_{n}\right\} _{n=1}^{\infty}$ is a Markov Chain taking real values. Are the following Markov Chains? $$Y_{n}=\sum_{i=1}^{n}X_{i} , Z_{n}=\left(X_{n},X_{n-1}\right)$$ Edit1 I ...
0
votes
1answer
18 views

Is there a relationship between the mean matrix and the transition matrix of a multi type branching process?

Let $\mathbf{M}$ be the mean matrix of a multi type branching process $(\mathbf{Z}^{(n)})_{n\geq1}=((Z^{(n)}_1,\ldots,Z^{(n)}_k))_{n\geq1}$. This matrix is defined as follows $$M_{i,j}=\mathbb ...
1
vote
1answer
57 views

First hitting time expectation and Markov property

Let $H_A$ be the first hitting time, such that $H_A\geqslant1$, so we have $X_0=i\notin A$. All texts I looked at, state without any further justification that $$ \mathbb E(H_A\mid X_1=j, ...
2
votes
0answers
35 views

Geometric ergodicity and mixing - stationary case

I have this theorem: The Markov Chain {$X_n$} is stationary and geometrically ergodic if and only if {$X_n$} is stationary and absolutely regular with $\beta_n=O(\gamma^n)$ for some $\gamma\in ...
0
votes
1answer
32 views

A discrete time Markov chain with such a transient state that $\mathbb P(T_i<\infty \ | \ X_0=i) \neq 0$

All examples of discrete time Markov chains my text provides are where $S$ is finite, and as far as I can tell, it makes all transient states have $$\mathbb P(T_i<\infty \ | \ X_0=i) = 0.$$ Are ...
1
vote
1answer
62 views

random walk with sticky barriers

Consider a random walk on the line 1,...,d. You start at point 1. At each step you flip a coin: heads means go left, tails means go right. If you're at 1 and get a heads, just stay where you are ...
0
votes
0answers
25 views

Does Markov property imply $\mathbb P (X_n=i \ | \ X_0=j)= \mathbb P (X_{n+1}=i \ | \ X_1=j)$?

If the future depends only on the present and not on the past (aka Markov property), one could expect $$\mathbb P (X_n=i \ | \ X_0=j)= \mathbb P (X_{n+1}=i \ | \ X_1=j)$$ to hold. Is that true? I've ...
0
votes
1answer
71 views

The expected number of visits before hitting zero in simple random walk

I am learning Markov chains and encounter the following problem: Suppose in simple random walk, we start from state k. What's the expected number of visits to k before we hit 0? The book does not ...
0
votes
0answers
15 views

action of transition operator on function

Let $P$ be the transition operator of a markov chain with discrete time and discrete state space $X$. The action of the transition operator on a function $X \to \mathbb{R}$ is defined by $Pf(x) = ...
0
votes
1answer
42 views

Markov Chains and Return Times

Let $(X_n)_{n≥0}$ be a Markov chain with transition kernel $p$ on a countable state space $S$, starting at $x∈S$ $T^{(1)}=\inf\{n≥1:X_n=x\} \quad \quad$ first return time to $x$ ...
1
vote
1answer
26 views

Characterization of conditional independence

Definition: Let $\mathcal{G},\mathcal{K},\mathcal{H}$ be $\sigma $-subalgebras of $\mathcal{F}$, where $\left( {\Omega ,\mathcal{F},\mathbb{P}} \right)$ is a given probability space. We say that ...
1
vote
1answer
28 views

Equality involving a sequence of independent exponentially distributed variables

I'm trying to prove the following statement: Let $\left( {{T_n}:n \geqslant 1} \right)$ be a sequence of independent, exponentially distributed random variables with ${T_n} \sim Exp\left( {{q_n}} ...
0
votes
0answers
19 views

How to use symmetry of transition rate matrix in a continuous-time Markov chain?

This is part of a bigger question, so I have to change the question a bit to focus on the point. We have a continuous- time Markov chain with the following transition rate matrix: $$Q= \begin{pmatrix} ...
0
votes
1answer
19 views

A problem on Markov process

Suppose, $\Pi_{\theta}$ be the transition probability function of a Markov chain. For any function $f$ define $$\Pi_{\theta}f_{\theta}(x) = \int f(y,\theta)\Pi_{\theta}(x,dy).$$ Is there any ...
1
vote
1answer
53 views

PRobability Markov chain, system of equations

I'm looking for techniques or tricks to solve a system of linear equations you get where you want to find the limiting probabilities. The system is this: $\pi_0 = 0.7\pi_0 + 0.2\pi_1 + 0.1\pi_2$ ( ...
0
votes
0answers
28 views

discrete time Markov chain, difference between absorbing and recurrent classes.

In a discrete time Markov chain, are there any differences between an absorbing and a recurrent class? Recurrence is that we with probability 1 will reenter a state that we are in, this is a class ...
0
votes
1answer
62 views

Markov chain notation

In a book of stochastic approximation, in the convergence of the ODE method chapter I see the following notation : the state vector of a system $X_n$ has a dynamic representation controlled by ...
2
votes
1answer
82 views

What does a customer see when it begins to be served in $M/M/1$ queue?

In queueing theory, the PASTA (Poisson Arrivals See Time Averages) principle [wiki] justifies $a_n = P_n$ where $$a_n = \text{proportion of customers that find } n \text{ customers in the system when ...
0
votes
2answers
52 views

Prove that something is a Markov chain

Let $\xi_0, \xi_1, \xi_2, ...$be independent, identically distributed, integer valued random variables. Define $Y_n$ = max{$\xi_i: 0 \leq i \leq n$}. Show that $(Y_{n)n\geq0}$ is a Markov chain and ...
2
votes
1answer
39 views

Is the following Markov Chain a martingale?

Say I have a finite, ergodic Markov chain with states ${0,1,2,3}$ and with the following transition matrix: $$\begin{bmatrix} \frac{7}{10} & \frac{3}{10} & 0 &0\\ \frac{1}{10} & ...
1
vote
0answers
54 views

Continuous transition kernel for Markov Chains

I am trying to show that the stationary distribution for a Markov Chain on a continuous state space can be obtained by building a transition density kernel, which obeys the detailed balance rule where ...
1
vote
0answers
28 views

Analysis of Steady State Probability for Markov Process

I have a balance equation, representing a Markov Chain, which yields $$ (K - z) \pi(Z_c = z) = (\lambda_c + (z+1)x) \pi(Z_c = z+1) $$ where K is the maximum state of the server. The term $\lambda_c$ ...
0
votes
0answers
26 views

Restarting a Markov chain

I'm reading an article and having difficulty understanding some basic stochastic processes (I'm new to the subject so please pardon my wording of the question). Let $S$ be a set of states and $Q$ be a ...
1
vote
1answer
53 views

Markov Chain Ergodic Theorem (Proof references)

Where can I find a proof of the erogidc theorem for Markov chains that doesn't use Birkhoff? The theorem states the following : Let $(X_n)_{n\in \mathbb{N}}$ be an irreducible and positively ...
1
vote
1answer
38 views

$X_n, n> 0$ is a Markov Chain, how to interpret $Z_n = (X_n,X_{n+1}), n > 0$?

Am a newbie to Markov Chain. So, this might be incredibly naive/stupid question. If $X_n, \, n > 0$ is MC, am having difficulty imagining/interpreting process $Z_n = (X_n,X_{n+1}), n > 0$. I ...
0
votes
0answers
42 views

Markov Chain Geometric Convergence

Perhaps due to my non-mathematician background (lack of Measure Theory knowledge), I have some difficulties about the Markov Chain Theory. Given an ergodic (irreducible and aperiodic) Markov Chain ...
0
votes
0answers
29 views

Markov chain definition: should the conditional probability also hold if $P(X_{n} = i_{n}, \ldots, X_0 = i_0) = 0$? Is $S$ a set of real numbers?

Definition of Markov Chain (as it is stated in my textbook): Let $S$ be a set of states and $\mathbb P = \{p_{i,j}\}$, $i,j \in S$ a transistion matrix . Then the sequence of RV's $(X_{n})_{n \ge 0}$ ...
4
votes
0answers
49 views

Markov chains and natural filtration

I have the following problem Consider a homogeneous Markov chain $(X_n)$ with countable state set $E$. Suppose that $A$ is a proper subset of $E$ and consider the stopping times $\tau^0=0 $ and ...
2
votes
1answer
64 views

continuous time Markov chain, something the book does not explain

I have a problem with something in my book, under the chapter of continuous time Markov chains. I will post a link to what the book does. They do something which they seem to take for granted, but I ...
1
vote
0answers
53 views

Markov chain question

Consider an irreducible, recurrent Markov Chain ($X_n$) on a countable state space $S$ with transition probability $p(x,y).$ Pick a sigma-algebra $A \subset S$ and let $T_k=\inf\{n>T_{k-1}:X_n \in ...
0
votes
1answer
56 views

Proving a chain is aperiodic, and finding a stationary distribution.

We have an irreducible Markov chain with a not necessarily finite state space. It has a transition matrix $P$ such that $P^2=P$. Prove (1) the chain is aperiodic, and (2) prove $p_{ij}=p_{jj}$ ...
0
votes
1answer
35 views

Question involving an invariant measure on a Markov chain

Suppose $\mu$ is an invariant measure for a Markov chain with state space $S$ with $\mu(i)p_{ij}=\mu(j)p_{ji}$ $\forall i,j \in S$. Describe a Markov chain with this property. Also, show that $\mu$ is ...