0
votes
0answers
10 views

A basic doubt on ergodic markov chain

Given an ergodic markov chain $\{X_n\}$ is there any easy way to calculate the following in terms of $i$ and transition probabilities ? $$ \inf(\lambda \in \Bbb R_+ : \sum_{n=0}^{\infty} \lambda^{-n} ...
0
votes
1answer
11 views

In an irreducible, aperiodic, null-recurrent Markov chain holds $\sum_n p_{ij}^{(n)} = \infty$

My lecture notes state the following theorem: Theorem 2. Let $(X_n)$ be an irreducible, aperiodic, null-recurrent Markov chain. Then $$\forall i,j \in S : p_{ij}^{(n)}\to 0 \text{ and } \sum_n ...
1
vote
1answer
77 views

A Markov Chain Problem.(Change the color of ball)

There are $n$ different color balls in a box. Take two balls in turns, and change color of the second ball to the first. (This is one operation). Let $k$ be the (random) number of operations needed to ...
-1
votes
0answers
14 views

Prove this is markov chain [on hold]

Let $Y_i = i$ with probability $p_i$. Let $X_i = max[Y_i , Y_i-1, ...]. $ We have $i={0,1,2,3,4,...}. $ Prove that $X_i$ is a Markov chain and write down its matrix.
0
votes
1answer
39 views

How to apply the strong Markov property in this case?

I'm trying to understand the following proof: Theorem: Let $(X_n)$ be an irreducible $(\alpha, \mathbf p)$-Markov chain with a finite state space $S$. Then $(X_n)$ is positive recurrent. ...
1
vote
1answer
19 views

Max of independent and identical random variables is Markov

I'm supposed to show that given a sequence $\{Y_n\}$ of i.i.d the stochastic process $$X_n=\max(Y_0, Y_1...,Y_n)$$ is a Markov of chain. I think I could do it by induction but I would rather see how ...
-1
votes
0answers
35 views

A question about a Markov Chain

I encountered a question about Markov Chains which looks interesting. Given a homogeneous, irreducible, non cyclic Markov Chain with $K$ possible states and a transition matrix $Q$. We define $T_i$ ...
0
votes
1answer
24 views

Supply the transition matrix for these (possible) Markov chains

Reading Grimmet, Stirzaker: Probability and Random Processes, which unfortunately doesn't have solutions. Trying to make sure I understand Markov chains. A die is rolled repeatedly. Which of these ...
0
votes
0answers
15 views

Probabilities in a Markov Model

I am reading a paper on Markov Models and I am trying to figure out how to compute the probabilities for the $\alpha$-pass. I am given an $N\times N$ matrix $A$, that has the probabilities of ...
0
votes
0answers
12 views

For a general absorbing markov chain, if we have that $I-Q$ can be inverted, is it possible to prove the chain covers all stationary distributions?

If I have a general absorbing markov chain, there are nice properties when $I-Q$ is invertible. In my book, it claims it can be shown that a vector: $(0,0,0,...,0,v_1,...,v_{N-r+1} \in ...
0
votes
1answer
38 views

Can the transition probabilities of an inhomogeneous Markov chain be written as an exponential?

If $Z_t$ is a homogeneous continuous-time Markov chain with finite state space $E=\{1,\ldots,p\}$, transition matrices $(P(t))$ and intensity matrix $Q$, it holds that $$ P(t) = \exp(tQ), $$ see for ...
2
votes
1answer
31 views

Finding the probability of ever visiting a transient state for a zero-seeking device for a Markov Chain?

A zero-seeking device operates as follows: if it is in state $j$ at time $n$, then at time $n+1$, its position is $0$ with probability $\frac{1}{j}$ or $k$ with probability $\frac{2k}{j^2}$, where $k$ ...
1
vote
1answer
19 views

Producing transient and recurrent examples for birth-death chains with mixed birth- and death-probabilities

Suppose we have a birth-death chain with a state space $$ S = \{0,1,2,\ldots\} $$ and transition probailities: $$p(x,y)=\begin{cases}q_x, &\text{if } y = x-1, &\text{i.e. death}\\ ...
1
vote
0answers
26 views

A question about Markov

There is a continuous-time markov chain,and we know the probability transition matrix P.The time between 2 states can be formulated as a exponential distribution whose u is related to the 2 states.Now ...
0
votes
1answer
28 views

Q-matrix vs. P-matrix description of a Markov chain

Consider a continuous time Markov chain $(X_t)_{t \geq 0}$ on some state space $S$ with transition matrix (P-matrix) $p_t(x,y)$, the probability density of jumping from $x$ to $y$ in time $t$. The ...
2
votes
2answers
23 views

Probability of returning to a given state after n transitions-Markov chains

Let us denote $f_j^{(n)}$ denote the probability of the first return to state $j $after n transitions. Let $p_{jj}^{(n)}$ be the probability of returning to the state $j$ after $n$ transitions when ...
0
votes
1answer
72 views

How to make sense out of this: Ergodic theorem for Markov chains

We had the ergodic theorem for Markov chains, stating that: For a state space $S \subset \mathbb{N}$ and all functions $f \in L^1$ (meaning that $\sum_{s \in S} |f(s)|\pi(s) < \infty$) and an ...
1
vote
1answer
57 views

Need help with a basic exercise about Markov chains

Suppose $\left\{ X_{n}\right\} _{n=1}^{\infty}$ is a Markov Chain taking real values. Are the following Markov Chains? $$Y_{n}=\sum_{i=1}^{n}X_{i} , Z_{n}=\left(X_{n},X_{n-1}\right)$$ Edit1 I ...
0
votes
1answer
31 views

Long run behavior of a absorbing markov chain

$$A=\begin{pmatrix}1&0&0&0\\0&1&0&0\\0.2&0&0.6&0.2\\0&0.2&0.2&0.6\end{pmatrix}.$$ In the above matrix how do I calculate the probability that in the ...
0
votes
0answers
40 views

Can ergodic Markov chains be periodic?

I found a statement in one of my notes which said If a state is persistent, aperiodic and not null the it is said to be ergodic Is it necessary that it should be aperiodic? This statement ...
1
vote
1answer
25 views

Markov chains: An issue in classification of states

I recently came across a lemma which goes as follows. Suppose a Markov chain has N states. Let i and j be pair of states. Then j can be reached from i iff there is an integer $ 0 ≤n< N$ such ...
0
votes
2answers
53 views

Expected number of steps to absorbtion - Markov chain

I want to calculate the expected number of steps (transitions) needed for absorbtion. So the transition probability matrix $P$ has exactly one (lets say it is the first one) column with a $1$ and the ...
0
votes
0answers
16 views

Solution to linear system around the topic of Markov-chains

Let $(X_n)_{n\geq 0}$ be a Markov-chain with the state space $S$ and transition matrix $P=(p_{xy})_{x, y \in S}$. For $A\subset S$ be $H^A:=\inf\{n = 0, 1, \dots | X_n \in A\}$ the first visit time ...
0
votes
1answer
30 views

Are random walk variations Markov-Chains?

Let $S_{n}:= S_0 + \sum_{i=1}^{n}X_i$ be a simple random walk, $X_i$ are independent random variables with $P[X_i=1] = p, P[X_i = -1] = 1-p$. Let $M_n:=\max\{S_0, \dots, S_n\}$. The task at hand is ...
0
votes
1answer
25 views

Writing down the transition matrix of a discrete Markov chain

Please consider the following scenario: One person is walking along a discrete circle induced by $\mathbb{Z}/n\mathbb{Z}$ In each round we roll a dice with $w\in\left\{2,\ldots n\right\}$ sides If ...
0
votes
1answer
30 views

A discrete time Markov chain with such a transient state that $\mathbb P(T_i<\infty \ | \ X_0=i) \neq 0$

All examples of discrete time Markov chains my text provides are where $S$ is finite, and as far as I can tell, it makes all transient states have $$\mathbb P(T_i<\infty \ | \ X_0=i) = 0.$$ Are ...
0
votes
1answer
25 views

Why does this hold for the mean hitting time?

Let $X$ be a Markov chain and $T_A$ the hitting time. My text uses this in a proof: $$\mathbb E[T_A \ | \ X_0=k ] = \sum_{l\in S} \mathbb P(X_1=l \ | X_0=k\ )(1+\mathbb E[T_A \ | \ X_0=l ])$$ and I ...
0
votes
1answer
30 views

Inferring transition rates from continuous markov chain question

A house has 2 rooms of similar sizes with identical air conditioners equipped with thermostats which turn on and off as needed to maintain the temperature in each room to a desired level of 22 ...
1
vote
1answer
59 views

random walk with sticky barriers

Consider a random walk on the line 1,...,d. You start at point 1. At each step you flip a coin: heads means go left, tails means go right. If you're at 1 and get a heads, just stay where you are ...
0
votes
0answers
22 views

Transition matrix in left-right hidden semi-Markov model

I'm developing a hidden semi-Markov model left-right . In a left-right model a sequence of $M$ states starts in state $1$ and ends in state $M$, with no repetition of states. Since the model is ...
1
vote
2answers
23 views

Can the ergodic theorem for Markov chains be proved with linear algebra?

This theorem is in my book, let me just say that it is for discrete-time Markov chains, that are time-homogeneous. Ergodic is defined in the book as being positive recurrent and aperiodic. The ...
0
votes
0answers
30 views

Probability of not reaching completion in Markov process

This question is supposed to be easy but is very hard for me. The Norwegian Skating Association has mass produced certain "collectors' cards" with all $N$ speedskaters (Norwegian as well as ...
0
votes
1answer
36 views

Markov chain exercise

Hello i have this Markov chain exercise: Basically we can always move up 1 step, but there is always a possibility that we will go down to the first state 0, the Markov chain consists of N states. ...
5
votes
1answer
90 views

Compute a probability in Random Walk by Martingales

Let $X_n$ be the state at time $n$ of a Markov chain with these transition probabilities : $$p_{i,i+1}=p_i\qquad,\qquad p_{i,i-1}=q_i=1-p_i$$ $(a)$ Show that $Z_n=g(X_n)\,;\,n\geq0$, is a ...
1
vote
0answers
18 views

Markov chains mixing time

Informally, the mixing time of a Markov chain is the time it takes to reach “nearly uniform” distribution from any arbitrary starting distribution. What does it mean by nearly uniform? I hope some one ...
0
votes
0answers
30 views

Perron Frobenius Theorem and Markov chains and more

I came across few ways of calculating convergence rates of Markov chains but I am a bit confused as to how these differ from each other and what may be the best way to calculate. The second ...
2
votes
1answer
63 views

Monotonicity and Convexity of Stochastic Matrices

The definition of stochastic monotonicity and convexity is given by "Stochastic Orders and Their Applications" by Moshe Shaked and George Shanthikumar (1994) as: Let $P = \{p_{i,j} \}$ be a ...
1
vote
2answers
50 views

Convergence time of a Markov chain

We know that a regular Markov chains converges to a unique matrix. The convergence time maybe finite or infinite. My interest is in the case where the convergence time is finite. How can we accurately ...
0
votes
3answers
89 views

How fast does this Markov chain converge?

Observe the above a Markov chain and limiting matrix of it. Finding the limiting matrix if it exists is easy but I am curious as to how fast this given matrix converges to its limiting matrix. Is ...
1
vote
1answer
128 views

Probability of a trajectory in Markov processes

I need help with a simple formula! (My question is taken from here, pag 26 eq 1.112. ) Consider a Markov Process with associated Master Equation: \begin{equation*} ...
1
vote
0answers
15 views

Changes in the transition matrix of a Markov chain

In most or all Markov chain theories that I know of assumes that the transition matrix does not change over time. But what if certain changes are expected to occur at certain times in the transition ...
1
vote
1answer
28 views

Optimization of a Sum of Variables

Let there be variables $A$, $B$, $C$, $D$, and $E$ such that a total of $N$ points is allocated among the variables: $A$+$B$+$C$+$D$+$E$=$N$, $N$∈$ℝ$. Let the corresponding point values returned by ...
1
vote
0answers
43 views

Problem with stochastic processes book - should I switch.

I've been reading "Essentials of Stochastic Processes" (second edition) by "Richard Durrett" and I quite liked it, it's a nice size book and it's very easy to read. However, and this is quite a big ...
1
vote
1answer
45 views

Continuous-time Markov Chain forward/backward equations and MLE

I have two questions: 1) Using Kolmogorov's forward and backward equations, show that $p_{11}(t) + p_{21}(t) + p_{31}(t) = 1$ and $p_{21}(t) = p_{31}(t)$ where $p_{ij}(t) = P(X(t) = j | X(0) = i)$. ...
0
votes
0answers
16 views

discrete time Markov chain, difference between absorbing and recurrent classes.

In a discrete time Markov chain, are there any differences between an absorbing and a recurrent class? Recurrence is that we with probability 1 will reenter a state that we are in, this is a class ...
1
vote
1answer
34 views

Ehrenfest urn model expectation question

Consider the Ehrenfest urn model in which $M$ molecules are distributed between two urns, and at each time point one of the molecules is chosen at random and is then removed from its urn and placed in ...
2
votes
0answers
57 views

What does a customer see when it begins to be served in $M/M/1$ queue?

In queueing theory, the PASTA (Poisson Arrivals See Time Averages) principle [wiki] justifies $a_n = P_n$ where $$a_n = \text{proportion of customers that find } n \text{ in the system when they ...
2
votes
1answer
62 views

Inverse of a regular stochastic matrix

Is it true that the inverse of a regular stochastic matrix is also regular? Are there any other interesting features that the inverse may have of a regular stochastic matrix? Hope someone could answer ...
1
vote
0answers
33 views

References for time-inhomogeneous Markov jump processes?

In some central models in life insurance mathematics, the state of the insured is modeled using a continuous-time time-inhomogeneous Markov process with finitely many states. While many results for ...
0
votes
0answers
36 views

relations between properties of stochastic processess

If we have an integer valued stochastic process, are these implications correct? independent increments $\rightarrow$ Markov property Markov property $\nrightarrow$ independent increments stationary ...