Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
12 views

Differences of Markov chain is Markov

In my studies of Markov chains, I was tackled with this tough problem: Let $ \{ X_n \}_{n=0}^{\infty} $ be a homogeneous Markov chain with transition probabilities satisfying $ | i-j | > 1 \to ...
0
votes
0answers
6 views

Drift analysis of an absorbing Markov chain

Consider a set $S$, and suppose we have a sequence of random subsets $$ \zeta_t = \{x_1, \dots, x_n\} $$ for $x_1, \dots, x_n \in S$. We do not know with which probability density the points of each $\...
0
votes
0answers
8 views

Basic Limit Theorem for Markov Chain (Knowing the odds)

In the book "Knowing the Odds", Basic Limit Theorem for Markov Chain is stated as follows. Theorem 7.41 (Basic Limit Theorem). Suppose j is a recurrent aperiodic state in an irreducible Markov chain....
0
votes
1answer
22 views

What is the difference between a reversible markov chain and a reversible in equilibrium markov chain?

In the text I'm using it says: Let X = {$X_n : 0 \leq n \leq N$} be an irreducible Markov chain such that $X_n$ has the stationary distribution $\pi$ for all $n$. The chain is called reversible if ...
-2
votes
0answers
19 views

Communicating classes of a power of the irreducible transition matrix? [on hold]

Suppose $P$ is an irreducible transition matrix, with period $d$. Consider the transition matrix $P^k$. In terms of $d$ and $k$, how many communicating classes does $P^k$ have, and what is the period ...
0
votes
0answers
15 views

stopping criteria for power-iteration to find rank-1 matrix

I start with B=I, A positive matrix, and compute B=(BA)/norm(B) by iterating until B is sufficiently close to rank-1 matrix. What is a good stopping criterion for this algorithm? There's Birkhoff ...
-1
votes
0answers
12 views

Simple Markov property on stopping times

Suppose $(S_n)_{n\geq1}$ is a Markov chain on the two dimensional lattice of the integers. Then define the stopping time $\tau_A'=\inf\{n\geq1:S_n\notin A'\}$ and consider the following for $A\subset ...
-1
votes
0answers
31 views

Entropy and Markov chain [on hold]

Assume that $X_n$ is a discrete Markov chain and $H$ is entropy function. I want to prove $$H\left(X_0\mid X_n\right) \geq H\left(X_0\mid X_{n-1}\right)$$ but I have no idea how to prove it. please ...
1
vote
1answer
25 views

Show that it is a Markov chain, determine the transition-probability matrix and reversibility

Certain machine has three possible states: $0=working,\,1=broken\,and\, awaiting\,repair,\,2=broken\,and\,being\,repaired$. The permanence times (in minutes) in each state have independent geometric ...
4
votes
2answers
33 views

Distribution of throws of die rigged to never produce twice in a row the same result

A die is “fixed” so that each time it is rolled the score cannot be the same as the preceding score, all other scores having probablity 1/5. If the first score is 6, what is the probability that the ...
0
votes
1answer
18 views

Random walk mean number of visits to state before absorption

This is from Stirzaker's book Random Processes. Suppose we have a simple random walk with probability going "up" p, "down" q. At time 0 it stats at 0, so $$S_0 = 0$$ Now let $u_b $ be the mean ...
1
vote
0answers
20 views

Extension of erdodic theorem with WLLN

Suppose you have a ergodic (or irreducible) Markov chain $(A_t)_{t\geq0}$ in continuous time. denote by $\pi$ the invariant distribution of $A$. If $f$ is a function of $A_s$ which is integrable w.r.t....
-1
votes
0answers
33 views

Determining the infinitesimal generator of a Markov chain [closed]

The infinitesimal generator of a Markov chain $X$ on a countable state space $S$ is defined by $$A(f)(x)=\lim_{t\downarrow 0} \frac{E^x(f(X_t))-f(x)}{t}.$$ Are there any ways of working out $A$ ...
1
vote
2answers
69 views

Expected steps to eliminate a character? [closed]

This came up in a game theory crafting exercise. Imagine a character has 195 hit points. You can shoot at them and there are three results: Critical - 100 damage - 40% of shots Body - 50 damage -...
1
vote
0answers
17 views

First time of passage, discrete random walk with disjoint absorbing regions

I have a sum $T^i$ of zero/one Bern$(p)$ random variables $T_i$ and multiple disjoint absorbing regions, i.e. the absorbing region is a union of disjoint, closed sets: $$T^i \in \bigcup_{u \in \...
1
vote
0answers
10 views

Filtering/MCMC methods for this HMM

I have a Discrete HMM with hidden Markovian signals of the form $\{X_t\}_{t \in [0, \infty)} \in \{ 1,2,3\}$ and observed outputs of the form $\{Y_n\}_{n \in \mathbb{N}} \in \{ 1,2\}$. Each ...
0
votes
0answers
30 views

Application Strong Markov Property

I am considering a random walk $S_n$ on a state space $\mathbb{Z}^d$. I want to show that $E_x\left[\sum_{n=0}^{\tau_A-1}{1_{\{S_n=y\}}}\right]=\frac{1}{P_x[\tau_A<\tau_y]}$, where $\tau_A=\inf\{n\...
1
vote
1answer
30 views

Steady state distribution needed

I have a chain $C_t$. At every instant $t$ an exponential random variable $X_t$ with parameter $\lambda$ is added to the chain or if the chain has a value greater than $Q$ then a value $Q$ is ...
-1
votes
0answers
16 views

how to find transition matrix given initial state and state after $n$ steps

Is it possible to reverse-engineer the transition matrix of a Markov Chain, knowing only the initial state, and the state after a number of steps? I realize that there may be an infinite number of ...
1
vote
0answers
33 views

Better to go first in Snakes and Ladders?

We consider the game as described in http://www.datagenetics.com/blog/november12011/ . Each person rolls a dice and the person who gets 6 on the face can start and the other keeps waiting. If the ...
-1
votes
0answers
14 views

how to check if a process satisfies the markovian property with continuous time?

as an example we have A source transmitting messages is alternately on and off. The off-times are independent random variables having a common exponential distribution with rate α and the on-...
0
votes
0answers
15 views

Is there a way to “normalise” probability values of Markov chain transitions for comparison?

Suppose I have a series of states and I've a database of frequencies (probabilities) of the states transiting from one to another. I've a set of states ${A...G}$. Let's say I've a state transition (...
-1
votes
0answers
46 views

Computing $\mathbb{E}[Z_n\mid Z_0=1]$ for a branching process [closed]

I came across a question whilst revising material to do with Branching processes. I am looking for help with part $(iii)$, here is my working: \begin{align*}&\mathbb{E}[Z_n\mid Z_0=1]\\&=\...
0
votes
0answers
17 views

Stationary distribution of finite-state Markov chain in terms of determinants/products of eigenvalues

I have an $M$-state continuous-time Markov chain with transition-rate matrix $K$ (the column sums are zero), which has $M$ distinct eigenvalues $\lambda_i$, $i=1,\dots,M$. $\lambda_M=0$, so $K$ has ...
0
votes
1answer
31 views

Equilibrium distribution exponentially fast

I need to prove that for an aperiodic, irreducible Markov Chain $X_n$ with stationary distribution $\pi$ holds that $P_x[X_n=j]\to\pi(j)$ exponentially fast. I found some proof of that statement but ...
1
vote
0answers
92 views

Probability of losing lotteries needed

A person earns $x_i$ amount of money every month where $x_i$ is an exponential random variable with parameter $\lambda_1$. The amount $x_i(1-p)y$, here $0 \leq p\leq 1$ and $y$ is exponentially ...
2
votes
1answer
40 views

Bayesian Estimator and Markov Chains

This is Exercise 6.1.14 from Dembo's notes found here. At this point, we are just beginning a discussion of Markov chains. I have no prior experience with estimators and so I am a bit lost with this ...
0
votes
1answer
27 views

Check that stopping time is a.s. finite

I have the following situation. Let $(X_i)_{i\geq1}$ be a sequence of iid random variables in $\mathbb{Z}$ and consider the random walk $S_n=\sum_{i=1}^n{X_i}$, $S_0=x$. Let $y>x$ and consider ...
2
votes
1answer
51 views

Find the mean number of steps in a Markov chain

Let $S = \{1,2,3,4\}$ be a state space like this $$\begin{array}\\ 1 & - & 2 \\ | & & |\\ 3 & - & 4 \end{array} $$ and let $P$ be the transition matrix given by $$P = \begin{...
0
votes
0answers
30 views

Error term in the definition of the transition rates of a continuous time Markov chain

I'm studying G.F.Lawler's stochastic process book. There he defines the transition rates $\alpha(x,y)$ from the state $x$ to state $y$ (the state space is countable) of a continuous time Markov chain $...
0
votes
1answer
35 views

Stopping times of random walk with time dependent absorbing barriers

I have a Bern$(p)$ random walk ($Y_i = 1$ with probability $p$ and Y_i = 0 with $1-p$) with two absorbing boundaries, $A: Y^i \leq t_i$ and $B:Y^i \geq d_i-t_i$. Now, both $d_i$ and $t_i$ are evolving ...
0
votes
2answers
45 views

Recurrence Relation with two parameters and Summation

This is a recurrence relation with two parameters which came up in a problem I was trying to solve. Given $$\begin{align}&A_n=pB_{n-1};\qquad &&B_n=q(A_{n-1}+B_{n-1})\\ &A_4=p; \...
0
votes
0answers
23 views

HMM limiting distribution

Consider a hidden markov model (HMM) with two hidden states $A$ and $B$ and emission support $1$ and $2$ fitted with initial state distribution $$\lambda = [\begin{array}{cc} .7&.3\end{array}]$$ ...
0
votes
1answer
22 views

Long-run fraction Markov Chains

A machine has three critical parts (1,2,3) but can function as long as two of these parts are functional. When two are broken, they are replaced and the machine is functional the next day. The state ...
1
vote
2answers
37 views

General Two-State Markov Chain: $P(X_{n}=1)=\frac{b}{a+b}+(1-a-b)^n \big(P(X_0=1)-\frac{b}{a+b}\big)$

Consider a general chain with the state space $S=\{1,2\}$ and write the transition probability as $$\begin{pmatrix} 1-a&a\\ b&1-b\end{pmatrix}$$ Use the Markov property to show that $$P(X_{n}=...
0
votes
0answers
36 views

Lebesgue Integral vs. Lebesgue Stieltjes Integral

Forgive me if this has been addressed in a question already on here (and for my lack of comfort with measure theory), but is there any difference between the Lebesgue integral and the Lebesgue-...
0
votes
0answers
22 views

implementing particle filters without a priory distribution

i am implrmrnting the particle filter, and i have some problem understanding the algorithm. given the state equations: $$ x_k = f(x_{k-1},v_k) $$ $$ z_k=h(x_k,u_k) $$ where $v_k, u_k$ are process ...
1
vote
1answer
26 views

Let $\mathbf{X}$ be a Markov chain on a square find $p_{1,1}(n)$

Consider a square like this $$\begin{array}\\ 1 & - & 2\\ | & & |\\ 3 & - & 4 \end{array} $$ such that you can go from each state with chance $\tfrac{1}{2}$ to the ...
0
votes
1answer
30 views

Why is this Markov chain's stationary distribution not (1/2, 0, 0, 1/2)?

I have the Markov chain 1, 0, 0, 0 1/2, 0, 1/2, 0 0, 1/2, 0, 1/2 0, 0, 0, 1 I understand how to build the system with which I am supposed to find that ...
1
vote
2answers
33 views

Help on some results with Markov chains

I currently have a markov chain represented by the following matrix: $$\left[\begin{array}{ccc} 0&1&0 \\ .99&0&.01 \\ 0&0&1 \end{array}\right],$$ i.e. a row-stochastic matrix, ...
-1
votes
0answers
22 views

Find a formula for $\mathbb{E}(T)$ in terms of $p_{11}$. [closed]

Let $X_0, X_1, X_2,\dots$ be a homogeneous Markov chain with state space $S=\{s_1,\dots,s_n\}$, and transition matrix $P=(p_{ij})$. Suppose further that the chain starts at the initial state $s_1$. ...
2
votes
1answer
34 views

How does a Markov process inherit its homogeneity to the embedded Markov chain?

A homogenous Markov process $\lbrace X(t),t\geq 0\rbrace $ is given and the embedded Markov chain $Y_0,Y_1,\ldots$ is defined as $Y_n:=X(T_n)$, where the $0=T_0<T_1<\ldots$ are the moments where ...
0
votes
0answers
15 views

Started Counts method for estimating transition probabilities of a discrete time markov chain

I would be very pleased if you could help me with a problem I'm having for my Bachelor's thesis. I'm working on some inventory forecasting methods and one of the method's I'd like to apply is a method ...
3
votes
0answers
30 views

Distribution on number of revisits in past $k$ steps of Markov chain

Consider a finite-state Markov chain with transition matrix $P$. The chain starts in a state chosen uniformly over all the states and runs indefinitely from there. We're going to examine only the $k ...
1
vote
1answer
18 views

Conditioning on invariant sigma algebra with respect to ergodic measure

So this question arose to me while applying the Ergodic theorem. If $X$ is a finite state (in $ \{1,\dots,d\}$) continuous-time Markov chain, which is ergodic, then $X$ has a unique invariant ...
2
votes
0answers
55 views

Is this transformation of a Markov process again Markovian?

Let $(X_t)_{t\in\mathbb{N}_0}$ be a stationary Markov process valued in $\mathbb{R}$ and $c\in\mathbb{R}$. Is the process $(Y_t)_{t\in\mathbb{N}_0}$ defined by $$ Y_t={\bf 1}{(X_t<c)} $$ again a ...