A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
0answers
9 views

How to calculate mean of busy-cylce

Let $\{X(t)\}$ be birth-death process on a finite state {0,1,2} with non negative birth rates $(\lambda_0,\lambda_1)$and death rates $(\mu_1,\mu_2)$. Suppose $\mathbb{P}(X(0)=0)=1$ and $s_0=\inf ...
0
votes
0answers
8 views

How to find expectation of birth-death process

Let $\{X(t)\}$ be birth-death process on two-state space {0,1}. Let birth rate $\lambda=2$ and death rate $\mu=12$. How to calculate $\lim_{t\to\infty} \mathbb{E}[X(t)]$?
2
votes
0answers
47 views

Norris exercise: Showing $P_0[X_n\neq0\forall n\geq1]=6/\pi^2$

Consider exercise 1.3.4 of Norris' Markov Chains. The question is as follows: Let $\{X_n\}_{n\geq 0}$ be a Markov Chain with state space $S=\{0,1,2,\dots\}$. Suppose the transition probabilities ...
1
vote
0answers
28 views

The patterns of markov chain [on hold]

I have one question let $X$ be a Markov chain that could take the values $1,2$ or $3$ with the same probability $1/3$. what is the probability that $(1,2,1)$ pattern occurs sooner than $(2,1,3)$?
2
votes
0answers
23 views

Moving Average of an Ergodic Markov processes.

Let $ \{X(t); t\geq 0\} $ be an ergodic Markov process, and let G be a positive integrable function with $\int_0^\infty G(x)dx=1$. Does $$F(t)\doteq G*X(t) = \int_0^t G(t-s)X(s)ds $$ converge in ...
0
votes
0answers
31 views

Identity for return times in continuous Markov chain

I need help with this problem about return times in continuous time Markov chains: We are given a continuous time Markov chain $(X_t)$. Define $T_i=\inf{\{t>0; X_t=i, X_{t-} \neq i\}}$, which ...
0
votes
0answers
10 views

Question about a type of continuous state Markov-process.

I was wondering if anyone knows whether a particular result has been proven or is indeed true. My problem is as follows. Suppose I have a stationary, ergodic Markov chain that follows a process of ...
3
votes
1answer
42 views

Rigorous argument of the Markov property used in discrete-time Markov chains

I am reading an example related to discrete-time Markov chains which I do not really understand rigorously. Suppose that $\{ X_n \}_{n \in \mathbb{N} }$ is a time-homogeneous discrete-time Markov ...
1
vote
0answers
15 views

non-ordered chapman kolmogorov equation

If $p(x)$ is a probability density one would normally have $\int dx_1 p(x_2|x_1)p(x_1)=p(x_2)$ However is there a straightforward way to interpret the following: $\int dx_1 ...
1
vote
0answers
26 views

An example of Markov chain with no closed class?

What is an example of Markov chain with no closed communicating class? Closed class means that once we are in that class, there would be no escape from it. I am thinking that an example would be ...
0
votes
0answers
20 views

For which p is the Markov chain recurrent (almost random walk)

We have a Markov chain on Z with matrix: $p_{ii+1}=p=1-p_{ii-1}$ for $i\leqslant-1$, $p_{ii-1}=p=1-p_{ii+1}$ for $i\geqslant1$, and $p_{00}=p_{01}=p_{-10}=\frac{1}{3}$. For which values of p ...
1
vote
1answer
18 views

Equality of probability of finite hitting time for irreducible states in Markov Chain

Suppose I have a finite state Markov Chain with state space $S=\{1,2,3,4,5,6\}$. Suppose I further have that $\{1,2\}$,$\{3,4\}$ and $\{5,6\}$ are irreducible classes where $\{1,2\}$ and $\{3,4\}$ are ...
0
votes
0answers
14 views

Deriving Time of extintion of a Small neural Network

I'm trying to derive the Expected Value of the Time of Extintion $\tau_{ext}$ of a small Neural Stochastic Network with the following dynamics, where I consider $\tau_{ext}$ to be the time of the last ...
0
votes
1answer
32 views

Show that $m_{ii}=\infty$ when $i$ is transient

Show that $m_{ii}=\infty$ when $i$ is transient, where $m_{ii}$ is the mean time to get from $i$ to $i$. if $i$ is transient I know that there is a positive probability of going to some ...
1
vote
1answer
28 views

Cont Time Markov Chains. Stationary Probability

A barber finishes haircuts at rate $3$, measured in hours, so on average it takes him 20 minutes to cut a person’s hair. Customers arrive at rate 2. There is, however, only a two chair waiting ...
1
vote
0answers
15 views

Applying Markov Decision Processes to an arrival forecasting problem

I have the following problem and I'd like to know if it's something that was already studied in the literature or not. I'm not sure about the naming conventions either. I have a system $S$ that can ...
0
votes
0answers
13 views

Equivalence between Label propagation and Likelihood estimation over a Markov Random Field

On page 13/14 of label propagation the near equivalence is set up between what they call 33/34 (reproduced below). This near equivalence is not obvious to me. $$P_{F'}(Y) = \frac{1}{Z} \exp \left( ...
0
votes
0answers
22 views

Conditional expectation in continous markov chains

I am trying to understand the double integral in calculating the conditional expectation. In calculating $V_i$, the second and third equalities are due to the law of total probability. I have the ...
1
vote
1answer
23 views

How to discretely stochastically simulate a continuous-time Markov chain?

A continuous-time markov chain describes a continuously varying process, such that future state only depends on the current state. A sampling of a continuous markov chain can be described in terms of ...
0
votes
0answers
26 views

Density function of absorption time in this Markov Chain

Let $X_t$ be a continuous time Markov Chain with state space $\{1,2,3\}$ with the following transition matrix: $$\left( \begin{matrix} -(\lambda+\delta) & \lambda & \delta \\ \mu & ...
2
votes
1answer
52 views

Markov Chain: flip coin 8 times and get 3 consecutive heads

I have confusion while reading the following example in the course material. Q: In a sequence of independent flips of a fair coin, let N denote the number of flips until there is a run of three ...
0
votes
2answers
64 views

Markov Chain: flip 8 coins and get 3 consecutive heads

I was reading the material and I am confused at the following example. Q: In a sequence of independent flips of a fair coin, let N denote the number of flips until there is a run of three ...
0
votes
0answers
10 views

Direct proof of the existence of optimal memoryless deterministic policies in MDP

It is well known that (finite-state, finite-action, discrete time) MDPs admit an optimal policy that is memoryless and deterministic (sometimes called pure). The proof of this fact for ...
0
votes
1answer
19 views

Mean time for the trajectory. Find mean

What is the mean of time when the trajectory of the wiener process, $W_t$, is over the line $y=t$? We need to find $\Bbb{E}\tau$, where $\tau=\sum\limits_{a,b:\forall t\in(a,b) ; ...
0
votes
0answers
15 views

HMM optimization: Lagrange multiplier problem

In David Barber's textbook "Bayesian Reasoning and Machine Learning" he hints at the derivation of the Baum-Welch algorithm for HMM parameter learning: Textbook excerpt, (cannot include images yet, ...
1
vote
1answer
30 views

Strong Markov property with two stopping times

I have a diffusion $X=(X_t)_{t\ge0}$ and a stopping time $\tau$. From the strong Markov property I know that for any time $t\ge0$ (or a random time independent of $X$) I get that ...
0
votes
0answers
11 views

Examples of state-space models that show strong homeostasis but also substantial change after critical threshold?

The question is, can can anyone provide examples of systems or math models that exhibit patterns of homeostasis but which can be exhibit substantial transitions or bifurcations after some critical ...
2
votes
1answer
29 views

How to find Kolmogorov Forward Equations, given generator matrix Q?

I am having difficulty in forming Kolmogorov Forward Equations. I understand how the KFE is derived and that $$\frac {d}{ds} p_{ij} (s) = \sum_{k \neq j} p_{ik} (s) \lambda_{k} r_{kj} - p_{ij} ...
1
vote
0answers
54 views

Does a Markov process have memory zero?

I have the following question: The Markov process with two states and a transition matrix $$P =\begin{pmatrix} 0.3 & 0.7 \\ 0.3 & 0.7 \end{pmatrix}$$ has memory zero. Is it true? My ...
2
votes
0answers
24 views

Good introductory book coupling methods

I am very interested in coupling methods, can you recommend me a good introductory books on this subject? Thanks
1
vote
0answers
40 views

Birth and death process. Total time spent in state i.

Question: Let $X(t)$ be a birth-death process with $\lambda_n = \lambda > 0$ and $\mu_n = \mu > 0,$ where $\lambda > \mu$ and $X(0) = 0$. Show that the total time $T_i$ spent in state $i$ is ...
1
vote
0answers
37 views

Gambler's ruin problem - expected time

I have troubles seeing the following. Consider the classical gambler's ruin problem, betting 1 at each time $t\in \mathbb{N}$, and losing or winning -1 respectively +1 at each time till the fortune of ...
2
votes
0answers
37 views

Markov chain limit problem

Let $X_n$ be a Markov chain on a countable state space, $\mathbb{S}$. Let $N_n(x) = \sum_{k=1}^n\mathbb{1}_{\{X_k=x\}}$ denote the number of times the chain visits state $x\in \mathbb{S}$. Let ...
0
votes
1answer
65 views

Renewal process with bimodal times

Suppose we have a stochastic process $X_t$ of a light using a single light bulb. When the light bulb burns out it is immedieatly replaced with a new one. Suppose that the time between failures is ...
0
votes
1answer
39 views

Has a Markov chain in compact metric space a stationary distribution (possibly non-unique)?

Let $Y_n$ be a Markov chain in a compact metric space. Is it true that it has a stationary distribution?
1
vote
0answers
35 views

Markov chain of transition probabilities

Let $P$ be a transition matrix on a discrete state space with $N$ elements. $P_{i,j}$ is the probability of going from state $i$ to state $j$. Let $\pi$ be the stationary distribution. Let $\{X_n\}$ ...
0
votes
0answers
19 views

steady state distribution of the following Markov jump process?

Consider a queueing process with the following rate transition matrix: $\mathbf{P}=\left( \begin{smallmatrix} -\lambda & \lambda & & & & & & &\\ \mu & ...
1
vote
1answer
26 views

Markov chain with dynamic higher orders

Let $X_i$ be the node visited by a random walk at step $i$, and the following equations be the transition probabilities. $Pr(X_n = x_n | X_{n-1} = x_{n-1}, \cdots, X_1 = x_1) = Pr(X_n = x_n | X_{n-1} ...
1
vote
1answer
27 views

optimal utility calculation for a simple discrete Markov chain

I am trying to calculate analytically the optimal decision rule for a simple discrete markov chain, following standard decision theory framework (slide 17 in ...
1
vote
1answer
76 views

Show that the probability of never hitting 0 on a birth-death chain is $6/\pi^2$.

In the question we have a birth-death chain on $\{0,1,2,...\}$ whose only non-zero transitions from $i$ are to $i+1$ and $i-1$, with probabilities $p_i$ and $q_i$, respectively. I have that $p_i$ and ...
2
votes
1answer
42 views

Markov Process: predict the weather using a stochastic matrix

I have the following stochastic matrix $$ P = \begin{pmatrix} P(S \mid S) = 0.5 & P(F \mid S) = 0.2 & P(R \mid S) = 0.3 \\ P(S \mid F) = 0.2 & P(F \mid F) = 0.7 & P(R \mid F) = 0.1 ...
1
vote
1answer
77 views

A property of Poisson process

Let $Y_t$ be a centered Poisson process, why \begin{equation} \lim_{n \to \infty} \sup_{s<t} |n^{-1}Y(ns)| = 0 \qquad a.s. \qquad \forall t\ge 0 \end{equation} This is a fundamental step in the ...
5
votes
1answer
70 views

Constructing Martingales from Markov Processes

I know that for a Markov process $X_t$ with generator $L$ and $f,f^2\in D(L)$, $$M_t=f(X_t)-\int_0^t Lf(X_s)\ ds$$ is a martingale (w.r.t. $P^x$). And I want to show that $$M_t^2-\int_0^t ...
0
votes
0answers
74 views

Open Jackson network expected customers and distribution

I have a stochastic modelling test tomorrow, I'm stuck on one practice question. We have an open Jackson network which is as follows: Arrivals in queue 1 are a Poisson process with rate $\lambda$. ...
2
votes
1answer
45 views

Find the steady state probability that both A and B catch a headache.

I have a question about Markov chain. Let A and B be patients, A has headache at the rate $1$ times/week and recovers from it at rate of $2$ times/week. The patient B has it at the rates $2$ and $4$ ...
0
votes
0answers
46 views

hitting time for a continuous time markov chain

Brad’s relationship with his girl friend Angelina changes between Amorous, Bickering, Confusion, and Depression according to the following transition rates when t is the time in months. They are ...
0
votes
1answer
63 views

Solve a problem using Markov chains

We have the following problem: At the beginning of every year, a gardener classifies his soil based on its quality: it's either good, mediocre or bad. Assume that the classification of the soil ...
0
votes
1answer
44 views

Probability distribution in 7th steps

Let's assume that there is a markov chain with a transition matrix $P$: $\begin{bmatrix} 0 &0 &0 &\frac{1}{2} & \frac{1}{2} & 0\\ 0& 0& 0& \frac{1}{2}& ...
1
vote
1answer
18 views

Understanding stochastic matrices

We start a game with 2 euros, i.e. at time 0 we have 2 euros. At time $t=1,2,...$ we play a game with a stake of 1 euro and with odds of winning $p$ (hence odds of losing $1-p$). We define $X_t$ at ...
0
votes
0answers
18 views

Stochastic calculus for continuous time Markov chains

I have absolved a course on stochastic analysis, i.e. integrals with respect to the brownian motion. Now I know that there is a theory of stochastic calculus for diskrete matringales, however I was ...