A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
1answer
23 views

Mean time for the trajectory. Find mean

What is the mean of time when the trajectory of the wiener process, $W_t$, is over the line $y=t$? We need to find $\Bbb{E}\tau$, where $\tau=\sum\limits_{a,b:\forall t\in(a,b) ; ...
0
votes
0answers
19 views

HMM optimization: Lagrange multiplier problem

In David Barber's textbook "Bayesian Reasoning and Machine Learning" he hints at the derivation of the Baum-Welch algorithm for HMM parameter learning: Textbook excerpt, (cannot include images yet, ...
1
vote
1answer
31 views

Strong Markov property with two stopping times

I have a diffusion $X=(X_t)_{t\ge0}$ and a stopping time $\tau$. From the strong Markov property I know that for any time $t\ge0$ (or a random time independent of $X$) I get that ...
0
votes
0answers
16 views

Examples of state-space models that show strong homeostasis but also substantial change after critical threshold?

The question is, can can anyone provide examples of systems or math models that exhibit patterns of homeostasis but which can be exhibit substantial transitions or bifurcations after some critical ...
2
votes
1answer
47 views

How to find Kolmogorov Forward Equations, given generator matrix Q?

I am having difficulty in forming Kolmogorov Forward Equations. I understand how the KFE is derived and that $$\frac {d}{ds} p_{ij} (s) = \sum_{k \neq j} p_{ik} (s) \lambda_{k} r_{kj} - p_{ij} ...
1
vote
0answers
64 views

Does a Markov process have memory zero?

I have the following question: The Markov process with two states and a transition matrix $$P =\begin{pmatrix} 0.3 & 0.7 \\ 0.3 & 0.7 \end{pmatrix}$$ has memory zero. Is it true? My ...
2
votes
0answers
28 views

Good introductory book coupling methods

I am very interested in coupling methods, can you recommend me a good introductory books on this subject? Thanks
2
votes
0answers
64 views

Gambler's ruin problem - expected time

I have troubles seeing the following. Consider the classical gambler's ruin problem, betting 1 at each time $t\in \mathbb{N}$, and losing or winning -1 respectively +1 at each time till the fortune of ...
2
votes
0answers
41 views

Markov chain limit problem

Let $X_n$ be a Markov chain on a countable state space, $\mathbb{S}$. Let $N_n(x) = \sum_{k=1}^n\mathbb{1}_{\{X_k=x\}}$ denote the number of times the chain visits state $x\in \mathbb{S}$. Let ...
0
votes
1answer
72 views

Renewal process with bimodal times

Suppose we have a stochastic process $X_t$ of a light using a single light bulb. When the light bulb burns out it is immedieatly replaced with a new one. Suppose that the time between failures is ...
0
votes
1answer
42 views

Has a Markov chain in compact metric space a stationary distribution (possibly non-unique)?

Let $Y_n$ be a Markov chain in a compact metric space. Is it true that it has a stationary distribution?
1
vote
0answers
43 views

Markov chain of transition probabilities

Let $P$ be a transition matrix on a discrete state space with $N$ elements. $P_{i,j}$ is the probability of going from state $i$ to state $j$. Let $\pi$ be the stationary distribution. Let $\{X_n\}$ ...
0
votes
0answers
23 views

steady state distribution of the following Markov jump process?

Consider a queueing process with the following rate transition matrix: $\mathbf{P}=\left( \begin{smallmatrix} -\lambda & \lambda & & & & & & &\\ \mu & ...
1
vote
1answer
29 views

Markov chain with dynamic higher orders

Let $X_i$ be the node visited by a random walk at step $i$, and the following equations be the transition probabilities. $Pr(X_n = x_n | X_{n-1} = x_{n-1}, \cdots, X_1 = x_1) = Pr(X_n = x_n | X_{n-1} ...
1
vote
1answer
35 views

optimal utility calculation for a simple discrete Markov chain

I am trying to calculate analytically the optimal decision rule for a simple discrete markov chain, following standard decision theory framework (slide 17 in ...
1
vote
1answer
90 views

Show that the probability of never hitting 0 on a birth-death chain is $6/\pi^2$.

In the question we have a birth-death chain on $\{0,1,2,...\}$ whose only non-zero transitions from $i$ are to $i+1$ and $i-1$, with probabilities $p_i$ and $q_i$, respectively. I have that $p_i$ and ...
2
votes
1answer
61 views

Markov Process: predict the weather using a stochastic matrix

I have the following stochastic matrix $$ P = \begin{pmatrix} P(S \mid S) = 0.5 & P(F \mid S) = 0.2 & P(R \mid S) = 0.3 \\ P(S \mid F) = 0.2 & P(F \mid F) = 0.7 & P(R \mid F) = 0.1 ...
1
vote
1answer
81 views

A property of Poisson process

Let $Y_t$ be a centered Poisson process, why \begin{equation} \lim_{n \to \infty} \sup_{s<t} |n^{-1}Y(ns)| = 0 \qquad a.s. \qquad \forall t\ge 0 \end{equation} This is a fundamental step in the ...
5
votes
1answer
93 views

Constructing Martingales from Markov Processes

I know that for a Markov process $X_t$ with generator $L$ and $f,f^2\in D(L)$, $$M_t=f(X_t)-\int_0^t Lf(X_s)\ ds$$ is a martingale (w.r.t. $P^x$). And I want to show that $$M_t^2-\int_0^t ...
0
votes
0answers
82 views

Open Jackson network expected customers and distribution

I have a stochastic modelling test tomorrow, I'm stuck on one practice question. We have an open Jackson network which is as follows: Arrivals in queue 1 are a Poisson process with rate $\lambda$. ...
2
votes
1answer
47 views

Find the steady state probability that both A and B catch a headache.

I have a question about Markov chain. Let A and B be patients, A has headache at the rate $1$ times/week and recovers from it at rate of $2$ times/week. The patient B has it at the rates $2$ and $4$ ...
0
votes
0answers
87 views

hitting time for a continuous time markov chain

Brad’s relationship with his girl friend Angelina changes between Amorous, Bickering, Confusion, and Depression according to the following transition rates when t is the time in months. They are ...
0
votes
1answer
70 views

Solve a problem using Markov chains

We have the following problem: At the beginning of every year, a gardener classifies his soil based on its quality: it's either good, mediocre or bad. Assume that the classification of the soil ...
0
votes
1answer
46 views

Probability distribution in 7th steps

Let's assume that there is a markov chain with a transition matrix $P$: $\begin{bmatrix} 0 &0 &0 &\frac{1}{2} & \frac{1}{2} & 0\\ 0& 0& 0& \frac{1}{2}& ...
1
vote
1answer
19 views

Understanding stochastic matrices

We start a game with 2 euros, i.e. at time 0 we have 2 euros. At time $t=1,2,...$ we play a game with a stake of 1 euro and with odds of winning $p$ (hence odds of losing $1-p$). We define $X_t$ at ...
0
votes
0answers
25 views

Stochastic calculus for continuous time Markov chains

I have absolved a course on stochastic analysis, i.e. integrals with respect to the brownian motion. Now I know that there is a theory of stochastic calculus for diskrete matringales, however I was ...
0
votes
0answers
22 views

Markov chain jump

It follows from applying the Markov property that if we start in some point $x \in S$ ($S$ is assumed to be finite here) where $(X_t)_{t \ge 0}$ is a Markov chain that the stopping time ...
0
votes
0answers
26 views

Interchange limit and partial derivative of two different variables

I have seen the following related questions: Interchange limit of one variable with partial derivative of another variable and Interchange of partial derivative and limit But my case is a bit ...
0
votes
0answers
28 views

Prove M\M\1 queue is in fact CTMC

As i understand the number of customers in the M\M\1 queue is expressed as $N(t)=A(t)-\int_0^tI_{\{N(s-1)\geq1\}}dD(s)$ Where A(t), D(t) are independent Poisson processes with rates $\lambda$ and ...
1
vote
0answers
25 views

Stochastic process $X(t)=W_tX(t-1)$ with $\left\{W_t\right\}_{t=1}^n$ iid row stochastic matrices

I have been struggling for a while with the following problem. Consider a sequence of iid row stochastic matrices $\left\{W_t\right\}_{t=1}^n$ and the linear dynamical system $X(t)=W_tX(t-1)$ with ...
2
votes
1answer
51 views

True or false: The process $\{ X(t), t \geq 0 \}$ at a $M/M/s$ queue is a reversible Markov process.

Let $X(t)$ denote the number of customers in a system at time $t$. The process $\{ X(t), t \geq 0 \}$ at a $M/M/s$ queue is a reversible Markov process. Is this statement true or false for: (a) ...
4
votes
0answers
109 views

Proof of strong Markov Property of double sided Levy Process

Let $(X_t)_{t\in\mathbb{R}}$ be a Levy Process, i.e. $X_0 = 0$ a.s., $X$ has independent and stationary increments, and almost all paths $t\mapsto X_t(\omega)$ are right continuous with left hand ...
2
votes
0answers
73 views

Markov process and filtration

I would like to restate the question. I'm reading Revuz/Yor's definition of Markov process (P81), they started from transition function, and define the $P_t f(x)$ as usual (let's only consider the ...
1
vote
1answer
39 views

Increments of birth-death process

Reading about birth-death processes i encountered a formula which i can't derive It's here, page 4, number (7)-(8). ...
0
votes
1answer
45 views

Markov chain with absorbing states?

Let's say I have $5$ states (state $2$ to $6$, state $1$ is missing) when time$=0$, and $6$ states (state $1$ to $6$) when time$=1$, and now I want to calculate the transition matrix. Does it mean ...
2
votes
0answers
35 views

Marginalized non Markov Chapman kolmogorov equation

The usual Chapman kolmogorov equation states $$\int dx_1 p(x_2|x_1)p(x_1|x_0)=p(x_2|x_0)$$ Which means we can also identify $$\int dx_0 p(x_2|x_1)p(x_1|x_0)p(x_0)=p(x_2|x_1)p(x_1)$$ Now, because ...
0
votes
0answers
22 views

Markov Random Field - compute joint factor psi?

I'm given a distribution on 3 discrete variables $x, y, z$ which is defined as $$p(x, y, z) = \frac{1}{Z} \psi(x, y, z) = \frac{1}{Z} \phi_1(x, y) \phi_2(y, z)$$ where $x, y$ can take up value among ...
0
votes
0answers
19 views

Solution of heat equation with cross terms on rectangle.

I would like to find the fundamental solution of the following PDE $$ u_t = \frac12 u_{xx} + \rho u_{xy} + \frac12 u_{yy} $$ on the rectangle $[-a,a]\times[-b,b]$, with $a,b>0$ and with homogeneous ...
0
votes
1answer
25 views

How do I read this equation: $ Pr(T < t \mid y) = \int_{0}^{\infty }\int_{\Omega }k(x)p(t, x\mid y)\,dx\,dt \ $?

How do I read this equation (especially the left side) in terms of a Continuous Markov Process model? $$ Pr(T < t \mid y) = \int_{0}^{\infty }\int_{\Omega }k(x)p(t, x\mid y)\,dx\,dt \\ $$ Where $ ...
2
votes
2answers
59 views

A Markov process which is not strong Markov process (follow up 2)

In http://mathoverflow.net/questions/43833/a-markov-process-which-is-not-a-strong-markov-process George Lowther's example: "Consider the following continuous Markov process $X$, starting from ...
0
votes
0answers
60 views

A stationary distribution of Markov chain

For a irreducible finite Markov chain, I know that the definition of a stationary distribution is as follows: $\pi P = \pi$, where $P$ is a transition matrix and $\pi$ is a stationary distribution ...
0
votes
0answers
32 views

First passage time of the Brownian motion

In an exercise (4.1 Krapinsky, "A kinetic view of statistical physics") I am asked to show that: The probability that a brownian motion on a 1D discrete lattice never reaches the site $n$ scales as ...
0
votes
1answer
33 views

Estimate the typical numeber of times a brownian motion on ℤ starting from $0$ does a particular transition

Consider an 1D infinite lattice. The lattice is fully occupied except from a vacancy in the origin which undergoes simple diffusion (in countinuous time). At position $n>0$ in the lattice there is ...
0
votes
0answers
24 views

Asymptotic variance for Markov chain

Let $\hat{\mu_t}(f)=\frac{1}{t} \sum_{i=1}^{t} f(X_i)$ is an estimator for a finite, irreducible Markov chain $\{X_t\}$ with its stationary distribution $\pi$. In addition, assume the estimator is ...
0
votes
0answers
24 views

use Markov property to bound expectation

Let $X_s, s\geq 0$ be a Markov process. Let $s<s^D<t_D<t$. Let $f$ be some suitable function and write $\hat{X}$ for another process that is not Markov. Denote by $X_{s,t}$ the increment of ...
1
vote
0answers
38 views

Uniform integrability of functional of an ergodic Markov Chain?

Suppose $\{X_n\}$ is an ergodic Markov Chain with general state space $\mathcal{X}$ and stationary distribution $\pi$. Suppose $\forall x\in\mathcal{X}, f(x)\geq 0$ and ...
0
votes
0answers
13 views

Markov process convergence and kernels

I have a sequence of Markov processes $\{X_t^n\}$ with the same starting distribution ($X_0^n \sim X_0$) for every $n$ and such that for every $t \geq 0$ I have convergencence in probability to some ...
0
votes
0answers
12 views

Example of Markov process that is not strong Markov process — Revisited

I've reading the discussion concerning the existence of a Markov process that is not a Strong markov process. In the Mathoverflow site, we can find a neat example from Byron here. The explanation is ...
0
votes
0answers
31 views

Strong Markov property and stopping time

In the book by Jeanblanc, Yor & Cheney, "mathematical methods for financial markets", on page 17 above Prop.1.1.14.3, there is a strange identity of a strong Markov process $X$ that reads $${\bf ...
1
vote
1answer
59 views

Markov Chain Application - flipping coin heads, tails, finished

Suppose that we are flipping coins iteratively, until we get tails two in a row. Define three states: Heads, Tails, and Finished. Suppose that the probability of getting a head is $p$, and ...