A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
1answer
81 views

A property of Poisson process

Let $Y_t$ be a centered Poisson process, why \begin{equation} \lim_{n \to \infty} \sup_{s<t} |n^{-1}Y(ns)| = 0 \qquad a.s. \qquad \forall t\ge 0 \end{equation} This is a fundamental step in the ...
5
votes
1answer
94 views

Constructing Martingales from Markov Processes

I know that for a Markov process $X_t$ with generator $L$ and $f,f^2\in D(L)$, $$M_t=f(X_t)-\int_0^t Lf(X_s)\ ds$$ is a martingale (w.r.t. $P^x$). And I want to show that $$M_t^2-\int_0^t (Lf^2(X_s)-...
0
votes
0answers
82 views

Open Jackson network expected customers and distribution

I have a stochastic modelling test tomorrow, I'm stuck on one practice question. We have an open Jackson network which is as follows: Arrivals in queue 1 are a Poisson process with rate $\lambda$. ...
2
votes
1answer
47 views

Find the steady state probability that both A and B catch a headache.

I have a question about Markov chain. Let A and B be patients, A has headache at the rate $1$ times/week and recovers from it at rate of $2$ times/week. The patient B has it at the rates $2$ and $4$ /...
0
votes
0answers
89 views

hitting time for a continuous time markov chain

Brad’s relationship with his girl friend Angelina changes between Amorous, Bickering, Confusion, and Depression according to the following transition rates when t is the time in months. They are ...
0
votes
1answer
70 views

Solve a problem using Markov chains

We have the following problem: At the beginning of every year, a gardener classifies his soil based on its quality: it's either good, mediocre or bad. Assume that the classification of the soil ...
0
votes
1answer
46 views

Probability distribution in 7th steps

Let's assume that there is a markov chain with a transition matrix $P$: $\begin{bmatrix} 0 &0 &0 &\frac{1}{2} & \frac{1}{2} & 0\\ 0& 0& 0& \frac{1}{2}& \...
1
vote
1answer
19 views

Understanding stochastic matrices

We start a game with 2 euros, i.e. at time 0 we have 2 euros. At time $t=1,2,...$ we play a game with a stake of 1 euro and with odds of winning $p$ (hence odds of losing $1-p$). We define $X_t$ at ...
0
votes
0answers
29 views

Stochastic calculus for continuous time Markov chains

I have absolved a course on stochastic analysis, i.e. integrals with respect to the brownian motion. Now I know that there is a theory of stochastic calculus for diskrete matringales, however I was ...
0
votes
0answers
22 views

Markov chain jump

It follows from applying the Markov property that if we start in some point $x \in S$ ($S$ is assumed to be finite here) where $(X_t)_{t \ge 0}$ is a Markov chain that the stopping time $\tau_x:=\inf\...
0
votes
0answers
27 views

Interchange limit and partial derivative of two different variables

I have seen the following related questions: Interchange limit of one variable with partial derivative of another variable and Interchange of partial derivative and limit But my case is a bit ...
0
votes
0answers
28 views

Prove M\M\1 queue is in fact CTMC

As i understand the number of customers in the M\M\1 queue is expressed as $N(t)=A(t)-\int_0^tI_{\{N(s-1)\geq1\}}dD(s)$ Where A(t), D(t) are independent Poisson processes with rates $\lambda$ and $\...
1
vote
0answers
26 views

Stochastic process $X(t)=W_tX(t-1)$ with $\left\{W_t\right\}_{t=1}^n$ iid row stochastic matrices

I have been struggling for a while with the following problem. Consider a sequence of iid row stochastic matrices $\left\{W_t\right\}_{t=1}^n$ and the linear dynamical system $X(t)=W_tX(t-1)$ with ...
2
votes
1answer
54 views

True or false: The process $\{ X(t), t \geq 0 \}$ at a $M/M/s$ queue is a reversible Markov process.

Let $X(t)$ denote the number of customers in a system at time $t$. The process $\{ X(t), t \geq 0 \}$ at a $M/M/s$ queue is a reversible Markov process. Is this statement true or false for: (a) $...
4
votes
0answers
111 views

Proof of strong Markov Property of double sided Levy Process

Let $(X_t)_{t\in\mathbb{R}}$ be a Levy Process, i.e. $X_0 = 0$ a.s., $X$ has independent and stationary increments, and almost all paths $t\mapsto X_t(\omega)$ are right continuous with left hand ...
2
votes
0answers
78 views

Markov process and filtration

I would like to restate the question. I'm reading Revuz/Yor's definition of Markov process (P81), they started from transition function, and define the $P_t f(x)$ as usual (let's only consider the ...
1
vote
1answer
39 views

Increments of birth-death process

Reading about birth-death processes i encountered a formula which i can't derive It's here, page 4, number (7)-(8). $P_{i,i+1}(h)=P(X(t+h)-X(t)=1|X(t)=1)=\frac{(\lambda_ih)^1e^{-\lambda_ih}}{1!}\frac{...
0
votes
1answer
45 views

Markov chain with absorbing states?

Let's say I have $5$ states (state $2$ to $6$, state $1$ is missing) when time$=0$, and $6$ states (state $1$ to $6$) when time$=1$, and now I want to calculate the transition matrix. Does it mean ...
2
votes
0answers
37 views

Marginalized non Markov Chapman kolmogorov equation

The usual Chapman kolmogorov equation states $$\int dx_1 p(x_2|x_1)p(x_1|x_0)=p(x_2|x_0)$$ Which means we can also identify $$\int dx_0 p(x_2|x_1)p(x_1|x_0)p(x_0)=p(x_2|x_1)p(x_1)$$ Now, because $$\...
0
votes
0answers
23 views

Markov Random Field - compute joint factor psi?

I'm given a distribution on 3 discrete variables $x, y, z$ which is defined as $$p(x, y, z) = \frac{1}{Z} \psi(x, y, z) = \frac{1}{Z} \phi_1(x, y) \phi_2(y, z)$$ where $x, y$ can take up value among ...
0
votes
0answers
19 views

Solution of heat equation with cross terms on rectangle.

I would like to find the fundamental solution of the following PDE $$ u_t = \frac12 u_{xx} + \rho u_{xy} + \frac12 u_{yy} $$ on the rectangle $[-a,a]\times[-b,b]$, with $a,b>0$ and with homogeneous ...
0
votes
1answer
26 views

How do I read this equation: $ Pr(T < t \mid y) = \int_{0}^{\infty }\int_{\Omega }k(x)p(t, x\mid y)\,dx\,dt \ $?

How do I read this equation (especially the left side) in terms of a Continuous Markov Process model? $$ Pr(T < t \mid y) = \int_{0}^{\infty }\int_{\Omega }k(x)p(t, x\mid y)\,dx\,dt \\ $$ Where $ ...
2
votes
2answers
62 views

A Markov process which is not strong Markov process (follow up 2)

In http://mathoverflow.net/questions/43833/a-markov-process-which-is-not-a-strong-markov-process George Lowther's example: "Consider the following continuous Markov process $X$, starting from ...
0
votes
0answers
62 views

A stationary distribution of Markov chain

For a irreducible finite Markov chain, I know that the definition of a stationary distribution is as follows: $\pi P = \pi$, where $P$ is a transition matrix and $\pi$ is a stationary distribution (...
0
votes
0answers
36 views

First passage time of the Brownian motion

In an exercise (4.1 Krapinsky, "A kinetic view of statistical physics") I am asked to show that: The probability that a brownian motion on a 1D discrete lattice never reaches the site $n$ scales as $\...
0
votes
1answer
34 views

Estimate the typical numeber of times a brownian motion on ℤ starting from $0$ does a particular transition

Consider an 1D infinite lattice. The lattice is fully occupied except from a vacancy in the origin which undergoes simple diffusion (in countinuous time). At position $n>0$ in the lattice there is ...
0
votes
0answers
28 views

Asymptotic variance for Markov chain

Let $\hat{\mu_t}(f)=\frac{1}{t} \sum_{i=1}^{t} f(X_i)$ is an estimator for a finite, irreducible Markov chain $\{X_t\}$ with its stationary distribution $\pi$. In addition, assume the estimator is ...
0
votes
0answers
24 views

use Markov property to bound expectation

Let $X_s, s\geq 0$ be a Markov process. Let $s<s^D<t_D<t$. Let $f$ be some suitable function and write $\hat{X}$ for another process that is not Markov. Denote by $X_{s,t}$ the increment of $...
1
vote
0answers
39 views

Uniform integrability of functional of an ergodic Markov Chain?

Suppose $\{X_n\}$ is an ergodic Markov Chain with general state space $\mathcal{X}$ and stationary distribution $\pi$. Suppose $\forall x\in\mathcal{X}, f(x)\geq 0$ and $\mathbb{E}_{\pi}[f(X)]<\...
0
votes
0answers
13 views

Markov process convergence and kernels

I have a sequence of Markov processes $\{X_t^n\}$ with the same starting distribution ($X_0^n \sim X_0$) for every $n$ and such that for every $t \geq 0$ I have convergencence in probability to some ...
0
votes
0answers
13 views

Example of Markov process that is not strong Markov process — Revisited

I've reading the discussion concerning the existence of a Markov process that is not a Strong markov process. In the Mathoverflow site, we can find a neat example from Byron here. The explanation is ...
0
votes
0answers
31 views

Strong Markov property and stopping time

In the book by Jeanblanc, Yor & Cheney, "mathematical methods for financial markets", on page 17 above Prop.1.1.14.3, there is a strange identity of a strong Markov process $X$ that reads $${\bf 1}...
1
vote
1answer
61 views

Markov Chain Application - flipping coin heads, tails, finished

Suppose that we are flipping coins iteratively, until we get tails two in a row. Define three states: Heads, Tails, and Finished. Suppose that the probability of getting a head is $p$, and ...
1
vote
0answers
25 views

When is a markov process not conservative where do we need $P(t,x;I)$ to be continuous to the right?

We find on Petr Mandl's book Analytical treatment of one-dimensional Markov processes page 10 the following definition of conservative markov process My question concerns the proof of theorem 4 ...
0
votes
1answer
35 views

Prove or disprove that the successive maxima of sums of i.i.d. increments are a Markov process

Let $\{\xi_n\}$ be independent, identically distributed, random variables. Define $S_k = \sum\limits_{i=0}^k \xi_i $ and $\eta_k = \max(S_0, ..., S_k)$. How to prove or disprove that $\{ \...
0
votes
0answers
22 views

Transience of random walks starting from each of the sites which have been already visited

Consider a simple random walk on $\mathbb{Z}^d$, $d \geq 3$, which starts from the origin. As $d \geq 3$, there is a positive probability that the random walk never visits the origin again. Now, let ...
0
votes
0answers
12 views

Is the value of the discrete Green's function on a box independent of the position of the points within the box?

I currently contemplate over the discrete Green's function on a box and am trying to gain an intuition for its behaviour. Consider the box $B := \{-N,\cdots,-1,0,1,\cdots,N\}^2$ and let $(X_i)_{i \in ...
0
votes
1answer
52 views

If $N_t$ is a Poisson process and $Y\in\{-1,1\}$, then $X_t = Y(-1)^{N_t}$ is a Markov process

Let $Y$ be a random variable with values in $\{-1,1\}$ independent of a Poisson process $\{N_t\}$ with intensity $\lambda>0$. Set the process $X = \{X_t\}$ by $X_t=Y(-1)^{N_t}$. Show that $X$ is a ...
0
votes
3answers
71 views

markov chains and coin flips

A coin that comes up heads with probability p is continually flipped until the pattern T T T H appears. Let X denote the number of flips, find EX. If I use Markov chains is there a simpler way to ...
2
votes
0answers
21 views

Why do we look only for the first and second derivative when dealing with diffusions?

I would like to understand, from an analytical point of view, why is it that we only take the first and second derivatives into account when computing the generator of a diffusion. This question is ...
0
votes
0answers
38 views

Importance of uniform stationary distribution

When I study Markov chain (or sampling) related papers, most of them emphasize "uniform stationary distribution". But, I can't sure why it is important for Markov chain problems or randomized ...
1
vote
1answer
28 views

Transition Probability of M/M/1 Queue given any constant observing period T

I am trying to find the transition probability for $M$/$M$/$1$ queue given any constant observing period $T$ (if $T$ go to infinity the transition matrix will degenerate to a matrix with identical ...
1
vote
0answers
35 views

The differences between the return times to a recurrent state of a discrete Markov chain are independent and identically distributed

Let $(\Omega,\mathcal A)$ be a measurable space and $\mathbb F=(\mathcal F_n)_{n\in\mathbb N_0}$ be a filtration on $(\Omega,\mathcal A)$ $E$ be an at most countable set equipped with the discrete ...
1
vote
1answer
39 views

Markov process, compute how much time spent in each state in average before absorption.

This problem has three states for a person, which are either employed, unemployed or early retirement. The probability that a working person goes unemployed is 0.2 (ie with intensity 0.2 per year). ...
1
vote
0answers
23 views

Fluctuation of a martingale conditioned to return

Consider a martingale $M_t$ on $\mathbb{Z}$ starting from $M_0=0$ and such that $Var[M_t] \leq C \, t$, where $C>0$ is some constant. For a given $n \in \mathbb{N}$ and $t \leq n$, define a process ...
0
votes
0answers
19 views

Property of an irreducible Markov Chain

How can we prove that if a Markov Chain is irreducible (does not contain any closed set), then every state can be reached from every other state in the chain ?
0
votes
1answer
28 views

model with markov chain

Suppose to have the following situation: At a bar at each time unit arrives a certain number of customer with probabilities $p_1,p_2,...,p_n$. In the bar there are 3 bartenders so 3 customer can be ...
0
votes
1answer
61 views

I think I've found an invariant distribution for a transient discrete Markov chain - Where is my mistake?

Let $E$ be an at most countable set equipped with the discrete topology $\mathcal E$ $X=(X_n)_{n\in\mathbb N_0}$ be a discrete Markov chain with values in $(E,\mathcal E)$, distributions $(\...
1
vote
0answers
40 views

An irreducible Markov chain is positive recurrent if and only if there is an invariant distribution

Let $E$ be an at most countable set equipped with the discrete topology $\mathcal E$ $X=(X_n)_{n\in\mathbb N_0}$ be a discrete Markov chain with values in $(E,\mathcal E)$, distributions $(\...
-1
votes
1answer
20 views

If the probability measure $\mu$ is a left-eigenvector to the eigenvalue $1$ of a stochastic matrix $p$, then $\mu p^n=\mu$

Let $E$ be an at most countable set and $\mathcal E$ be the discrete topology on $E$ $p=\left(p(x,y)\right)_{x,y\in E}$ be a stochastic matrix $\mu$ be a probability measure on $(E,\mathcal E)$ ...