A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
0answers
46 views

hitting time for a continuous time markov chain

Brad’s relationship with his girl friend Angelina changes between Amorous, Bickering, Confusion, and Depression according to the following transition rates when t is the time in months. They are ...
0
votes
1answer
63 views

Solve a problem using Markov chains

We have the following problem: At the beginning of every year, a gardener classifies his soil based on its quality: it's either good, mediocre or bad. Assume that the classification of the soil ...
0
votes
1answer
44 views

Probability distribution in 7th steps

Let's assume that there is a markov chain with a transition matrix $P$: $\begin{bmatrix} 0 &0 &0 &\frac{1}{2} & \frac{1}{2} & 0\\ 0& 0& 0& \frac{1}{2}& ...
1
vote
1answer
18 views

Understanding stochastic matrices

We start a game with 2 euros, i.e. at time 0 we have 2 euros. At time $t=1,2,...$ we play a game with a stake of 1 euro and with odds of winning $p$ (hence odds of losing $1-p$). We define $X_t$ at ...
0
votes
0answers
18 views

Stochastic calculus for continuous time Markov chains

I have absolved a course on stochastic analysis, i.e. integrals with respect to the brownian motion. Now I know that there is a theory of stochastic calculus for diskrete matringales, however I was ...
0
votes
0answers
16 views

Markov chain jump

It follows from applying the Markov property that if we start in some point $x \in S$ ($S$ is assumed to be finite here) where $(X_t)_{t \ge 0}$ is a Markov chain that the stopping time ...
0
votes
0answers
21 views

Interchange limit and partial derivative of two different variables

I have seen the following related questions: Interchange limit of one variable with partial derivative of another variable and Interchange of partial derivative and limit But my case is a bit ...
0
votes
0answers
20 views

Prove M\M\1 queue is in fact CTMC

As i understand the number of customers in the M\M\1 queue is expressed as $N(t)=A(t)-\int_0^tI_{\{N(s-1)\geq1\}}dD(s)$ Where A(t), D(t) are independent Poisson processes with rates $\lambda$ and ...
1
vote
0answers
21 views

Stochastic process $X(t)=W_tX(t-1)$ with $\left\{W_t\right\}_{t=1}^n$ iid row stochastic matrices

I have been struggling for a while with the following problem. Consider a sequence of iid row stochastic matrices $\left\{W_t\right\}_{t=1}^n$ and the linear dynamical system $X(t)=W_tX(t-1)$ with ...
2
votes
0answers
34 views

True or false: The process $\{ X(t), t \geq 0 \}$ at a $M/M/s$ queue is a reversible Markov process.

Let $X(t)$ denote the number of customers in a system at time $t$. The process $\{ X(t), t \geq 0 \}$ at a $M/M/s$ queue is a reversible Markov process. Is this statement true or false for: (a) ...
3
votes
0answers
84 views

Proof of strong Markov Property of double sided Levy Process

Let $(X_t)_{t\in\mathbb{R}}$ be a Levy Process, i.e. $X_0 = 0$ a.s., $X$ has independent and stationary increments, and almost all paths $t\mapsto X_t(\omega)$ are right continuous with left hand ...
2
votes
0answers
54 views

Markov process and filtration

I would like to restate the question. I'm reading Revuz/Yor's definition of Markov process (P81), they started from transition function, and define the $P_t f(x)$ as usual (let's only consider the ...
1
vote
1answer
30 views

Increments of birth-death process

Reading about birth-death processes i encountered a formula which i can't derive It's here, page 4, number (7)-(8). ...
0
votes
1answer
40 views

Markov chain with absorbing states?

Let's say I have $5$ states (state $2$ to $6$, state $1$ is missing) when time$=0$, and $6$ states (state $1$ to $6$) when time$=1$, and now I want to calculate the transition matrix. Does it mean ...
2
votes
0answers
29 views

Marginalized non Markov Chapman kolmogorov equation

The usual Chapman kolmogorov equation states $$\int dx_1 p(x_2|x_1)p(x_1|x_0)=p(x_2|x_0)$$ Which means we can also identify $$\int dx_0 p(x_2|x_1)p(x_1|x_0)p(x_0)=p(x_2|x_1)p(x_1)$$ Now, because ...
0
votes
0answers
18 views

Markov Random Field - compute joint factor psi?

I'm given a distribution on 3 discrete variables $x, y, z$ which is defined as $$p(x, y, z) = \frac{1}{Z} \psi(x, y, z) = \frac{1}{Z} \phi_1(x, y) \phi_2(y, z)$$ where $x, y$ can take up value among ...
0
votes
0answers
17 views

Solution of heat equation with cross terms on rectangle.

I would like to find the fundamental solution of the following PDE $$ u_t = \frac12 u_{xx} + \rho u_{xy} + \frac12 u_{yy} $$ on the rectangle $[-a,a]\times[-b,b]$, with $a,b>0$ and with homogeneous ...
0
votes
1answer
24 views

How do I read this equation: $ Pr(T < t \mid y) = \int_{0}^{\infty }\int_{\Omega }k(x)p(t, x\mid y)\,dx\,dt \ $?

How do I read this equation (especially the left side) in terms of a Continuous Markov Process model? $$ Pr(T < t \mid y) = \int_{0}^{\infty }\int_{\Omega }k(x)p(t, x\mid y)\,dx\,dt \\ $$ Where $ ...
2
votes
2answers
57 views

A Markov process which is not strong Markov process (follow up 2)

In http://mathoverflow.net/questions/43833/a-markov-process-which-is-not-a-strong-markov-process George Lowther's example: "Consider the following continuous Markov process $X$, starting from ...
0
votes
0answers
49 views

A stationary distribution of Markov chain

For a irreducible finite Markov chain, I know that the definition of a stationary distribution is as follows: $\pi P = \pi$, where $P$ is a transition matrix and $\pi$ is a stationary distribution ...
0
votes
0answers
17 views

First passage time of the Brownian motion

In an exercise (4.1 Krapinsky, "A kinetic view of statistical physics") I am asked to show that: The probability that a brownian motion on a 1D discrete lattice never reaches the site $n$ scales as ...
0
votes
1answer
32 views

Estimate the typical numeber of times a brownian motion on ℤ starting from $0$ does a particular transition

Consider an 1D infinite lattice. The lattice is fully occupied except from a vacancy in the origin which undergoes simple diffusion (in countinuous time). At position $n>0$ in the lattice there is ...
0
votes
0answers
15 views

Asymptotic variance for Markov chain

Let $\hat{\mu_t}(f)=\frac{1}{t} \sum_{i=1}^{t} f(X_i)$ is an estimator for a finite, irreducible Markov chain $\{X_t\}$ with its stationary distribution $\pi$. In addition, assume the estimator is ...
0
votes
0answers
23 views

use Markov property to bound expectation

Let $X_s, s\geq 0$ be a Markov process. Let $s<s^D<t_D<t$. Let $f$ be some suitable function and write $\hat{X}$ for another process that is not Markov. Denote by $X_{s,t}$ the increment of ...
1
vote
0answers
35 views

Uniform integrability of functional of an ergodic Markov Chain?

Suppose $\{X_n\}$ is an ergodic Markov Chain with general state space $\mathcal{X}$ and stationary distribution $\pi$. Suppose $\forall x\in\mathcal{X}, f(x)\geq 0$ and ...
0
votes
0answers
12 views

Markov process convergence and kernels

I have a sequence of Markov processes $\{X_t^n\}$ with the same starting distribution ($X_0^n \sim X_0$) for every $n$ and such that for every $t \geq 0$ I have convergencence in probability to some ...
0
votes
0answers
9 views

Example of Markov process that is not strong Markov process — Revisited

I've reading the discussion concerning the existence of a Markov process that is not a Strong markov process. In the Mathoverflow site, we can find a neat example from Byron here. The explanation is ...
0
votes
0answers
29 views

Strong Markov property and stopping time

In the book by Jeanblanc, Yor & Cheney, "mathematical methods for financial markets", on page 17 above Prop.1.1.14.3, there is a strange identity of a strong Markov process $X$ that reads $${\bf ...
1
vote
1answer
43 views

Markov Chain Application - flipping coin heads, tails, finished

Suppose that we are flipping coins iteratively, until we get tails two in a row. Define three states: Heads, Tails, and Finished. Suppose that the probability of getting a head is $p$, and ...
1
vote
0answers
15 views

When is a markov process not conservative where do we need $P(t,x;I)$ to be continuous to the right?

We find on Petr Mandl's book Analytical treatment of one-dimensional Markov processes page 10 the following definition of conservative markov process My question concerns the proof of theorem 4 ...
0
votes
1answer
27 views

Prove or disprove that the successive maxima of sums of i.i.d. increments are a Markov process

Let $\{\xi_n\}$ be independent, identically distributed, random variables. Define $S_k = \sum\limits_{i=0}^k \xi_i $ and $\eta_k = \max(S_0, ..., S_k)$. How to prove or disprove that ...
0
votes
0answers
20 views

Transience of random walks starting from each of the sites which have been already visited

Consider a simple random walk on $\mathbb{Z}^d$, $d \geq 3$, which starts from the origin. As $d \geq 3$, there is a positive probability that the random walk never visits the origin again. Now, let ...
0
votes
0answers
7 views

Is the value of the discrete Green's function on a box independent of the position of the points within the box?

I currently contemplate over the discrete Green's function on a box and am trying to gain an intuition for its behaviour. Consider the box $B := \{-N,\cdots,-1,0,1,\cdots,N\}^2$ and let $(X_i)_{i \in ...
0
votes
1answer
50 views

If $N_t$ is a Poisson process and $Y\in\{-1,1\}$, then $X_t = Y(-1)^{N_t}$ is a Markov process

Let $Y$ be a random variable with values in $\{-1,1\}$ independent of a Poisson process $\{N_t\}$ with intensity $\lambda>0$. Set the process $X = \{X_t\}$ by $X_t=Y(-1)^{N_t}$. Show that $X$ is a ...
0
votes
3answers
62 views

markov chains and coin flips

A coin that comes up heads with probability p is continually flipped until the pattern T T T H appears. Let X denote the number of flips, find EX. If I use Markov chains is there a simpler way to ...
2
votes
0answers
20 views

Why do we look only for the first and second derivative when dealing with diffusions?

I would like to understand, from an analytical point of view, why is it that we only take the first and second derivatives into account when computing the generator of a diffusion. This question is ...
0
votes
0answers
31 views

Importance of uniform stationary distribution

When I study Markov chain (or sampling) related papers, most of them emphasize "uniform stationary distribution". But, I can't sure why it is important for Markov chain problems or randomized ...
1
vote
1answer
22 views

Transition Probability of M/M/1 Queue given any constant observing period T

I am trying to find the transition probability for $M$/$M$/$1$ queue given any constant observing period $T$ (if $T$ go to infinity the transition matrix will degenerate to a matrix with identical ...
1
vote
0answers
30 views

The differences between the return times to a recurrent state of a discrete Markov chain are independent and identically distributed

Let $(\Omega,\mathcal A)$ be a measurable space and $\mathbb F=(\mathcal F_n)_{n\in\mathbb N_0}$ be a filtration on $(\Omega,\mathcal A)$ $E$ be an at most countable set equipped with the discrete ...
1
vote
1answer
34 views

Markov process, compute how much time spent in each state in average before absorption.

This problem has three states for a person, which are either employed, unemployed or early retirement. The probability that a working person goes unemployed is 0.2 (ie with intensity 0.2 per year). ...
1
vote
0answers
20 views

Fluctuation of a martingale conditioned to return

Consider a martingale $M_t$ on $\mathbb{Z}$ starting from $M_0=0$ and such that $Var[M_t] \leq C \, t$, where $C>0$ is some constant. For a given $n \in \mathbb{N}$ and $t \leq n$, define a process ...
0
votes
0answers
18 views

Property of an irreducible Markov Chain

How can we prove that if a Markov Chain is irreducible (does not contain any closed set), then every state can be reached from every other state in the chain ?
0
votes
1answer
26 views

model with markov chain

Suppose to have the following situation: At a bar at each time unit arrives a certain number of customer with probabilities $p_1,p_2,...,p_n$. In the bar there are 3 bartenders so 3 customer can be ...
0
votes
1answer
56 views

I think I've found an invariant distribution for a transient discrete Markov chain - Where is my mistake?

Let $E$ be an at most countable set equipped with the discrete topology $\mathcal E$ $X=(X_n)_{n\in\mathbb N_0}$ be a discrete Markov chain with values in $(E,\mathcal E)$, distributions ...
1
vote
0answers
31 views

An irreducible Markov chain is positive recurrent if and only if there is an invariant distribution

Let $E$ be an at most countable set equipped with the discrete topology $\mathcal E$ $X=(X_n)_{n\in\mathbb N_0}$ be a discrete Markov chain with values in $(E,\mathcal E)$, distributions ...
-1
votes
1answer
19 views

If the probability measure $\mu$ is a left-eigenvector to the eigenvalue $1$ of a stochastic matrix $p$, then $\mu p^n=\mu$

Let $E$ be an at most countable set and $\mathcal E$ be the discrete topology on $E$ $p=\left(p(x,y)\right)_{x,y\in E}$ be a stochastic matrix $\mu$ be a probability measure on $(E,\mathcal E)$ ...
0
votes
0answers
29 views

Find the backward equation for the random walk on the integers in the form $f_n(x) = \alpha_n + (x - \beta_n)^2$

Consider the homogeneous Markov process given by the random walk on the integers with probabilities $a$, $b$, and $c$ of moving one step backward, staying in the same place, and one step forward. ...
0
votes
1answer
17 views

Many Markov chains with one graph

I am studying Markov chain as a beginner. When I read some documents, I often find a sentence as follows. "For an undirected graph, many finite irreducible Markov chains can be generated." But, it ...
2
votes
0answers
43 views

Excursion of random walk conditioning on return

Consider a simple random walk in one dimension starting from the origin. Let $\epsilon>0$. How to prove that, conditioning on the event that the random walk is at the origin at time $n$, the ...
0
votes
0answers
47 views

how to show if {$Z_n$} is a Markov chain given {$X_n$}

I'm currently working on an practice question from my notes. But I'm not quite understanding the idea of how to prove that something is a Markov chain. Let {$X_i$}, $i = 1,2,...$, be a Markov chain ...