A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

21
votes
2answers
2k views

What is the importance of the infinitesimal generator of Brownian motion?

I have read that the infinitesimal generator of Brownian motion is $\frac{1}{2}\small\triangle$. Unfortunately, I have no background in semigroup theory, and the expositions of semigroup theory I have ...
14
votes
1answer
562 views

Hilbert's Barber Shop

Hilbert opens a barber shop with an infinite number of chairs and an infinite number of barbers. Customers arrive via a Poisson random process with an expected 1 person every 10 minutes. Upon arrival, ...
13
votes
1answer
5k views

Markov process vs. markov chain vs. random process vs. stochastic process vs. collection of random variables

I'm trying to understand each of the above terms, and I'm having a lot of trouble deciphering the difference between them (note, my mathematics training isn't very strong - so please go easy on the ...
13
votes
2answers
440 views

Difference in probability distributions from two different kernels

I wonder if the probability kernels of Markov processes on the same state space are close enough, does it also hold for the probabilities of the event that depend only on first $n$ values of the ...
12
votes
3answers
285 views

Select a new value from last $N$ values; how long until the last $N$ are all the same?

Say first we have N distinct numbers in a line, like 1,2,3,...,N, in each round, we choose a ...
11
votes
1answer
654 views

Motivation of Feynman-Kac formula and its relation to Kolmogorov backward/forward equations?

Kolmogorov backward/forward equations are pdes, derived for the semigroups constructed from the Markov transition kernels. Feynman-Kac formula is also a pde corresponding to a stochastic process ...
10
votes
1answer
111 views

Exploiting the Markov property

I've encountered the following problem when dealing with short-rate models in finance and applying the Feynman-Kac theorem to relate conditional expectations to PDEs. Let $(\Omega,\mathcal{F},\{\...
9
votes
0answers
210 views

Uniqueness of an infinite system of linear ODEs

How to prove that $\dot{x}=ax,\space x(0)=1$ has a unique solution if $a,x$ are infinite dimensional matrices? More specifically, let $Q$ be a bounded infinitesimal generator, i.e. $Q=(q_{i,j})_{i,j\...
7
votes
4answers
9k views

Why Markov matrices always have 1 as an eigenvalue

Also called stochastic matrix. Let $A=[a_{ij}]$ - matrix over $\mathbb{R}$ $0\le a_{ij} \le 1 \forall i,j$ $\sum_{j}a_{ij}=1 \forall i$ i.e the sum along each column of $A$ is 1. I ...
7
votes
1answer
103 views

If $A$ is the generator of $(P_t)$, then $A+f$ is the generator of $(P_t^f)$

Let $X=(X_t)_{t\geq0}$ be a Markov process on a state space $\Gamma$ (a Hausdorff topological vector space), let $A$ be the infinitesimal generator of $X$ and let $\mathcal C(\Gamma)$ the space of ...
7
votes
4answers
307 views

Discover where Bob is sleeping using hidden Markov chains

Bob lives in four different houses $A, B, C$ and $D$ that are connected like the following graph shows: Bob likes to sleep in any of his houses, but they are far apart so he only sleeps in a house ...
6
votes
4answers
320 views

Probability of going into an absorbing state

If I have a random walk Markov chain whose transition probability matrix is given by $$ \mathbf{P} = \matrix{~ & 0 & 1 & 2 & 3 \\ 0 & 1 & 0 & 0 & 0 \\ ...
6
votes
2answers
634 views

Interpretation for the determinant of a stochastic matrix?

Is there a probabilistic interpretation for the determinant of a stochastic matrix (i.e. an $n \times n$ matrix whose columns sum to unity)?
6
votes
1answer
269 views

How is the Chapman-Kolmogorov Equation not a fancy name for matrix multiplication?

The Chapman-Kolmogorov Equation: $$p^{m+n}(i,j)=\sum_kp^m(i,k)p^n(k,j)$$ Matrix Multiplication (with $[A]_{i,j}=a_{i,j}$ where $A$ is a linear map "" for B) $$[AB]_{i,j}=\sum_ka_{i,k}b_{k,j}$$ In ...
6
votes
1answer
4k views

Transition Kernel of a Markov Chain

Supposing $X_t$ is a Markov Process, can the transition kernel be defined by $$K_t(x,A):= P(X_{t+1} \in A | X_t = x)?$$ Assume that $X_t : \Omega \to \mathbb{R}^n$. The issue is that under the normal ...
6
votes
1answer
81 views

Martingale converges to the boundary

I asked an almost same question before and it is solved by considering adjacent $Z_n$ can not be far away and obtain a contradiction. However, if the setting is altered a bit, I wonder whether it is ...
6
votes
0answers
147 views

Limit distributions for Markov chains $X\to\sqrt{U+X}$

This question spawned from a recent, very interesting problem. Let $\varphi=\frac{1+\sqrt{5}}{2}$ and $T$ denote the map on the space of continuous probability density functions supported over $\...
5
votes
1answer
7k views

Kendall notation's “General distribution”, what does that mean?

The first and second parameters for the Kendall's notation may have a G value, which stands for General distribution, see here. But what does that mean? What is a ...
5
votes
1answer
2k views

Why does a time-homogeneous Markov process possess the Markov property?

Klenke defines (Definition 17.3, p. 346) a time-homogeneous Markov process independently, rather than as a special case of a stochastic process that possesses the Markov property (Definition 17.1, p. ...
5
votes
2answers
181 views

Convergence of the powers of a Markov transition matrix

I have a Markov matrix $$P=\begin{bmatrix}1&0&0&0&0&0\\\frac{1}{2}&0&\frac{1}{2}&0&0&0\\\frac{1}{4}&0&\frac{1}{4}&\frac{1}{2}&0&0\\\frac{1}{...
5
votes
1answer
94 views

Constructing Martingales from Markov Processes

I know that for a Markov process $X_t$ with generator $L$ and $f,f^2\in D(L)$, $$M_t=f(X_t)-\int_0^t Lf(X_s)\ ds$$ is a martingale (w.r.t. $P^x$). And I want to show that $$M_t^2-\int_0^t (Lf^2(X_s)-...
5
votes
3answers
238 views

Does an n-order Markov chain still represent a Markov process?

I am trying to understand Markov processes but am still confused by their definition. In particular, the Wikipedia page gives this example of a non-Markov process. The example is of pulling different ...
5
votes
1answer
133 views

Log-likelihood function

I'm not sure if this could be asked here, or in math overflow... In the following paper Cho, Jin Seo, and Halbert White. "Testing for regime switching." Econometrica 75.6 (2007): 1671-1720. doi: ...
5
votes
1answer
108 views

Is a Markov process uniquely determined?

Let $E$ be a Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $I\subseteq[0,\infty)$ be closed under addition and $0\in I$ Please consider the following result: Let $(\...
5
votes
1answer
195 views

Show that Brownian motion on the unit circle is exponentially ergodic and has the uniform measure as its invariant distribution.

My search results keep bring up planar Brownian motion on the unit disk. However, I am specifically referring to $e^{jW_{t}} = [\cos(W_t),\sin(W_t)]^{T}$ where $W_t$ is Brownian motion. I am at a ...
5
votes
1answer
155 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is $$P\{X_{n+1}=j|X_0=i_0,\ldots,X_n=i\}=P\{X_{n+1}=...
5
votes
2answers
1k views

Probability of Extinction in a simple Birth and Death Process

We are asked to show that the probability of extinction $\zeta=\lim_{t\to \infty} P\left(X(t)=0\right)$ given by: $$\zeta=\begin{cases}1&\text{if }\lambda\le \mu,\\ \left(\frac \mu\lambda \right)^...
5
votes
1answer
55 views

Branching Process: generation survival or extinction?

Let $p\in [0,1]$, and consider a branching process where the number of offspring of an individual is zero with probability $p$, and is two with probability $1-p$. Initially there is one ...
5
votes
1answer
154 views

Conditions for birth and death process having only finitely many deaths.

Consider a birth and death process on $\mathbb{N}=\left\{0,1,2,\ldots\right\}$, given by the transition probabilities $p(n,n+1)=\lambda_n$ and $p(n,n-1)=\mu_n$ (those are the birth and death rates, ...
5
votes
0answers
45 views

Poisson: Conditional Probability on Pizza order

I am not sure about my answer. In particular, part b of the following question. Pizza orders arrive according to a Poisson process of rate 20 per hour. Orders are independently for a vegetarian ...
5
votes
0answers
63 views

markov chain: 2 state chain

I have a machine. It has two states, broken or working. If it is working, then it will be broken with probability $q=0.1$. If the machine is working, I will make \$1000 dollar a day. If it is broken, ...
5
votes
1answer
382 views

Convergence of Ornstein-Uhlenbeck process as a scaled Brownian Motion

Let $W$ be a standard Brownian motion. Let $\alpha,\sigma^2 >0$, and let $X_0$ be a $\mathbb{R}$-valued random variable with distibution $\nu$ that is independent of $\sigma(W_t,t\geq 0)$. Now ...
5
votes
0answers
231 views

Question on Conditional expectation [closed]

Let $X_1$ and $X_2$ be two random variables on $(\Omega,\mathcal{B},P)$. Suppose there is a function $g:\mathcal{B}\times\mathbb{R}\rightarrow[0,1]$ such that for any $x$, $g(\cdot,x)$ is a ...
4
votes
2answers
3k views

Transformation of state-space that preserves Markov property

I am solving a problem in Mathematical Statistics by Jun Shao Let $\{X_n \}$ be a Markov chain. Show that if $g$ is a one-to-one Borel function, then $\{g(X_n )\}$ is also a Markov chain. Give an ...
4
votes
1answer
199 views

A die whose score cannot be as before (Markov chains)

A die is "fixed" so that each time it is rolled the score cannot be the same as the preceding score, all other scores having probability $1/5$. Given that the first score is 6, what is the probability ...
4
votes
2answers
939 views

Diffusion process. Distribution vs transition probability.

I need confirmation on the following problem: Take a SDE of the form: \begin{equation} dX_t=a(X_t,t)dt+b(X_t,t)dW_t \end{equation} where all the conditions, such that the solution $X_t$ is defined ...
4
votes
1answer
619 views

Covariance of Brownian-motion-like processes

We know that $\operatorname{Cov}(B_s,B_t)=\min(s,t)$ if $B_t$ is Brownian motion. What is $\operatorname{Cov}(B_{f(s)},B_{f(t)})$ for some injective $f$? How can I write $B_{f(t)}$ in an Ito ...
4
votes
1answer
277 views

Find the Stationary Distribution of an infinite state Markov chain

A Markov Chain on states 0,1,..... has transition probabilities $P_{ij}=1/(i+2)$ for j=0,1,....,i,i+1. I'm supposed to find the stationary distribution. So do I take the limit as n goes to ...
4
votes
1answer
95 views

random walk in a certain environment

Consider the following random walk in one dimension, starting from $r(0)=0$. $$ r(i+1) = r(i) + \xi, $$ where $\xi(i, r(i))$ is an increment with distribution $P(\xi=1) = \frac{c^{r(i)}}{i-r(i)+1}$ ...
4
votes
1answer
125 views

Showing a process is not markov

I keep searching but I can't find any place that gives a good method of showing a process is NOT Markov. The definition I am using is that for every $s<t$ and $g$ bounded borel there is $f$ borel ...
4
votes
1answer
108 views

Fixed point of transition kernel generates martingale

Let $P^{h}, h \geqslant 0$ be a transition kernel for some homogenous Markov process $X_t$, $\mathbb{E}|X_t|<\infty$: $$ P_{X_{t+h},X_t}(A,B) = \int\limits_{A}P^h(x,B)P_{X_t}(dx) $$ where $P_{X_{...
4
votes
2answers
335 views

Question about an exercise in Revuz/Yor

I'm solving exercise 2.28 in Revuz/Yor. I was able to prove 1). Unfortunately at 2) I got stuck. I have to show: Let $B$ be a d-dimensional Brownian motion and $A\in \mathcal{A}:=\cap_t \mathcal{A}...
4
votes
1answer
83 views

Ehrenfest Chain: stationary distribution

In the Ehrenfest Chain model: There are M balls which are divided between urn A and urn B. At each stage, if a ball is chosen, then it would be moved into a different urn. Let $X_n$ be the # of ...
4
votes
1answer
71 views

If $τ_x^k$ is the time of the $k$-th entrance of a Markov chain into $x$, then $\text P_x[τ_y^k<∞]=\text P_x[τ_y^1<∞](\text P_y[τ_y^1<∞])^{k-1}$

Let $E$ be at most countable and equipped with the discrete topology and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_n)_{n\in\mathbb N_0}$ be a discrete Markov chain with values in $(E,\...
4
votes
1answer
100 views

Expectation and limit of a stop-and-go traveler markov chain

The velocity $V(t)$ of a stop and go traveler is a two-state Markov chain whose generator is given by $$ \begin{array}{cc} &\begin{matrix}0&1\end{matrix}\\ \ \begin{matrix}0\\ 1\end{matrix} &...
4
votes
1answer
861 views

Multidimensional infinitesimal generator of a jump-diffusion

Let $X=\{X_t\}_{t\geq0}$ be an $n$-dimensional Markov process, defined by the SDE $$dX_t = \mu(t, X_t) \, dt + \sigma(t,X_t) \, dB_t+\beta(t-,X_{t-}) \, dN_t,$$ where $\mu, \sigma$ and $\beta$ are ...
4
votes
1answer
720 views

Recursively Solving a Bellman Equation

Problem Overview I am to figure out $v_\pi$ of a certain Markov state. Given Information A set of actions, $a$ containing ${up, down, left, right}$ $v_\pi(12), v_\pi(13), v_\pi(14)$ (I am given ...
4
votes
0answers
26 views

Question about a type of continuous state Markov-process.

EDIT: Solved! It turns out that if the function is continuous and various regularity conditions hold then the statement is true. This has been established in the 'stochastic approximation' literature, ...
4
votes
0answers
110 views

Proof of strong Markov Property of double sided Levy Process

Let $(X_t)_{t\in\mathbb{R}}$ be a Levy Process, i.e. $X_0 = 0$ a.s., $X$ has independent and stationary increments, and almost all paths $t\mapsto X_t(\omega)$ are right continuous with left hand ...
4
votes
2answers
100 views

What is the probability this Markov chain does not reach state $r$?

Consider a random walk on the non-negative integers. You start at $0$, and in each step you either move $1$ higher, or $2$ lower (but can't go below $0$). The direction is picked w.p. $1/2$ ...