A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

14
votes
1answer
481 views

Hilbert's Barber Shop

Hilbert opens a barber shop with an infinite number of chairs and an infinite number of barbers. Customers arrive via a Poisson random process with an expected 1 person every 10 minutes. Upon arrival, ...
12
votes
3answers
274 views

Select a new value from last $N$ values; how long until the last $N$ are all the same?

Say first we have N distinct numbers in a line, like 1,2,3,...,N, in each round, we choose a ...
12
votes
2answers
419 views

Difference in probability distributions from two different kernels

I wonder if the probability kernels of Markov processes on the same state space are close enough, does it also hold for the probabilities of the event that depend only on first $n$ values of the ...
11
votes
1answer
399 views

Motivation of Feynman-Kac formula and its relation to Kolmogorov backward/forward equations?

Kolmogorov backward/forward equations are pdes, derived for the semigroups constructed from the Markov transition kernels. Feynman-Kac formula is also a pde corresponding to a stochastic process ...
7
votes
2answers
241 views

What is the importance of the infinitesimal generator of Brownian motion?

I have read that the infinitesimal generator of Brownian motion is $\frac{1}{2}\small\triangle$. Unfortunately, I have no background in semigroup theory, and the expositions of semigroup theory I have ...
7
votes
0answers
129 views

Uniqueness of an infinite system of linear ODEs

How to prove that $\dot{x}=ax,\space x(0)=1$ has a unique solution if $a,x$ are infinite dimensional matrices? More specifically, let $Q$ be a bounded infinitesimal generator, i.e. ...
5
votes
3answers
172 views

Does an n-order Markov chain still represent a Markov process?

I am trying to understand Markov processes but am still confused by their definition. In particular, the Wikipedia page gives this example of a non-Markov process. The example is of pulling different ...
5
votes
1answer
113 views

Log-likelihood function

I'm not sure if this could be asked here, or in math overflow... In the following paper Cho, Jin Seo, and Halbert White. "Testing for regime switching." Econometrica 75.6 (2007): 1671-1720. doi: ...
5
votes
0answers
160 views

Question on Conditional expectation

Let $X_1$ and $X_2$ be two random variables on $(\Omega,\mathcal{B},P)$. Suppose there is a function $g:\mathcal{B}\times\mathbb{R}\rightarrow[0,1]$ such that for any $x$, $g(\cdot,x)$ is a ...
4
votes
1answer
4k views

Kendall notation's “General distribution”, what does that mean?

The first and second parameters for the Kendall's notation may have a G value, which stands for General distribution, see here. But what does that mean? What is a ...
4
votes
1answer
337 views

Covariance of Brownian-motion-like processes

We know that $\operatorname{Cov}(B_s,B_t)=\min(s,t)$ if $B_t$ is Brownian motion. What is $\operatorname{Cov}(B_{f(s)},B_{f(t)})$ for some injective $f$? How can I write $B_{f(t)}$ in an Ito ...
4
votes
1answer
72 views

How is the Chapman-Kolmogorov Equation not a fancy name for matrix multiplication?

The Chapman-Kolmogorov Equation: $$p^{m+n}(i,j)=\sum_kp^m(i,k)p^n(k,j)$$ Matrix Multiplication (with $[A]_{i,j}=a_{i,j}$ where $A$ is a linear map "" for B) $$[AB]_{i,j}=\sum_ka_{i,k}b_{k,j}$$ In ...
4
votes
1answer
77 views

random walk in a certain environment

Consider the following random walk in one dimension, starting from $r(0)=0$. $$ r(i+1) = r(i) + \xi, $$ where $\xi(i, r(i))$ is an increment with distribution $P(\xi=1) = \frac{c^{r(i)}}{i-r(i)+1}$ ...
4
votes
1answer
2k views

Transition Kernel of a Markov Chain

Supposing $X_t$ is a Markov Process, can the transition kernel be defined by $$K_t(x,A):= P(X_{t+1} \in A | X_t = x)?$$ Assume that $X_t : \Omega \to \mathbb{R}^n$. The issue is that under the normal ...
4
votes
1answer
87 views

Showing a process is not markov

I keep searching but I can't find any place that gives a good method of showing a process is NOT Markov. The definition I am using is that for every $s<t$ and $g$ bounded borel there is $f$ borel ...
4
votes
1answer
86 views

Fixed point of transition kernel generates martingale

Let $P^{h}, h \geqslant 0$ be a transition kernel for some homogenous Markov process $X_t$, $\mathbb{E}|X_t|<\infty$: $$ P_{X_{t+h},X_t}(A,B) = \int\limits_{A}P^h(x,B)P_{X_t}(dx) $$ where ...
4
votes
2answers
2k views

Why Markov matrices always have 1 as an eigenvalue

Also called stochastic matrix. Let $A=[a_{ij}]$ - matrix over $\mathbb{R}$ $0\le a_{ij} \le 1 \forall i,j$ $\sum_{j}a_{ij}=1 \forall i$ i.e the sum along each column of $A$ is 1. I ...
4
votes
1answer
365 views

Recursively Solving a Bellman Equation

Problem Overview I am to figure out $v_\pi$ of a certain Markov state. Given Information A set of actions, $a$ containing ${up, down, left, right}$ $v_\pi(12), v_\pi(13), v_\pi(14)$ (I am given ...
4
votes
0answers
66 views

A Continuous-Time Markov Process Taking All Possible Values

Let $\mathbb{N}$ be the set of positive integers. For each $n \in \mathbb{N}$, let $X^{(n)}=\{ X^{(n)}(t): t \geq 0 \}$ be a Markov chain with state-space the two point set $\{0,1\}$ and $Q$-matrix ...
4
votes
0answers
128 views

Conditional probability and integrating out part of a random walk

Suppose that I have a random walk process defined by $\alpha_{t+1}$ ~ N$(\alpha_t, \omega^2)$. Given $\alpha_t$ and $\alpha_{t+2}$, I understand why the conditional formula for ...
4
votes
0answers
169 views

estimation of transition probabilities from aggregate data

Please, O mathematicians, help me understand the approach to the problem of estimating transition probabilities given only aggregate data in Kalbfleisch & Lawless' 1984 paper "Least-Squares ...
3
votes
1answer
2k views

Markov process vs. markov chain vs. random process vs. stochastic process vs. collection of random variables

I'm trying to understand each of the above terms, and I'm having a lot of trouble deciphering the difference between them (note, my mathematics training isn't very strong - so please go easy on the ...
3
votes
3answers
106 views

Introduction to Markov Random Fields

I'm looking for a gentle introduction to this topic. The material I've found so far is substantially related to physics, and requires some background in such field. Is there anything more general and ...
3
votes
1answer
46 views

Probability of a sequence of events in a Poisson process.

I am starting to study Poisson processes and I came up with this question: Let there be two Poisson processes with rates $\lambda$ and $\mu$ respectively, monitoring the occurrence of events (e.g. ...
3
votes
1answer
143 views

Applying equation to Markov process

This seems as an easy question, but however I can't handle it. In the following I need this fact: If $X=(X_t)$ is a Markov process with transition semigroup $(K_t)$ and initial distribution $\mu$ ...
3
votes
1answer
133 views

Levy processes with no positive jumps

Let X be a Levy process with no positive jumps and $\tau_y:=\inf\{t> 0: X_t > y\}$ then we have $$X_{\tau_y}=y\text{ on }\{\tau_y <\infty\}.$$ Could you explain that why? and does it hold ...
3
votes
1answer
76 views

What is the value of this game?

We have 3 black and 2 red balls in an urn. If we pick a black ball, we lose 1 USD. If we pick a red ball we win 1 USD. We can chose to start the game or not. If we start the game we can stop after ...
3
votes
1answer
77 views

Strong Markov property of Brownian motion

I was able to understand Brownian Motion $\{B(t):t\geq0\}$ has Strong Markov Property i.e. For any stopping time $\tau$, $P(B(t+\tau)\leq y | \mathcal{F}_{\tau})=P(B(t+\tau)\leq y|B(\tau))$ a.s. , $y ...
3
votes
1answer
872 views

Why does a time-homogeneous Markov process possess the Markov property?

Klenke defines (Definition 17.3, p. 346) a time-homogeneous Markov process independently, rather than as a special case of a stochastic process that possesses the Markov property (Definition 17.1, p. ...
3
votes
2answers
33 views

Always null recurrence at the boundary between positive recurrence and transience?

I have the following theorem: Let $\rho$ be the traffic intensity. a) If $\rho<1$, then $X$ is positive recurrent. b) If $\rho>1$, then $X$ is transient. c) If $\rho=1$, ...
3
votes
1answer
168 views

Elementary proof of geometric / negative binomial distribution in birth-death processes

The birth-death process concerns a population of $n_0$ individuals, each of which reproduce and die at a constant rate as time $t$ increases from $t=0$. Each individual splits into two individuals ...
3
votes
1answer
66 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is ...
3
votes
1answer
64 views

Random walk where increment depend on current position

Consider the following stochastic process, $$b(i+1) = b(i) + \xi_i (b_i),$$ where $\xi_i(b_i) \in \{-1, k \}$ are the independent increments having the following distribution: $$\begin{align} P (\xi ...
3
votes
1answer
63 views

For a generator $G$ of a Markov process in continuous time and finite state space, how would one prove that the entries of $e^{tG}$ are non-negative?

I have a generator matrix G for a Markov chain in continuous time and finite state space and I am looking to prove that the entries of $e^{tG} \geq 0 $ By definition $G = P'(0)$ with entries $g_{ij} ...
3
votes
1answer
103 views

Limit of a probability regarding a random walk

Consider a random walk on the integers starting at 0, where in each step you move either 1 or 2 meters (back or forth alike). As soon as you reach either the $N$ or $N+1$ meter mark, you stop. What is ...
3
votes
1answer
281 views

In a continuous-time Markov process, is the waiting time between jumps a function of the current state?

Two books construct Markov processes from Q-matrices using waiting times and jump chains but differ in whether the waiting times depend on the current state. Can the two be reconciled? Klenke claims ...
3
votes
2answers
452 views

Probability of Extinction in a simple Birth and Death Process

We are asked to show that the probability of extinction $\zeta=\lim_{t\to \infty} P\left(X(t)=0\right)$ given by: $$\zeta=\begin{cases}1&\text{if }\lambda\le \mu,\\ \left(\frac \mu\lambda ...
3
votes
1answer
450 views

Multidimensional infinitesimal generator of a jump-diffusion

Let $X=\{X_t\}_{t\geq0}$ be an $n$-dimensional Markov process, defined by the SDE $$dX_t = \mu(t, X_t) \, dt + \sigma(t,X_t) \, dB_t+\beta(t-,X_{t-}) \, dN_t,$$ where $\mu, \sigma$ and $\beta$ are ...
3
votes
1answer
71 views

A equivalent definition of the Feller Process.

I saw this on Liggett's Book (P.95). Let $S=% %TCIMACRO{\U{2115} }% %BeginExpansion \mathbb{N} %EndExpansion ,$ and suppose $\left( X_{t}\right) _{t\geq 0}$ is a continuous-time Markov process with ...
3
votes
0answers
133 views

Generalization of Dobrushin's Ergodic Decomposition for continuous Markov Chains

Let $T$ be the shift transformation. Let $P$ be invariant for $T$ and also define a discrete state space Markov Chain. Let $C_{1},\ldots,C_{n}$ be the connected components of the Markov Chain. It ...
2
votes
2answers
54 views

Question on MIT Markov Matrices video

Markov matrices are pretty new to me and I'm a little rusty with my linear algebra. My question stems from watching this video from YouTube on Markov matrices. For those who wish to skip the video, ...
2
votes
1answer
106 views

A question about a stochastic process being Markov

Let $(X_{s},\mathcal{F}_{s})$ be a stochastic process adapted to a given filtration. I was told that, in order to prove that $X$ is Markov, it suffice to prove that for any nonnegative, ...
2
votes
1answer
68 views

transition kernel

I've got some trouble with transition kernels. We look at Markov process with statspace $(S,\mathcal{S})$ and initial distribution $\mu^0$. We have a transition kernel $P:S\times ...
2
votes
2answers
23 views

Probability of returning to a given state after n transitions-Markov chains

Let us denote $f_j^{(n)}$ denote the probability of the first return to state $j $after n transitions. Let $p_{jj}^{(n)}$ be the probability of returning to the state $j$ after $n$ transitions when ...
2
votes
1answer
51 views

Inverse of a regular stochastic matrix

Is it true that the inverse of a regular stochastic matrix is also regular? Are there any other interesting features that the inverse may have of a regular stochastic matrix? Hope someone could answer ...
2
votes
1answer
53 views

time-homogeneous continuous time Markov chain

I have a question about the continuous time Markov chain. In the Poisson process we have independent and stationary increments. Do we have this in a continuous time Markov chain that is ...
2
votes
1answer
103 views

Markov chain problem with finite states

Consider any Markov chain on a state space with exactly $r$ states. We want to find the largest $N>0$ such that there exists states $i,j$ for every $n < N$ we have $p_{ij}^{(N)} > 0$ and ...
2
votes
1answer
154 views

Markov chains: is “aperiodic + irreducible” equivalent to “regular”?

I have two books on stochastic processes. In one book, it says that the limiting matrix is possible to find if the matrix is regular, that is, if for some $n$ $P^n$ has only positive values. The ...
2
votes
1answer
114 views

Proof that Markov Chains converges to the stationary distribution

Let $P$ is a transition matrix of a Markov Chain, which is irreducible, aperiodic and lets assume $\pi$ is its stationary distribution: $\pi = \pi P$. Does anyone knows the proof for the following ...
2
votes
1answer
49 views

Markov property of a random process (a solution of piece-wise deterministic equations)

Consider a piece-wise deterministic (Markov!) process