A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

9
votes
0answers
210 views

Uniqueness of an infinite system of linear ODEs

How to prove that $\dot{x}=ax,\space x(0)=1$ has a unique solution if $a,x$ are infinite dimensional matrices? More specifically, let $Q$ be a bounded infinitesimal generator, i.e. $Q=(q_{i,j})_{i,j\...
6
votes
0answers
149 views

Limit distributions for Markov chains $X\to\sqrt{U+X}$

This question spawned from a recent, very interesting problem. Let $\varphi=\frac{1+\sqrt{5}}{2}$ and $T$ denote the map on the space of continuous probability density functions supported over $\...
5
votes
0answers
46 views

Poisson: Conditional Probability on Pizza order

I am not sure about my answer. In particular, part b of the following question. Pizza orders arrive according to a Poisson process of rate 20 per hour. Orders are independently for a vegetarian ...
5
votes
0answers
63 views

markov chain: 2 state chain

I have a machine. It has two states, broken or working. If it is working, then it will be broken with probability $q=0.1$. If the machine is working, I will make \$1000 dollar a day. If it is broken, ...
4
votes
0answers
26 views

Question about a type of continuous state Markov-process.

EDIT: Solved! It turns out that if the function is continuous and various regularity conditions hold then the statement is true. This has been established in the 'stochastic approximation' literature, ...
4
votes
0answers
111 views

Proof of strong Markov Property of double sided Levy Process

Let $(X_t)_{t\in\mathbb{R}}$ be a Levy Process, i.e. $X_0 = 0$ a.s., $X$ has independent and stationary increments, and almost all paths $t\mapsto X_t(\omega)$ are right continuous with left hand ...
4
votes
0answers
111 views

A Continuous-Time Markov Process Taking All Possible Values

Let $\mathbb{N}$ be the set of positive integers. For each $n \in \mathbb{N}$, let $X^{(n)}=\{ X^{(n)}(t): t \geq 0 \}$ be a Markov chain with state-space the two point set $\{0,1\}$ and $Q$-matrix \...
4
votes
0answers
157 views

Conditional probability and integrating out part of a random walk

Suppose that I have a random walk process defined by $\alpha_{t+1}$ ~ N$(\alpha_t, \omega^2)$. Given $\alpha_t$ and $\alpha_{t+2}$, I understand why the conditional formula for $(\alpha_{t+1}|\alpha_{...
4
votes
0answers
278 views

estimation of transition probabilities from aggregate data

Please, O mathematicians, help me understand the approach to the problem of estimating transition probabilities given only aggregate data in Kalbfleisch & Lawless' 1984 paper "Least-Squares ...
3
votes
0answers
30 views

Distribution on number of revisits in past $k$ steps of Markov chain

Consider a finite-state Markov chain with transition matrix $P$. The chain starts in a state chosen uniformly over all the states and runs indefinitely from there. We're going to examine only the $k ...
3
votes
0answers
24 views

How to handle Finite-state-machine with correlated inputs?

My system can be represented by the following state-diagram. where each arch represents Input/Output when a transition is made from one state to the other. The inputs to this FSM are correlated. ...
3
votes
0answers
52 views

This is a Markov Chain?

Consider two irreducible ergodic Markov chains with the same state space $\{0, 1, . . . , N\}$, with transition matrices $P$ and $Q$ and respective stationary distributions $\pi$ and $\rho$. We ...
3
votes
0answers
109 views

Moment Generating Function for Brownian motion's exit of interval.

Let $B(t)$ be a standard BM. Consider the stopping time $T = \inf\{ t > 0: |B(t)| = a\},$ the usual first exit time of the interval $(-a, a).$ We can see that $\mathbb{E} e^{tT} < \infty$ for $...
3
votes
0answers
51 views

Correct steps in rewriting expectation to a probability

My knowledge of measure theory and probability spaces is limited, so please keep it relatively simple. Let $\{X(t), ~ t \ge 0\}$ be a Markov process on the countable state space $\mathbb{N}_0$ with ...
3
votes
0answers
205 views

Why are inter arrival times in the continuous version of discrete-time Markov chains always exponentially distributed?

I am curious whether there exist continuous time Markov processes for which the times between jumping times (which I call inter arrival times) are not exponentially distributed, but have some other ...
3
votes
0answers
77 views

Discrete Laplacian

I have the following question and I can't figure out how to do the proof. Could you give me some hints in both directions of the equivalence? Suppose $A$ is a bounded subset of $\mathbb{Z}^d$. Then $\...
3
votes
0answers
75 views

Simple random walk conditioning on non-return

Consider a simple symmetric random walk on $\mathbb{Z}$, $(S_t)_{t \geq 0}$, with $S_0=0$. Let $k$ and $j$ be two positive integers. Let $P_{k,j}$ be the probability that the walker hits the vertex $k$...
3
votes
0answers
157 views

Generalization of Dobrushin's Ergodic Decomposition for continuous Markov Chains

Let $T$ be the shift transformation. Let $P$ be invariant for $T$ and also define a discrete state space Markov Chain. Let $C_{1},\ldots,C_{n}$ be the connected components of the Markov Chain. It ...
3
votes
0answers
328 views

Is every killed Markov process still a Markov process?

Suppose we've got $X=(X(t))_{t\geq 0}$. $X$ is a strong Markov process with respect to filtration $\mathcal{F}_t$, taking values in some subset of $\mathbb{R}$. We take $\tau$ - a stopping time w.r.t $...
2
votes
0answers
33 views

Fractional powers of Markov generators

Let $H$ be the generator of a symmetric Markov semigroup on $L^2(\mathbb{R}^n).$ Why the fractional power $H^\alpha$ (defined on a proper domain) with $0 < \alpha < 1$ turn out to be the ...
2
votes
0answers
22 views

Is the infinitesimal generator for Lie groups the same as the infinitesimal generator of a Markov semigroup?

Is the infinitesimal generator for Lie groups related to the infinitesimal generator of a Markov semigroup? Or are they totally different concepts? https://en.wikipedia.org/wiki/Lie_group#...
2
votes
0answers
13 views

Uniform convergence of the action of a Feller semigroup in one variable.

Assume we have two subsets of the some euclidean spaces $X\subset \mathbb{R}^m$ and $Y\subset\mathbb{R}^n$ and a a Feller semigroup $(Q_t)_{t\geq 0}$ on $Y$. Suppose also that we have a continuous ...
2
votes
0answers
55 views

Is this transformation of a Markov process again Markovian?

Let $(X_t)_{t\in\mathbb{N}_0}$ be a stationary Markov process valued in $\mathbb{R}$ and $c\in\mathbb{R}$. Is the process $(Y_t)_{t\in\mathbb{N}_0}$ defined by $$ Y_t={\bf 1}{(X_t<c)} $$ again a ...
2
votes
0answers
25 views

Strategy for selling/buying a stock by average reward value iteration

At beginning of any day $t$, I may own $0$ or $1$ share. The price of the share follows the Markov chain in the table below. At the beginning of a day where I own a share, I may either sell at today’s ...
2
votes
0answers
56 views

Strong markov property vector valued process from independent strong markov components

I want to prove that if $X$ and $Y$ are (continuous time) independent strong markov $\mathbb{R}$-valued processes w.r.t. their natural filtrations $\mathcal{F}^X_t$ and $\mathcal{F}^Y_t$, that the ...
2
votes
0answers
32 views

Moving Average of an Ergodic Markov processes.

Let $ \{X(t); t\geq 0\} $ be an ergodic Markov process, and let G be a positive integrable function with $\int_0^\infty G(x)dx=1$. Does $$F(t)\doteq G*X(t) = \int_0^t G(t-s)X(s)ds $$ converge in ...
2
votes
0answers
30 views

Good introductory book coupling methods

I am very interested in coupling methods, can you recommend me a good introductory books on this subject? Thanks
2
votes
0answers
68 views

Gambler's ruin problem - expected time

I have troubles seeing the following. Consider the classical gambler's ruin problem, betting 1 at each time $t\in \mathbb{N}$, and losing or winning -1 respectively +1 at each time till the fortune of ...
2
votes
0answers
41 views

Markov chain limit problem

Let $X_n$ be a Markov chain on a countable state space, $\mathbb{S}$. Let $N_n(x) = \sum_{k=1}^n\mathbb{1}_{\{X_k=x\}}$ denote the number of times the chain visits state $x\in \mathbb{S}$. Let $x,y,...
2
votes
0answers
78 views

Markov process and filtration

I would like to restate the question. I'm reading Revuz/Yor's definition of Markov process (P81), they started from transition function, and define the $P_t f(x)$ as usual (let's only consider the ...
2
votes
0answers
37 views

Marginalized non Markov Chapman kolmogorov equation

The usual Chapman kolmogorov equation states $$\int dx_1 p(x_2|x_1)p(x_1|x_0)=p(x_2|x_0)$$ Which means we can also identify $$\int dx_0 p(x_2|x_1)p(x_1|x_0)p(x_0)=p(x_2|x_1)p(x_1)$$ Now, because $$\...
2
votes
0answers
21 views

Why do we look only for the first and second derivative when dealing with diffusions?

I would like to understand, from an analytical point of view, why is it that we only take the first and second derivatives into account when computing the generator of a diffusion. This question is ...
2
votes
0answers
50 views

Excursion of random walk conditioning on return

Consider a simple random walk in one dimension starting from the origin. Let $\epsilon>0$. How to prove that, conditioning on the event that the random walk is at the origin at time $n$, the ...
2
votes
0answers
41 views

Compute the value of $\mathbb{P}_{0}\left[\tau\geq n\right]$ and $\mathbb{P}_{0}\left[\tau=+\infty\right]$

We consider the Markov chain $\left(X_{n}\right)_{n\geq0}$ on $\mathbb{N}$ whose transition matrix $P=\left(p_{k,l}\right)_{k,l\geq0}$ given by $$\forall k\in\mathbb{N} p_{k,0}=q_{k},p_{k,k+1}...
2
votes
0answers
56 views

Expected response time of Continuous time Markov chain

I'm studying CTMC (Continuous Time Markov Chains). I came across the following slide I don't understand how they got $M(t+h) = M(t) + \alpha h + M(t)\lambda h - M(t) \mu h +o(h)$ Could anyone ...
2
votes
0answers
69 views

Why is the stationary distribution a distribution?

Suppose we have a time-homogeneous, discrete-time, aperiodic, positive recurrent, irreducible Markov chain $(X_t)_{t \geq 0}$ on a discrete state space $E$. It is known that its stationary ...
2
votes
0answers
113 views

How to prove a set is a core for an infinitesimal generator

I am trying to prove that the set $D=\bigcap_{n\geq 1} \{f\in C^2 (\mathbb{R}) :f(0)=\sum_{k\geq 0} f(\frac{k}{\sqrt{n}})g_n (k)\}$ is a core for the infinitesimal generator of reflecting brownian ...
2
votes
0answers
61 views

Semigroup associated to a Markov process

I'm studying the transition semigroup associated to a Markov Process, in particular the Hille-Yosida theorem and the Martingale Problem. In my notes I found : "If $\{T_t\}_t$ is a strongly continuous ...
2
votes
0answers
49 views

Simple Markov property

I want to prove the simple Markov property but I come to a point where I do not see how to conclude. I want to prove $\mathbb{E}_\nu[Z\circ\Theta_t\mid \mathcal{Y}^0_t]=\mathbb{E}_{Y_t}[Z]:=\mathbb{E}...
2
votes
0answers
72 views

Electrostatic capacity of two spheres with changing radii

Although I have read a lot of questions and answers here, this is my first time actually posting. Feel free to suggest needed edits. My question is the following (in a simplified setting). All this ...
2
votes
0answers
36 views

Discrete-Time Stochastic Calculus and Stopping Times: Resources

In my measure-theoretic probability course we covered what the professor called "discrete-time stochastic calculus". Essentially, it was a three part method for computing certain quantities such as ...
2
votes
0answers
50 views

Limit of decreasing sequences of markov time (stopping time) is markov time?

Let $(\Omega, \mathcal{F}, \{\mathcal{F}_t\}_{t \geqslant 0}, \mathbb{P})$ be a filtered probability space and let $\tau_n \geqslant \tau_{n+1}$ be a markov time (stopping time) with respect to ...
2
votes
0answers
38 views

Example of Markov process not strong Markov

This is again a question about this example (also see here). It seems we can write this process as $$X_t := \bigl(t - \text{Exp}(1) \mathbf{1}_{\{X_0 = 0\}}\bigr)^+ + \mathbf{1}_{\{X_0 \neq 0 \}}X_0\,...
2
votes
0answers
99 views

Absorbed brownian motion is a Markov process

I have been asked to prove that the Brownian motion absorbed at the origin is a Markov process. Formally, let $B_t^x$ be a Brownian motion originating from $x>0$ and let $\tau^x_0 = \inf\{t>0 : ...
2
votes
0answers
100 views

Transition function for absorbed Brownian motion

I need an help with the following exercise. I've already seen this question Prove that Brownian Motion absorbed at the origin is Markov but I don't understand the answer. Also I would like to prove ...
2
votes
0answers
34 views

Showing which classes are recurrent and which are transient

If I have a Markov chain on states {0,1,2,3,4,5} $$ \mathbf{a} = \matrix{~ & 0 & 1 & 2 & 3 & 4 & 5 \\ 0 & 1/3 & 0 & 2/3 & 0 & 0 & 0 \\ ...
2
votes
0answers
41 views

Define Markov chain and rewrite to recursively solve

Customers arrive at a server with rate $\lambda$ and are served at rate $\mu$. The server breaks down with rate $\gamma$, which causes all customers to leave. New customers can only arrive once the ...
2
votes
0answers
130 views

Model as a continuous time Markov Chain

A system consists of two machines, of which one works and the other is standby. Only the working machine can break down (with rate $\lambda$). If it breaks down the other machine takes over (if it not ...
2
votes
0answers
28 views

Probability of going from a set $S$ to its complement on a Markov chain

I need to show that if $\pi$ is the stationary distribution of a Markov chain $M$, then for every set of vertices $S$, the probability to choose a random node in $S$ according to $\pi$ and then going ...
2
votes
0answers
31 views

Stochastic scheduling to maximize the expected number of customers arrived at the root of a Jackson tree

In a Jackson network, organized as a tree rooted at queue r, several customers are queued at time 0 and there is no new customer arrival. The service time of each customer in queue i is geometrically ...