A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
0answers
18 views

A Characterization of the Strong Markov Property

I have a question concerning the strong Markov property: For a strong Markov process $(X_u)_{u\ge 0}$, a real time $t\in \mathbb{R}$ and an optional stopping time $T$ with $t< T$ \begin{align*} ...
0
votes
1answer
33 views

Markov processes: Hitting times for a point form an i.i.d. sequence

Say that I have a recurrent time-homogenous diffusion process X (continuous strong markov process) and two points $x,y$. If $X_t$ goes from $y$, to $x$ and then back to $y$ again we denote it as a ...
-1
votes
0answers
31 views

Showing that $(X_n)$ obeys the Markov Property. [closed]

Consider a process $(X_n)_{n\geq0}$ where we define $X_0 = 0$ and for $n \geq 1$: $$X_n = X_{n-1} + Z_n$$ where $Z_n$ for $n \geq 1$ are independent random variables on $\{ -1, 1 \}$ with ...
1
vote
0answers
21 views

Process Transition Algorithm

I have a process with 100 possible states and independent entities going through the process. All the Entities have been observed through a span of 5 years at the end of each month. When the ...
0
votes
0answers
25 views

How is the Laplace transform of the density of a specific point process computed?

I am trying to understand a little of this thesis by Anna Rudas. In particular the continuous model presented in Section 2.2.2. We are given a weight function $w: \mathbb{N} \rightarrow ...
3
votes
2answers
99 views

Stochastic variables independent given Tau

Say we have a filtration $(\mathbb{F}_s)$, and a stopping time $\tau$ w.r.t. to that filtration.Let $X_t$ be a continuous stochastic process (not required to be adapted to the mentioned filtration), ...
1
vote
1answer
29 views

Joint distribution of Brownian motion and its running maximum

$B$ being standard Brownian motion, its running maximum is defined as $M_t = \sup_{0\leq s\leq t} B_s$. I am trying to follow the proof of the following result but I don't understand some of the steps ...
3
votes
1answer
63 views

Markov and strong Markov properties

In my study of strong Markov property of an RCLL canonical Markov process I encounter the following definition: Suppose $Y_t:\omega\rightarrow \omega(t)$ is canonical Markov process with respect to ...
1
vote
0answers
34 views

Transition density of an AR(1) process?

If we have an AR(1) process, i.e: $X_{t+1} = \alpha X_t + e_{t+1}$ with $X_0=0$ then what is its Markov Chain transition density? We know that for a Markov chain, the following holds: $P(X_{t+1}\leq ...
3
votes
0answers
61 views

A Markov Chain probability, conditioned on a random time.

My question: Upon reading theory about diffusion processes, i came across an argument which i believe simplifies to this: Say we have a Borel measurable set $A$ (if it matters you can set $A=\lbrace ...
1
vote
3answers
43 views

Confused about definition of absorption probability

My confusion can probably most easily be explained with an example. Consider the following one step transition matrix : $$ P=\matrix{% & 0 & 1 & 2 & 3 & 4 \\ 0 & ...
0
votes
0answers
17 views

Reference for General state space Markov chain

What is a good reference for general state space Markov chains? Is there a reference which assumes only familiarity with finite/countable state space Markov chains and then extends the results (e.g., ...
0
votes
1answer
59 views

Why is $f(X_t)-\int_0^t Af(X_s) \, ds$ a martingale for a Markov process $(X_t)_{t \geq 0}$?

I think if $A$ is the usual generator for the Markov process $(X_t)_t$ $$A f (x) = \lim_{t \downarrow 0} \frac{\mathbb{E}^{x} [f(X_{t})] - f(x)}{t}$$ then we get that for any "nice" $f$ the process ...
1
vote
1answer
50 views

Understanding the Markov property of Brownian motion

I'm trying to understand the Markov property for Brownian motions in full generality. The textbook I'm following states it like this: Recall that we have a family of measures $P_x, x \in ...
2
votes
0answers
23 views

Expected response time of Continuous time Markov chain

I'm studying CTMC (Continuous Time Markov Chains). I came across the following slide I don't understand how they got $M(t+h) = M(t) + \alpha h + M(t)\lambda h - M(t) \mu h +o(h)$ Could anyone ...
0
votes
2answers
35 views

Markovian systems: Why must controls be independent of state?

I am currently working my way through Probabilistic Robotics by Thrun, Burgard, and Fox. On p. 91, I encountered the following statement: The Markovian assumption implies independence between ...
0
votes
0answers
13 views

Joint Markov Chain (Two Correlated Markov Processes)

I have two Markov Chains, and they exhibit some correlation between them. For instance, when Chain A moves to state i, there is a high likelihood that Chain B moves to state j. How would I go about ...
1
vote
0answers
42 views

Defining a function over time in terms of a random variable that is undefined at a certain time

Let $X_n$ be a random variable taking on one of three values $a,b$ or $c$ over time. That is, for each $n \in \mathbb{N}$, we have $X_n \in \{a,b,c\}$. Also, for each $n \in \mathbb{N}$, let $F_n$ be ...
2
votes
0answers
38 views

Why is the stationary distribution a distribution?

Suppose we have a time-homogeneous, discrete-time, aperiodic, positive recurrent, irreducible Markov chain $(X_t)_{t \geq 0}$ on a discrete state space $E$. It is known that its stationary ...
1
vote
0answers
91 views

How to prove a set is a core for an infinitesimal generator

I am trying to prove that the set $D=\bigcap_{n\geq 1} \{f\in C^2 (\mathbb{R}) :f(0)=\sum_{k\geq 0} f(\frac{k}{\sqrt{n}})g_n (k)\}$ is a core for the infinitesimal generator of reflecting brownian ...
0
votes
1answer
31 views

Build a Markov process from a transition semigroup

Suppose that we have a Markov process $X$ with time-homogeneus transition probability function given by $p(t,x,A)$, with $t>0, x\in E, A \in \mathcal E$, where $(E,\mathcal E)$ is the state space. ...
2
votes
0answers
34 views

Moment Generating Function for Brownian motion's exit of interval.

Let $B(t)$ be a standard BM. Consider the stopping time $T = \inf\{ t > 0: |B(t)| = a\},$ the usual first exit time of the interval $(-a, a).$ We can see that $\mathbb{E} e^{tT} < \infty$ for ...
0
votes
1answer
21 views

A question involving Markov processes

Let $(S, \mathcal{B}, m)$ be a measurable space and $X_p := L^p(S, \mathcal{B}, m)$. Let $T_t \in \mathcal{L}(X_p, X_p)$ be a bounded linear operator defined by $$(T_t f)(x) = \int\limits_S P(t, x, ...
1
vote
0answers
40 views

Krylov-Bogoliubov theorem in discrete and continuous time

Consider the following two similar frameworks (the first one in discrete time, the second one in continuous time) in which I am trying to apply the Krylov-Bogoliubov theorem. 1) (Discrete time.) ...
1
vote
1answer
38 views

Distribution Stopping time under Brownian motions

Considering $W$ the canonical process on $C([0,1],\mathbb{R})$ and the row filtration generated by the coordinate process of $W$, I want to prove that ...
1
vote
0answers
48 views

Semigroup associated to a Markov process

I'm studying the transition semigroup associated to a Markov Process, in particular the Hille-Yosida theorem and the Martingale Problem. In my notes I found : "If $\{T_t\}_t$ is a strongly continuous ...
2
votes
1answer
40 views

Properties of a transient state in a Markov Chain

I have been trying to solve this problem for a while now Prove that if $j$ is transient state, then $\displaystyle\sum_{n=1}^\infty p_{ij}^{(n)}<\infty \ \forall i \in S$, with $S$ the state ...
1
vote
1answer
33 views

Proving technique used to show an equivalence to the definition of a Markov process

Let $X=(X_t)_{t\in I}$ be Markov process with values in a Polish space $E$. I want to show, that there exists a stochastic kernel $\kappa:E\times\mathcal{B}(E)^{\otimes I}\to [0,1]$ such that ...
0
votes
1answer
49 views

Assume a die is rolled repeatedly. Find the markov matrix $P$ for the random variable of the time until the next $6$.

Assume a die is rolled repeatedly. Find the markov/transition matrix $P$ for the random variable $X_r$ = the time until the next six at time $r$. My solution was: For $i,j \geq 0$, $P$ is given ...
3
votes
2answers
75 views

Markov Chains - Strong Markov Property

I'm struck with an exercise. I tried, but the results don't seem to fit to those proposed. Exercise: Two players play the following game. The one who begins draws two cards from a deck of 40 cards ...
2
votes
1answer
73 views

Autocorrelated, discrete, bounded and symmetric random walk with no edge attraction

I need to move over a discrete set of linearly organized.. let's say "Japan steps" $S=\{0,\dots,c\}, c \in \mathbb{N}^*$. My current position is given by $d \in S$. On each time step, I need to draw ...
1
vote
0answers
20 views

Factorization of the Fokker-Planck semigroup

"In the classical theory of Markov processes, the Fokker-Planck semigroup $\{T_t:t\ge 0\}$ can be factored as $T_t=N\circ U_t \circ j$, $t\ge 0$, where $j$ is an embedding, $U_t$ is a group of ...
2
votes
1answer
38 views

Markov processes and $C_0$-semigroups

A Markov process $(X_t)_{t \geq 0}$ in continuous time on $\mathbb{R}^d$ can be described by a semigroup of Markov kernels $(p_t(x,A))_{t \geq 0}$ with $p_0(x,A) = 1_{A}(x)$ and which fulfill the ...
2
votes
0answers
33 views

Simple Markov property

I want to prove the simple Markov property but I come to a point where I do not see how to conclude. I want to prove $\mathbb{E}_\nu[Z\circ\Theta_t\mid ...
0
votes
0answers
18 views

model a system with finite users as a Markov Chain

I have to model a system M/M/2 with finite users (4 users) as a Markov Chain (and then find the probality an incoming users would enter the queue being the servers busy but that's not the problem). I ...
0
votes
0answers
7 views

Looking for resources on Harris recurrence

I'm working on a problem (in a not countable space) and it seems that I could get much further with it if I can prove that a certain Markov chain is Harris recurrent (I strongly suspect that it is). I ...
0
votes
1answer
27 views

How to understand this kind of Markov chain?

There are two unidirectionally coupled processes $X_t$ and $Y_t$. The coupling is $Y_t=g(X_{t-\delta},\dots)$. The Bayesian network of the two process is described in the following figure: Now this ...
0
votes
0answers
12 views

rigorous definition of adjoint infinitesimal generator for Markov process?

A Markov process on a $\sigma$-addive measurable space $(E,\Sigma,\mu)$ can be described by a family of operators $(P_t)_{t \geq 0} ,\, P_t: L^\infty(\mu) \to L^\infty(\mu)$. These operators define a ...
0
votes
0answers
44 views

Markov Chains and transition semigoups

I'm trying to figure out what the following statement refers to. A process $X$ is markov with transitions semigroup $(K_t)_{t\geq0}$ and initial distribution $\mu$ if and only if for all ...
0
votes
0answers
15 views

How does one estimate the order of a Markov chain empirically (given the data)?

I have a string of symbols $x_1, x_2, ...., x_n$, ($n$ very large), belonging to a finite alphabet. I know that they are a result of a Markov process, but I want to find out the order of the process. ...
3
votes
0answers
41 views

Correct steps in rewriting expectation to a probability

My knowledge of measure theory and probability spaces is limited, so please keep it relatively simple. Let $\{X(t), ~ t \ge 0\}$ be a Markov process on the countable state space $\mathbb{N}_0$ with ...
2
votes
0answers
46 views

Electrostatic capacity of two spheres with changing radii

Although I have read a lot of questions and answers here, this is my first time actually posting. Feel free to suggest needed edits. My question is the following (in a simplified setting). All this ...
1
vote
0answers
69 views

Doubt Concerning Markov Property

Given a Markovian process $(X_t )_{t\geq 0 }$, is the following property accurate? $$\mathbb E \left[ f(X_{t_1}, X_{t_2},X_{t_3}) \mid \mathcal F ^X_{t_2}\right] = \mathbb E \left[ f(X_{t_1}, ...
1
vote
1answer
29 views

Condition for the existence of a Markov process from the properties of a semigroup $\{T_t\}$

In the article Diffusion processes with continuous coefficients I (1969, Stroock and Varadhan) one finds the following arguments in pages 26-27 "$(\cdots)$ for any $ \epsilon >0, \sup_{x \in ...
0
votes
0answers
44 views

Calculating the generator of a weighted transition function

Let $(P_t)_t$ be the transition function of a Feller-Dynkin process $X$. The usual Banach space of functions that the semigroup $(P_t)_t$ is working on is $C_0(E)$, i.e. continuous functions that ...
0
votes
0answers
19 views

Ergodicity property for continuous-time Harris positive Markov process

The following theorem is Theorem 13.3.3 of Meyn and Tweedie's Markov Chains and Stochastic Stability on page 328 Theorem 13.3.3. If $\Phi$ is positive Harris and aperiodic, then for every initial ...
2
votes
1answer
78 views

Expression for the quotient between two stationary states in a Markov process

I've been thinking about this problem and I would appreciate some help. Consider a finite number of states ($n$) Markov process with transition matrix $Q_{n\times n}$ with the usual properties and ...
0
votes
0answers
27 views

Probability of hitting zero

Suppose time is discrete. $X_{t+1} = X_t + x_t$. $x_t$ is of continuous value, iid with mean zero and finite variance. Let initial condition $X_0>0$, how can I prove that the probability of $X_t$ ...
0
votes
0answers
32 views

Can solution to Rubik's cube be seen from the point of view of Markov Decision Process?

Solving Rubik's cube can be thought of as a Planning problem which has : a state space $S$ a set $G \subseteq S$ of goal states (in this case singleton) actions $A(s) \subseteq A$ applicable in ...
1
vote
1answer
55 views

Is this PMF or PDF?

I am reading a technical report on expectation-maximization (EM) algorithm (http://melodi.ee.washington.edu/people/bilmes/mypapers/em.pdf) and I am confused about something. For HMMs, it defines ...