A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
1answer
33 views

Markov processes: Hitting times for a point form an i.i.d. sequence

Say that I have a recurrent time-homogenous diffusion process X (continuous strong markov process) and two points $x,y$. If $X_t$ goes from $y$, to $x$ and then back to $y$ again we denote it as a ...
0
votes
1answer
618 views

Expected state of a Markov chain

Let's start with a slightly trivial Markov chain defined as follows: the beginning state is called $1$ and the set of states is $\mathbb{N}$. At each step, when the current state is $n$, the ...
-1
votes
0answers
31 views

Showing that $(X_n)$ obeys the Markov Property. [on hold]

Consider a process $(X_n)_{n\geq0}$ where we define $X_0 = 0$ and for $n \geq 1$: $$X_n = X_{n-1} + Z_n$$ where $Z_n$ for $n \geq 1$ are independent random variables on $\{ -1, 1 \}$ with ...
3
votes
2answers
99 views

Stochastic variables independent given Tau

Say we have a filtration $(\mathbb{F}_s)$, and a stopping time $\tau$ w.r.t. to that filtration.Let $X_t$ be a continuous stochastic process (not required to be adapted to the mentioned filtration), ...
-1
votes
0answers
13 views

Three State Model [closed]

Okay hey everyone I am given 6 transition intensities are go as follows : \mu _12 = 0.1 \mu _13 = 0.6 \mu _22 = 0.4 \mu _23 = 0.5 \mu _31 = 0.2 \mu _32 = 0.6 Okay now 'using the fact that you can ...
1
vote
0answers
21 views

Process Transition Algorithm

I have a process with 100 possible states and independent entities going through the process. All the Entities have been observed through a span of 5 years at the end of each month. When the ...
0
votes
0answers
25 views

How is the Laplace transform of the density of a specific point process computed?

I am trying to understand a little of this thesis by Anna Rudas. In particular the continuous model presented in Section 2.2.2. We are given a weight function $w: \mathbb{N} \rightarrow ...
1
vote
3answers
590 views

Expected number of visits to state $j$ between successive visits to a state $i$ in a Markov chain given conditional information

Say I have a Markov chain $\{X_n: n \geq 1\}$ with state space $E = \{1,2,3,4,5\}$ and transition matrix, $$ P = \begin{bmatrix} 0 & 1/2 & 1/2 & 0 & 0 \\\ 1/2 & 0 & 1/2 & ...
1
vote
1answer
29 views

Joint distribution of Brownian motion and its running maximum

$B$ being standard Brownian motion, its running maximum is defined as $M_t = \sup_{0\leq s\leq t} B_s$. I am trying to follow the proof of the following result but I don't understand some of the steps ...
0
votes
0answers
49 views

A Counter Example of Doeblin Condition

The question is to prove that the following Markov process doesn't satisfy the Doeblin Condition. Let $X=\{\ldots,-n,\ldots,-1,0,1,\ldots,n,\ldots \} $, the Markov Transition Matrix $P$ is defined as ...
3
votes
1answer
75 views

Resolvent of a Markov process

I have a question about theory of Markov processes. Let $(\Omega,\mathcal{F},P)$ be a probability space. Let $E$ be a Hausdorff topological space and $\mathcal{B}(E)$ be its Borel $\sigma$-algebra ...
3
votes
1answer
60 views

Markov and strong Markov properties

In my study of strong Markov property of an RCLL canonical Markov process I encounter the following definition: Suppose $Y_t:\omega\rightarrow \omega(t)$ is canonical Markov process with respect to ...
1
vote
1answer
158 views

Ornstein-Uhlenbeck process: Markov, but not martingale?

I'm puzzled about properties of the Ornstein-Uhlenbeck process, given by the Itō integral $$ X_t = x e^{-\lambda t} + \sigma \int_0^t e^{-\lambda(t-s)} d W_s \,. $$ I compute that $\{X_t\}$ is not ...
0
votes
1answer
59 views

Why is $f(X_t)-\int_0^t Af(X_s) \, ds$ a martingale for a Markov process $(X_t)_{t \geq 0}$?

I think if $A$ is the usual generator for the Markov process $(X_t)_t$ $$A f (x) = \lim_{t \downarrow 0} \frac{\mathbb{E}^{x} [f(X_{t})] - f(x)}{t}$$ then we get that for any "nice" $f$ the process ...
1
vote
0answers
34 views

Transition density of an AR(1) process?

If we have an AR(1) process, i.e: $X_{t+1} = \alpha X_t + e_{t+1}$ with $X_0=0$ then what is its Markov Chain transition density? We know that for a Markov chain, the following holds: $P(X_{t+1}\leq ...
6
votes
1answer
62 views

Martingale converges to the boundary

I asked an almost same question before and it is solved by considering adjacent $Z_n$ can not be far away and obtain a contradiction. However, if the setting is altered a bit, I wonder whether it is ...
3
votes
0answers
61 views

A Markov Chain probability, conditioned on a random time.

My question: Upon reading theory about diffusion processes, i came across an argument which i believe simplifies to this: Say we have a Borel measurable set $A$ (if it matters you can set $A=\lbrace ...
1
vote
3answers
43 views

Confused about definition of absorption probability

My confusion can probably most easily be explained with an example. Consider the following one step transition matrix : $$ P=\matrix{% & 0 & 1 & 2 & 3 & 4 \\ 0 & ...
0
votes
0answers
17 views

Reference for General state space Markov chain

What is a good reference for general state space Markov chains? Is there a reference which assumes only familiarity with finite/countable state space Markov chains and then extends the results (e.g., ...
1
vote
1answer
50 views

Understanding the Markov property of Brownian motion

I'm trying to understand the Markov property for Brownian motions in full generality. The textbook I'm following states it like this: Recall that we have a family of measures $P_x, x \in ...
0
votes
2answers
35 views

Markovian systems: Why must controls be independent of state?

I am currently working my way through Probabilistic Robotics by Thrun, Burgard, and Fox. On p. 91, I encountered the following statement: The Markovian assumption implies independence between ...
2
votes
0answers
23 views

Expected response time of Continuous time Markov chain

I'm studying CTMC (Continuous Time Markov Chains). I came across the following slide I don't understand how they got $M(t+h) = M(t) + \alpha h + M(t)\lambda h - M(t) \mu h +o(h)$ Could anyone ...
0
votes
0answers
13 views

Joint Markov Chain (Two Correlated Markov Processes)

I have two Markov Chains, and they exhibit some correlation between them. For instance, when Chain A moves to state i, there is a high likelihood that Chain B moves to state j. How would I go about ...
1
vote
0answers
42 views

Defining a function over time in terms of a random variable that is undefined at a certain time

Let $X_n$ be a random variable taking on one of three values $a,b$ or $c$ over time. That is, for each $n \in \mathbb{N}$, we have $X_n \in \{a,b,c\}$. Also, for each $n \in \mathbb{N}$, let $F_n$ be ...
2
votes
0answers
38 views

Why is the stationary distribution a distribution?

Suppose we have a time-homogeneous, discrete-time, aperiodic, positive recurrent, irreducible Markov chain $(X_t)_{t \geq 0}$ on a discrete state space $E$. It is known that its stationary ...
2
votes
1answer
73 views

Autocorrelated, discrete, bounded and symmetric random walk with no edge attraction

I need to move over a discrete set of linearly organized.. let's say "Japan steps" $S=\{0,\dots,c\}, c \in \mathbb{N}^*$. My current position is given by $d \in S$. On each time step, I need to draw ...
1
vote
0answers
91 views

How to prove a set is a core for an infinitesimal generator

I am trying to prove that the set $D=\bigcap_{n\geq 1} \{f\in C^2 (\mathbb{R}) :f(0)=\sum_{k\geq 0} f(\frac{k}{\sqrt{n}})g_n (k)\}$ is a core for the infinitesimal generator of reflecting brownian ...
0
votes
1answer
31 views

Build a Markov process from a transition semigroup

Suppose that we have a Markov process $X$ with time-homogeneus transition probability function given by $p(t,x,A)$, with $t>0, x\in E, A \in \mathcal E$, where $(E,\mathcal E)$ is the state space. ...
2
votes
0answers
34 views

Moment Generating Function for Brownian motion's exit of interval.

Let $B(t)$ be a standard BM. Consider the stopping time $T = \inf\{ t > 0: |B(t)| = a\},$ the usual first exit time of the interval $(-a, a).$ We can see that $\mathbb{E} e^{tT} < \infty$ for ...
0
votes
1answer
21 views

A question involving Markov processes

Let $(S, \mathcal{B}, m)$ be a measurable space and $X_p := L^p(S, \mathcal{B}, m)$. Let $T_t \in \mathcal{L}(X_p, X_p)$ be a bounded linear operator defined by $$(T_t f)(x) = \int\limits_S P(t, x, ...
1
vote
0answers
40 views

Krylov-Bogoliubov theorem in discrete and continuous time

Consider the following two similar frameworks (the first one in discrete time, the second one in continuous time) in which I am trying to apply the Krylov-Bogoliubov theorem. 1) (Discrete time.) ...
1
vote
1answer
38 views

Distribution Stopping time under Brownian motions

Considering $W$ the canonical process on $C([0,1],\mathbb{R})$ and the row filtration generated by the coordinate process of $W$, I want to prove that ...
0
votes
2answers
57 views

Markov process which is not martingale

I have seen the examples of a discrete time martingale that is not a Markov Process. Can you construct me an example of discrete time Markov Process that is not a martingale?
1
vote
0answers
48 views

Semigroup associated to a Markov process

I'm studying the transition semigroup associated to a Markov Process, in particular the Hille-Yosida theorem and the Martingale Problem. In my notes I found : "If $\{T_t\}_t$ is a strongly continuous ...
2
votes
1answer
40 views

Properties of a transient state in a Markov Chain

I have been trying to solve this problem for a while now Prove that if $j$ is transient state, then $\displaystyle\sum_{n=1}^\infty p_{ij}^{(n)}<\infty \ \forall i \in S$, with $S$ the state ...
1
vote
1answer
33 views

Proving technique used to show an equivalence to the definition of a Markov process

Let $X=(X_t)_{t\in I}$ be Markov process with values in a Polish space $E$. I want to show, that there exists a stochastic kernel $\kappa:E\times\mathcal{B}(E)^{\otimes I}\to [0,1]$ such that ...
5
votes
1answer
3k views

Transition Kernel of a Markov Chain

Supposing $X_t$ is a Markov Process, can the transition kernel be defined by $$K_t(x,A):= P(X_{t+1} \in A | X_t = x)?$$ Assume that $X_t : \Omega \to \mathbb{R}^n$. The issue is that under the normal ...
0
votes
1answer
49 views

Assume a die is rolled repeatedly. Find the markov matrix $P$ for the random variable of the time until the next $6$.

Assume a die is rolled repeatedly. Find the markov/transition matrix $P$ for the random variable $X_r$ = the time until the next six at time $r$. My solution was: For $i,j \geq 0$, $P$ is given ...
2
votes
1answer
38 views

Markov processes and $C_0$-semigroups

A Markov process $(X_t)_{t \geq 0}$ in continuous time on $\mathbb{R}^d$ can be described by a semigroup of Markov kernels $(p_t(x,A))_{t \geq 0}$ with $p_0(x,A) = 1_{A}(x)$ and which fulfill the ...
3
votes
2answers
73 views

Markov Chains - Strong Markov Property

I'm struck with an exercise. I tried, but the results don't seem to fit to those proposed. Exercise: Two players play the following game. The one who begins draws two cards from a deck of 40 cards ...
0
votes
1answer
27 views

How to understand this kind of Markov chain?

There are two unidirectionally coupled processes $X_t$ and $Y_t$. The coupling is $Y_t=g(X_{t-\delta},\dots)$. The Bayesian network of the two process is described in the following figure: Now this ...
1
vote
0answers
20 views

Factorization of the Fokker-Planck semigroup

"In the classical theory of Markov processes, the Fokker-Planck semigroup $\{T_t:t\ge 0\}$ can be factored as $T_t=N\circ U_t \circ j$, $t\ge 0$, where $j$ is an embedding, $U_t$ is a group of ...
2
votes
0answers
33 views

Simple Markov property

I want to prove the simple Markov property but I come to a point where I do not see how to conclude. I want to prove $\mathbb{E}_\nu[Z\circ\Theta_t\mid ...
2
votes
1answer
78 views

Expression for the quotient between two stationary states in a Markov process

I've been thinking about this problem and I would appreciate some help. Consider a finite number of states ($n$) Markov process with transition matrix $Q_{n\times n}$ with the usual properties and ...
0
votes
0answers
18 views

model a system with finite users as a Markov Chain

I have to model a system M/M/2 with finite users (4 users) as a Markov Chain (and then find the probality an incoming users would enter the queue being the servers busy but that's not the problem). I ...
0
votes
0answers
7 views

Looking for resources on Harris recurrence

I'm working on a problem (in a not countable space) and it seems that I could get much further with it if I can prove that a certain Markov chain is Harris recurrent (I strongly suspect that it is). I ...
2
votes
0answers
46 views

Electrostatic capacity of two spheres with changing radii

Although I have read a lot of questions and answers here, this is my first time actually posting. Feel free to suggest needed edits. My question is the following (in a simplified setting). All this ...
0
votes
0answers
12 views

rigorous definition of adjoint infinitesimal generator for Markov process?

A Markov process on a $\sigma$-addive measurable space $(E,\Sigma,\mu)$ can be described by a family of operators $(P_t)_{t \geq 0} ,\, P_t: L^\infty(\mu) \to L^\infty(\mu)$. These operators define a ...
0
votes
0answers
44 views

Markov Chains and transition semigoups

I'm trying to figure out what the following statement refers to. A process $X$ is markov with transitions semigroup $(K_t)_{t\geq0}$ and initial distribution $\mu$ if and only if for all ...
0
votes
0answers
15 views

How does one estimate the order of a Markov chain empirically (given the data)?

I have a string of symbols $x_1, x_2, ...., x_n$, ($n$ very large), belonging to a finite alphabet. I know that they are a result of a Markov process, but I want to find out the order of the process. ...