A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
1answer
40 views

Expected value of visits in a state of a discrete Markov chain [duplicate]

Let $X=(X_n)_{n\in\mathbb N_0}$ be a Markov chain with values in a at most countable Polish space $E$ and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $(\operatorname P_x)_{x\in E}$ be the ...
0
votes
1answer
67 views

Periodicity of a Markov Chain.

A class property of Markov Chain is periodicity. But I do not understand how is to calculate the period of a state from a transition probability matrix. I am following the book "An Introduction to ...
4
votes
1answer
70 views

If $τ_x^k$ is the time of the $k$-th entrance of a Markov chain into $x$, then $\text P_x[τ_y^k<∞]=\text P_x[τ_y^1<∞](\text P_y[τ_y^1<∞])^{k-1}$

Let $E$ be at most countable and equipped with the discrete topology and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_n)_{n\in\mathbb N_0}$ be a discrete Markov chain with values in ...
3
votes
1answer
41 views

Does the measurability of $x\mapsto\operatorname P_x[A]$ imply the measurability of $x\mapsto\operatorname E_x[X]$?

Let $(\Omega,\mathcal A)$ and $(E,\mathcal E)$ be measurable spaces $(\operatorname P_x)_{x\in E}$ be a family of probability measures on $(\Omega,\mathcal A)$ such that $$E\ni x\mapsto\operatorname ...
0
votes
0answers
81 views

Waiting times independence and distribution

I am struggling with that: We have irreducible and aperodic Markov Chain on finite state space. There is a state $\alpha$ which is recurrent. We define $\tau_n = \min (m >{\tau_{n-1} : X_m = ...
1
vote
2answers
40 views

Possibly broken definition of the strong Markov property

Let $I\subseteq [0,\infty)$ be closed under addition and $0\in E$ $E$ be a Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_t)_{t\in I}$ be a Markov process with values in ...
0
votes
1answer
91 views

Flea on a triangle

"A flea hops randomly on the vertices of a triangle with vertices labeled 1,2 and 3, hopping to each of the other vertices with equal probability. If the flea starts at vertex 1, find the probability ...
0
votes
0answers
19 views

Steady-state sensitivity analysis based on semi-Markov process with absorbing state

I'm looking for some references in which steady-state sensitivity analysis through perturbation and based on semi-Markov process with absorbing state, have been studied. Is there any references about ...
1
vote
1answer
38 views

Markov processes in paper “Recent Contributions to The Mathematical Theory of Communication”

I was reading the well-known paper by Warren Weaver, "Recent Contributions to The Mathematical Theory of Communication", I stumpled upon the following sentence(p. 5)" A system which produces a ...
0
votes
1answer
37 views

Transition intensities of Markov process

In Markov process, transition intensities from state i to j are defined as derivatives of transition probabilities at zero: $$q_{ij}=p_{ij}'(0)$$ However I can't somehow catch the interpretation of ...
0
votes
2answers
66 views

How do I construct transition matrix for the following?

A shopkeeper runs his shop in an area that typically gets heavy rains. He has three umbrellas. Every day, he goes to his shop in the morning and comes back home in the evening. If it is raining in ...
0
votes
0answers
19 views

A question involving kernels on measurable spaces

Let $(E, \mathcal{B}(E)), (F, \mathcal{B}(F))$ measurable spaces. A $\it{kernel}$ from $(E, \mathcal{B}(E))$ to $(F, \mathcal{B}(F))$ is a map $N : p\mathcal{B}(E) \to p\mathcal{B}(F)$ such that: $$N ...
1
vote
0answers
87 views

Determining if a stochastic process is a markov chain

Let $\{X_t\}_{t \geq 0}$ be two-state Markov Chain with state space $S=\{0,1\}$, transition matrix $$ P= \begin{bmatrix} 1-p & p \\ q & 1-q \end{bmatrix} $$ and initial ...
0
votes
1answer
23 views

Recurrence of $\pi$ irreducible chains with invariant distribution

In Tierney's paper (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.110.5995&rep=rep1&type=pdf) on page 1712 in the first paragraph there's this proposition that if a chain $P$ is ...
4
votes
2answers
97 views

What is the probability this Markov chain does not reach state $r$?

Consider a random walk on the non-negative integers. You start at $0$, and in each step you either move $1$ higher, or $2$ lower (but can't go below $0$). The direction is picked w.p. $1/2$ ...
0
votes
0answers
25 views

Continuous Markov Decision Processes: approximate value iteration vs least squares fitting of sampled value function

This is my first question here and I just started getting interested in MDPs, so forgive me for both inaccuracies or unclear questions. Let's talk about continuous MDPs. Sampled based approaches ...
2
votes
0answers
41 views

Compute the value of $\mathbb{P}_{0}\left[\tau\geq n\right]$ and $\mathbb{P}_{0}\left[\tau=+\infty\right]$

We consider the Markov chain $\left(X_{n}\right)_{n\geq0}$ on $\mathbb{N}$ whose transition matrix $P=\left(p_{k,l}\right)_{k,l\geq0}$ given by $$\forall k\in\mathbb{N} ...
0
votes
0answers
37 views

Definition of Past in Markov Property

Usually in textbooks, the definition of the Markov Property reads as: \begin{equation} P(X_n\leqslant x_n|X_{n-1}\leqslant x_{n-1},...,X_{0}\leqslant x_{0})=P(X_n\leqslant x_n|X_{n-1}\leqslant ...
1
vote
1answer
58 views

Prove that this Markov chain is irreducible if and only if there exist infinitely many $k\geq0$ such that $q_{k}>0$

We consider the Markov chain $\left(X_{n}\right)_{n\geq0}$ on $\mathbb{N}$ whose transition matrix $P=\left(p_{k,l}\right)_{k,l\geq0}$ given by $$\forall k\in\mathbb{N} ...
0
votes
1answer
65 views

Gambler's Ruin with changing probabilities

I have the following Markov Chain and am trying to evaluate the probability that the Chain reaches state 4 before it returns to state 1, given it starts in state 1. I've seen many typical problems ...
2
votes
1answer
100 views

Markov chain with infinitely many states

I am stumped on the following infinite Markov Chain. Given the this transition matrix for a Markov chain, how do I determine what values of $x$ the chain is positive recurrent/null ...
0
votes
1answer
127 views

How to compute transition matrix for the following Markov chain?

Each morning a runner leaves his house and goes for a jog. He is equally likely to leave either from his front or back door. Upon leaving the house, he chooses a pair of sports shoes (or goes for a ...
0
votes
1answer
24 views

Explicit transition matrix

An urn $U$ contains always $N$ balls, some white and some black balls. Fix $p \in ]0,1[$; at each stage a coin having probability $p$ of landing heads is flipped. If heads appear, then a ball is ...
3
votes
1answer
112 views

Irreducible Markov chain and invariant measure

We consider a Markov chain $\left(X,P\right)$ on a finite state space $X$. We denote $P:=\left(p_{x,y}\right)_{x,y\in X}$ and for $n\in\mathbb{N}$ $P^{n}:=\left(p_{x,y}^{(n)}\right)_{x,y\in ...
0
votes
1answer
57 views

A Characterization of the Strong Markov Property

I have a question concerning the strong Markov property: For a strong Markov process $(X_u)_{u\ge 0}$, a real time $t\in \mathbb{R}$ and an optional stopping time $T$ with $t< T$ \begin{align*} ...
0
votes
1answer
46 views

Markov processes: Hitting times for a point form an i.i.d. sequence

Say that I have a recurrent time-homogenous diffusion process X (continuous strong markov process) and two points $x,y$. If $X_t$ goes from $y$, to $x$ and then back to $y$ again we denote it as a ...
1
vote
0answers
23 views

Process Transition Algorithm

I have a process with 100 possible states and independent entities going through the process. All the Entities have been observed through a span of 5 years at the end of each month. When the ...
5
votes
1answer
167 views

Show that Brownian motion on the unit circle is exponentially ergodic and has the uniform measure as its invariant distribution.

My search results keep bring up planar Brownian motion on the unit disk. However, I am specifically referring to $e^{jW_{t}} = [\cos(W_t),\sin(W_t)]^{T}$ where $W_t$ is Brownian motion. I am at a ...
0
votes
0answers
48 views

How is the Laplace transform of the density of a specific point process computed?

I am trying to understand a little of this thesis by Anna Rudas. In particular the continuous model presented in Section 2.2.2. We are given a weight function $w: \mathbb{N} \rightarrow ...
3
votes
2answers
110 views

Stochastic variables independent given Tau

Say we have a filtration $(\mathbb{F}_s)$, and a stopping time $\tau$ w.r.t. to that filtration.Let $X_t$ be a continuous stochastic process (not required to be adapted to the mentioned filtration), ...
1
vote
1answer
147 views

Joint distribution of Brownian motion and its running maximum

$B$ being standard Brownian motion, its running maximum is defined as $M_t = \sup_{0\leq s\leq t} B_s$. I am trying to follow the proof of the following result but I don't understand some of the steps ...
3
votes
1answer
101 views

Markov and strong Markov properties

In my study of strong Markov property of an RCLL canonical Markov process I encounter the following definition: Suppose $Y_t:\omega\rightarrow \omega(t)$ is canonical Markov process with respect to ...
1
vote
0answers
41 views

Transition density of an AR(1) process?

If we have an AR(1) process, i.e: $X_{t+1} = \alpha X_t + e_{t+1}$ with $X_0=0$ then what is its Markov Chain transition density? We know that for a Markov chain, the following holds: $P(X_{t+1}\leq ...
3
votes
0answers
71 views

A Markov Chain probability, conditioned on a random time.

My question: Upon reading theory about diffusion processes, i came across an argument which i believe simplifies to this: Say we have a Borel measurable set $A$ (if it matters you can set $A=\lbrace ...
1
vote
3answers
63 views

Confused about definition of absorption probability

My confusion can probably most easily be explained with an example. Consider the following one step transition matrix : $$ P=\matrix{% & 0 & 1 & 2 & 3 & 4 \\ 0 & ...
0
votes
0answers
24 views

Reference for General state space Markov chain

What is a good reference for general state space Markov chains? Is there a reference which assumes only familiarity with finite/countable state space Markov chains and then extends the results (e.g., ...
0
votes
1answer
81 views

Why is $f(X_t)-\int_0^t Af(X_s) \, ds$ a martingale for a Markov process $(X_t)_{t \geq 0}$?

I think if $A$ is the usual generator for the Markov process $(X_t)_t$ $$A f (x) = \lim_{t \downarrow 0} \frac{\mathbb{E}^{x} [f(X_{t})] - f(x)}{t}$$ then we get that for any "nice" $f$ the process ...
2
votes
1answer
84 views

Understanding the Markov property of Brownian motion

I'm trying to understand the Markov property for Brownian motions in full generality. The textbook I'm following states it like this: Recall that we have a family of measures $P_x, x \in ...
2
votes
0answers
48 views

Expected response time of Continuous time Markov chain

I'm studying CTMC (Continuous Time Markov Chains). I came across the following slide I don't understand how they got $M(t+h) = M(t) + \alpha h + M(t)\lambda h - M(t) \mu h +o(h)$ Could anyone ...
0
votes
2answers
46 views

Markovian systems: Why must controls be independent of state?

I am currently working my way through Probabilistic Robotics by Thrun, Burgard, and Fox. On p. 91, I encountered the following statement: The Markovian assumption implies independence between ...
0
votes
0answers
21 views

Joint Markov Chain (Two Correlated Markov Processes)

I have two Markov Chains, and they exhibit some correlation between them. For instance, when Chain A moves to state i, there is a high likelihood that Chain B moves to state j. How would I go about ...
1
vote
0answers
44 views

Defining a function over time in terms of a random variable that is undefined at a certain time

Let $X_n$ be a random variable taking on one of three values $a,b$ or $c$ over time. That is, for each $n \in \mathbb{N}$, we have $X_n \in \{a,b,c\}$. Also, for each $n \in \mathbb{N}$, let $F_n$ be ...
2
votes
0answers
63 views

Why is the stationary distribution a distribution?

Suppose we have a time-homogeneous, discrete-time, aperiodic, positive recurrent, irreducible Markov chain $(X_t)_{t \geq 0}$ on a discrete state space $E$. It is known that its stationary ...
2
votes
0answers
109 views

How to prove a set is a core for an infinitesimal generator

I am trying to prove that the set $D=\bigcap_{n\geq 1} \{f\in C^2 (\mathbb{R}) :f(0)=\sum_{k\geq 0} f(\frac{k}{\sqrt{n}})g_n (k)\}$ is a core for the infinitesimal generator of reflecting brownian ...
0
votes
1answer
42 views

Build a Markov process from a transition semigroup

Suppose that we have a Markov process $X$ with time-homogeneus transition probability function given by $p(t,x,A)$, with $t>0, x\in E, A \in \mathcal E$, where $(E,\mathcal E)$ is the state space. ...
3
votes
0answers
99 views

Moment Generating Function for Brownian motion's exit of interval.

Let $B(t)$ be a standard BM. Consider the stopping time $T = \inf\{ t > 0: |B(t)| = a\},$ the usual first exit time of the interval $(-a, a).$ We can see that $\mathbb{E} e^{tT} < \infty$ for ...
0
votes
1answer
21 views

A question involving Markov processes

Let $(S, \mathcal{B}, m)$ be a measurable space and $X_p := L^p(S, \mathcal{B}, m)$. Let $T_t \in \mathcal{L}(X_p, X_p)$ be a bounded linear operator defined by $$(T_t f)(x) = \int\limits_S P(t, x, ...
1
vote
0answers
45 views

Krylov-Bogoliubov theorem in discrete and continuous time

Consider the following two similar frameworks (the first one in discrete time, the second one in continuous time) in which I am trying to apply the Krylov-Bogoliubov theorem. 1) (Discrete time.) ...
1
vote
1answer
45 views

Distribution Stopping time under Brownian motions

Considering $W$ the canonical process on $C([0,1],\mathbb{R})$ and the row filtration generated by the coordinate process of $W$, I want to prove that ...
2
votes
0answers
58 views

Semigroup associated to a Markov process

I'm studying the transition semigroup associated to a Markov Process, in particular the Hille-Yosida theorem and the Martingale Problem. In my notes I found : "If $\{T_t\}_t$ is a strongly continuous ...