A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
0answers
48 views

Blackwell's example in Markov process theory and Kolmogorov's extension theorem

I'm reading Continuous Time Markov Processes: An Introduction by Thomas M. Liggett. Chapter 2.4 is devoted to Blackwell's example. Let $E=\left\{0,1\right\}$, $\mathcal E:=2^E$ and $X$ be the (...
1
vote
0answers
17 views

Distribution of $(X(t_1),X(t_2))$ of a diffusion process $X(t)$

I am very new to SDE's and diffusion processes, I came across this diffusion process given by $dX(t)=[\alpha-(\alpha + \beta)X(t)]dt + \sqrt{2X(t)(1-X(t))}dB(t)$ where $B(t)$ is a continuous Brownian ...
0
votes
2answers
54 views

Convergence of Markov Chain

Could you give me an intuition for the statement: "The Markov chain converges to its stationary distribution"? I know the math behind it. I'm asking for an intuition without using mathematical ...
0
votes
1answer
37 views

For a discrete Markov process $X$, the probability that $X$ started in $x$ returns to $x$ is always positive. So, there are no absorbing states?!

Let $E$ be an at most countable set equipped with the discrete topology and $\mathcal E=2^E$ $X=(X_t)_{t\ge 0}$ be a discrete Markov process with values in $(E,\mathcal E)$ and distributions $(\...
0
votes
1answer
33 views

How can I find out the missing Markov transition probabilities given an incomplete transition graph?

I'm given a transition graph as shown below. I need to fill in the two missing probabilities. Is there a general method for doing this?
3
votes
2answers
100 views

If $X$ is a right-continuous, discrete Markov process, then $\displaystyle\lim_{t\downarrow 0}\operatorname P_x\left[X_t=x\right]=1$

Let $E$ be an at most countable Polish space and $\mathcal E$ be the discrete topology on $E$ $X=(X_t)_{t\ge 0}$ be a discrete Markov process with values $(E,\mathcal E)$ and distributions $(\...
0
votes
0answers
19 views

Deriving/Fitting Origin-Destination Matrix of Directed Graph Flow

Let me preface this by saying that my this area of Mathematics is not my specialty (so pardon me if this is an easy question that I just cannot articulate correctly). I am trying to find a way to ...
5
votes
1answer
109 views

Is a Markov process uniquely determined?

Let $E$ be a Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $I\subseteq[0,\infty)$ be closed under addition and $0\in I$ Please consider the following result: Let $(\...
0
votes
0answers
42 views

Under which conditions on a Markov process $X$ does $\frac 1t\lim_{t\downarrow 0}\operatorname P_x\left[X_t\in B\right]$ exist?

Let $I\subseteq[0,\infty)$ be closed under addition and $0\in I$ $(\Omega,\mathcal A)$ be a measurable space $E$ be a Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_t)_{t\...
2
votes
2answers
43 views

If $x$ is a recurrent state of a discrete Markov chain and the probability to go from $x$ to $y$ is positive, then $y$ is recurrent

Let $E$ be an at most countable Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_n)_{n\in\mathbb N_0}$ be a Markov chain with values $(E,\mathcal E)$, distributions $(\...
1
vote
0answers
30 views

A non-absorbing state of a discrete Markov chain is recurrent if and only if the Green's function explodes at this state

Let $E$ be an at most countable Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_n)_{n\in\mathbb N_0}$ be a Markov chain with values $(E,\mathcal E)$, distributions $(\...
0
votes
1answer
45 views

Expected value of visits in a state of a discrete Markov chain [duplicate]

Let $X=(X_n)_{n\in\mathbb N_0}$ be a Markov chain with values in a at most countable Polish space $E$ and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $(\operatorname P_x)_{x\in E}$ be the ...
0
votes
1answer
73 views

Periodicity of a Markov Chain.

A class property of Markov Chain is periodicity. But I do not understand how is to calculate the period of a state from a transition probability matrix. I am following the book "An Introduction to ...
4
votes
1answer
71 views

If $τ_x^k$ is the time of the $k$-th entrance of a Markov chain into $x$, then $\text P_x[τ_y^k<∞]=\text P_x[τ_y^1<∞](\text P_y[τ_y^1<∞])^{k-1}$

Let $E$ be at most countable and equipped with the discrete topology and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_n)_{n\in\mathbb N_0}$ be a discrete Markov chain with values in $(E,\...
3
votes
1answer
41 views

Does the measurability of $x\mapsto\operatorname P_x[A]$ imply the measurability of $x\mapsto\operatorname E_x[X]$?

Let $(\Omega,\mathcal A)$ and $(E,\mathcal E)$ be measurable spaces $(\operatorname P_x)_{x\in E}$ be a family of probability measures on $(\Omega,\mathcal A)$ such that $$E\ni x\mapsto\operatorname ...
0
votes
0answers
81 views

Waiting times independence and distribution

I am struggling with that: We have irreducible and aperodic Markov Chain on finite state space. There is a state $\alpha$ which is recurrent. We define $\tau_n = \min (m >{\tau_{n-1} : X_m = \...
1
vote
2answers
40 views

Possibly broken definition of the strong Markov property

Let $I\subseteq [0,\infty)$ be closed under addition and $0\in E$ $E$ be a Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_t)_{t\in I}$ be a Markov process with values in $(...
0
votes
1answer
104 views

Flea on a triangle

"A flea hops randomly on the vertices of a triangle with vertices labeled 1,2 and 3, hopping to each of the other vertices with equal probability. If the flea starts at vertex 1, find the probability ...
0
votes
0answers
20 views

Steady-state sensitivity analysis based on semi-Markov process with absorbing state

I'm looking for some references in which steady-state sensitivity analysis through perturbation and based on semi-Markov process with absorbing state, have been studied. Is there any references about ...
1
vote
1answer
38 views

Markov processes in paper “Recent Contributions to The Mathematical Theory of Communication”

I was reading the well-known paper by Warren Weaver, "Recent Contributions to The Mathematical Theory of Communication", I stumpled upon the following sentence(p. 5)" A system which produces a ...
0
votes
1answer
39 views

Transition intensities of Markov process

In Markov process, transition intensities from state i to j are defined as derivatives of transition probabilities at zero: $$q_{ij}=p_{ij}'(0)$$ However I can't somehow catch the interpretation of ...
0
votes
2answers
69 views

How do I construct transition matrix for the following?

A shopkeeper runs his shop in an area that typically gets heavy rains. He has three umbrellas. Every day, he goes to his shop in the morning and comes back home in the evening. If it is raining in ...
0
votes
0answers
19 views

A question involving kernels on measurable spaces

Let $(E, \mathcal{B}(E)), (F, \mathcal{B}(F))$ measurable spaces. A $\it{kernel}$ from $(E, \mathcal{B}(E))$ to $(F, \mathcal{B}(F))$ is a map $N : p\mathcal{B}(E) \to p\mathcal{B}(F)$ such that: $$N ...
1
vote
0answers
91 views

Determining if a stochastic process is a markov chain

Let $\{X_t\}_{t \geq 0}$ be two-state Markov Chain with state space $S=\{0,1\}$, transition matrix $$ P= \begin{bmatrix} 1-p & p \\ q & 1-q \end{bmatrix} $$ and initial ...
0
votes
1answer
24 views

Recurrence of $\pi$ irreducible chains with invariant distribution

In Tierney's paper (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.110.5995&rep=rep1&type=pdf) on page 1712 in the first paragraph there's this proposition that if a chain $P$ is $\...
4
votes
2answers
100 views

What is the probability this Markov chain does not reach state $r$?

Consider a random walk on the non-negative integers. You start at $0$, and in each step you either move $1$ higher, or $2$ lower (but can't go below $0$). The direction is picked w.p. $1/2$ ...
0
votes
0answers
30 views

Continuous Markov Decision Processes: approximate value iteration vs least squares fitting of sampled value function

This is my first question here and I just started getting interested in MDPs, so forgive me for both inaccuracies or unclear questions. Let's talk about continuous MDPs. Sampled based approaches ...
2
votes
0answers
41 views

Compute the value of $\mathbb{P}_{0}\left[\tau\geq n\right]$ and $\mathbb{P}_{0}\left[\tau=+\infty\right]$

We consider the Markov chain $\left(X_{n}\right)_{n\geq0}$ on $\mathbb{N}$ whose transition matrix $P=\left(p_{k,l}\right)_{k,l\geq0}$ given by $$\forall k\in\mathbb{N} p_{k,0}=q_{k},p_{k,k+1}...
0
votes
0answers
37 views

Definition of Past in Markov Property

Usually in textbooks, the definition of the Markov Property reads as: \begin{equation} P(X_n\leqslant x_n|X_{n-1}\leqslant x_{n-1},...,X_{0}\leqslant x_{0})=P(X_n\leqslant x_n|X_{n-1}\leqslant x_{n-1}...
1
vote
1answer
60 views

Prove that this Markov chain is irreducible if and only if there exist infinitely many $k\geq0$ such that $q_{k}>0$

We consider the Markov chain $\left(X_{n}\right)_{n\geq0}$ on $\mathbb{N}$ whose transition matrix $P=\left(p_{k,l}\right)_{k,l\geq0}$ given by $$\forall k\in\mathbb{N} p_{k,k-1}=q_{k},p_{k,k+...
0
votes
1answer
69 views

Gambler's Ruin with changing probabilities

I have the following Markov Chain and am trying to evaluate the probability that the Chain reaches state 4 before it returns to state 1, given it starts in state 1. I've seen many typical problems ...
2
votes
1answer
104 views

Markov chain with infinitely many states

I am stumped on the following infinite Markov Chain. Given the this transition matrix for a Markov chain, how do I determine what values of $x$ the chain is positive recurrent/null recurrent/...
0
votes
1answer
149 views

How to compute transition matrix for the following Markov chain?

Each morning a runner leaves his house and goes for a jog. He is equally likely to leave either from his front or back door. Upon leaving the house, he chooses a pair of sports shoes (or goes for a ...
0
votes
1answer
24 views

Explicit transition matrix

An urn $U$ contains always $N$ balls, some white and some black balls. Fix $p \in ]0,1[$; at each stage a coin having probability $p$ of landing heads is flipped. If heads appear, then a ball is ...
3
votes
1answer
115 views

Irreducible Markov chain and invariant measure

We consider a Markov chain $\left(X,P\right)$ on a finite state space $X$. We denote $P:=\left(p_{x,y}\right)_{x,y\in X}$ and for $n\in\mathbb{N}$ $P^{n}:=\left(p_{x,y}^{(n)}\right)_{x,y\in X}$....
0
votes
1answer
57 views

A Characterization of the Strong Markov Property

I have a question concerning the strong Markov property: For a strong Markov process $(X_u)_{u\ge 0}$, a real time $t\in \mathbb{R}$ and an optional stopping time $T$ with $t< T$ \begin{align*} \...
0
votes
1answer
46 views

Markov processes: Hitting times for a point form an i.i.d. sequence

Say that I have a recurrent time-homogenous diffusion process X (continuous strong markov process) and two points $x,y$. If $X_t$ goes from $y$, to $x$ and then back to $y$ again we denote it as a "...
1
vote
0answers
23 views

Process Transition Algorithm

I have a process with 100 possible states and independent entities going through the process. All the Entities have been observed through a span of 5 years at the end of each month. When the ...
5
votes
1answer
196 views

Show that Brownian motion on the unit circle is exponentially ergodic and has the uniform measure as its invariant distribution.

My search results keep bring up planar Brownian motion on the unit disk. However, I am specifically referring to $e^{jW_{t}} = [\cos(W_t),\sin(W_t)]^{T}$ where $W_t$ is Brownian motion. I am at a ...
0
votes
0answers
52 views

How is the Laplace transform of the density of a specific point process computed?

I am trying to understand a little of this thesis by Anna Rudas. In particular the continuous model presented in Section 2.2.2. We are given a weight function $w: \mathbb{N} \rightarrow \mathbb{R}_{&...
3
votes
2answers
111 views

Stochastic variables independent given Tau

Say we have a filtration $(\mathbb{F}_s)$, and a stopping time $\tau$ w.r.t. to that filtration.Let $X_t$ be a continuous stochastic process (not required to be adapted to the mentioned filtration), ...
1
vote
1answer
167 views

Joint distribution of Brownian motion and its running maximum

$B$ being standard Brownian motion, its running maximum is defined as $M_t = \sup_{0\leq s\leq t} B_s$. I am trying to follow the proof of the following result but I don't understand some of the steps ...
3
votes
1answer
108 views

Markov and strong Markov properties

In my study of strong Markov property of an RCLL canonical Markov process I encounter the following definition: Suppose $Y_t:\omega\rightarrow \omega(t)$ is canonical Markov process with respect to ...
1
vote
0answers
46 views

Transition density of an AR(1) process?

If we have an AR(1) process, i.e: $X_{t+1} = \alpha X_t + e_{t+1}$ with $X_0=0$ then what is its Markov Chain transition density? We know that for a Markov chain, the following holds: $P(X_{t+1}\leq ...
1
vote
3answers
64 views

Confused about definition of absorption probability

My confusion can probably most easily be explained with an example. Consider the following one step transition matrix : $$ P=\matrix{% & 0 & 1 & 2 & 3 & 4 \\ 0 & ...
0
votes
0answers
26 views

Reference for General state space Markov chain

What is a good reference for general state space Markov chains? Is there a reference which assumes only familiarity with finite/countable state space Markov chains and then extends the results (e.g., ...
0
votes
1answer
87 views

Why is $f(X_t)-\int_0^t Af(X_s) \, ds$ a martingale for a Markov process $(X_t)_{t \geq 0}$?

I think if $A$ is the usual generator for the Markov process $(X_t)_t$ $$A f (x) = \lim_{t \downarrow 0} \frac{\mathbb{E}^{x} [f(X_{t})] - f(x)}{t}$$ then we get that for any "nice" $f$ the process ...
2
votes
1answer
92 views

Understanding the Markov property of Brownian motion

I'm trying to understand the Markov property for Brownian motions in full generality. The textbook I'm following states it like this: Recall that we have a family of measures $P_x, x \in \mathbb{R}...
2
votes
0answers
56 views

Expected response time of Continuous time Markov chain

I'm studying CTMC (Continuous Time Markov Chains). I came across the following slide I don't understand how they got $M(t+h) = M(t) + \alpha h + M(t)\lambda h - M(t) \mu h +o(h)$ Could anyone ...
0
votes
2answers
46 views

Markovian systems: Why must controls be independent of state?

I am currently working my way through Probabilistic Robotics by Thrun, Burgard, and Fox. On p. 91, I encountered the following statement: The Markovian assumption implies independence between $x_{...