# Tagged Questions

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

48 views

### Blackwell's example in Markov process theory and Kolmogorov's extension theorem

I'm reading Continuous Time Markov Processes: An Introduction by Thomas M. Liggett. Chapter 2.4 is devoted to Blackwell's example. Let $E=\left\{0,1\right\}$, $\mathcal E:=2^E$ and $X$ be the (...
17 views

### Distribution of $(X(t_1),X(t_2))$ of a diffusion process $X(t)$

I am very new to SDE's and diffusion processes, I came across this diffusion process given by $dX(t)=[\alpha-(\alpha + \beta)X(t)]dt + \sqrt{2X(t)(1-X(t))}dB(t)$ where $B(t)$ is a continuous Brownian ...
54 views

### Convergence of Markov Chain

Could you give me an intuition for the statement: "The Markov chain converges to its stationary distribution"? I know the math behind it. I'm asking for an intuition without using mathematical ...
37 views

19 views

### Deriving/Fitting Origin-Destination Matrix of Directed Graph Flow

Let me preface this by saying that my this area of Mathematics is not my specialty (so pardon me if this is an easy question that I just cannot articulate correctly). I am trying to find a way to ...
109 views

43 views

45 views

### Expected value of visits in a state of a discrete Markov chain [duplicate]

Let $X=(X_n)_{n\in\mathbb N_0}$ be a Markov chain with values in a at most countable Polish space $E$ and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $(\operatorname P_x)_{x\in E}$ be the ...
73 views

### Periodicity of a Markov Chain.

A class property of Markov Chain is periodicity. But I do not understand how is to calculate the period of a state from a transition probability matrix. I am following the book "An Introduction to ...
71 views

100 views

### What is the probability this Markov chain does not reach state $r$?

Consider a random walk on the non-negative integers. You start at $0$, and in each step you either move $1$ higher, or $2$ lower (but can't go below $0$). The direction is picked w.p. $1/2$ ...
30 views

### Continuous Markov Decision Processes: approximate value iteration vs least squares fitting of sampled value function

This is my first question here and I just started getting interested in MDPs, so forgive me for both inaccuracies or unclear questions. Let's talk about continuous MDPs. Sampled based approaches ...
41 views

69 views

### Gambler's Ruin with changing probabilities

I have the following Markov Chain and am trying to evaluate the probability that the Chain reaches state 4 before it returns to state 1, given it starts in state 1. I've seen many typical problems ...
104 views

### Markov chain with infinitely many states

I am stumped on the following infinite Markov Chain. Given the this transition matrix for a Markov chain, how do I determine what values of $x$ the chain is positive recurrent/null recurrent/...
149 views

### How to compute transition matrix for the following Markov chain?

Each morning a runner leaves his house and goes for a jog. He is equally likely to leave either from his front or back door. Upon leaving the house, he chooses a pair of sports shoes (or goes for a ...
24 views

### Explicit transition matrix

An urn $U$ contains always $N$ balls, some white and some black balls. Fix $p \in ]0,1[$; at each stage a coin having probability $p$ of landing heads is flipped. If heads appear, then a ball is ...
115 views

### Irreducible Markov chain and invariant measure

We consider a Markov chain $\left(X,P\right)$ on a finite state space $X$. We denote $P:=\left(p_{x,y}\right)_{x,y\in X}$ and for $n\in\mathbb{N}$ $P^{n}:=\left(p_{x,y}^{(n)}\right)_{x,y\in X}$....
57 views

### A Characterization of the Strong Markov Property

I have a question concerning the strong Markov property: For a strong Markov process $(X_u)_{u\ge 0}$, a real time $t\in \mathbb{R}$ and an optional stopping time $T$ with $t< T$ \begin{align*} \...
46 views

### Markov processes: Hitting times for a point form an i.i.d. sequence

Say that I have a recurrent time-homogenous diffusion process X (continuous strong markov process) and two points $x,y$. If $X_t$ goes from $y$, to $x$ and then back to $y$ again we denote it as a "...
23 views

### Process Transition Algorithm

I have a process with 100 possible states and independent entities going through the process. All the Entities have been observed through a span of 5 years at the end of each month. When the ...
196 views

### Show that Brownian motion on the unit circle is exponentially ergodic and has the uniform measure as its invariant distribution.

My search results keep bring up planar Brownian motion on the unit disk. However, I am specifically referring to $e^{jW_{t}} = [\cos(W_t),\sin(W_t)]^{T}$ where $W_t$ is Brownian motion. I am at a ...
52 views