# Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

1,842 questions
Filter by
Sorted by
Tagged with
12 views

### Questions on the Kantorovich-Rubinstein duality

Let $\mu,\nu$ be probability measures on a metric space $(E,d)$ endowed with the Borel $\sigma$-algebra and $$\operatorname W_d(\mu,\nu):=\inf_{\gamma\in\mathcal C(\mu,\:\nu)}\int d\:{\rm d}\gamma,$$ ...
8 views

### When does a Markov semigroup preserve differentiability?

Let $E$ be a $\mathbb R$-Banach space (for simplicity, assume $E=\mathbb R$, if you like) and $(\kappa_t)_{t\ge0}$ be a Markov semigroup on $(E,\mathcal B(E))$. I would like to know under which ...
28 views

### About measures acting on measurable functions.

If $\phi : X \rightarrow \mathbb{R}$ is a measurable function and $\mu$ is a measure on $X$ then what is $\mu(\phi)"$? Is this a notation for some function? I came across this notation for the ...
19 views

### The quadratic variation of the Brownian motion almost certainly tends to T

On the segment $[0, T]$, choose $n$ independent points $t_{n,k}$ (each distributed evenly). Prove that the quadratic variation of the Brownian motion on the sequence of random partitions of the ...
41 views

13 views

50 views

### Gamblers ruin problem where I got the same answer for section e and f. My answer for Alice's Probability of winning is 0.02030134814

Consider the gambler’s ruin problem. Alice starts with £a; her opponent Bob starts with £(m − a). In each round, Alice wins £1 from Bob with probability p = 0.4, or loses £1 to Bob with probability 1 −...
30 views

7 views

### Markov decision Processes - Optimal state value function

I want to know how an optimal state value function defined for Markov decision Processes Could anyone be kind enough to define the Optimal State value function for MDP?
50 views

### Question on circular random walk [closed]

A truck transports goods among $10$ points located on a circular route. These goods are carried only from one point to the next with probability $p$, or to the preceding point with probability $q=1-p$....
31 views

### Let $(X_t)$ be a continuous-time Markov chain and $\tau$ the first jump time. Compute $\mathbb E_x [a^{\tau} \phi (X_\tau)]$

Let $(X_t)$ be a continuous-time Markov chain such that The state space $V$ is finite and endowed with discrete topology. The infintesimal generator is $L: V^2 \to \mathbb R$. Let $\alpha \in (0,1)$...
22 views

### How to get $\mathbb E[a^{\tau_1} \phi(X_{\tau_1}) | X_0 =x] = \mathbb E[a^{\tau_2} \phi(X_{\tau_2}) | X_0 =x]$ from Strong Markov property?

Consider a continuous-time Markov chain $(X_t)_{t \ge 0}$ with respect to a completed right-continuous filtration $(\mathcal G_t)_{t \ge 0}$. Suppose that The state space $V$ is finite and endowed ...
23 views

### Does this martingale have right-continuous (or cadlag) sample paths?

Let $(X_t)$ be an irreducible continuous-time Markov chain $(X_t)_{t \ge 0}$ with respect to its canonical filtration $(\mathcal G_t)_{t \ge 0}$. Suppose $(\mathcal G_t)_{t \ge 0}$ is completed and ...
9 views

### Does irreducibility imply this stopping time is almost surely finite?

Let $(X_t)_{t \ge 0}$ be an irreducible continuous-time Markov chain with finite state space $V$. Let $D \subseteq V$ be open and consider the stopping time $T = \inf \{t \ge 0 \mid X_t \in D\}$. The ...