# Tagged Questions

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

151 views

### Ornstein-Uhlenbeck process: Markov, but not martingale?

I'm puzzled about properties of the Ornstein-Uhlenbeck process, given by the Itō integral $$X_t = x e^{-\lambda t} + \sigma \int_0^t e^{-\lambda(t-s)} d W_s \,.$$ I compute that $\{X_t\}$ is not ...
47 views

### Why is $f(X_t)-\int_0^t Af(X_s) \, ds$ a martingale for a Markov process $(X_t)_{t \geq 0}$?

I think if $A$ is the usual generator for the Markov process $(X_t)_t$ $$A f (x) = \lim_{t \downarrow 0} \frac{\mathbb{E}^{x} [f(X_{t})] - f(x)}{t}$$ then we get that for any "nice" $f$ the process ...
23 views

35 views

36 views

### Krylov-Bogoliubov theorem in discrete and continuous time

Consider the following two similar frameworks (the first one in discrete time, the second one in continuous time) in which I am trying to apply the Krylov-Bogoliubov theorem. 1) (Discrete time.) ...
35 views

### Distribution Stopping time under Brownian motions

Considering $W$ the canonical process on $C([0,1],\mathbb{R})$ and the row filtration generated by the coordinate process of $W$, I want to prove that ...
56 views

### Markov process which is not martingale

I have seen the examples of a discrete time martingale that is not a Markov Process. Can you construct me an example of discrete time Markov Process that is not a martingale?
47 views

### Semigroup associated to a Markov process

I'm studying the transition semigroup associated to a Markov Process, in particular the Hille-Yosida theorem and the Martingale Problem. In my notes I found : "If $\{T_t\}_t$ is a strongly continuous ...
39 views

### Properties of a transient state in a Markov Chain

I have been trying to solve this problem for a while now Prove that if $j$ is transient state, then $\displaystyle\sum_{n=1}^\infty p_{ij}^{(n)}<\infty \ \forall i \in S$, with $S$ the state ...
31 views

### Proving technique used to show an equivalence to the definition of a Markov process

Let $X=(X_t)_{t\in I}$ be Markov process with values in a Polish space $E$. I want to show, that there exists a stochastic kernel $\kappa:E\times\mathcal{B}(E)^{\otimes I}\to [0,1]$ such that ...
3k views

### Transition Kernel of a Markov Chain

Supposing $X_t$ is a Markov Process, can the transition kernel be defined by $$K_t(x,A):= P(X_{t+1} \in A | X_t = x)?$$ Assume that $X_t : \Omega \to \mathbb{R}^n$. The issue is that under the normal ...
47 views

### Assume a die is rolled repeatedly. Find the markov matrix $P$ for the random variable of the time until the next $6$.

Assume a die is rolled repeatedly. Find the markov/transition matrix $P$ for the random variable $X_r$ = the time until the next six at time $r$. My solution was: For $i,j \geq 0$, $P$ is given ...
37 views

### Markov processes and $C_0$-semigroups

A Markov process $(X_t)_{t \geq 0}$ in continuous time on $\mathbb{R}^d$ can be described by a semigroup of Markov kernels $(p_t(x,A))_{t \geq 0}$ with $p_0(x,A) = 1_{A}(x)$ and which fulfill the ...
70 views

### Markov Chains - Strong Markov Property

I'm struck with an exercise. I tried, but the results don't seem to fit to those proposed. Exercise: Two players play the following game. The one who begins draws two cards from a deck of 40 cards ...
26 views

### How to understand this kind of Markov chain?

There are two unidirectionally coupled processes $X_t$ and $Y_t$. The coupling is $Y_t=g(X_{t-\delta},\dots)$. The Bayesian network of the two process is described in the following figure: Now this ...
18 views

### Factorization of the Fokker-Planck semigroup

"In the classical theory of Markov processes, the Fokker-Planck semigroup $\{T_t:t\ge 0\}$ can be factored as $T_t=N\circ U_t \circ j$, $t\ge 0$, where $j$ is an embedding, $U_t$ is a group of ...
26 views

42 views

### Calculating the generator of a weighted transition function

Let $(P_t)_t$ be the transition function of a Feller-Dynkin process $X$. The usual Banach space of functions that the semigroup $(P_t)_t$ is working on is $C_0(E)$, i.e. continuous functions that ...
17 views

### Ergodicity property for continuous-time Harris positive Markov process

The following theorem is Theorem 13.3.3 of Meyn and Tweedie's Markov Chains and Stochastic Stability on page 328 Theorem 13.3.3. If $\Phi$ is positive Harris and aperiodic, then for every initial ...
31 views

### Can solution to Rubik's cube be seen from the point of view of Markov Decision Process?

Solving Rubik's cube can be thought of as a Planning problem which has : a state space $S$ a set $G \subseteq S$ of goal states (in this case singleton) actions $A(s) \subseteq A$ applicable in ...
27 views

### Probability of hitting zero

Suppose time is discrete. $X_{t+1} = X_t + x_t$. $x_t$ is of continuous value, iid with mean zero and finite variance. Let initial condition $X_0>0$, how can I prove that the probability of $X_t$ ...
55 views

### Is this PMF or PDF?

I am reading a technical report on expectation-maximization (EM) algorithm (http://melodi.ee.washington.edu/people/bilmes/mypapers/em.pdf) and I am confused about something. For HMMs, it defines ...
17 views

### Discrete-Time Stochastic Calculus and Stopping Times: Resources

In my measure-theoretic probability course we covered what the professor called "discrete-time stochastic calculus". Essentially, it was a three part method for computing certain quantities such as ...
The question looks like a gambling problem. But I am not sure whether it is the same or similar to gambler's ruin problem. Assume I have wealth $W_t$. At each step $t$, I encounter a random shock ...
Consider the following server farm model. Customers arrive at a server farm according to a Poisson process at rate $\lambda$, each requesting a machine for an amount of time that has an exponential ...