As of May 31, 2023, we have updated our Code of Conduct.

# Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

2,448 questions
Filter by
Sorted by
Tagged with
89 views
+250

### Asymptotic behavior of a Markov process on the set of $\{0,1\}$-polynomials.

I am trying to study the asymptotic behavior of a stochastic process defined on the space of single variable polynomials whose coefficients are either $0$ or $1$. Letting $\mathbb{B}=\{0,1\}$, I will ...
26 views
+50

20 views

### Explosion for a birth and death process

I understand that for a birth process with rate $\exp(\lambda_n)$, explosion occurs if there are infinitely many individuals born in finite time, which is iff $\sum \frac{1}{\lambda_n}<\infty$. My ...
18 views

### Does there exist an MDP policy with this property?

Consider a discrete-time MDP with finite states and actions. For any policy $\pi$ and state $s$, let $u_t^{\pi}(s)$ be the expected total reward for using $\pi$ at times $t, t+1, ..., N$ if the ...
53 views

### Can a continuous markov chain explode with probability strictly between 0 and 1

A continuous markov process $(X_{t})_{t\geq_{0}}$ with corresponding Q matrix Q defined on a countable space $I$ is said to be explosive if $\mathbb{P}_{i}(\xi < \infty) >0$ for some $i \in I$ ...
31 views

### probability question in a linear lattice and biased coin toss [duplicate]

So consider this question: Suppose you are on the integral number line then if you toss a biased coin with probability "p" for heads and "q" for tails. You start at 0 and you toss ...
21 views

### Question about an inequality for Levy Processes

Let $(X_t)_t$ be a standard Levy process. We assume $t\mapsto X_t$ is continuous in probability or a.s. cadlag. In this case, why does the continuity of $x\mapsto e^{i\xi \cdot x}$ implies that there ...
17 views

19 views

### Gaussian process with scalar input and 2D output

So simulating a $nD$ Weiner process is easy enough, just generate points from a standard $nD$ Gaussian distribution and take the cumulative sum. If $n=1$, I know that such a Weiner process is just a ...
16 views

### Conditional probabilities for coupled Markov processes

I have been working with a continuous-time birth and death process in my research, of the linear type discussed in this paper of Karlin: https://www.jstor.org/stable/24900526. I am struggling to ...
1 vote
28 views

26 views

18 views

40 views

### Brownian motion reflection principle under the general stopping time

For a general stopping time $\tau$, we define $\hat{B}(t)=B(t)1_{\{t\leq \tau\}}+(2B(\tau)-B(t))1_{\{t>\tau\}}$. I know the strong Markov property $B'(t)=(t+\tau)-B(\tau)$ is also a Brownian motion ...
45 views

### A consequence of Itô's lemma

In its simplest form, for any twice continuously differentiable function $f$ on the reals and Itô process $X_t$, it states that $f(t, X_t)$ is itself an Itô process. Every Itô process, for suitable ...
79 views

### The solution $X$ of the SDE $\mathrm d X_t = f(t, X_t) \mathrm d t + g(t, X_t) \mathrm d B_t$ is a Markov process
I'm reading Section 5.4 Markov property from these notes, i.e., $\newcommand{\Ex}{\mathbb E}\newcommand{\diff}{\,\mathrm d}$Let $B$ be a standard Brownian motion and $(\mathcal F_t, t \ge 0)$ its ...