As of May 31, 2023, we have updated our Code of Conduct.

Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

Filter by
Sorted by
Tagged with
2 votes
1 answer
89 views
+250

Asymptotic behavior of a Markov process on the set of $\{0,1\}$-polynomials.

I am trying to study the asymptotic behavior of a stochastic process defined on the space of single variable polynomials whose coefficients are either $0$ or $1$. Letting $\mathbb{B}=\{0,1\}$, I will ...
Francesco Bilotta's user avatar
0 votes
0 answers
26 views
+50

If $\nu$ is a measure with density $q$, which is invariant with respect to the Markov kernel $\kappa$, why is $\kappa^\ast(fq)=fq$ for all $f$?

Let $(E,\mathcal E)$ be a measurable space and $\kappa$ be a Markov kernel with density $p$ with respect to a measure $\lambda$ on $(E,\mathcal E)$; i.e. $$\kappa(x,B):=\int_Bp(x,y)\;\;\;\text{for }(x,...
0xbadf00d's user avatar
  • 13.4k
0 votes
0 answers
11 views

How did the Markov property reformulated from $X_t|F_s$ to $x_t|x_{t-1}$ in discrete case, what about the continuous case?

I'm reading the wikipedia page on Markov property. It mentioned that the Markov property $$P(X_t\in A|F_s)=P(X_t\in A| X_s)$$ for $s,t\in I$ with $s<t$, where $(F_s,s\in I)$ was a filtration , ...
ShoutOutAndCalculate's user avatar
18 votes
2 answers
1k views

Probability of Sisyphus laboring forever

Zeus has decreed that Sisyphus must spend each day removing all the rocks in a certain valley and transferring them to Mount Olympus. Each night, each rock Sisyphus places on Mount Olympus is subject ...
Danjx's user avatar
  • 309
6 votes
2 answers
186 views

Time Reversal of an Ornstein-Uhlenbeck Process

Suppose we are given a stochastic process $(X_t)_{0\leq t\leq 1}$ which satisfies $$dX_t=-\theta X_tdt+\sigma dB_t,$$ also known as the Ornstein-Uhlenbeck process, where $\theta>0,\sigma\in\mathbb ...
Small Deviation's user avatar
3 votes
1 answer
107 views

Is integer part of Markov random process Markovian

I have this problem to solve. Is the process [$\xi_k$], $k \in N:=\{n1<n2<...\}$ Markovian, where $\xi_t, t \in \mathbb{R}$ is random Markov process. [y] is the integer part of y. I don't know, ...
Chris Smith's user avatar
0 votes
0 answers
27 views

Understanding formulation for Markov chain

There is a formulation for Markov chains that I don't understand. Consider a Markov Chain $\textbf{X}=\{X_n:n=0,1,...\}$, with arbitrary state space $S$ with countably generated $\sigma$-field $\...
potfire's user avatar
  • 11
0 votes
0 answers
15 views

Computing the stationary distribution of a Markov chain (unique eigenvector of a matrix)

What's the fastest (maybe just the fastest known) algorithm for computing the stationary distribution of a Markov chain (assuming it exists)? Equivalently, the fastest way to compute the unique ...
Closed Limelike Curves's user avatar
0 votes
1 answer
35 views

Fourier transform of one dimensional markov chain

I'm working through Keener and Sneyd's Mathematical Physiology I: Cellular Physiology and had trouble solving a derivation. To summarize, we have a 1D Markov chain (illustration in screenshot below). ...
user3277533's user avatar
1 vote
0 answers
28 views

Is the empty Markov Chain a valid Markov Chain? [closed]

I was revising for my exam on Stochastic Processes today and when looking at the definitions it seemed as if the empty set/matrix satisfies the conditions of a Stochastic Matrix. Therefore, I was ...
Matt Helm's user avatar
1 vote
0 answers
15 views

Prove that family of kernels is a family of Markovian transition functions

I have to show that the following family of kernels is a family of Markovian transition functions on $(\mathbb{R}_+,\mathcal{B}(\mathbb{R}_+))$: \begin{equation}\tag{1}\label{eq:identity} P_t f(x) = e^...
user515933's user avatar
0 votes
0 answers
46 views

Calculating long-run average probability for a coin toss when probability of head varies over time

Re-phrased question and edit: I guess people find the question confusing, so following the suggestion of @Henry, this an attempt to rephrase. So there are $n$ biased coins $x = \{1,2,..,n\}$, with ...
Neijal Kanderbalt's user avatar
1 vote
0 answers
37 views

Convergence of all stopped Markov processes vs Convergence of the Markov process

Suppose we are given a continuous-time (strong) Markov process $(X_t)_{t\geq 0}$ on $\mathbb R^d$, perhaps Feller if you will. For any compact subset $K\subset \mathbb R^d$ we may construct a "...
Small Deviation's user avatar
0 votes
0 answers
25 views

Doubly stochastic matrix in Markov and distributions

The transition matrix $\bf P$ of a Markov chain be a doubly stochastic matrix that is all entries are non-negative and all row sums as well as all column sums are equal to $1$. How can I prove that ...
sneha_jerin's user avatar
0 votes
0 answers
21 views

$E(X_n | X_m) = X_m \mu^{n-m}$ for $ m\leq n$ for branching process $X_n$

Let $X_n$ be the size of the $n$th generation in an ordinary branching process with $X_0 = 1, E (X_1) = \mu$, and Var$(X_1) > 0$. It is claimed that for $m \leq n$ $$E(X_n | X_m) = X_m \mu^{n-m}$$ ...
Morcus's user avatar
  • 571
1 vote
0 answers
34 views

Determining if a stochastic process forms a Markov chain

I believe this is a simple problem but im not sure if im satisfied with my solution. We are tossing a fair coin. Let $X_n$ be the number of heads thrown in the $n+1$ and $n+2$ throw. Does the sequence ...
VlakecTomaz's user avatar
0 votes
0 answers
25 views

Distribution of $X_t$ for birth death processes

Given a continuous Markov chain that is a birth-death process on countable space $I$, suppose it starts at state $ i_{0}$, what do we know about the distribution of $X_{t}$ specifically, $\mathbb{E}_{...
wsz_fantasy's user avatar
0 votes
0 answers
31 views

Independent increments implies Markov property?

I am trying to prove the following: If $X =\{X_t : t\geq 0\}$ is a real-valued stochastic process with independent increments then it is also a Markov process: $$ p\{X_t \in A | \mathcal{F}_s\} = ...
KolmogorovTierGod's user avatar
0 votes
0 answers
20 views

Explosion for a birth and death process

I understand that for a birth process with rate $\exp(\lambda_n)$, explosion occurs if there are infinitely many individuals born in finite time, which is iff $\sum \frac{1}{\lambda_n}<\infty$. My ...
orangecat's user avatar
  • 303
0 votes
0 answers
18 views

Does there exist an MDP policy with this property?

Consider a discrete-time MDP with finite states and actions. For any policy $\pi$ and state $s$, let $u_t^{\pi}(s)$ be the expected total reward for using $\pi$ at times $t, t+1, ..., N$ if the ...
AMfrn's user avatar
  • 11
2 votes
1 answer
53 views

Can a continuous markov chain explode with probability strictly between 0 and 1

A continuous markov process $(X_{t})_{t\geq_{0}}$ with corresponding Q matrix Q defined on a countable space $I$ is said to be explosive if $\mathbb{P}_{i}(\xi < \infty) >0$ for some $i \in I$ ...
wsz_fantasy's user avatar
0 votes
0 answers
31 views

probability question in a linear lattice and biased coin toss [duplicate]

So consider this question: Suppose you are on the integral number line then if you toss a biased coin with probability "p" for heads and "q" for tails. You start at 0 and you toss ...
Razz's user avatar
  • 101
0 votes
1 answer
21 views

Question about an inequality for Levy Processes

Let $(X_t)_t$ be a standard Levy process. We assume $t\mapsto X_t$ is continuous in probability or a.s. cadlag. In this case, why does the continuity of $x\mapsto e^{i\xi \cdot x}$ implies that there ...
nomadicmathematician's user avatar
0 votes
2 answers
17 views

A question about a form of Markov ineqauality

I am studying Levy processes, and met the argument that if $X_t$ is a.s. cadlag then $X_t$ is continuous in probability. The proof goes by showing that $$\lim_{u\to t} P(|X_u - X_t|>\epsilon)=\lim_{...
nomadicmathematician's user avatar
0 votes
0 answers
31 views

TV Distance between Limiting Distribution and One Step Transition Probability Distribution of Markov Chains

Edit 05/01/2023: This question may seem obtrusive at first glance, but it is warranted by some reinforcement learning algorithms, such as TD algorithm. In fact, consider building the Bellman Equation ...
Sizhe Ding's user avatar
0 votes
0 answers
20 views

What if the reward of an action in MDP(Markov Decision Process) is not immediately known.

We know that in MDP the reward of an action is required to be immediately known. What if the reward of an action in MDP(Markov Decision Process) is dependent on the later state. For example, consider ...
koko's user avatar
  • 21
1 vote
0 answers
28 views

Markov process characterization and application of monotone class theorem

I'm studying from Continuous Martingales and Brownian Motion, Daniel Revuz & Marc Yor the following proposition about the characterization of Markov process: Where $X_t \colon (\Omega, \mathcal{...
user515933's user avatar
0 votes
0 answers
22 views

Sum of conditions in conditional probablity

I'm taking a class on Markov Processes and my professor claimed that we can sum up $P(A|X_{1}=x_{1},X_{2}=x_{2})$ over all possible $x_{2}$ and thus eliminate term $X_{2}$. I assume he meant that $$ \...
wsz_fantasy's user avatar
0 votes
0 answers
19 views

Gaussian process with scalar input and 2D output

So simulating a $nD$ Weiner process is easy enough, just generate points from a standard $nD$ Gaussian distribution and take the cumulative sum. If $n=1$, I know that such a Weiner process is just a ...
markusas's user avatar
0 votes
0 answers
16 views

Conditional probabilities for coupled Markov processes

I have been working with a continuous-time birth and death process in my research, of the linear type discussed in this paper of Karlin: https://www.jstor.org/stable/24900526. I am struggling to ...
hulsey's user avatar
  • 151
1 vote
1 answer
28 views

induction proof for transition probability matrix

Consider a Markov chain $\left(X_{n}\right)_{n \geq 0}$ on the state space $S=$ $\{1, \ldots, m+1\}, m \geq 1$, with transition probability matrix of dimension $(m+1) \times(m+1)$ $$ P=\left(\begin{...
math math's user avatar
0 votes
0 answers
9 views

Point process with rate function driven by a Markov model

Is there such a thing as a Poisson point process with a rate function λ driven by a discrete-time Markov chain? So, a point process where the observations are generated according to a sequential model?...
BadgerBadgerBadger's user avatar
0 votes
0 answers
23 views

Markov Property for time Homogenous diffusion

I am reading Affine Diffusions and Related Processes: Simulation, Theory and Applications of Alfonsi, at some point (Chapter 1.2.2), the author writes [simplified]: Given: $$ \mathbb{E}\left[\exp(...
DjibSA's user avatar
  • 1
0 votes
1 answer
26 views

Calculating entries in a transition probability matrix

Consider a Markov chain $\left(X_{n}\right)_{n \geq 0}$ on the state space $S=$ $\{1, \ldots, m+1\}, m \geq 1$, with transition probability matrix of dimension $(m+1) \times(m+1)$ $$ P=\left(\begin{...
math math's user avatar
0 votes
0 answers
13 views

Markov property with random walks

I am studying for my course on Simple Symmetric Random Walks and in one section of the notes the following step is take: $$ \mathbb{P}(S_{2n}^{\prime}\neq 0,\dots,S_{2n-2k+1}^{\prime}\neq 0\mid S_{2n-...
TK99's user avatar
  • 395
0 votes
0 answers
18 views

What is the differential form of an invariant probability measure?

The transition probability $p(t,x,dy)$ of a Markov process is defined on a Polish space $X$. An invariant probability distribution for the process is a distribution $\mu$ on $X$ that satisfies $\int p(...
AlexanderGrey's user avatar
1 vote
1 answer
37 views

Variance of event count in a two-state Markov Modulated Poisson Process (2-MMPP) during a given time period

I have a two-state Markov Modulated Poisson Process (2-MMPP). It's a two-state continuous time Markov process, with $\mu_{12}$ the switching rate from state 1 to state 2 ; $\mu_{21}$ the switching ...
Léa's user avatar
  • 31
5 votes
1 answer
115 views

Are harmonic functions measurable for the invariant sigma-algebra?

Let $X$ be a measurable space, and let $\Phi$ be a stationary, discrete-time Markov process with state space $X$ and stationary measure $p$ on $X$. Let $f:X\to R$ be a bounded harmonic function for $\...
geodude's user avatar
  • 7,601
0 votes
0 answers
40 views

Brownian motion reflection principle under the general stopping time

For a general stopping time $\tau$, we define $\hat{B}(t)=B(t)1_{\{t\leq \tau\}}+(2B(\tau)-B(t))1_{\{t>\tau\}}$. I know the strong Markov property $B'(t)=(t+\tau)-B(\tau)$ is also a Brownian motion ...
Tobikawaii's user avatar
0 votes
0 answers
45 views

A consequence of Itô's lemma

In its simplest form, for any twice continuously differentiable function $f$ on the reals and Itô process $X_t$, it states that $f(t, X_t)$ is itself an Itô process. Every Itô process, for suitable ...
ric.san's user avatar
  • 97
5 votes
0 answers
79 views

Converse of Dynkin's formula

It is a well known (see of theorem 19.21 of this book) that any Feller stochastic process $X$ with infinitesimal generator $(\mathcal{L},\mathcal{D})$ satisfies the Dynkin's formula i.e. for any $f \...
Marco's user avatar
  • 371
0 votes
1 answer
38 views

Manually Performing Calculations in a Continuous Time Markov Chain

As a learning exercise, I am trying to learn how to fit and simulate from Continuous Time Markov Chains. Suppose I have a Stochastic Process that can assume 3 States: S1, S2 or S3. Lets say I have ...
stats_noob's user avatar
  • 2,446
1 vote
1 answer
49 views

Harmonic function is almost surely constant

It is a well-known fact of finite-state Markov chains that every harmonic function on a stationary Markov chain is almost-surely constant. Is this true also in continuous space? More precisely, let $X$...
geodude's user avatar
  • 7,601
1 vote
1 answer
56 views

Markov Chain transition probabilities using proof by induction

Let $\left(X_{n}\right)_{n \geq 0}$ be a Markov chain on the state space $S=\mathbb{N}$ with transition probabilities $p_{12}=1, p_{i, i+1}=$ $i(i+2) /(i+1)^{2}$, and $p_{i 1}=1 /(i+1)^{2}, i>1$, ...
math math's user avatar
1 vote
0 answers
21 views

Sampling from Kolmogorov forward equation

I've asked this question over in cross-validated over here:https://stats.stackexchange.com/q/610492/383970 But no answers. I was curious if somebody in this stack has an answer. I've been learning ...
troutman314's user avatar
0 votes
0 answers
11 views

Expectation of discrete time, continuous increments process with a drift

Let $(X_t)_{t=1}^\infty$ be a discrete time process such that for every $t$, If $X_{t-1} \geq 1 $, then $X_t=X_{t-1} + \delta_t$. All $\delta_1,...\delta_i,...$ are iid, $\delta_i \in [-1,1]$ almost ...
AvidLearner's user avatar
1 vote
0 answers
67 views

Expected idle time of a server in an M/M/N queue

Consider a standard M/M/N queue where the jobs' arrival rate is $N\lambda$, the service rate of each of the $N$ servers is $1$, and $\lambda < 1$. When a new job arrives, assign it to an idle ...
zxzx179's user avatar
  • 1,443
0 votes
0 answers
41 views

Markov semigroup: the mathematical definition of $\mathcal{L}(X_t | \mathcal{L}(X_0)=\nu)$

I'm reading about Markov semigroups from these slides, i.e., We consider a Markov process $(X_t)_{t \geq 0}$, with state space $E$, assumed to be ergodic with unique invariant probability measure $\...
Akira's user avatar
  • 14.8k
0 votes
1 answer
42 views

Gambler's ruin markov chain variation

If we play a game where I start with 2 dollars and you start with one dollar, and I have a probability of 1/3 of winning a dollar from you and you have a 1/3 probability of winning a dollar from me, ...
quantrader23's user avatar
0 votes
1 answer
92 views

The solution $X$ of the SDE $\mathrm d X_t = f(t, X_t) \mathrm d t + g(t, X_t) \mathrm d B_t$ is a Markov process

I'm reading Section 5.4 Markov property from these notes, i.e., $\newcommand{\Ex}{\mathbb E}\newcommand{\diff}{\,\mathrm d}$Let $B$ be a standard Brownian motion and $(\mathcal F_t, t \ge 0)$ its ...
Akira's user avatar
  • 14.8k

1
2 3 4 5
49