Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

0
votes
0answers
11 views

Proof that thin sets are finely separated

I would like to justify the following claim concerning thin sets found in Chung, Lectures from Markov Processes to Brownian Motion, page 112: Such a set is finely separated in the sense that each ...
1
vote
0answers
9 views

Markov property for unbounded function

Let $(X_t)$ be a Markov process with respect to a filtration $\mathcal{F}_t$. Assume that $P(X_t>0 \, \forall t\geq 0) = 1 $. Denote $E_x$ the expectation under the measure where $X_0=x$. Is it ...
1
vote
1answer
27 views

Markov process: The population distribution of the system after $n$-transitions

I am familiar with discrete markov processes and idea of their convergence towards a stable distribution after some number of steps. But my question is much simpler--more like a validation of some ...
0
votes
0answers
12 views

Residence times of the telegraph process?

The telegraph process is a two state stochastic process defined by the master equation $$ \dot{\pi}_0(t) = \tau^{-1} \pi_1(t) - \sigma^{-1} \pi_0(t) $$ $$ \dot{\pi}_1(t) = \sigma^{-1} \pi_0(t) - \...
-1
votes
1answer
28 views

Invariant measure for Itō diffusion

Let $f\in C^2(\mathbb R)$ be positive and $h\ge 0$. Assume that $g:=f'/f$ is Lipschitz continuous and let $U$ be a strong solution of $${\rm d}U_t=\frac h2g(U_t){\rm d}t+\sqrt h{\rm d}W_t$$ ($W$ being ...
0
votes
0answers
12 views

Second-Order Continuous Time Markov Process (incorporating memory)

It is a simple procedure to convert a second-order discrete Markov chain into a first order chain that represents it. As an example, one could have two states: A and B. One wants to model the fact ...
0
votes
0answers
17 views

Number of returns simple random walk $\mathbb{Z}^d$

I am interested to know the precise numerical value of the expected number of returns to the origin of a simple random walk on $\mathbb{Z}^d$, when $d \geq 3$. Does anyone know where I can find such a ...
0
votes
0answers
16 views

Markov Chain first order applied to attribution modelling

Based on this article I'm using within R the Channel Attribution package to leverage on the Markov Chain in order to attribute conversion between several marketing channels. However, the computation ...
1
vote
0answers
17 views

Show $(B_t )^2$ i.e. square of a Brownian motion is a Markov process. [duplicate]

Problem: Show $(B_t )^2$ i.e. square of a Brownian motion is a Markov process. To do this, I want to show $$P( B_t ^2 | B_{t_1} ^2 , ... , B_{t_n} ^2 )= P( B_t ^2 | B_{t_n} ^2 ) $$ where $0<...
-1
votes
0answers
17 views

Non-unique stationary distribution but all states inter-communicate?

I have a Continuous Time Markov Chain with the following probability transition matrix: $$P_t= \begin{bmatrix} 1-\lambda t e^{-\lambda t} & \lambda t e^{-\lambda t} \\ \mu t e^{-\mu t} &...
0
votes
1answer
18 views

Why is $P^b(B_{T-T_b} \in (-\infty,b))=1/2$

Why is $P^b(B_{T-T_b} \in (-\infty,b))=1/2$ on the set $\{T_b<t\} $ where $T_b=\inf\{t \ge 0 :B_t=b\}$ and $T=t 1_{\{T_b<t\}}+\infty 1_{\{T_b \ge 0\}}$. I am trying to understand Proposition 2....
-1
votes
0answers
16 views

Derive the Conditional Distribution of a Brownian Motion Process [closed]

Let $\ X(t),t \ge 0$ be a Brownian motion process. That is, $\ X(t)$ is a process with independent increments such that: $$\ X(t) - X(s) \sim N(0,t-s), 0\le s \lt t $$ and $\ X(0)= 0$. ...
0
votes
0answers
13 views

When is it necessary to solve Kolmogorov forward equations (KFE) for a Markov Chain?

Say I have a continuous time markov chain, time homogeneous $X$ with a few states (say, 2). I want to know the distribution of where $X$ is at time $t$, call it $\mu_t$, which will be a vector of 2 ...
0
votes
1answer
11 views

Why does $P(Q_t = q | X_{0:L} = i_{0:L}) = P(Q_t = q, X_{0:L} = i_{0:L})$?

This is a derivation of an equation used to maximize the posterior probability that $Q_m = i_m$ given a model and a sequence of observations. $Q_m$ is a RV which maps to some $q \in S$, the state ...
1
vote
1answer
18 views

Conditions for Markov process not to reach point at infinity

My question concerns the book Lectures from Markov Processes to Brownian Motion by Kai Lai Chung, more precisely the remark at the bottom of page 76: We prove later in paragraph 3.3 that on $\{ t &...
1
vote
0answers
29 views

If $Y$ is a Markov chain and $h>0$, why is $(Y_{\lfloor t/h\rfloor})_{t\ge0}$ not a Markov process?

Let $\left(Y^{(n)}_k\right)_{k\in\mathbb N_0}$ be a Markov chain for $n\in\mathbb N$, $(h_n)_{n\in\mathbb N}\subseteq(0,\infty)$ with $h_n\xrightarrow{n\to\infty}\infty$ and $$X^{(n)}_t:=Y^{(n)}_{\...
3
votes
0answers
36 views

How to find probability transition matrix for continuous time markov chain?

In Grimmet and Stirzaker, on page 258 it explains how to find transition probabilities, given a generator matrix: (a) nothing happens during $(t,t+h)$ with probability $1+g_{ii}*h+o(h)$ (b) ...
2
votes
0answers
44 views

Random walk on $\{0,1,…,k\}$, find the average gain in 10 000 steps

I have the following problem which I can't seem to figure out. The problem is as follows. Consider simple random walk on {0, 1, ... , k} with reflecting boundaries at 0 and k, that is, random ...
1
vote
1answer
27 views

Period of each state

I am trying to determine the period of each state $ j = 0, 1, 2$ for this irreducible Markov Chain with transition probability matrix $$P=\begin{bmatrix}0&0&1\\1&0&0\\\frac{1}{2}&\...
0
votes
0answers
5 views

Why conditional independence assumption in GP models is usually valid? Or it isn't?

I'm interested in the ground for making conditional independence assumption (e.g. that different target dimensions do not covary for given input $x$) when we are modelling some multidimensional signal ...
0
votes
0answers
15 views

Markov property for mixed joint densities

Let $X \rightarrow Y \rightarrow Z$ be a Markov chain in that order, $X$ and $Y$ be jointly Gaussian, and $Z$ be a discrete random variable with finite alphabet $\mathcal{Z}$. Denote their mixed ...
2
votes
1answer
66 views

Steps of a Markov chain subordinated to a Poisson process

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $\left(Y^{(n)}_k\right)_{k\in\mathbb N_0}$ be a time-homogeneous Markov chain on $(\Omega,\mathcal A,\operatorname P)$ $D([0,1]):=\...
2
votes
0answers
30 views

Backward Kolmogorov equation for simple markov process

The following exercise is from a course on SDE's and I am a bit stumped. Consider the process. $dX_t=\lambda\left(\xi-X_t \right)dt+\gamma\sqrt{|X_t|}dB_t$ $\lambda,\xi,\gamma>0$ Find $\mathbb{P}...
1
vote
1answer
22 views

Transition rate matrix of a combined birth-death processes.

If the transition rate matrix X of one birth-death process is defined by \begin{bmatrix} -\lambda & \mu \\ \mu & -\lambda \end{bmatrix} and another transition rate matrix Y of a second birth-...
0
votes
0answers
15 views

How do I interpret the following $E^{X_S}[f(X_t)]$?

Given a Markov family $X=\{X_t, \mathcal{F}_t, t\ge 0\}$ on some $(\Omega,\mathcal{F})$,together with a family of probability measures $\{P^x\}_{x \in \mathbb{R}^d}$ on $(\Omega,\mathcal{F})$ and a ...
1
vote
0answers
54 views

Branching process and posterior distribution.

I have the following branching process: (I added the $Z$'s myself for the number of individuals in each gen, hope it's correct.) Assume the offspring distribution is Poisson with expectation $\...
0
votes
1answer
30 views

Stopping time of Feller process

Let $X$ be a Feller process on $\mathbb{R}$ with generator $Gf=\frac{1}{2}f''-f'$ on $C_c^2$. Let $\tau_b$ be the first time that $X$ hits $b\in\mathbb{R}$. Show that for $x > 0$, $bP_x(\tau_b < ...
0
votes
1answer
14 views

What exactly does it mean to multiply a vector by the transition matrix of a Markov process?

I know that given a stationary distribution and 2 state transition matrix that $\begin{pmatrix} \Pi _{1} & \Pi _{2} \end{pmatrix}\begin{pmatrix} P_{00} & P_{01}\\ P_{10}& P_{11} \end{...
1
vote
1answer
24 views

Probability that a random walk does not leave a subgraph after $k$ steps

Suppose I have a connected, finite, graph $G = (V,E)$, and I have some vertex set $U$ such that the subgraph of $G$ induced by the vertices $Y = V \setminus U$ is still connected. Now, suppose I have ...
0
votes
0answers
10 views

Alternative formulation of a markov process

I'm wondering how the markov property can be specified as follows, if anyone can provide more details (this looks awfully like the definition for a martingale): $$E[f(X_t)|\mathcal{F}_s]=E[f(X_t)|\...
1
vote
1answer
14 views

Why $P^\mu[X_{S+t} \in \Gamma \mid \mathcal{F}_{S+}]=P^\mu[X_{S+t} \in \Gamma \mid X_{S}]=0 \text{ ,} P^\mu\text{-a.s. on } \{S=\infty\}$

For any progressively measurable process $X$ and any optional time $S$ of $\{\mathcal{F}_t\}$ why do we have that $$P^\mu[X_{S+t} \in \Gamma \mid \mathcal{F}_{S+}]=P^\mu[X_{S+t} \in \Gamma \mid X_{S}]=...
2
votes
0answers
106 views

Convergence of discrete-time Markov chain to Feller processes

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(X_t)_{t\ge0}$ be a Feller process on $(\Omega,\mathcal A,\operatorname P)$ $(h_d)_{d\in\mathbb N}\subseteq(0,\infty)$ with $$h_d\...
1
vote
0answers
28 views

Backward Kolmogorov equation to find probability

From lecture notes in a course on SDE's. We are tasked with using the backward Kolmogorov equation to find. $\mathbb{P}^{X_t=x}\left(X_T\geq2 \right)$ I am confused by the terminology here. We are ...
0
votes
0answers
15 views

Birth-death process Expected waiting time in when in invariant distribution

Say I have a birth-death process, with birth having Poisson distribution with parameter $\lambda$ and death having poisson distribution $\mu$. Assuming that both stochastic processes, birth and death, ...
0
votes
1answer
20 views

A question on the defintion of Markov process

Included in the defintion of the Markov process is the following $\text{ for } x \in \mathbb{R}^d,s,t \ge 0, \Gamma \in \mathcal{B}(\mathbb{R}^d)$ $$, P^x[X_{t+s} \in \Gamma \mid X_s=y]=P^y[X_t \in \...
0
votes
0answers
11 views

Choosing between non-determinism and probabilistic models

I have a stochastic system such that there are discrete states. At each discrete state, one or more probabilistic transition rules apply. For example, when Sam is in house he can go to school with a ...
0
votes
0answers
75 views

Question related to Discrete Markov Chains

Consider a dual core processor which operates using a cache memory. Two processes cannot simultaneously access the cache i.e. if a process is accessing the cache and another process tries to access it,...
1
vote
2answers
47 views

Markov Chain with Unique Stationary Distribution [closed]

Why is it that a non-negative integers Markov chain with transition prob. matrix (t.p.m) $P$ given by $p_{i,i+1} = p$ and $p_{i,0} = 1 − p$, has a unique stationary distribution $π$?
2
votes
1answer
38 views

A problem on equivalent definitions of Markov property

Suppose that $X, (\Omega,\mathcal{F}),\{P^x\}_{x \in \mathbb{R}^d}$ is a Markov family with shift operators $\{\theta_s\}_{s \ge 0}$ and for every $x \in \mathbb{R}^d,s \ge 0, G \in \mathcal{F}_s$ ...
0
votes
0answers
18 views

When does the semigroup corresponding to a stochastic process has the Feller property?

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(X_t)_{t\ge0}$ be a real-valued process on $(\Omega,\mathcal A,\operatorname P)$ $E$ be a closed subspace of $\left\{f:\mathbb R\to\...
0
votes
0answers
12 views

Mean Time in Illness Death Model

I would like to calculate the mean time "alive" for a progressive, time homogenous, three state Markov model in continous time. It is an illness-death model with three states: healthy, ill, dead. ...
0
votes
0answers
25 views

Pick Balls with Unequal Probabilities

There are some balls with 4 different colors in a black box. The ratio of A color ball is 10%, B is 20%, C is 30% and D is 40%. The rule of this game is that you randomly pick one ball and mark the ...
1
vote
1answer
43 views

How to analyze this type of queue

The setup is as follows: Families arrive at a taxi stand according to a Poisson process with rate $\lambda$. An arriving family finding $N$ other families waiting for a taxi does not wait. Taxis ...
-1
votes
0answers
36 views
+50

Show that the solution of an autonomous SDE is a time-homogeneous Markov process

Let $(\Omega,\mathcal A,\operatorname P)$ be a complete probability space and $$\mathcal N:=\left\{N\in\mathcal A:\operatorname P[N]=0\right\}$$ $(W_t)_{t\ge0}$ be a Brownian motion on $(\Omega,\...
0
votes
0answers
8 views

Which is the markovian process associated to the Dirichlet form given by the inner product?

I know that there is a relation between Dirichlet forms and Markov processes. In particular, for every regular Dirichlet form, there exist a Markovian process associated. I would like to understand ...
0
votes
1answer
21 views

What is the relation between markov process , markov chain and poisson process?

I am unable to understand these concepts clearly. I am getting lost while reading about these. I have googled about these concepts but still nothing helped me to understand these concepts. according ...
0
votes
0answers
13 views

Using Time Homogenous Markov Chain to maximise profit

Suppose there is a company with 30 printing machines and they need to hire staff to operate the machines such that the profit is maximised. The model is that each machine is either 1. Idle state (not ...
1
vote
1answer
37 views

Show that a random walk on a finite grid graph is recurrent

I want to show that a random walk on a graph is recurrent. The graph is a network of nodes which connect together to make a $N \times M$ rectangular grid, such as this, my first thought was to ...
0
votes
0answers
34 views

What's exactly meant if one is saying that the closure of a multivalued operator is the generator of a $C^0$-semigroup?

Let $E$ be $\mathbb R$-Banach space $A$ be a subspace of $E\times E$ and $$\mathcal D(A):=\left\{x\in E\mid\exists y\in E:(x,y)\in A\right\}$$ $(T(t))_{t\ge0}$ be a contractive $C^0$-semigroup on $\...
0
votes
0answers
9 views

reference request: multivariate discrete state space markov processes in continuous time

I'm interested in the stochastic population dynamics of $N$ interacting species. For example, Cox and Miller (1965) outline a birth death process with $N=2$ sexes: $$ p_{ij}'(t) = -(i\mu_1 + j\mu_2 +...