A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
0answers
22 views

Differences of Markov chain is Markov

In my studies of Markov chains, I was tackled with this tough problem: Let $ \{ X_n \}_{n=0}^{\infty} $ be a homogeneous Markov chain with transition probabilities satisfying $ | i-j | > 1 \to ...
1
vote
0answers
24 views

ergodic theorem for expectation of positive recurrent diffusion

Suppose $X_t$ is a positive recurrent diffusion on $\mathbb{R}$ with invariant probability measure $\mu$. There is an ergodic theorem (see V.53. in Rogers & Williams volume II) that states $$\lim_{...
-2
votes
0answers
20 views

Communicating classes of a power of the irreducible transition matrix? [on hold]

Suppose $P$ is an irreducible transition matrix, with period $d$. Consider the transition matrix $P^k$. In terms of $d$ and $k$, how many communicating classes does $P^k$ have, and what is the period ...
-1
votes
0answers
13 views

Simple Markov property on stopping times

Suppose $(S_n)_{n\geq1}$ is a Markov chain on the two dimensional lattice of the integers. Then define the stopping time $\tau_A'=\inf\{n\geq1:S_n\notin A'\}$ and consider the following for $A\subset ...
0
votes
0answers
27 views

Markov Process converges intuition

To explain my question, I think best to start with the example: assume Markov matrix like this: $ 0 < a < 1$ $$ P = \begin{bmatrix} a & (1-a) \\ (1-a) & a\end{bmatrix}$$ The question is ...
0
votes
1answer
19 views

Optimal average utility of the processing network needed

In "Utility Optimal Scheduling in Processing Networks" by Michael J. Neely et al an example of processing network is provided. There are three queues ($q_1,q_2,q_3$) in the network and two processors (...
-1
votes
0answers
33 views

Determining the infinitesimal generator of a Markov chain [closed]

The infinitesimal generator of a Markov chain $X$ on a countable state space $S$ is defined by $$A(f)(x)=\lim_{t\downarrow 0} \frac{E^x(f(X_t))-f(x)}{t}.$$ Are there any ways of working out $A$ ...
2
votes
0answers
35 views

Fractional powers of Markov generators

Let $H$ be the generator of a symmetric Markov semigroup on $L^2(\mathbb{R}^n).$ Why the fractional power $H^\alpha$ (defined on a proper domain) with $0 < \alpha < 1$ turn out to be the ...
0
votes
0answers
14 views

Coupling a “partially” stationary process?

Take the stationary process $X$ on $\{0,1\}$ with distribution $\pi=(\pi_0,\pi_1).$ Then introduce the rates: $$ \begin{aligned} 0\rightarrow2 & \quad \text{ at rate } \quad \gamma_{02} \\ 1\...
1
vote
1answer
22 views

concise book on MDPs with stress on solving them using DP

What is a good book for MDP with a stress on solving them using DP? However, the book should stress on the theorems and proofs and make a case for why DP is the most popular tool to solve MDPs. I am ...
2
votes
0answers
22 views

Is the infinitesimal generator for Lie groups the same as the infinitesimal generator of a Markov semigroup?

Is the infinitesimal generator for Lie groups related to the infinitesimal generator of a Markov semigroup? Or are they totally different concepts? https://en.wikipedia.org/wiki/Lie_group#...
1
vote
0answers
10 views

Filtering/MCMC methods for this HMM

I have a Discrete HMM with hidden Markovian signals of the form $\{X_t\}_{t \in [0, \infty)} \in \{ 1,2,3\}$ and observed outputs of the form $\{Y_n\}_{n \in \mathbb{N}} \in \{ 1,2\}$. Each ...
1
vote
1answer
30 views

Steady state distribution needed

I have a chain $C_t$. At every instant $t$ an exponential random variable $X_t$ with parameter $\lambda$ is added to the chain or if the chain has a value greater than $Q$ then a value $Q$ is ...
0
votes
0answers
67 views

Convergence of Markov process as some rates tend to infinity

Take the simple two state Markov process characterized by transitions $$ \begin{aligned} 0\rightarrow1 & \quad \text{ at rate } \quad \alpha\lambda \\ 1\rightarrow0 & \quad \text{ at rate } ...
6
votes
0answers
151 views

Limit distributions for Markov chains $X\to\sqrt{U+X}$

This question spawned from a recent, very interesting problem. Let $\varphi=\frac{1+\sqrt{5}}{2}$ and $T$ denote the map on the space of continuous probability density functions supported over $\...
-1
votes
0answers
14 views

how to check if a process satisfies the markovian property with continuous time?

as an example we have A source transmitting messages is alternately on and off. The off-times are independent random variables having a common exponential distribution with rate α and the on-...
1
vote
0answers
12 views

Maximum property, Resolvent, Markov process

I have a question about Markov processes and related topics. Let $E$ be a locally compact separable metric space and $(X_{t},P_{x})$ a Markov process on $E$. For a bounded measurable function $f : E \...
0
votes
0answers
18 views

What is the extended generator of a Heston process?

There is a lot of literature about infinitesimal generators, however I find almost nothing about extended generators. I have the following definition: An operator $\mathcal{G}$ with domain $\mathcal{...
0
votes
0answers
11 views

Markov Decision Process - Value Iteration

I am trying to do the following problem. I have derived the following equation for $ V_{1}(S) $ but it's incorrect and I'm not sure where I am going wrong. $ \gamma $ = 1 $$ V_{1}(S) = max\{0.9*[0+ ...
1
vote
0answers
93 views

Probability of losing lotteries needed

A person earns $x_i$ amount of money every month where $x_i$ is an exponential random variable with parameter $\lambda_1$. The amount $x_i(1-p)y$, here $0 \leq p\leq 1$ and $y$ is exponentially ...
1
vote
0answers
41 views

Showing that the following process is a martingale [closed]

Let $Nf(x) = \frac{1}{\sqrt{2\pi}}\int_\mathbb{R} e^{-\frac{|x-y|^2}{2}}f(y)dy, \;\; f \in b\mathcal{B}(\mathbb{R}), x \in \mathbb{R}.$ Let $X = (X_t, \mathcal{F}_t, \mathbb{P}^x)$ a pure jumps Markov ...
0
votes
1answer
18 views

Prove that Wiener process is Markov process [closed]

Prove that the Wiener process $\xi(t),T\ni t $ that starts from $0$ is the Markov process. I have no idea...
0
votes
0answers
24 views

A markovian transition function

Let $(P_t)_{t \geq 0}$ be a transition function on $(E, \mathcal{B})$ such that $P_t \; 1 \leq 1, \; \forall t \geq 0$. Show that if there is $t_0 > 0$ such that $P_{t_0} 1 = 1$, then $(P_t)_{t \...
0
votes
1answer
42 views

A Markov process with right continuous trajectories and left limits

Let $Nf(x) = \frac{1}{\sqrt{2\pi}}\int_\mathbb{R} e^{-\frac{|x-y|^2}{2}}f(y)dy, \;\; f \in b\mathcal{B}(\mathbb{R}), x \in \mathbb{R}.$ Let $X = (X_t, \mathcal{F}_t, \mathbb{P}^x)$ be a pure jump ...
1
vote
1answer
49 views

Population in a Galton Watson process

Consider a Galton-Watson process, $W_0$, $W_1$, $W_2$ $\ldots$, where $W_0=1$ and the next random variables are defined by the following recursion, $$ W_t = \sum\limits_{i=0}^{W_{t-1}} \xi_i, $$ where ...
0
votes
0answers
14 views

Does EM can drive the complete data likelihood to decrease but likelihood increase?

I tried to apply the Expectation-Maximization(EM) algorithm on Hidden Markov Model(HMM). The data is simulated from the HMM exactly. Initialization: parameters not so far from the true one. E-step: ...
1
vote
2answers
28 views

conditions in which the repair shop process is recurrent (null\positive) or transient

here's the Story: Let $\epsilon_1.\epsilon_2,... $ be i.i.d numbers of machines for repair to the repair shop on mornings of days $1, 2,...$ . Assume that the shop is capable of repairing exactly K ...
1
vote
1answer
26 views

Let $\mathbf{X}$ be a Markov chain on a square find $p_{1,1}(n)$

Consider a square like this $$\begin{array}\\ 1 & - & 2\\ | & & |\\ 3 & - & 4 \end{array} $$ such that you can go from each state with chance $\tfrac{1}{2}$ to the ...
2
votes
1answer
34 views

How does a Markov process inherit its homogeneity to the embedded Markov chain?

A homogenous Markov process $\lbrace X(t),t\geq 0\rbrace $ is given and the embedded Markov chain $Y_0,Y_1,\ldots$ is defined as $Y_n:=X(T_n)$, where the $0=T_0<T_1<\ldots$ are the moments where ...
0
votes
0answers
15 views

Started Counts method for estimating transition probabilities of a discrete time markov chain

I would be very pleased if you could help me with a problem I'm having for my Bachelor's thesis. I'm working on some inventory forecasting methods and one of the method's I'd like to apply is a method ...
1
vote
0answers
16 views

Equation with the expectation of a assessed Markov process

In my book about Markov processes there is following equation in a proof and I don't see why it's right, I already ask some people in the university, but I had no success, can somebody help me? $$E(\...
3
votes
0answers
30 views

Distribution on number of revisits in past $k$ steps of Markov chain

Consider a finite-state Markov chain with transition matrix $P$. The chain starts in a state chosen uniformly over all the states and runs indefinitely from there. We're going to examine only the $k ...
1
vote
1answer
80 views

Is the mapping “positive stochastic matrix onto its Perron-projection” continuous?

I am dealing with a topological question concerning the mapping that maps a positive stochastic matrix onto its invariant distribution. I am asking myself if such a mapping is continuous (or ...
2
votes
0answers
13 views

Uniform convergence of the action of a Feller semigroup in one variable.

Assume we have two subsets of the some euclidean spaces $X\subset \mathbb{R}^m$ and $Y\subset\mathbb{R}^n$ and a a Feller semigroup $(Q_t)_{t\geq 0}$ on $Y$. Suppose also that we have a continuous ...
2
votes
0answers
56 views

Is this transformation of a Markov process again Markovian?

Let $(X_t)_{t\in\mathbb{N}_0}$ be a stationary Markov process valued in $\mathbb{R}$ and $c\in\mathbb{R}$. Is the process $(Y_t)_{t\in\mathbb{N}_0}$ defined by $$ Y_t={\bf 1}{(X_t<c)} $$ again a ...
0
votes
1answer
36 views

Proving specific formula for stationary markov process [closed]

In my probability class, right now we are dealing with Markov chains and I was stumbled by parts of this problem: Given a $ \{ X_n \}_{n=0}^{\infty} $ be a homogeneous Markov chain (the transition ...
1
vote
0answers
30 views

Statement of the strong Markov property in Norris' book

In J.R.Norris' Markov chains book, the strong Markov property for discrete-time, Markov chains is stated and proved as follows: Let $(X_n)_{n \geqslant 0}$ be a Markov chain with transition ...
1
vote
0answers
20 views

Transition functions and Markov processes

I am wondering whether there is a one-to-one correspondence between transition functions and homogeneous Markov processes? We say that $(X_t,\mathcal{F}_t)_{t\geq 0}$ is a Markov process if $\mathbb{...
-1
votes
0answers
23 views

Show there exist $N $ such that $p_{s,s}(n) > 0$ for all $n \geq N$ for an aperiodic state $s$ of a Markov chain

Show there exist $N \geq 1$ such that $p_{s,s}(n) > 0$ for all $n \geq N$ for an aperiodic state $s$ of a Markov chain. Where $p_{s,s}(n)$ is a transition probability. My approach: because $s$ is ...
1
vote
0answers
30 views

Is the process Markov or not?

Consider the stochastic process with $X_0=0$ and $$ X_t= \begin{cases} 0 & \text{ for } \ \ t<\tau_1 \\ 1 & \text{ for } \ \ \tau_1\leq t < \tau_1+\tau_2 \\ 2 & \text{ for } \ \ \...
0
votes
1answer
30 views

Do Markov generators form a linear space?

Let $G_1$ and $G_2$ be generators for two distinct continuous-time Markov processes $X^{(1)}$ and $X^{(2)}$ on a common probability space $\Omega$ (with Markov semigroups $S^{(1)}$ and $S^{(2)}$) so ...
0
votes
1answer
33 views

About the expected transitions in Markov Chain

The problem is here: The given answer is here: K = $2+ X_1 + X_2$, where $X_1$ and $X_2$ are independent exponential random variables with parameters $2/3$ and $3/5$. $$ E[K] = 2=2+1/p_1 +1/p_2 = ...
3
votes
2answers
84 views

Crossing of Brownian Motion Sample Paths

I would like to ask for a more rigorous statement and proof of Lemma on page 5 of this paper. In essence, it states that two distinct sample paths of a Brownian motion does not strictly cross (meaning ...
3
votes
0answers
24 views

How to handle Finite-state-machine with correlated inputs?

My system can be represented by the following state-diagram. where each arch represents Input/Output when a transition is made from one state to the other. The inputs to this FSM are correlated. ...
0
votes
1answer
24 views

$\sum_{i=1}^{n-k}\frac{i}{n-k}\binom{2n-2k}{n-k+i}\frac{1}{2}^{2(n-k)}=\frac{1}{2}^{2n-2k}\binom{2(n-k)-1}{n-k}$

2$\sum_{i=1}^{n-k}\frac{i}{n-k}\binom{2n-2k}{n-k+i}\frac{1}{2}^{2(n-k)}=2\frac{1}{2}^{2n-2k}\binom{2(n-k)-1}{n-k}$. This is an identity in a note for a class in Markov Processes, but I can't ...
1
vote
1answer
22 views

Strong Markov Property for Markov Chains - Statement Verification

I suspect that my handwritten lecture notes for the Strong Markov Property are wrong. I'd appreciate corrections to them. We first define the following: A random variable $\tau$ is called a ...
10
votes
1answer
111 views

Exploiting the Markov property

I've encountered the following problem when dealing with short-rate models in finance and applying the Feynman-Kac theorem to relate conditional expectations to PDEs. Let $(\Omega,\mathcal{F},\{\...
0
votes
1answer
30 views

Finding Initial state vector with given values

I am not sure how to use a given values to form a initial state vector. There are 30% of customers from Company A switch to company B every month, and 35% customers from Company B switch to company A ...
0
votes
2answers
31 views

$\mathcal{P}$ stochastic matrix. If there is $k > 0$ st $\mathcal{P}^k(j, i) > 0$, then there is $r \leq (n-1)$ st $\mathcal{P}^r(j, i) > 0$

Let $\mathcal{P}$ be stochastic matrix of order n. If there is $k > 0$ such that $\mathcal{P}^k(j, i) > 0$, then there is $r \leq (n-1)$ such that $\mathcal{P}^r(j, i) > 0$. My attempt: ...
3
votes
1answer
40 views

Probability that two random walks on $\mathbb{Z}^2$ meet at the origin

Suppose $X,Y$ are symmetric, independent random walks on the lattice $\mathbb{Z}^2$. I am trying to find the probability: $$\mathbb{P}\big(X_n=Y_n=(0,0)\;\text{for some}\;n\,\big|\,X_0=Y_0=(0,N)\big)$$...