A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
0answers
15 views

Preposition about the Entries of the Product of Markov Matrices.

Definition: A Markov matrix is an $n \times n$ complex matrix with the sum of the elements in every column equal to 1. My task is to prove that: If A, B are Markov matrices such that $|a_{ij}|\leq1$ ...
2
votes
0answers
29 views

Random walk on d-dimensional torus

I am reading the following paper: http://arxiv.org/pdf/1602.03849v2.pdf I will explain the general setup below. Let $x\in X=\mathbb{T}^d$, where $\mathbb{T}^d$ is the d dimensional torus. Let $\rho$ ...
0
votes
0answers
9 views

Is Markov Chain property true for correlated inputs?

I have a finite state machine (FSM). At time $k$, state is $\theta^k$ and input is $x^k$. The next state $\theta^{k+1}$ and output $y^k$ are completely determined by \begin{align} \theta^{k+1} &=...
0
votes
0answers
15 views

Markov process and Doob-Meyer decomposition

$(\Omega,\mathcal{F},(\mathcal{F}_t)_{t\geq0}),\mathbb{P})$ - a filtered probability space supporting a 1-dimensional Brownian motion $B=(B_t)_{t\geq0}$, where \begin{equation} \mathcal{F}_t=\sigma(...
0
votes
1answer
41 views

the proof for weak law of large numbers

This is the text of proof of the law I don't understand why when $n\to \infty$, $\frac{\sigma^2}{n\epsilon^2} \to +\infty$? isn't $\frac{\sigma^2}{n\epsilon^2}=\frac{1}{k^2}$? how come $\frac{1}{k^2}...
0
votes
0answers
108 views

Calculating probabilities in a Markov chain process [on hold]

I have 3 variables A, B and C with each variable having a probability of 0.6 and 0.4 i.e. A can have states (ON) with probability of 0.6 as well as can remain in certain states (OFF) with probability ...
1
vote
0answers
20 views

Does a linear operator on probability measures determine a Markov kernel?

Let $(\Omega, \mathcal{F})$ be a measurable space. Let $M$ be a linear operator on the space of probability measures on $(\Omega, \mathcal{F})$, i.e. for $\alpha \in [0,1]$ and probability measures $...
0
votes
1answer
45 views

Finding the stationary distribution

For a Markov process with state space $S= \{ 0,1,2,\dots\}$ The one step probabilities are: $p_{0,0}=q$, $ p_{0,2}=p$ and $p_{i,i-1}=q$, $ p_{i,i+1}=p$ for $i \geq1$ where $p+q=1$. The one step ...
0
votes
0answers
18 views

showing that a matrix has repetitive values?

Here my primary aim is to calculate the stationary distribution of a DTMC using left-eigen values i.e, $ \pi = \pi*P$. But for some matrices, I observe that some states a same stationary probability. ...
1
vote
1answer
26 views

What is the main difference between Markov renewal process and Semi-Markov Process?

In The literature, it was said that Semi Markov processes are a continuous-time extension of Markov Renewal Process. We know ...
0
votes
1answer
15 views

Is this a Markov chain when dealing with minimum?

So in my probability studies I just encountered this: For a Markov chain $ X_n $ we have a finite state space $ \{1,2,...,k\} $ such that we can transit from one index to the next and from k only ...
1
vote
0answers
29 views

Differences of Markov chain is Markov

In my studies of Markov chains, I was tackled with this tough problem: Let $ \{ X_n \}_{n=0}^{\infty} $ be a homogeneous Markov chain with transition probabilities satisfying $ | i-j | > 1 \to ...
1
vote
0answers
32 views

ergodic theorem for expectation of positive recurrent diffusion

Suppose $X_t$ is a positive recurrent diffusion on $\mathbb{R}$ with invariant probability measure $\mu$. There is an ergodic theorem (see V.53. in Rogers & Williams volume II) that states $$\lim_{...
-1
votes
0answers
57 views

Simple Markov property on stopping times [on hold]

Suppose $(S_n)_{n\geq1}$ is a Markov chain on the two dimensional lattice of the integers. Then define the stopping time $\tau_A'=\inf\{n\geq1:S_n\notin A'\}$ and consider the following for $A\subset ...
0
votes
1answer
21 views

Optimal average utility of the processing network needed

In "Utility Optimal Scheduling in Processing Networks" by Michael J. Neely et al an example of processing network is provided. There are three queues ($q_1,q_2,q_3$) in the network and two processors (...
2
votes
0answers
39 views

Fractional powers of Markov generators

Let $H$ be the generator of a symmetric Markov semigroup on $L^2(\mathbb{R}^n).$ Why the fractional power $H^\alpha$ (defined on a proper domain) with $0 < \alpha < 1$ turn out to be the ...
0
votes
0answers
16 views

Coupling a “partially” stationary process?

Take the stationary process $X$ on $\{0,1\}$ with distribution $\pi=(\pi_0,\pi_1).$ Then introduce the rates: $$ \begin{aligned} 0\rightarrow2 & \quad \text{ at rate } \quad \gamma_{02} \\ 1\...
1
vote
1answer
24 views

concise book on MDPs with stress on solving them using DP

What is a good book for MDP with a stress on solving them using DP? However, the book should stress on the theorems and proofs and make a case for why DP is the most popular tool to solve MDPs. I am ...
3
votes
0answers
54 views

Is the infinitesimal generator for Lie groups the same as the infinitesimal generator of a Markov semigroup?

Is the infinitesimal generator for Lie groups related to the infinitesimal generator of a Markov semigroup? Or are they totally different concepts? https://en.wikipedia.org/wiki/Lie_group#...
1
vote
0answers
12 views

Filtering/MCMC methods for this HMM

I have a Discrete HMM with hidden Markovian signals of the form $\{X_t\}_{t \in [0, \infty)} \in \{ 1,2,3\}$ and observed outputs of the form $\{Y_n\}_{n \in \mathbb{N}} \in \{ 1,2\}$. Each ...
1
vote
1answer
30 views

Steady state distribution needed

I have a chain $C_t$. At every instant $t$ an exponential random variable $X_t$ with parameter $\lambda$ is added to the chain or if the chain has a value greater than $Q$ then a value $Q$ is ...
0
votes
0answers
69 views

Convergence of Markov process as some rates tend to infinity

Take the simple two state Markov process characterized by transitions $$ \begin{aligned} 0\rightarrow1 & \quad \text{ at rate } \quad \alpha\lambda \\ 1\rightarrow0 & \quad \text{ at rate } ...
7
votes
0answers
185 views

Limit distributions for Markov chains $X\to\sqrt{U+X}$

This question spawned from a recent, very interesting problem. Let $\varphi=\frac{1+\sqrt{5}}{2}$ and $T$ denote the map on the space of continuous probability density functions supported over $\...
1
vote
0answers
14 views

Maximum property, Resolvent, Markov process

I have a question about Markov processes and related topics. Let $E$ be a locally compact separable metric space and $(X_{t},P_{x})$ a Markov process on $E$. For a bounded measurable function $f : E \...
0
votes
0answers
22 views

What is the extended generator of a Heston process?

There is a lot of literature about infinitesimal generators, however I find almost nothing about extended generators. I have the following definition: An operator $\mathcal{G}$ with domain $\mathcal{...
0
votes
0answers
12 views

Markov Decision Process - Value Iteration

I am trying to do the following problem. I have derived the following equation for $ V_{1}(S) $ but it's incorrect and I'm not sure where I am going wrong. $ \gamma $ = 1 $$ V_{1}(S) = max\{0.9*[0+ ...
1
vote
0answers
93 views

Probability of losing lotteries needed

A person earns $x_i$ amount of money every month where $x_i$ is an exponential random variable with parameter $\lambda_1$. The amount $x_i(1-p)y$, here $0 \leq p\leq 1$ and $y$ is exponentially ...
-1
votes
1answer
21 views

Prove that Wiener process is Markov process [closed]

Prove that the Wiener process $\xi(t),T\ni t $ that starts from $0$ is the Markov process. I have no idea...
-1
votes
0answers
26 views

A markovian transition function [on hold]

Let $(P_t)_{t \geq 0}$ be a transition function on $(E, \mathcal{B})$ such that $P_t \; 1 \leq 1, \; \forall t \geq 0$. Show that if there is $t_0 > 0$ such that $P_{t_0} 1 = 1$, then $(P_t)_{t \...
0
votes
1answer
44 views

A Markov process with right continuous trajectories and left limits

Let $Nf(x) = \frac{1}{\sqrt{2\pi}}\int_\mathbb{R} e^{-\frac{|x-y|^2}{2}}f(y)dy, \;\; f \in b\mathcal{B}(\mathbb{R}), x \in \mathbb{R}.$ Let $X = (X_t, \mathcal{F}_t, \mathbb{P}^x)$ be a pure jump ...
1
vote
1answer
52 views

Population in a Galton Watson process

Consider a Galton-Watson process, $W_0$, $W_1$, $W_2$ $\ldots$, where $W_0=1$ and the next random variables are defined by the following recursion, $$ W_t = \sum\limits_{i=0}^{W_{t-1}} \xi_i, $$ where ...
0
votes
0answers
14 views

Does EM can drive the complete data likelihood to decrease but likelihood increase?

I tried to apply the Expectation-Maximization(EM) algorithm on Hidden Markov Model(HMM). The data is simulated from the HMM exactly. Initialization: parameters not so far from the true one. E-step: ...
1
vote
2answers
28 views

conditions in which the repair shop process is recurrent (null\positive) or transient

here's the Story: Let $\epsilon_1.\epsilon_2,... $ be i.i.d numbers of machines for repair to the repair shop on mornings of days $1, 2,...$ . Assume that the shop is capable of repairing exactly K ...
1
vote
1answer
27 views

Let $\mathbf{X}$ be a Markov chain on a square find $p_{1,1}(n)$

Consider a square like this $$\begin{array}\\ 1 & - & 2\\ | & & |\\ 3 & - & 4 \end{array} $$ such that you can go from each state with chance $\tfrac{1}{2}$ to the ...
2
votes
1answer
36 views

How does a Markov process inherit its homogeneity to the embedded Markov chain?

A homogenous Markov process $\lbrace X(t),t\geq 0\rbrace $ is given and the embedded Markov chain $Y_0,Y_1,\ldots$ is defined as $Y_n:=X(T_n)$, where the $0=T_0<T_1<\ldots$ are the moments where ...
0
votes
0answers
17 views

Started Counts method for estimating transition probabilities of a discrete time markov chain

I would be very pleased if you could help me with a problem I'm having for my Bachelor's thesis. I'm working on some inventory forecasting methods and one of the method's I'd like to apply is a method ...
1
vote
0answers
20 views

Equation with the expectation of a assessed Markov process

In my book about Markov processes there is following equation in a proof and I don't see why it's right, I already ask some people in the university, but I had no success, can somebody help me? $$E(\...
3
votes
0answers
32 views

Distribution on number of revisits in past $k$ steps of Markov chain

Consider a finite-state Markov chain with transition matrix $P$. The chain starts in a state chosen uniformly over all the states and runs indefinitely from there. We're going to examine only the $k ...
1
vote
1answer
89 views

Is the mapping “positive stochastic matrix onto its Perron-projection” continuous?

I am dealing with a topological question concerning the mapping that maps a positive stochastic matrix onto its invariant distribution. I am asking myself if such a mapping is continuous (or ...
2
votes
0answers
14 views

Uniform convergence of the action of a Feller semigroup in one variable.

Assume we have two subsets of the some euclidean spaces $X\subset \mathbb{R}^m$ and $Y\subset\mathbb{R}^n$ and a a Feller semigroup $(Q_t)_{t\geq 0}$ on $Y$. Suppose also that we have a continuous ...
2
votes
0answers
58 views

Is this transformation of a Markov process again Markovian?

Let $(X_t)_{t\in\mathbb{N}_0}$ be a stationary Markov process valued in $\mathbb{R}$ and $c\in\mathbb{R}$. Is the process $(Y_t)_{t\in\mathbb{N}_0}$ defined by $$ Y_t={\bf 1}{(X_t<c)} $$ again a ...
0
votes
1answer
36 views

Proving specific formula for stationary markov process [closed]

In my probability class, right now we are dealing with Markov chains and I was stumbled by parts of this problem: Given a $ \{ X_n \}_{n=0}^{\infty} $ be a homogeneous Markov chain (the transition ...
1
vote
0answers
32 views

Statement of the strong Markov property in Norris' book

In J.R.Norris' Markov chains book, the strong Markov property for discrete-time, Markov chains is stated and proved as follows: Let $(X_n)_{n \geqslant 0}$ be a Markov chain with transition ...
1
vote
0answers
26 views

Transition functions and Markov processes

I am wondering whether there is a one-to-one correspondence between transition functions and homogeneous Markov processes? We say that $(X_t,\mathcal{F}_t)_{t\geq 0}$ is a Markov process if $\mathbb{...
1
vote
0answers
32 views

Is the process Markov or not?

Consider the stochastic process with $X_0=0$ and $$ X_t= \begin{cases} 0 & \text{ for } \ \ t<\tau_1 \\ 1 & \text{ for } \ \ \tau_1\leq t < \tau_1+\tau_2 \\ 2 & \text{ for } \ \ \...
0
votes
1answer
30 views

Do Markov generators form a linear space?

Let $G_1$ and $G_2$ be generators for two distinct continuous-time Markov processes $X^{(1)}$ and $X^{(2)}$ on a common probability space $\Omega$ (with Markov semigroups $S^{(1)}$ and $S^{(2)}$) so ...
0
votes
1answer
33 views

About the expected transitions in Markov Chain

The problem is here: The given answer is here: K = $2+ X_1 + X_2$, where $X_1$ and $X_2$ are independent exponential random variables with parameters $2/3$ and $3/5$. $$ E[K] = 2=2+1/p_1 +1/p_2 = ...
3
votes
2answers
91 views

Crossing of Brownian Motion Sample Paths

I would like to ask for a more rigorous statement and proof of Lemma on page 5 of this paper. In essence, it states that two distinct sample paths of a Brownian motion does not strictly cross (meaning ...
3
votes
0answers
30 views

How to handle Finite-state-machine with correlated inputs?

My system can be represented by the following state-diagram. where each arch represents Input/Output when a transition is made from one state to the other. The inputs to this FSM are correlated. ...
0
votes
1answer
25 views

$\sum_{i=1}^{n-k}\frac{i}{n-k}\binom{2n-2k}{n-k+i}\frac{1}{2}^{2(n-k)}=\frac{1}{2}^{2n-2k}\binom{2(n-k)-1}{n-k}$

2$\sum_{i=1}^{n-k}\frac{i}{n-k}\binom{2n-2k}{n-k+i}\frac{1}{2}^{2(n-k)}=2\frac{1}{2}^{2n-2k}\binom{2(n-k)-1}{n-k}$. This is an identity in a note for a class in Markov Processes, but I can't ...