A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
0answers
70 views

“To every Q-matrix corresponds a unique Markov process.” Proving uniqueness

"To every Q-matrix corresponds a unique Markov process." I'm trying to understand Klenke's proof of the uniqueness part of this proposition. Klenke's proof Following is an adapted version of ...
1
vote
1answer
275 views

Question on the proof of the Blumenthal 0-1 law in textbook.

I'm studying Stochastic Processes by Richard F. Bass. Within this book I encountered the definition of a Markov process, which is given as follows: We are given a separable metric space $S$ endowed ...
3
votes
1answer
245 views

In a continuous-time Markov process, is the waiting time between jumps a function of the current state?

Two books construct Markov processes from Q-matrices using waiting times and jump chains but differ in whether the waiting times depend on the current state. Can the two be reconciled? Klenke claims ...
1
vote
0answers
34 views

To every Markov process corresponds a Q-matrix? [duplicate]

Possible Duplicate: Logarithm of a Markov Matrix It is known that to every Q-matrix corresponds a unique Markov process. Does the converse hold? Specifically, Can a (discrete state, ...
1
vote
0answers
32 views

Books about Markov Models

I am looking about books on Markov chains, with recent findings such as autoregressive HMM, HMM with inputs, multiple HMM connected together. Is there anything I can look at?
1
vote
1answer
131 views

To every Q-matrix corresponds a unique Markov process

"To every Q-matrix $q$ corresponds a unique Markov process." I'm trying to understand Klenke's proof of the "existence" part of this proposition, namely that given a Q-Matrix $q$, there exists a ...
1
vote
2answers
102 views

A Poisson process's distributions

The Poisson process $\left(N_t\right)_{t\in\left[0,\infty\right)}$ is supposed to be a Markov process, but a Markov process $\left(X_t\right)_{t\in I}$ should be coupled with a family of distributions ...
2
votes
1answer
233 views

Showing that the Poisson process is characterized by five properties

Klenke defines the Poisson process as a family of non-negative integer valued random variables with independent increments, whose increments are distributed according to the Poisson distribution ...
0
votes
1answer
103 views

Reversibility of Markov Process and Exponential Distribution of Transition Rates

I am reading the paper Towards Utility-optimal Random Access Without Message Passing by J. Liu, Y. Yi, A. Proutiere, M. Chiang, H. V. Poor. A sentence in Section 3.2 can be paraphrased as follows: ...
0
votes
1answer
87 views

The strong Markov property with an uncountable index set

The following definition of the strong Markov property, from Klenke's book, supposes an index set $I$ that is not necessarily countable. However, it is explicitly mentioned previously (following Lemma ...
0
votes
2answers
1k views

How to find expected time to reach a state in a CTMC?

Given a simple CTMC with three states 0,1,2. There are three transitions $0 \rightarrow 1$ (with rate $2u$), $1 \rightarrow 2$ (with rate $u$), $1 \rightarrow 0$ (rate $v$).So $2$ is an absorbing ...
12
votes
3answers
272 views

Select a new value from last $N$ values; how long until the last $N$ are all the same?

Say first we have N distinct numbers in a line, like 1,2,3,...,N, in each round, we choose a ...
2
votes
1answer
150 views

Prove that Brownian Motion absorbed at the origin is Markov

I'm trying to prove that Brownian motion absorbed at the origin is a Markov process with respect to the original filtration $\{\mathcal{F}_{t}\}$. To be more specific, let $(B_{t},\mathcal{F}_{t})_{t ...
3
votes
1answer
704 views

Why does a time-homogeneous Markov process possess the Markov property?

Klenke defines (Definition 17.3, p. 346) a time-homogeneous Markov process independently, rather than as a special case of a stochastic process that possesses the Markov property (Definition 17.1, p. ...
2
votes
1answer
100 views

A question about a stochastic process being Markov

Let $(X_{s},\mathcal{F}_{s})$ be a stochastic process adapted to a given filtration. I was told that, in order to prove that $X$ is Markov, it suffice to prove that for any nonnegative, ...
1
vote
1answer
632 views

Using max/min operators in linear programming.

I'm currently implementing a Markov Decision Process using the solver GLPK, I'm following the lecture by Vincent Conitzer, and there is a step I don't understand between the theoretical problem and ...
2
votes
1answer
82 views

Markov property w.r.t. a countable state space

Background Let $\left(X_t\right)_{t \in I}$ ($I\subseteq\mathbb R$) be an $E$-valued stochastic process ($E$ being a Polish space with the Borel $\sigma$-algebra $\mathcal{B}\left(E\right)$) equipped ...
2
votes
1answer
96 views

Expected number of jumps in regular jump HMC

Consider a homogeneous Markov Chain $X$ on a countable state space, ie a jump process. It is said to be regular (does not explode) if there are only a finite number of jumps in every finite interval. ...
1
vote
1answer
77 views

Martingale Problem and PDE's

Let $X$ be a RCLL Markov Process with generator $A$. Then I know that $$ M^f = f(X)-f(X_0)-\int Af(X_s)ds $$ is a martingal for every $f\in \mathcal{D}_A$. If we suppose that $Af=0$, we see that ...
4
votes
1answer
1k views

Transition Kernel of a Markov Chain

Supposing $X_t$ is a Markov Process, can the transition kernel be defined by $$K_t(x,A):= P(X_{t+1} \in A | X_t = x)?$$ Assume that $X_t : \Omega \to \mathbb{R}^n$. The issue is that under the normal ...
0
votes
1answer
280 views

How are the pairs of two independent pure-birth processes a Markov process?

A pure-birth Process is a generalization of a homogeneous Poission process. Whereas in the Poisson process the holding times between jumps are iid exponentially distributed random variables with ...
2
votes
1answer
391 views

Understanding a Markov Chain

I am using a Markov Chain to get the 10 best search results from the union of 3 different search engines. The top 10 results are taken from each engine to form a set of 30 results. The chain starts ...
3
votes
1answer
140 views

Applying equation to Markov process

This seems as an easy question, but however I can't handle it. In the following I need this fact: If $X=(X_t)$ is a Markov process with transition semigroup $(K_t)$ and initial distribution $\mu$ ...
2
votes
1answer
70 views

local variance of Markov decision processes

Does anybody know the notion of "local variance" of Markov decision processes? Any reference would be appreciated. Thanks.
0
votes
1answer
122 views

Steps to convert one linear algebra equation to another

I'm reading a paper that jumps from eq (1) to eq (2) without describing the steps taken. I'd like to understand how/why/what steps allow the transform. (My linear algebra is a bit rusty...) (1) ...
2
votes
2answers
169 views

Markov chain basic positive recurrency question

If a discrete markov chain is stationary (as far as I know: doesn't modify itself with time), irreducible (doesn't have transient states) and aperiodic (no periodic states), is it positive recurrent? ...
2
votes
0answers
190 views

Is every killed Markov process still a Markov process?

Suppose we've got $X=(X(t))_{t\geq 0}$. $X$ is a strong Markov process with respect to filtration $\mathcal{F}_t$, taking values in some subset of $\mathbb{R}$. We take $\tau$ - a stopping time w.r.t ...
4
votes
1answer
3k views

Kendall notation's “General distribution”, what does that mean?

The first and second parameters for the Kendall's notation may have a G value, which stands for General distribution, see here. But what does that mean? What is a ...
1
vote
1answer
339 views

Markov Process: Have you seen this notation and do you know what it means?

Ok, I've already posted this a minute ago, but my text deleted itself while I was editing it :-( So next try: Can you help me to understand the notation my professor uses to describe Markov ...
0
votes
0answers
191 views

How to calculate the limit kernel of a non-ergodic Markov Chain?

This question is about finding the limit kernel $P^\infty$ of a non-ergodic Markov Chain. The problem We consider a TDMC (Time Discrete Markov Chain) $(X_t)_{t \geq 0}$ with $X \in \mathcal{X}$ ...
1
vote
0answers
147 views

Continuous-time Markov process - need help with flux and discrete-time equivalent

I have a continuous-time markov process and I need to calculate the following transition frequencies matrix (aka intensity matrix) transition probabilities all parameters which define permanence ...
0
votes
1answer
113 views

Manipulating ergodic Markov chains in order to make them non-ergodic

Consider a Markov chain, for simplicity let us consider time discrete chains. The problem We consider a TDMC (Time Discrete Markov Chain) $(X_t)_{t \geq 0}$ with $X \in \mathcal{X}$ (having ...
11
votes
2answers
414 views

Difference in probability distributions from two different kernels

I wonder if the probability kernels of Markov processes on the same state space are close enough, does it also hold for the probabilities of the event that depend only on first $n$ values of the ...
1
vote
1answer
225 views

How does this follow? Markov Chain and conditional expectation question.

I have the following from a book: Assume that $$ P_x(\tau_C \circ \theta_{(k-1)N} > N|F_{(k-1)N}) = P_{X_{(k-1)N}}(\tau_C > N). $$ Integrating over $\{ \tau_C > (k-1)N\}$ using the ...
2
votes
1answer
71 views

The property preserved under the perturbations of the kernel

For a measurable space $(E,\mathcal E)$ and a Markov kernel $P:E\times \mathcal E\to[0,1]$ there is a unique homogeneous Markov chain $X$. The first return time is defined as $$ \tau_A = \inf\{k\geq ...
3
votes
1answer
132 views

Levy processes with no positive jumps

Let X be a Levy process with no positive jumps and $\tau_y:=\inf\{t> 0: X_t > y\}$ then we have $$X_{\tau_y}=y\text{ on }\{\tau_y <\infty\}.$$ Could you explain that why? and does it hold ...