A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

7
votes
0answers
142 views

Uniqueness of an infinite system of linear ODEs

How to prove that $\dot{x}=ax,\space x(0)=1$ has a unique solution if $a,x$ are infinite dimensional matrices? More specifically, let $Q$ be a bounded infinitesimal generator, i.e. ...
6
votes
0answers
178 views

Question on Conditional expectation

Let $X_1$ and $X_2$ be two random variables on $(\Omega,\mathcal{B},P)$. Suppose there is a function $g:\mathcal{B}\times\mathbb{R}\rightarrow[0,1]$ such that for any $x$, $g(\cdot,x)$ is a ...
4
votes
0answers
76 views

A Continuous-Time Markov Process Taking All Possible Values

Let $\mathbb{N}$ be the set of positive integers. For each $n \in \mathbb{N}$, let $X^{(n)}=\{ X^{(n)}(t): t \geq 0 \}$ be a Markov chain with state-space the two point set $\{0,1\}$ and $Q$-matrix ...
4
votes
0answers
138 views

Conditional probability and integrating out part of a random walk

Suppose that I have a random walk process defined by $\alpha_{t+1}$ ~ N$(\alpha_t, \omega^2)$. Given $\alpha_t$ and $\alpha_{t+2}$, I understand why the conditional formula for ...
4
votes
0answers
191 views

estimation of transition probabilities from aggregate data

Please, O mathematicians, help me understand the approach to the problem of estimating transition probabilities given only aggregate data in Kalbfleisch & Lawless' 1984 paper "Least-Squares ...
3
votes
0answers
135 views

Generalization of Dobrushin's Ergodic Decomposition for continuous Markov Chains

Let $T$ be the shift transformation. Let $P$ be invariant for $T$ and also define a discrete state space Markov Chain. Let $C_{1},\ldots,C_{n}$ be the connected components of the Markov Chain. It ...
2
votes
0answers
60 views

Markov Chain with Normal Transition Matrix

Consider a (sub)-stochastic matrix $P$, and the associated Markov chain $X$ with \begin{align*} \mathbf P [X_n =y | X_0 = x] = P_{xy}^n. \end{align*} Suppose we have the condition $P^T P = P P^T$, ...
2
votes
0answers
48 views

A theorem in the paper “Noncommuting Random Products” by Furstenberg

I have a question concerning the proof of theorem 2.5 at page 395 of the paper Noncommuting Random Products, by H. Furstenberg, Trans. Amer. Math. Soc., 1963. The statement is as follows: Let $\mu$ ...
2
votes
0answers
49 views

Examples of decreasing-in-some-time-interval variance of a time homogeneous Markovian process

Let $x_t$ be a zero mean, time homogeneous Markovian process over time $t$ starting from $x_0=0$. What are the examples of $x_t$ where the variance at $t$ decrease over some interval of $t$? The ...
2
votes
0answers
31 views

Exsistence and uniqueness of stationary density for Markov Chain

Suppose we're given a function $f:\mathbb{R}^2\to\mathbb{R}$. We define a Markov Chain $(X_n)$ by \begin{align} X_0&\sim f_X, \\ X_n&=f(X_{n-1},Y_{n-1}), \end{align} where $(Y_n)$ is a ...
2
votes
0answers
70 views

Interpretation of potential kernels for Markov processes

One can associate a strongly continuous contraction semi group (SCCSG) to a Markov process with state space $S$ through its transition function, say $P_t$. Now one can interpret $P_t$ as a linear ...
2
votes
0answers
77 views

References for basics of Piecewise-Deterministic Markov Processes

I am looking for introductory/pedagogical material to Piecewise-Deterministic Markov Processes (see http://en.wikipedia.org/wiki/Piecewise-deterministic_Markov_process) (For the moment I am interested ...
2
votes
0answers
236 views

Is every killed Markov process still a Markov process?

Suppose we've got $X=(X(t))_{t\geq 0}$. $X$ is a strong Markov process with respect to filtration $\mathcal{F}_t$, taking values in some subset of $\mathbb{R}$. We take $\tau$ - a stopping time w.r.t ...
1
vote
0answers
9 views

Proportion of arrivals taking a particular path in a Routing Matrix

I have a routing matrix with Node-0 being the source/sink (outside world) and there are service Nodes 1,2..k in the system. The matrix has entries R_ij = Probability of an arrival at Node-i moving to ...
1
vote
0answers
20 views

Stochastic scheduling to maximize the expected number of customers arrived at the root of a Jackson tree

In a Jackson network, organized as a tree rooted at queue r, several customers are queued at time 0 and there is no new customer arrival. The service time of each customer in queue i is geometrically ...
1
vote
0answers
26 views

Kolmogorov backward equations for Birth-Death

I'm trying to solve the Kolmogorov backward equations for a Birth-Death Markov chain with three states. I have 2 equations: $$P_{00}'(t) = \lambda_0 (P_{10}(t)-P_{00}(t))$$ $$P_{10}'(t) = ...
1
vote
0answers
32 views

Chapman-Kolmogorov equations of time inhomogenous Markov chains

Let us assume that we are given a time inhomogenous Markov chain in continuous time (ICTMC) $(X(t))_{t \geq0}$ with a finite state space $\{1,\ldots,n\}$. Set $P(t)_{i,j} := \mathbb{P}(X(t) = j \mid ...
1
vote
0answers
54 views

Can we find a correlation between states of a Markov chain?

I have a fair bit of knowledge on Markov chains but I recently wondered if there is a way to find out a correlation between the states of a finite Markov chain. I could not find any material on this. ...
1
vote
0answers
13 views

Application of Strong Markov Property

Theorem SMP (Strong Markov Property) Let $X$ be a time homogenous Markov process with $T=\mathbb R_+$ or $\mathbb Z_+$ and let $\tau$ be a stopping time taking countably many values. Then ...
1
vote
0answers
44 views

Linear Filtering Problem (Keynman Fac/Particle Model)

$lienar Filtering Problem $$X_n^1 = X_{n-1}^1 + \epsilon_n *W_n $$ $$X_n^2 = (1-\alpha* \delta) X_{n-1}^2 + \beta*\delta X_n^1 $$ $$X_n^3 = X_{n-1}^3 + \delta*X_n^2$$ above is $$\approx$$ $$dX_n^1 ...
1
vote
0answers
34 views

Stochastic process using Markov chain (thief on the run!!)

I'm given an exercise where we are to simulate a thief escaping from an officer. The thief (let's call him/her T for simplicity) and an officer (O) have four cities to be in. Let's call the cities A, ...
1
vote
0answers
209 views

Hidden Markov Model, transition probabilities which are modeled with an exponential distribution

I'm looking at implementing an algorithm described in a paper, but I'm having trouble understanding how the transition probabilities for a Hidden Markov Model are defined. In the first sections, I ...
1
vote
0answers
9 views

A simple question on the inverse z-transformation of $\frac{z}{1-z}\mathscr{T}(z)\mathbf{q}$

I'm wondering if anyone who is familiar with the book Dynamic Programming and Markov Processes by Ronald Howard or simply z-transform can help me figure out an inverse z-transformation on page 23 of ...
1
vote
0answers
18 views

Simple random walk conditioning on non-return

Consider a simple symmetric random walk on $\mathbb{Z}$, $(S_t)_{t \geq 0}$, with $S_0=0$. Let $P_{k,j}$ be the probability that the walker hits the point $k$ without returning to the origin in ...
1
vote
0answers
56 views

Follow-up on solution to Markov process equation

I asked a question here about solving a system related to an absorbing Markov chain. I now have a variation where there are $m$ types (of student, job seeker, etc) each of which applies to ...
1
vote
0answers
26 views

Branching Brownian Motion and KPP equation

I have troubles understanding the proof of the connection between BBM and KPP equation. I mean the proof of the next lemma from the lecture notes of Anton Bovier about BBM, link. This is almost whole ...
1
vote
0answers
50 views

References for time-inhomogeneous Markov jump processes?

In some central models in life insurance mathematics, the state of the insured is modeled using a continuous-time time-inhomogeneous Markov process with finitely many states. While many results for ...
1
vote
0answers
29 views

Analysis of Steady State Probability for Markov Process

I have a balance equation, representing a Markov Chain, which yields $$ (K - z) \pi(Z_c = z) = (\lambda_c + (z+1)x) \pi(Z_c = z+1) $$ where K is the maximum state of the server. The term $\lambda_c$ ...
1
vote
0answers
21 views

When are the marginals of an extremal invariant measure also extremal invariant?

Let's suppose that $X$ is a compact metric space, and thus as is $X \times X$. If given a Markov process on $X \times X$ with marginals that are Markov processes on $X$, then we know that the ...
1
vote
0answers
62 views

How's the damping factor in Google PageRank algorithm calculated

I'm doing some researches about Google's PageRank algorithm for my thesis, I've found that the damping factor x (for example), where x is in : P` = x.P + (1-x)Q where P is the original ...
1
vote
0answers
25 views

showing a condition implies convergence to invariant distribution

For an arbitrary Markov chain, I'm trying to show that $\lVert{ P_{\mu}\circ \theta_n^{-1} - \nu \rVert}\to 0$ iff $\sum_j \lvert{ P_{ij}^n - \nu(j)\rvert}\to 0$, where $\theta_n^{-1}$ is a shift ...
1
vote
0answers
20 views

Three-State Markov Process with Differential Equations

This question is from a take home quiz and I could really use the help. Thanks in advance
1
vote
0answers
44 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
1
vote
0answers
28 views

Prove equilibrium theorem without irreducibility and aperiodicity

I have to solve the following question: Consider a random walk Markov chain on $S = \{1, 2, \ldots, 100\}$. If the chain is between 2 and 99, it selects one of the adjacent states with equal ...
1
vote
0answers
35 views

What does stationarity of the point process entail in a Markovian setting?

Let $(\Omega,\mathcal{F},P)$ be a probability space where $\Omega$ is the set of cadlag trajectories from $\mathbb{R}$ to a countable state space $S$. Let $X$ be the coordinate process $X_t(\omega) = ...
1
vote
0answers
81 views

MDP problem - How is the expected cost calculated?

I have been stuck with a problem for a while regarding Markov Decision Processes for a Policy improvement algorithm. Assume that I have probabilities for certain states to evolve the system into, ...
1
vote
0answers
43 views

State Space Difference Linear Dynamic System

I am interested in finding the DIFFERENCE in the state space distributions for two linear dynamical systems (System A and System B). I am able to solve for this using the matrix exponential. But the ...
1
vote
0answers
20 views

Probabilistic event triggered on a Markov Process transition

I would like to assess a system disponibility using a Markov Process. This system has two states : a functionning state 0 and a failure state 1, with a fault rate $\lambda$ and a mean time to repair ...
1
vote
0answers
20 views

Markov operators

Transition probability functions can always be used to generate Markov operators, correct? So is it correct to say that a Markov process is a collection of Markov operators? On the other hand, are ...
1
vote
0answers
100 views

First and second moments of recurrence time in a finite two-dimensional Markov chain

I have a two dimensional finite Markov chain with $(m+1)^2$ states, and with transition rates: $q_x((x,y)\to (x+1,y))=(m-x)\lambda,\quad 0\leq x< m, 0\leq y \leq m$, $q_x((x,y)\to ...
1
vote
0answers
75 views

Find Markov policy that minimizes(maximizes) the expected discounted cost(reward)

It's an exam problem I found online.Here's a link to the pastpaper. The problem is stated as follows. A repairman who services Q facilities moves between location s and location j according to the ...
1
vote
0answers
64 views

Proof for a Markov process example (using measure theory)

Consider the probability space $(\mathbb{R},B(\mathbb{R}),\delta_x)$ for a given $x\in\mathbb{R}$ (where $\delta_x$ is the Dirac measure) and define the process $X_t(\omega)=\omega - t$, for $t\geq ...
1
vote
0answers
28 views

Advanced reference in Markov processes

I am interested in a book which covers the more in depth stuff on continuous time Markov processes e.g. semi-groups, generators... Preferably such book would also contain a list of analysis results ...
1
vote
0answers
29 views

Single evaluation for using exponential sampling until past a point

I am trying to improve an algorithm that looks like the following (and am getting stumped): I am provided with a starting time, rate, and a target time. I then use an exponential distribution to ...
1
vote
0answers
169 views

Is there monotone class theorem used in one of these steps?

IN Rogers & Williams "Diffusions, Markov Process and Martingales" they introduce the resolvent as: $$R_\lambda f(x):=\int_{[0,\infty)}e^{-\lambda t}P_tf(x)dt=\int_ER_\lambda(x,dy)f(y)$$ where ...
1
vote
0answers
80 views

Conditional distributions of (higher-order) autoregressive Markov processes

If we specify an $p$-th order autoregressive process in discrete time by its transition distribution $F_{t|t-1,\ldots,t-p}$, what can be said about lower order conditional distribution where we ...
1
vote
0answers
80 views

How to prove ergodic property from aperiodicity and positive recurrence

How to prove that in case of an irreducible, aperiodic and positive recurrent Markov Chain time average along sample paths is equal to the ensemble average ? i.e. $$\lim_{n\to \infty ...
1
vote
0answers
51 views

Efficient random number generation for sojourn times in semi-Markov processes

I'm doing a self-study of semi-Markov processes and was wondering if there are efficient methods for generating random numbers for sojourn times. For example, generating a bunch of random numbers from ...
1
vote
0answers
31 views

Reference on Generators of Markov Processes

I came across these regime switching geometric Brownian Motions where the drift and volatility switches a number of states which is driven by a Markov Chain. Can someone please point me to a ...
1
vote
0answers
75 views

“To every Q-matrix corresponds a unique Markov process.” Proving uniqueness

"To every Q-matrix corresponds a unique Markov process." I'm trying to understand Klenke's proof of the uniqueness part of this proposition. Klenke's proof Following is an adapted version of ...