A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

8
votes
0answers
166 views

Uniqueness of an infinite system of linear ODEs

How to prove that $\dot{x}=ax,\space x(0)=1$ has a unique solution if $a,x$ are infinite dimensional matrices? More specifically, let $Q$ be a bounded infinitesimal generator, i.e. ...
6
votes
0answers
201 views

Question on Conditional expectation

Let $X_1$ and $X_2$ be two random variables on $(\Omega,\mathcal{B},P)$. Suppose there is a function $g:\mathcal{B}\times\mathbb{R}\rightarrow[0,1]$ such that for any $x$, $g(\cdot,x)$ is a ...
4
votes
0answers
84 views

A Continuous-Time Markov Process Taking All Possible Values

Let $\mathbb{N}$ be the set of positive integers. For each $n \in \mathbb{N}$, let $X^{(n)}=\{ X^{(n)}(t): t \geq 0 \}$ be a Markov chain with state-space the two point set $\{0,1\}$ and $Q$-matrix ...
4
votes
0answers
145 views

Conditional probability and integrating out part of a random walk

Suppose that I have a random walk process defined by $\alpha_{t+1}$ ~ N$(\alpha_t, \omega^2)$. Given $\alpha_t$ and $\alpha_{t+2}$, I understand why the conditional formula for ...
4
votes
0answers
203 views

estimation of transition probabilities from aggregate data

Please, O mathematicians, help me understand the approach to the problem of estimating transition probabilities given only aggregate data in Kalbfleisch & Lawless' 1984 paper "Least-Squares ...
3
votes
0answers
55 views

Simple random walk conditioning on non-return

Consider a simple symmetric random walk on $\mathbb{Z}$, $(S_t)_{t \geq 0}$, with $S_0=0$. Let $k$ and $j$ be two positive integers. Let $P_{k,j}$ be the probability that the walker hits the vertex ...
3
votes
0answers
139 views

Generalization of Dobrushin's Ergodic Decomposition for continuous Markov Chains

Let $T$ be the shift transformation. Let $P$ be invariant for $T$ and also define a discrete state space Markov Chain. Let $C_{1},\ldots,C_{n}$ be the connected components of the Markov Chain. It ...
2
votes
0answers
53 views

Why are inter arrival times in the continuous version of discrete-time Markov chains always exponentially distributed?

I am curious whether there exist continuous time Markov processes for which the times between jumping times (which I call inter arrival times) are not exponentially distributed, but have some other ...
2
votes
0answers
18 views

Showing which classes are recurrent and which are transient

If I have a Markov chain on states {0,1,2,3,4,5} $$ \mathbf{a} = \matrix{~ & 0 & 1 & 2 & 3 & 4 & 5 \\ 0 & 1/3 & 0 & 2/3 & 0 & 0 & 0 \\ ...
2
votes
0answers
30 views

Define Markov chain and rewrite to recursively solve

Customers arrive at a server with rate $\lambda$ and are served at rate $\mu$. The server breaks down with rate $\gamma$, which causes all customers to leave. New customers can only arrive once the ...
2
votes
0answers
55 views

Model as a continuous time Markov Chain

A system consists of two machines, of which one works and the other is standby. Only the working machine can break down (with rate $\lambda$). If it breaks down the other machine takes over (if it ...
2
votes
0answers
23 views

Probability of going from a set $S$ to its complement on a Markov chain

I need to show that if $\pi$ is the stationary distribution of a Markov chain $M$, then for every set of vertices $S$, the probability to choose a random node in $S$ according to $\pi$ and then going ...
2
votes
0answers
38 views

An irreducible Markov chain is a martingale

Let $\{X_n\}$ be an irreducible Markov chain. Does exist example of such $\{X_n\}$ which is also a martingale given that: a. $\{X_n\}$ is recurrent with finite number of states (but bigger ...
2
votes
0answers
89 views

Markov Chain with Normal Transition Matrix

Consider a (sub)-stochastic matrix $P$, and the associated Markov chain $X$ with \begin{align*} \mathbf P [X_n =y | X_0 = x] = P_{xy}^n. \end{align*} Suppose we have the condition $P^T P = P P^T$, ...
2
votes
0answers
55 views

A theorem in the paper “Noncommuting Random Products” by Furstenberg

I have a question concerning the proof of theorem 2.5 at page 395 of the paper Noncommuting Random Products, by H. Furstenberg, Trans. Amer. Math. Soc., 1963. The statement is as follows: Let $\mu$ ...
2
votes
0answers
58 views

Examples of decreasing-in-some-time-interval variance of a time homogeneous Markovian process

Let $x_t$ be a zero mean, time homogeneous Markovian process over time $t$ starting from $x_0=0$. What are the examples of $x_t$ where the variance at $t$ decrease over some interval of $t$? The ...
2
votes
0answers
37 views

Exsistence and uniqueness of stationary density for Markov Chain

Suppose we're given a function $f:\mathbb{R}^2\to\mathbb{R}$. We define a Markov Chain $(X_n)$ by \begin{align} X_0&\sim f_X, \\ X_n&=f(X_{n-1},Y_{n-1}), \end{align} where $(Y_n)$ is a ...
2
votes
0answers
89 views

Interpretation of potential kernels for Markov processes

One can associate a strongly continuous contraction semi group (SCCSG) to a Markov process with state space $S$ through its transition function, say $P_t$. Now one can interpret $P_t$ as a linear ...
2
votes
0answers
88 views

References for basics of Piecewise-Deterministic Markov Processes

I am looking for introductory/pedagogical material to Piecewise-Deterministic Markov Processes (see http://en.wikipedia.org/wiki/Piecewise-deterministic_Markov_process) (For the moment I am interested ...
2
votes
0answers
82 views

“To every Q-matrix corresponds a unique Markov process.” Proving uniqueness

"To every Q-matrix corresponds a unique Markov process." I'm trying to understand Klenke's proof of the uniqueness part of this proposition. Klenke's proof Following is an adapted version of ...
2
votes
0answers
262 views

Is every killed Markov process still a Markov process?

Suppose we've got $X=(X(t))_{t\geq 0}$. $X$ is a strong Markov process with respect to filtration $\mathcal{F}_t$, taking values in some subset of $\mathbb{R}$. We take $\tau$ - a stopping time w.r.t ...
1
vote
0answers
15 views

Merging rates on a CTMC model

first time question here. I'm having a rough time trying to represent the following CTMC. Any help would be gladly appreciated. We consider a server with a infinite buffer connected to a network. ...
1
vote
0answers
19 views

Examples of Non-Markov process with continuous time and finite set of states.

What is the best real world examples of non-Markov process with continuous time, but with finite set of states?
1
vote
0answers
33 views

Markov factorization of the density of an AR(1) process

Suppose we have a causal, stationary AR$(1)$ process with i.i.d. innovations $Z_t$. Then we know that it is a Markov as future value $X_{t+1} = \phi X_t + Z_{t+1}$ given the past $X_1,\ldots X_t$ ...
1
vote
0answers
29 views

capacity of biased random walk in $\mathbb{Z}^2$

Let $P_{x,y}$ the probability that a random walk starting from $x$ will ever visit $y$. Consider a biased random walk in $\mathbb{Z}^2$. Let $A_k$ be the set of vertices having a distance less than ...
1
vote
0answers
9 views

Is the steady state of a uniform markov chain always a vector of proportions?

Given that all edges in a markov chain are bi-directional (though not necessarily equally weighted), and each edge for a given node has equal probability, does the steady state always converge to a ...
1
vote
0answers
25 views

Preservation of the Markov Property for SDEs

Let $X$ be a continuous Markov process on $\mathbb{R}^d$ that is also a semimartingale. Let $V=(V_1,...,V_d)$ be a collection of suitably nice vector fields on $\mathbb{R}^d$ such that there exists a ...
1
vote
0answers
33 views

Is {Yt} a Brownian motion?

Suppose {B(t)} and {B˜(t)} are two independent standard Brownian motions and ρ is a constant, −1 < ρ < 1. The process Y(t) = ρB(t) + sqrt(1- ρ^2)*B˜(t) is distributed as a normal random variable ...
1
vote
0answers
44 views

Discrete Laplacian

I have the following question and I can't figure out how to do the proof. Could you give me some hints in both directions of the equivalence? Suppose $A$ is a bounded subset of $\mathbb{Z}^d$. Then ...
1
vote
0answers
18 views

Opposite of Absorbing State

This should be fairly standard, but I fail to google it, and nothing on the matter is on Math.SE. How do we call the opposite of an absorbing state? If we think about Markov chains/systems, that ...
1
vote
0answers
30 views

How to Simplify a Markov chain in order to estimate the average number of transitions to reach to a final state?

Is there any approach to approximate the expected number of transitions to complete a Markov chain without knowing the exact transition probabilities? The reason I ask this is because I want to ...
1
vote
0answers
48 views

I have to show that the following stochastic process is a Markov process

I don't understand how to show that some stochastic processes have the Markov property. For example, if I have the following process: $$(\Omega, \mathcal{F}, (X_t)_{t \geq 0}, P^y)$$ where $\Omega = ...
1
vote
0answers
32 views

Construction of pure birth process

I am considering a Markov chain $\lbrace X(t) \rbrace_{t≥0}$ in continunous time on the countable state space $S=\lbrace 0 \rbrace\cup \lbrace (i,j) \mid i \in \mathcal{A} , j \in \mathbb{N} \rbrace, ...
1
vote
0answers
27 views

Invariant Distribution of Two Dependent $\cdot/M/\infty$ Queues Running in Parallel

This is in preparation for an exam I have coming up. We have two $\cdot / M / \infty$ queues with external arrivals occurring according to a Poisson Process of rate $\lambda$. Service occurs with ...
1
vote
0answers
62 views

Semigroup of operators: weak continuity at 0+ implies weak continuity at any t > 0

Let ($E$, $d$) be a metric space. Consider the semigroup $\{P(t)\}_{t\geq 0}$ of bounded linear operators on the Banach space $\hat{C}(E)$ of continuous real functions on ($E$, $d$) vanishing at ...
1
vote
0answers
49 views

What is a convolution kernel?

What is a convolution kernel? (in measure theory, probability theory) In which book can I read about kernels on measurable spaces and convolution kernels? Thank you!
1
vote
0answers
29 views

Generator of a Feller semigroup on a coutable space

Let $E$ be a countable set in the discrete topology. Let $(T_t)_{t \geq 0}$ be a Feller semigroup on $E$, i.e. a strongly continuous semigroup of operators on $\mathcal{C}_0(E)$ (in the topology of ...
1
vote
0answers
12 views

Proportion of arrivals taking a particular path in a Routing Matrix

I have a routing matrix with Node-0 being the source/sink (outside world) and there are service Nodes 1,2..k in the system. The matrix has entries R_ij = Probability of an arrival at Node-i moving to ...
1
vote
0answers
24 views

Stochastic scheduling to maximize the expected number of customers arrived at the root of a Jackson tree

In a Jackson network, organized as a tree rooted at queue r, several customers are queued at time 0 and there is no new customer arrival. The service time of each customer in queue i is geometrically ...
1
vote
0answers
65 views

Kolmogorov backward equations for Birth-Death

I'm trying to solve the Kolmogorov backward equations for a Birth-Death Markov chain with three states. I have 2 equations: $$P_{00}'(t) = \lambda_0 (P_{10}(t)-P_{00}(t))$$ $$P_{10}'(t) = ...
1
vote
0answers
63 views

Chapman-Kolmogorov equations of time inhomogenous Markov chains

Let us assume that we are given a time inhomogenous Markov chain in continuous time (ICTMC) $(X(t))_{t \geq0}$ with a finite state space $\{1,\ldots,n\}$. Set $P(t)_{i,j} := \mathbb{P}(X(t) = j \mid ...
1
vote
0answers
114 views

Can we find a correlation between states of a Markov chain?

I have a fair bit of knowledge on Markov chains but I recently wondered if there is a way to find out a correlation between the states of a finite Markov chain. I could not find any material on this. ...
1
vote
0answers
49 views

Linear Filtering Problem (Keynman Fac/Particle Model)

$lienar Filtering Problem $$X_n^1 = X_{n-1}^1 + \epsilon_n *W_n $$ $$X_n^2 = (1-\alpha* \delta) X_{n-1}^2 + \beta*\delta X_n^1 $$ $$X_n^3 = X_{n-1}^3 + \delta*X_n^2$$ above is $$\approx$$ $$dX_n^1 ...
1
vote
0answers
47 views

Stochastic process using Markov chain (thief on the run!!)

I'm given an exercise where we are to simulate a thief escaping from an officer. The thief (let's call him/her T for simplicity) and an officer (O) have four cities to be in. Let's call the cities A, ...
1
vote
0answers
247 views

Hidden Markov Model, transition probabilities which are modeled with an exponential distribution

I'm looking at implementing an algorithm described in a paper, but I'm having trouble understanding how the transition probabilities for a Hidden Markov Model are defined. In the first sections, I ...
1
vote
0answers
14 views

A simple question on the inverse z-transformation of $\frac{z}{1-z}\mathscr{T}(z)\mathbf{q}$

I'm wondering if anyone who is familiar with the book Dynamic Programming and Markov Processes by Ronald Howard or simply z-transform can help me figure out an inverse z-transformation on page 23 of ...
1
vote
0answers
61 views

Follow-up on solution to Markov process equation

I asked a question here about solving a system related to an absorbing Markov chain. I now have a variation where there are $m$ types (of student, job seeker, etc) each of which applies to ...
1
vote
0answers
43 views

Branching Brownian Motion and KPP equation

I have troubles understanding the proof of the connection between BBM and KPP equation. I mean the proof of the next lemma from the lecture notes of Anton Bovier about BBM, link. This is almost whole ...
1
vote
0answers
70 views

References for time-inhomogeneous Markov jump processes?

In some central models in life insurance mathematics, the state of the insured is modeled using a continuous-time time-inhomogeneous Markov process with finitely many states. While many results for ...
1
vote
0answers
30 views

Analysis of Steady State Probability for Markov Process

I have a balance equation, representing a Markov Chain, which yields $$ (K - z) \pi(Z_c = z) = (\lambda_c + (z+1)x) \pi(Z_c = z+1) $$ where K is the maximum state of the server. The term $\lambda_c$ ...