A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

2
votes
2answers
649 views

Mean recurrence time and stationary distribution of a Markov chain?

In a Markov chain is there a theorem relating the existence of the stationary distribution and the mean recurrence time? E.g. impossible for stationary distribution to exist therefore mean recurrence ...
3
votes
1answer
173 views

What is the value of this game?

We have 3 black and 2 red balls in an urn. If we pick a black ball, we lose 1 USD. If we pick a red ball we win 1 USD. We can chose to start the game or not. If we start the game we can stop after ...
0
votes
1answer
29 views

Show $P(S_{2n}=x|S_0=x) \ge \frac{1}{N}$

Let $X_n$ be an aperiodic, discrete-time Markov chain so $S=\{1,...,N\}$ whose transition probability is symmetric. How can I show that for all $x \in S$ and all integers $n$, $P(S_{2n}=x|S_0=x) \ge ...
0
votes
1answer
167 views

Transition rate matrix from transition probability matrix

If I have a $2 \times 2$ continuous time Markov chain transition probability matrix (generated from a financial time series data), is it possible to get the transition rate matrix from this and if ...
0
votes
1answer
85 views

Markov processes and semimartingales

Semimartingales and Markov processes are two fundamental families in probability theory. There are many specific processes that belongs to the intersection of those two families, e.g. Levy processes. ...
1
vote
1answer
78 views

What is the Deterministic Traffic Generation Model?

I am studying Markov chains and queuing theory. I was curious about traffic generation models and actually happened to see the Deterministic Traffic Model, referred to as $D$ in Kendall's notation. ...
0
votes
1answer
84 views

Queue system with queue-triggered input process

I have a queue system, a classic system with an input generator, a queue and a servant. The servant is a $M$-servant with a certain serving rate $\mu$. The queue can contain an infinite number of ...
0
votes
1answer
73 views

Average time of permanence in a state of a Markov-chain

I know that in a Markov-chain the mean permanence time in a state is a random variable distributed accordingly to: Geometric distribution for Time Discrete Markov Chains Exponential distribution for ...
0
votes
0answers
28 views

Is the exit time independent of the state jumped to in a Markov chain?

Let $X$ be a continuous time Markov chain on a countable state space $S$, and let $\tau_n$ be the $n^{th}$ time at which the chain jumps out of a set $D$ (i.e. times $t$ at which, for some $\epsilon ...
0
votes
0answers
108 views

Markov renewal process vs Markov Jump process

The venerable wikipedia for "Markov renewal process" says that: "a Markov renewal process is a random process that generalizes the notion of Markov jump processes". So what's the definition of a ...
3
votes
1answer
367 views

Elementary proof of geometric / negative binomial distribution in birth-death processes

The birth-death process concerns a population of $n_0$ individuals, each of which reproduce and die at a constant rate as time $t$ increases from $t=0$. Each individual splits into two individuals ...
10
votes
2answers
840 views

What is the importance of the infinitesimal generator of Brownian motion?

I have read that the infinitesimal generator of Brownian motion is $\frac{1}{2}\small\triangle$. Unfortunately, I have no background in semigroup theory, and the expositions of semigroup theory I have ...
1
vote
2answers
176 views

Expected number of steps between states in a Markov Chain

Suppose I am given a state space $S=\{0,1,2,3\}$ with transition probability matrix $\mathbf{P}= \begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 \\[0.3em] \frac{2}{3} ...
0
votes
1answer
85 views

Exercise on Markov chain

Prove, or give an explicit counterexample to refute, the following assertion: if $\{X_n\}$ is a Markov chain, then $\{X_n^2\}$ is also a Markov chain. It's easy to show that ...
3
votes
1answer
103 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is ...
3
votes
1answer
126 views

Markov chain problem with finite states

Consider any Markov chain on a state space with exactly $r$ states. We want to find the largest $N>0$ such that there exists states $i,j$ for every $n < N$ we have $p_{ij}^{(N)} > 0$ and ...
1
vote
1answer
90 views

Stochastic processes with independent increments

If $\{X_{t}:t\geq 0\}$ is a real-valued stochastic process with independent increments then $\{X_{t}:t\geq0\}$ is a Markov process? Let $\{ \mathcal{F}_{t} \}_{t\geq0} $ be a natural filtration of ...
1
vote
0answers
58 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
3
votes
1answer
108 views

Strong Markov property of Brownian motion

I was able to understand Brownian Motion $\{B(t):t\geq0\}$ has Strong Markov Property i.e. For any stopping time $\tau$, $P(B(t+\tau)\leq y | \mathcal{F}_{\tau})=P(B(t+\tau)\leq y|B(\tau))$ a.s. , $y ...
1
vote
2answers
51 views

Calculation of the Gallavotti-Cohen fluctuation theorem made by Lebowitz

I have a problem understanding a calculation in this paper (another form of the theorem an be found here at equation 11). For those who want to read the paper, I have difficulties with formula 2.14 in ...
1
vote
1answer
47 views

Consider a Stochastic process X which moves between the corners of a regular tetrahedron. Find P(Xn = 1)

Consider a Stochastic process $X$ which moves between the corners of a regular tetrahedron. The process starts at time 0 in node 1. At each time step, the process chooses an one of the connected edges ...
3
votes
1answer
81 views

Random walk where increment depend on current position

Consider the following stochastic process, $$b(i+1) = b(i) + \xi_i (b_i),$$ where $\xi_i(b_i) \in \{-1, k \}$ are the independent increments having the following distribution: $$\begin{align} P (\xi ...
1
vote
1answer
81 views

Library chain stationary distribution

This is an exercise 1.47 from Richard Durrett's Essentials of Stochastic Processes p.85 (doi: 10.1007/978-1-4614-3615-7_1 or Google Books). On each request the ith of the $n$ possible books is the ...
0
votes
1answer
64 views

Poisson Process (Easy)

I'm stuck at the following question: Customers with items to repair arrive at a repair facility according to a Poisson process with rate λ. The repair time of an item has a uniform distribution on ...
1
vote
2answers
72 views

Markov chain simulation

I'm wonder if there is an algorithm to simulate a discrete Markov chains with a specific number of occurence of state knowing the transition matrixway. For example, how to simualte in $\mathbb R$ a ...
1
vote
1answer
257 views

Equivalence Classes of a Markov Chain with Transition Matrix

I have the following transition probability matrix for a markov chain with state space S={0,1,2,3,4,5,6}: $\begin{bmatrix} \frac13 & \frac13 &0 & 0 & \frac16 & 0 & \frac16\\ ...
4
votes
1answer
88 views

random walk in a certain environment

Consider the following random walk in one dimension, starting from $r(0)=0$. $$ r(i+1) = r(i) + \xi, $$ where $\xi(i, r(i))$ is an increment with distribution $P(\xi=1) = \frac{c^{r(i)}}{i-r(i)+1}$ ...
3
votes
1answer
62 views

Probability of a sequence of events in a Poisson process.

I am starting to study Poisson processes and I came up with this question: Let there be two Poisson processes with rates $\lambda$ and $\mu$ respectively, monitoring the occurrence of events (e.g. ...
1
vote
0answers
33 views

Prove equilibrium theorem without irreducibility and aperiodicity

I have to solve the following question: Consider a random walk Markov chain on $S = \{1, 2, \ldots, 100\}$. If the chain is between 2 and 99, it selects one of the adjacent states with equal ...
2
votes
1answer
511 views

Markov chains: is “aperiodic + irreducible” equivalent to “regular”?

I have two books on stochastic processes. In one book, it says that the limiting matrix is possible to find if the matrix is regular, that is, if for some $n$ $P^n$ has only positive values. The ...
1
vote
1answer
28 views

Markov Decision Process - Optimal policy invariance to scaling in the Utility Function

The title says it all. If i use a discounted Utility Function, why is the optimal policy invariant with respect tot the scaling of the Utility Function by a positive Factor?
2
votes
0answers
54 views

A theorem in the paper “Noncommuting Random Products” by Furstenberg

I have a question concerning the proof of theorem 2.5 at page 395 of the paper Noncommuting Random Products, by H. Furstenberg, Trans. Amer. Math. Soc., 1963. The statement is as follows: Let $\mu$ ...
1
vote
0answers
37 views

What does stationarity of the point process entail in a Markovian setting?

Let $(\Omega,\mathcal{F},P)$ be a probability space where $\Omega$ is the set of cadlag trajectories from $\mathbb{R}$ to a countable state space $S$. Let $X$ be the coordinate process $X_t(\omega) = ...
5
votes
1answer
123 views

Log-likelihood function

I'm not sure if this could be asked here, or in math overflow... In the following paper Cho, Jin Seo, and Halbert White. "Testing for regime switching." Econometrica 75.6 (2007): 1671-1720. doi: ...
1
vote
0answers
101 views

MDP problem - How is the expected cost calculated?

I have been stuck with a problem for a while regarding Markov Decision Processes for a Policy improvement algorithm. Assume that I have probabilities for certain states to evolve the system into, ...
3
votes
1answer
223 views

Equivalent defining Markov property

Consider the stochastic process $(X_t)_{t \in \mathbb{R}}$ and show the equivalence of the following two Markov properties: (a) $P(X_t \in A \mid X_u, u \leq s) = P(X_t \in A\mid X_s) \qquad ...
1
vote
1answer
60 views

Strong Markov property given transition functions

Suppose we are given family of transition functions satisfying Chapman-Kolmogorov equation, what conditions will ensure that there exists a continuous or cadlag Markov process with given transition ...
1
vote
0answers
45 views

State Space Difference Linear Dynamic System

I am interested in finding the DIFFERENCE in the state space distributions for two linear dynamical systems (System A and System B). I am able to solve for this using the matrix exponential. But the ...
0
votes
0answers
82 views

transition matrix for Markov chain

Can any one help me to solve this home work please? The city of Sacramento recently completed a new light rail system to bring commuters and shoppers into the downtown area and relieve freeway ...
1
vote
1answer
59 views

Estimate the probability using Markov chains

please consider this question: A study using Markov chains to estimate a patient's prognosis for improving under various treatment plans gives the following transition matrix as an example ...
0
votes
1answer
75 views

Stationary distribution for a Markov chain which is not irreducible

I have a Markov chain with $K$ states $S$: {$s_1,s_2,...,s_K$}. $s_1$ is reachable from any state in $S$; however not all the states can be reached from $s_1$. What does the stationary distribution ...
0
votes
0answers
50 views

Does time homogeneity imply strong Markov property in a Markovian process

Does a time homogeneous Markovian process necessarily have strong Markovian property? Does continuity in state space, time, or path make a difference? What are the examples if it does not?
2
votes
0answers
57 views

Examples of decreasing-in-some-time-interval variance of a time homogeneous Markovian process

Let $x_t$ be a zero mean, time homogeneous Markovian process over time $t$ starting from $x_0=0$. What are the examples of $x_t$ where the variance at $t$ decrease over some interval of $t$? The ...
2
votes
1answer
240 views

Proof that Markov Chains converges to the stationary distribution

Let $P$ is a transition matrix of a Markov Chain, which is irreducible, aperiodic and lets assume $\pi$ is its stationary distribution: $\pi = \pi P$. Does anyone knows the proof for the following ...
1
vote
0answers
25 views

Probabilistic event triggered on a Markov Process transition

I would like to assess a system disponibility using a Markov Process. This system has two states : a functionning state 0 and a failure state 1, with a fault rate $\lambda$ and a mean time to repair ...
2
votes
0answers
37 views

Exsistence and uniqueness of stationary density for Markov Chain

Suppose we're given a function $f:\mathbb{R}^2\to\mathbb{R}$. We define a Markov Chain $(X_n)$ by \begin{align} X_0&\sim f_X, \\ X_n&=f(X_{n-1},Y_{n-1}), \end{align} where $(Y_n)$ is a ...
0
votes
1answer
29 views

Asymmetric Markov process

The limiting distribution $\xi(x)$ of a Markov process $$x_0=1\text{ and }x_{i+1}=x_i+\Delta x_i,\tag1$$ where $\Delta x_i=-ax_i$ and $\Delta x_i=a$ occur with equal probability for every $i$, and ...
1
vote
0answers
25 views

Markov operators

Transition probability functions can always be used to generate Markov operators, correct? So is it correct to say that a Markov process is a collection of Markov operators? On the other hand, are ...
0
votes
0answers
44 views

How to prove theorem about consistency of Markov edge process?

How to prove such theorem: Markov edge process $p_E(y_E)$ with respect to DAG $G=(V,E)$ defined as $p_E(y_E) = \prod_{v \in V} p_E\left(y_{E_{\rm out}(v)} \,\big|\, y_{E_{\rm in}(v) } \right) = ...
0
votes
1answer
54 views

Help With Eigenvectors and Dynamical Systems

I have the following system of differential equations: $ \frac{d}{dt} \left[ \begin{array}{c} A(t)\\ N(t)\\\end{array} \right] = \left[ \begin{array}{c c} -(a+b) & 0\\ a & -(a+b)\\ ...