A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
0answers
9 views

Rigorous Derivation of Metropolis-Hastings Transition Density

The Metropolis-Hastings MCMC algorithm is as follows. Set $X_0$ to some initial value in the support of the target density $f$ and choose a proposal density $q(y \mid x)$; a density in $y$ for each ...
1
vote
1answer
22 views

Proving linear operators are Markov Generators

I am trying to do the following question from Liggett. Let $A$ be a linear operator defined on the space of continuous functions $C(E)$ for a compact set $E$. Define $A$ by $A=T-I$ where $T$ is a ...
5
votes
0answers
35 views

markov chain: 2 state chain

I have a machine. It has two states, broken or working. If it is working, then it will be broken with probability $q=0.1$. If the machine is working, I will make \$1000 dollar a day. If it is broken, ...
0
votes
0answers
9 views

Unconditional COvariance in markov switching model

I'm trying to do a portfolio optimisation within a Markov switching framework for some risky asset returns. My utility function ideally is CRRA (power) utility. However maximising a linear sum of two ...
0
votes
0answers
11 views

Invariant distributions: Applications in the real World

I'm studying about invariant distributions for Markov processes; say in the context of dynamics of Random Neural Networks (biological Networks). I can't fully understand what does an invariant ...
1
vote
0answers
27 views

The “how many pieces do you have buy on average” problem, a markov problem?

I recently discovered a problem similar to this one in a book about Markov chains: Assume you can buy $n-$ different set of cards in a store, but you do not know which one you'll buy: What is the ...
0
votes
0answers
11 views

When does a periodic but positive recurrent markov chain have a limiting distribution

So I know it's a fact that an aperiodic, finite state, irreducible (so positive recurrent) markov chain has a unique stationary distribution which is limiting. However, I am curious if there is a ...
4
votes
1answer
22 views

Ehrenfest Chain: stationary distribution

In the Ehrenfest Chain model: There are M balls which are divided between urn A and urn B. At each stage, if a ball is chosen, then it would be moved into a different urn. Let $X_n$ be the # of ...
0
votes
0answers
28 views

Explicit formula for return probability of simple random walk

Is there an explicit formula for the probability that a simple symmetric random walk on $\mathbb{Z}$ starting from $1$ will not hit $0$ before time $t$?
0
votes
0answers
17 views

“Mean-field results” in Probability theory

I'm studying a paper on (biological) Neural Networks, and the paper studies some stability properties of an $N$-sized network, and then, as $N$ tends to infinity, it is proven that a "mean-field ...
0
votes
0answers
10 views

How to calculate mean of busy-cylce

Let $\{X(t)\}$ be birth-death process on a finite state {0,1,2} with non negative birth rates $(\lambda_0,\lambda_1)$and death rates $(\mu_1,\mu_2)$. Suppose $\mathbb{P}(X(0)=0)=1$ and $s_0=\inf ...
-1
votes
0answers
11 views

How to find expectation of birth-death process [closed]

Let $\{X(t)\}$ be birth-death process on two-state space {0,1}. Let birth rate $\lambda=2$ and death rate $\mu=12$. How to calculate $\lim_{t\to\infty} \mathbb{E}[X(t)]$?
2
votes
0answers
23 views

Moving Average of an Ergodic Markov processes.

Let $ \{X(t); t\geq 0\} $ be an ergodic Markov process, and let G be a positive integrable function with $\int_0^\infty G(x)dx=1$. Does $$F(t)\doteq G*X(t) = \int_0^t G(t-s)X(s)ds $$ converge in ...
0
votes
0answers
32 views

Identity for return times in continuous Markov chain

I need help with this problem about return times in continuous time Markov chains: We are given a continuous time Markov chain $(X_t)$. Define $T_i=\inf{\{t>0; X_t=i, X_{t-} \neq i\}}$, which ...
1
vote
0answers
11 views

Question about a type of continuous state Markov-process.

I was wondering if anyone knows whether a particular result has been proven or is indeed true. My problem is as follows. Suppose I have a stationary, ergodic Markov chain that follows a process of ...
3
votes
1answer
42 views

Rigorous argument of the Markov property used in discrete-time Markov chains

I am reading an example related to discrete-time Markov chains which I do not really understand rigorously. Suppose that $\{ X_n \}_{n \in \mathbb{N} }$ is a time-homogeneous discrete-time Markov ...
1
vote
0answers
15 views

non-ordered chapman kolmogorov equation

If $p(x)$ is a probability density one would normally have $\int dx_1 p(x_2|x_1)p(x_1)=p(x_2)$ However is there a straightforward way to interpret the following: $\int dx_1 ...
1
vote
0answers
28 views

An example of Markov chain with no closed class?

What is an example of Markov chain with no closed communicating class? Closed class means that once we are in that class, there would be no escape from it. I am thinking that an example would be ...
0
votes
0answers
20 views

For which p is the Markov chain recurrent (almost random walk)

We have a Markov chain on Z with matrix: $p_{ii+1}=p=1-p_{ii-1}$ for $i\leqslant-1$, $p_{ii-1}=p=1-p_{ii+1}$ for $i\geqslant1$, and $p_{00}=p_{01}=p_{-10}=\frac{1}{3}$. For which values of p ...
1
vote
1answer
18 views

Equality of probability of finite hitting time for irreducible states in Markov Chain

Suppose I have a finite state Markov Chain with state space $S=\{1,2,3,4,5,6\}$. Suppose I further have that $\{1,2\}$,$\{3,4\}$ and $\{5,6\}$ are irreducible classes where $\{1,2\}$ and $\{3,4\}$ are ...
0
votes
0answers
14 views

Deriving Time of extintion of a Small neural Network

I'm trying to derive the Expected Value of the Time of Extintion $\tau_{ext}$ of a small Neural Stochastic Network with the following dynamics, where I consider $\tau_{ext}$ to be the time of the last ...
0
votes
1answer
33 views

Show that $m_{ii}=\infty$ when $i$ is transient

Show that $m_{ii}=\infty$ when $i$ is transient, where $m_{ii}$ is the mean time to get from $i$ to $i$. if $i$ is transient I know that there is a positive probability of going to some ...
1
vote
1answer
29 views

Cont Time Markov Chains. Stationary Probability

A barber finishes haircuts at rate $3$, measured in hours, so on average it takes him 20 minutes to cut a person’s hair. Customers arrive at rate 2. There is, however, only a two chair waiting ...
2
votes
0answers
17 views

Applying Markov Decision Processes to an arrival forecasting problem

I have the following problem and I'd like to know if it's something that was already studied in the literature or not. I'm not sure about the naming conventions either. I have a system $S$ that can ...
0
votes
0answers
13 views

Equivalence between Label propagation and Likelihood estimation over a Markov Random Field

On page 13/14 of label propagation the near equivalence is set up between what they call 33/34 (reproduced below). This near equivalence is not obvious to me. $$P_{F'}(Y) = \frac{1}{Z} \exp \left( ...
0
votes
0answers
23 views

Conditional expectation in continous markov chains

I am trying to understand the double integral in calculating the conditional expectation. In calculating $V_i$, the second and third equalities are due to the law of total probability. I have the ...
1
vote
1answer
23 views

How to discretely stochastically simulate a continuous-time Markov chain?

A continuous-time markov chain describes a continuously varying process, such that future state only depends on the current state. A sampling of a continuous markov chain can be described in terms of ...
0
votes
0answers
26 views

Density function of absorption time in this Markov Chain

Let $X_t$ be a continuous time Markov Chain with state space $\{1,2,3\}$ with the following transition matrix: $$\left( \begin{matrix} -(\lambda+\delta) & \lambda & \delta \\ \mu & ...
2
votes
1answer
57 views

Markov Chain: flip coin 8 times and get 3 consecutive heads

I have confusion while reading the following example in the course material. Q: In a sequence of independent flips of a fair coin, let N denote the number of flips until there is a run of three ...
0
votes
2answers
72 views

Markov Chain: flip 8 coins and get 3 consecutive heads

I was reading the material and I am confused at the following example. Q: In a sequence of independent flips of a fair coin, let N denote the number of flips until there is a run of three ...
0
votes
0answers
12 views

Direct proof of the existence of optimal memoryless deterministic policies in MDP

It is well known that (finite-state, finite-action, discrete time) MDPs admit an optimal policy that is memoryless and deterministic (sometimes called pure). The proof of this fact for ...
0
votes
1answer
19 views

Mean time for the trajectory. Find mean

What is the mean of time when the trajectory of the wiener process, $W_t$, is over the line $y=t$? We need to find $\Bbb{E}\tau$, where $\tau=\sum\limits_{a,b:\forall t\in(a,b) ; ...
0
votes
0answers
15 views

HMM optimization: Lagrange multiplier problem

In David Barber's textbook "Bayesian Reasoning and Machine Learning" he hints at the derivation of the Baum-Welch algorithm for HMM parameter learning: Textbook excerpt, (cannot include images yet, ...
1
vote
1answer
30 views

Strong Markov property with two stopping times

I have a diffusion $X=(X_t)_{t\ge0}$ and a stopping time $\tau$. From the strong Markov property I know that for any time $t\ge0$ (or a random time independent of $X$) I get that ...
0
votes
0answers
11 views

Examples of state-space models that show strong homeostasis but also substantial change after critical threshold?

The question is, can can anyone provide examples of systems or math models that exhibit patterns of homeostasis but which can be exhibit substantial transitions or bifurcations after some critical ...
2
votes
1answer
29 views

How to find Kolmogorov Forward Equations, given generator matrix Q?

I am having difficulty in forming Kolmogorov Forward Equations. I understand how the KFE is derived and that $$\frac {d}{ds} p_{ij} (s) = \sum_{k \neq j} p_{ik} (s) \lambda_{k} r_{kj} - p_{ij} ...
1
vote
0answers
54 views

Does a Markov process have memory zero?

I have the following question: The Markov process with two states and a transition matrix $$P =\begin{pmatrix} 0.3 & 0.7 \\ 0.3 & 0.7 \end{pmatrix}$$ has memory zero. Is it true? My ...
2
votes
0answers
24 views

Good introductory book coupling methods

I am very interested in coupling methods, can you recommend me a good introductory books on this subject? Thanks
1
vote
0answers
40 views

Birth and death process. Total time spent in state i.

Question: Let $X(t)$ be a birth-death process with $\lambda_n = \lambda > 0$ and $\mu_n = \mu > 0,$ where $\lambda > \mu$ and $X(0) = 0$. Show that the total time $T_i$ spent in state $i$ is ...
1
vote
0answers
40 views

Gambler's ruin problem - expected time

I have troubles seeing the following. Consider the classical gambler's ruin problem, betting 1 at each time $t\in \mathbb{N}$, and losing or winning -1 respectively +1 at each time till the fortune of ...
2
votes
0answers
37 views

Markov chain limit problem

Let $X_n$ be a Markov chain on a countable state space, $\mathbb{S}$. Let $N_n(x) = \sum_{k=1}^n\mathbb{1}_{\{X_k=x\}}$ denote the number of times the chain visits state $x\in \mathbb{S}$. Let ...
0
votes
1answer
65 views

Renewal process with bimodal times

Suppose we have a stochastic process $X_t$ of a light using a single light bulb. When the light bulb burns out it is immedieatly replaced with a new one. Suppose that the time between failures is ...
0
votes
1answer
40 views

Has a Markov chain in compact metric space a stationary distribution (possibly non-unique)?

Let $Y_n$ be a Markov chain in a compact metric space. Is it true that it has a stationary distribution?
1
vote
0answers
36 views

Markov chain of transition probabilities

Let $P$ be a transition matrix on a discrete state space with $N$ elements. $P_{i,j}$ is the probability of going from state $i$ to state $j$. Let $\pi$ be the stationary distribution. Let $\{X_n\}$ ...
0
votes
0answers
19 views

steady state distribution of the following Markov jump process?

Consider a queueing process with the following rate transition matrix: $\mathbf{P}=\left( \begin{smallmatrix} -\lambda & \lambda & & & & & & &\\ \mu & ...
1
vote
1answer
26 views

Markov chain with dynamic higher orders

Let $X_i$ be the node visited by a random walk at step $i$, and the following equations be the transition probabilities. $Pr(X_n = x_n | X_{n-1} = x_{n-1}, \cdots, X_1 = x_1) = Pr(X_n = x_n | X_{n-1} ...
1
vote
1answer
27 views

optimal utility calculation for a simple discrete Markov chain

I am trying to calculate analytically the optimal decision rule for a simple discrete markov chain, following standard decision theory framework (slide 17 in ...
1
vote
1answer
77 views

Show that the probability of never hitting 0 on a birth-death chain is $6/\pi^2$.

In the question we have a birth-death chain on $\{0,1,2,...\}$ whose only non-zero transitions from $i$ are to $i+1$ and $i-1$, with probabilities $p_i$ and $q_i$, respectively. I have that $p_i$ and ...
2
votes
1answer
43 views

Markov Process: predict the weather using a stochastic matrix

I have the following stochastic matrix $$ P = \begin{pmatrix} P(S \mid S) = 0.5 & P(F \mid S) = 0.2 & P(R \mid S) = 0.3 \\ P(S \mid F) = 0.2 & P(F \mid F) = 0.7 & P(R \mid F) = 0.1 ...
1
vote
1answer
78 views

A property of Poisson process

Let $Y_t$ be a centered Poisson process, why \begin{equation} \lim_{n \to \infty} \sup_{s<t} |n^{-1}Y(ns)| = 0 \qquad a.s. \qquad \forall t\ge 0 \end{equation} This is a fundamental step in the ...