A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
1answer
29 views

Up-to-date or Behind - [Markov Chain]

Alex is taking a bioinformatics class and in each week he can be either up-to-date or he may have fallen behind. If he is up-to-date in a given week, the probability that he will be up-to-date (or ...
0
votes
0answers
48 views

Basic questions on Markov Chain

I'm a beginner of Markov processes and I have some basic questions. Consider two sequences of real-valued random variables $\{X_t\}_t, \{Y_t\}_t$ where $t$ is a discrete time index, $t=0,1,...$, all ...
0
votes
1answer
51 views

Mickey mouse travels in a maze with nine $3 × 3$ cells. Markov Chain involved?

Mickey mouse travels in a maze with nine $3 × 3$ cells. The cells are numbered as $0, 1, ..., 8$ from left to right and top down. Each step Mickey travels from where it is to one of the surrounding ...
0
votes
0answers
32 views

$ (X_n : n = 0, 1, 2, …) $ is a Markov chain with state space $(1,2,…,10)$.

$ (X_n : n = 0, 1, 2, ...) $ is a Markov chain with state space $(1,2,...,10)$. Then which of the following is the correct answer? ($X_n+X_{n-1} : n = 1, 2, ...$) is a Markov chain. ($X_n-X_{n-1} : ...
1
vote
2answers
37 views

Busy times for a M/M/1 queue

I have an M/M/1 queue with people arriving Poisson with parameter $\lambda$ and a service time exponentially distributed rate $\mu$. I have been asked to find the average time between the first ...
3
votes
1answer
63 views

Proof of semigroup property for family of operators

I'm studying a proof of a large deviations principle and I'm having trouble with a part that is concerned with the semigroup property of a family of operators. Assumptions $(X_t)_{t\geq0}$ is a ...
1
vote
0answers
31 views

Irreducible and positive recurrent CTMC — first passage times are finite?

Consider a continuous-time Markov chain (CTMC) $X$ on a countably infinite state space $S$. The CTMC is irreducible and all the states are positive recurrent. Let $T(i,j)$ be the first passage time to ...
0
votes
0answers
22 views

Distribution of right censored observation in an absorbing Markov Chain

Consider a $3$ state Continuous time Process $\{X_t\}_{t \geq0}$ with state space $\mathcal{S} = \{0,1,2\}$ where state $0$ denotes the absorbing state. Let the generator of this process be: $$Q = ...
5
votes
2answers
176 views
0
votes
0answers
11 views

how to resolve the infinite nesting of interactive POMDP

I am reading papers about I-POMDP. I cant understand the finitely nested I-POMDPs given in these papers. The belief update of the algorithm has a problem that agents' belief updates mutually depend ...
0
votes
1answer
57 views

Equality for transient Markov chains

Let $S_n$ be a transient, irreducible random walk starting from $0$. Then I want to prove that $$\sum_{n\geq0}{p_n(0,x)}=P_0[S_n=x\text{ for some }n\geq0]\sum_{n\geq0}{p_n(0,0)}$$ where $p_n$ is the ...
1
vote
0answers
13 views

Rigorous Derivation of Metropolis-Hastings Transition Density

The Metropolis-Hastings MCMC algorithm is as follows. Set $X_0$ to some initial value in the support of the target density $f$ and choose a proposal density $q(y \mid x)$; a density in $y$ for each ...
1
vote
1answer
26 views

Proving linear operators are Markov Generators

I am trying to do the following question from Liggett. Let $A$ be a linear operator defined on the space of continuous functions $C(E)$ for a compact set $E$. Define $A$ by $A=T-I$ where $T$ is a ...
5
votes
0answers
61 views

markov chain: 2 state chain

I have a machine. It has two states, broken or working. If it is working, then it will be broken with probability $q=0.1$. If the machine is working, I will make \$1000 dollar a day. If it is broken, ...
3
votes
1answer
122 views

Simple random walk on $\mathbb Z^d$ and its generator

I'm still trying to figure out definitions and properties of random walks on $\mathbb Z^d$. My goal is to work up to understanding some large deviation principles for the local times of such random ...
0
votes
0answers
10 views

Unconditional COvariance in markov switching model

I'm trying to do a portfolio optimisation within a Markov switching framework for some risky asset returns. My utility function ideally is CRRA (power) utility. However maximising a linear sum of two ...
1
vote
1answer
14 views

Invariant distributions: Applications in the real World

I'm studying about invariant distributions for Markov processes; say in the context of dynamics of Random Neural Networks (biological Networks). I can't fully understand what does an invariant ...
1
vote
0answers
31 views

The “how many pieces do you have buy on average” problem, a markov problem?

I recently discovered a problem similar to this one in a book about Markov chains: Assume you can buy $n-$ different set of cards in a store, but you do not know which one you'll buy: What is the ...
0
votes
0answers
18 views

When does a periodic but positive recurrent markov chain have a limiting distribution

So I know it's a fact that an aperiodic, finite state, irreducible (so positive recurrent) markov chain has a unique stationary distribution which is limiting. However, I am curious if there is a ...
4
votes
1answer
73 views

Ehrenfest Chain: stationary distribution

In the Ehrenfest Chain model: There are M balls which are divided between urn A and urn B. At each stage, if a ball is chosen, then it would be moved into a different urn. Let $X_n$ be the # of ...
0
votes
0answers
36 views

Explicit formula for return probability of simple random walk

Is there an explicit formula for the probability that a simple symmetric random walk on $\mathbb{Z}$ starting from $1$ will not hit $0$ before time $t$?
1
vote
0answers
21 views

“Mean-field results” in Probability theory

I'm studying a paper on (biological) Neural Networks, and the paper studies some stability properties of an $N$-sized network, and then, as $N$ tends to infinity, it is proven that a "mean-field ...
1
vote
1answer
18 views

How to calculate mean of busy-cylce

Let $\{X(t)\}$ be birth-death process on a finite state {0,1,2} with non negative birth rates $(\lambda_0,\lambda_1)$and death rates $(\mu_1,\mu_2)$. Suppose $\mathbb{P}(X(0)=0)=1$ and $s_0=\inf ...
2
votes
0answers
32 views

Moving Average of an Ergodic Markov processes.

Let $ \{X(t); t\geq 0\} $ be an ergodic Markov process, and let G be a positive integrable function with $\int_0^\infty G(x)dx=1$. Does $$F(t)\doteq G*X(t) = \int_0^t G(t-s)X(s)ds $$ converge in ...
1
vote
0answers
45 views

Identity for return times in continuous Markov chain

I need help with this problem about return times in continuous time Markov chains: We are given a continuous time Markov chain $(X_t)$. Define $T_i=\inf{\{t>0; X_t=i, X_{t-} \neq i\}}$, which ...
4
votes
0answers
26 views

Question about a type of continuous state Markov-process.

EDIT: Solved! It turns out that if the function is continuous and various regularity conditions hold then the statement is true. This has been established in the 'stochastic approximation' literature, ...
3
votes
1answer
44 views

Rigorous argument of the Markov property used in discrete-time Markov chains

I am reading an example related to discrete-time Markov chains which I do not really understand rigorously. Suppose that $\{ X_n \}_{n \in \mathbb{N} }$ is a time-homogeneous discrete-time Markov ...
1
vote
0answers
15 views

non-ordered chapman kolmogorov equation

If $p(x)$ is a probability density one would normally have $\int dx_1 p(x_2|x_1)p(x_1)=p(x_2)$ However is there a straightforward way to interpret the following: $\int dx_1 ...
1
vote
0answers
34 views

An example of Markov chain with no closed class?

What is an example of Markov chain with no closed communicating class? Closed class means that once we are in that class, there would be no escape from it. I am thinking that an example would be ...
0
votes
0answers
26 views

For which p is the Markov chain recurrent (almost random walk)

We have a Markov chain on Z with matrix: $p_{ii+1}=p=1-p_{ii-1}$ for $i\leqslant-1$, $p_{ii-1}=p=1-p_{ii+1}$ for $i\geqslant1$, and $p_{00}=p_{01}=p_{-10}=\frac{1}{3}$. For which values of p ...
1
vote
1answer
28 views

Equality of probability of finite hitting time for irreducible states in Markov Chain

Suppose I have a finite state Markov Chain with state space $S=\{1,2,3,4,5,6\}$. Suppose I further have that $\{1,2\}$,$\{3,4\}$ and $\{5,6\}$ are irreducible classes where $\{1,2\}$ and $\{3,4\}$ are ...
1
vote
0answers
15 views

Deriving Time of extintion of a Small neural Network

I'm trying to derive the Expected Value of the Time of Extintion $\tau_{ext}$ of a small Neural Stochastic Network with the following dynamics, where I consider $\tau_{ext}$ to be the time of the last ...
0
votes
1answer
34 views

Show that $m_{ii}=\infty$ when $i$ is transient

Show that $m_{ii}=\infty$ when $i$ is transient, where $m_{ii}$ is the mean time to get from $i$ to $i$. if $i$ is transient I know that there is a positive probability of going to some ...
1
vote
1answer
34 views

Cont Time Markov Chains. Stationary Probability

A barber finishes haircuts at rate $3$, measured in hours, so on average it takes him 20 minutes to cut a person’s hair. Customers arrive at rate 2. There is, however, only a two chair waiting ...
3
votes
1answer
29 views

Applying Markov Decision Processes to an arrival forecasting problem

I have the following problem and I'd like to know if it's something that was already studied in the literature or not. I'm not sure about the naming conventions either. I have a system $S$ that can ...
0
votes
0answers
20 views

Equivalence between Label propagation and Likelihood estimation over a Markov Random Field

On page 13/14 of label propagation the near equivalence is set up between what they call 33/34 (reproduced below). This near equivalence is not obvious to me. $$P_{F'}(Y) = \frac{1}{Z} \exp \left( ...
0
votes
0answers
27 views

Conditional expectation in continous markov chains

I am trying to understand the double integral in calculating the conditional expectation. In calculating $V_i$, the second and third equalities are due to the law of total probability. I have the ...
1
vote
1answer
29 views

How to discretely stochastically simulate a continuous-time Markov chain?

A continuous-time markov chain describes a continuously varying process, such that future state only depends on the current state. A sampling of a continuous markov chain can be described in terms of ...
0
votes
1answer
45 views

Density function of absorption time in this Markov Chain

Let $X_t$ be a continuous time Markov Chain with state space $\{1,2,3\}$ with the following transition matrix: $$\left( \begin{matrix} -(\lambda+\delta) & \lambda & \delta \\ \mu & ...
2
votes
1answer
95 views

Markov Chain: flip coin 8 times and get 3 consecutive heads

I have confusion while reading the following example in the course material. Q: In a sequence of independent flips of a fair coin, let N denote the number of flips until there is a run of three ...
1
vote
2answers
124 views

Markov Chain: flip 8 coins and get 3 consecutive heads

I was reading the material and I am confused at the following example. Q: In a sequence of independent flips of a fair coin, let N denote the number of flips until there is a run of three ...
1
vote
0answers
33 views

Direct proof of the existence of optimal memoryless deterministic policies in MDP

It is well known that (finite-state, finite-action, discrete time) MDPs admit an optimal policy that is memoryless and deterministic (sometimes called pure). The proof of this fact for ...
0
votes
1answer
23 views

Mean time for the trajectory. Find mean

What is the mean of time when the trajectory of the wiener process, $W_t$, is over the line $y=t$? We need to find $\Bbb{E}\tau$, where $\tau=\sum\limits_{a,b:\forall t\in(a,b) ; ...
0
votes
0answers
19 views

HMM optimization: Lagrange multiplier problem

In David Barber's textbook "Bayesian Reasoning and Machine Learning" he hints at the derivation of the Baum-Welch algorithm for HMM parameter learning: Textbook excerpt, (cannot include images yet, ...
1
vote
1answer
31 views

Strong Markov property with two stopping times

I have a diffusion $X=(X_t)_{t\ge0}$ and a stopping time $\tau$. From the strong Markov property I know that for any time $t\ge0$ (or a random time independent of $X$) I get that ...
0
votes
0answers
16 views

Examples of state-space models that show strong homeostasis but also substantial change after critical threshold?

The question is, can can anyone provide examples of systems or math models that exhibit patterns of homeostasis but which can be exhibit substantial transitions or bifurcations after some critical ...
2
votes
1answer
45 views

How to find Kolmogorov Forward Equations, given generator matrix Q?

I am having difficulty in forming Kolmogorov Forward Equations. I understand how the KFE is derived and that $$\frac {d}{ds} p_{ij} (s) = \sum_{k \neq j} p_{ik} (s) \lambda_{k} r_{kj} - p_{ij} ...
1
vote
0answers
63 views

Does a Markov process have memory zero?

I have the following question: The Markov process with two states and a transition matrix $$P =\begin{pmatrix} 0.3 & 0.7 \\ 0.3 & 0.7 \end{pmatrix}$$ has memory zero. Is it true? My ...
2
votes
0answers
28 views

Good introductory book coupling methods

I am very interested in coupling methods, can you recommend me a good introductory books on this subject? Thanks
2
votes
0answers
64 views

Gambler's ruin problem - expected time

I have troubles seeing the following. Consider the classical gambler's ruin problem, betting 1 at each time $t\in \mathbb{N}$, and losing or winning -1 respectively +1 at each time till the fortune of ...