Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
10 views

emmission probabilities in a hidden markov model with 2 states and an alphabet of 4 characters

I'm reading through a text that is describing how to use use hidden markov models to identify areas of biological sequences that correspond to specific biological features. It starts with a simple ...
1
vote
1answer
39 views

Expected steps in a Markov chain

I have two jars; initially one is empty while the other holds $n$ red and $n$ blue marbles. Every minute, I do the following procedure: take any pair of marbles of different colors but from the same ...
-1
votes
0answers
28 views

Markov chains- construction of the stationary distribution of an irreducible, aperiodic and positive recurrent Markov chain

Proof In the excerpt of a proof (above) that proves that If markov chains are irreducible, aperiodic and positive recurrent. Then the distribution $\mathbf{\pi}$ with the entries of $\mathbf{\pi}$: ...
0
votes
0answers
41 views

Random walk with reflection and skips in linear system

Let's take a case of simple and linear Random walk (0, 1...n) with only one absorbing state n and reflecting state -1, which we can define as: P (move right at state i) = 1/2 and P (moving left at ...
5
votes
0answers
41 views

Poisson: Conditional Probability on Pizza order

I am not sure about my answer. In particular, part b of the following question. Pizza orders arrive according to a Poisson process of rate 20 per hour. Orders are independently for a vegetarian ...
0
votes
0answers
27 views

Getting different results with two methods for a Markov Chain

Given the below Markov transition matrix, calculate $p_{0,1}^9$ \begin{matrix} 0.5 & 0.5 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \\ \end{matrix} Method ...
3
votes
1answer
34 views

Poisson: $P(N(4)-N(2)=5|N(4)=8)$

Let {N(t), t ≥ 0} be a Poisson process of rate 2. Determine: a) $P(N(4)-N(2)=5|N(4)=8)$ Attempt: The conditional poisson distribution is uniformly distributed between the interval [0,4]. Therefore, ...
3
votes
0answers
41 views

This is a Markov Chain?

Consider two irreducible ergodic Markov chains with the same state space $\{0, 1, . . . , N\}$, with transition matrices $P$ and $Q$ and respective stationary distributions $\pi$ and $\rho$. We ...
1
vote
0answers
17 views

Harris Chain and Stationary Distribution

Consider a Harris chain given by $\{X_n\}$ with the following transition function, $X_{n+1}=\max \{0,X_n-b\} $ with probability $p$ and $X_{n+1}=\max \{0,a-\tau\} $ with probability $1-p$, where $\tau ...
1
vote
1answer
24 views

M/M/1 queue derivation: how to “recursively solve in dependence on $p_0$”

I want to sketch out the derivation of the equations for an M/M/1 queue for a presentation I'm giving. I can understand most of the derivation from Willig but I don't understand this section from p10 ...
0
votes
0answers
33 views

What is the difference between “state-transition-matrix” and a transition matrix?

What's the difference between a state-transition-matrix and a transition matrix (say, for an ergodic Markov Chain) that is typically taught in a basic probability theory course? This is the first ...
3
votes
1answer
33 views

Random walk on the positive integers with reflecting boundary

Consider a Markov chain $X$ on the positive integers where for each $n$: $$n\longrightarrow 1,\;2,\;3\;\dots \;n,\;n+1$$ with equal probability, and $n\longrightarrow m$ with zero probability if ...
0
votes
1answer
16 views

Random Walk Markov Chain Long run distribution

In the question above do I have to calculate the stationary distribution? I've been learning about the ergodic theorem but I'm not sure if it's applicable here. I know that the probability that Xn ...
1
vote
2answers
36 views

Expected time between successive visits in a Markov Chain?

This is a pretty basic question and I know the answer is probably really obvious, but I am having trouble reasoning as to why the following is true: (From my lecture notes): """ Expected time ...
1
vote
1answer
27 views

Transition matrix and communicating classes

Firstly I wanted to check if they have a mistake in the solution. So for the transition matrix, the element p11 should be 0 and p12 should be 1, but they have it the other way round so I just ...
1
vote
1answer
28 views

Constructing transition graph from transition matrix

Ok so for this question I'm having trouble understanding how the transition graph has been drawn from the given transition matrix. This is what I understand and hopefully someone can correct the ...
0
votes
1answer
28 views

Computation of n-step transition matrix : method of matching coefficients

For the third step I don't understand how they have worked out $C_1^1=0$? where did they get the value of 0 from? p11(n)=(c01+c11n)lamda 1^n. Using p11(1)=1-a and c01=1, n=1,lamda 1=1 and lamba ...
0
votes
2answers
29 views

Soft Question - book recommendation - Stochastic Processes

My mother language book on stochastic processes is pretty much complete(~500 pages) but would like another one in English, to have in my library. I'm looking for a similar book containing the ...
1
vote
0answers
10 views

How to determine the transition probability in Sequential Importance Sampling (SIS) for Particle Filter

Given a state-space model \begin{align} x_k &= f_k(x_{k-1}, v_{k-1}),\\ z_k &= h_k(x_k, w_k), \end{align} where $x_k \in {\mathbb R}^{n}$ and $y_k \in {\mathbb R}^{m}$ are the system state ...
-2
votes
1answer
23 views

Markov's matrix into stationary Distribution [closed]

How do I know if this Markov's Transition Matrix converges into a stationary distribution? $$P= \begin{bmatrix} .8 & .2 & 0 \\ .3 & .4 & .3 \\ .2 & .1 & .7 \\ ...
0
votes
1answer
42 views

Construction of continuous-time markov chain and finding stationary distribution

There are 15 lily pads and 6 frogs. Each frog, with rate 1, jumps to one of the other 9 unoccupied pads chosen uniformly at random. What is the stationary distribution for the set of occupied lily ...
1
vote
1answer
59 views

Prove that $Y_n=X_{n-1}X_n$ is a markov chain

Let $\{X_n\}_{n=0}^\infty$ a sequence of discrete random variables independent identically distributed. Let $Y_n$ such that $Y_n=X_{n-1}X_n$ for all $n\ge 1$ Is $\{Y_n\}_{n=0}^\infty$ a markov ...
1
vote
2answers
35 views

Find the general expression for the values of a steady state vector of an $n\times n$ transition matrix

I have a question that is asking to find the values of the elements in the steady state vector for a regular transition matrix P of size $n \times n$. All I'm given is that the the elements in each ...
0
votes
1answer
22 views

What is the (practical) difference between a stationary distribution and an equilibrium distribution of a MC?

I know that, for a Markov Chain, a stationary distribution is the (row) vector $\pi$ such that $\pi \cdot P = \pi$, where $P$ is the one-step transition matrix for the MC. Intuitively, I assume that ...
0
votes
1answer
66 views

prove homogeneous markov chain

$Y_0, Y_1,Y_2,\dots$ are independent and identically distributed random non-negative integer outcomes. Let X_0 = Y_0 $ Let $X_0 = Y_0$ and $X_n = X_{n-1} - Y_n$ if $X_{n-1}>0$, else $X_n = ...
0
votes
0answers
18 views

Exercises on the following topics on Markov Chains

We are being taught the following topics in Markov Chains: 1) Markov Chain Monte Carlo: Hard Core model, Counting random q-colourings of a graph 2) Total variation distance for a Simple Symmetric ...
1
vote
1answer
27 views

Transience, recurrence and null recurrence of markov chain

I am trying to get an intuition for transience, recurrence and null recurrence. I constructed an example MC for myself represented by this graph below: I'm thinking that all of the states are ...
0
votes
0answers
29 views

Defective Markov transition matrix and relation to its limiting distributions

Im trying to come to grips with what the physical interpretation of a non diagonalisable Markov Matrix means in terms of what we can deduce about it having a limiting distribution/ what potential ...
0
votes
1answer
41 views

Leaving time of a set

I want to prove the following result. Let $S_n$ be a symmetric irreducible random walk on the integers (d=dimension). Claim: If $x\in A$ and $P_x(T_A=\infty)>0$ then $\forall \epsilon>0\exists ...
1
vote
0answers
30 views

Variation on the classic ABRACADABRA problem

Suppose letters are chosen randomly from the alphabet, one at a time. It is well known that the expected time until ABRACADABRA is spelled out is $26^{11}+26^{4}+26$. The proof uses discrete time ...
2
votes
2answers
58 views

Renewal process problem, where $X_i$'s are i.i.d. with exponential distribution.

A room is lit by $2$ bulbs. Bulbs are replaced only when both bulbs burn out. Lifetimes of bulb's are i.i.d exponentially distributed with parameter $λ=1$. What fraction of the time is the room only ...
1
vote
2answers
20 views

Attitude true for Markov Chains(Maybe duplicate)

Suppose, that we have a Markov-chain with finite domain. For all $i,j$ elements $P_{i,j}>0$, where $P$ is our matrix. Show, that reversible(sorry, I forgot that out) stationary distribution exists ...
1
vote
2answers
58 views

Prove that $i$ is an accesible state in a markov chain

Let $i$ be a recurrent state of an homogeneous markov chain such that the state $j$ is accesible from $i$ (that is $\exists$ $k\ge 1$ such that $p_{ij}(k)>0$) Prove that $i$ is accesible from $j$ ...
0
votes
0answers
41 views

Computing hitting times from the stationary distribution

My question is whether it is possible to compute hitting times from the previously calculated stationary distribution $\pi$ of a continuous-time Markov process $(X_t)_{t \geq 0}$. I know that, from ...
2
votes
2answers
38 views

Mean number of tosses of a fair dice to get a sum of outcomes being a multiple of $5$

Let $S_n$ denote the sum of the outcomes of the $n$ tosses of a fair dice. Let $T=\inf\{n>0: S_n$ is a multiple of $5\}$. Compute $E(T)$ (by means of markov chains). Attempt. Instead of ...
1
vote
0answers
23 views

How is the following attitude true?(Markov chains)

Let us have an $X_n$ Markov-chain with finite $S$ set as domain. $A \subset S$ is given, so that $P_x(T_A < \infty) > 0$ for all $x \in S$. $T_A=\inf \{ n \geq 1: X_n \in A\}$. Then we take a ...
0
votes
1answer
21 views

Mean number of steps needed to reach a recurrent state in a finite irreducible Markov chain

Let $\mathbb{X}$ be a finite state space of an irreducible markov chain $\{X_n\}$ and let $T_x=\inf\{k\geq 0\mid X_k=x\}$ be the number of steps until $\{X_n\}$ reaches state $x\in \mathbb{X}$. ...
1
vote
1answer
36 views

Markov chains and queues

I do not understand how may I use the Markov Chain $Y$ and and describe the system $X$ using the states that the exercise suggest. I was searching queue's examples and -i understand this is a M/M/1 ...
0
votes
0answers
21 views

Binary Hidden Markov Model

Consider a binary HMM with 2 observed variables $O_n \in \{0,1\} \; \forall n \in \mathbb{N}$. Suppose that the hidden Markov process $X_n$ is characterised by a known transition probability matrix ...
1
vote
2answers
37 views

Continuous time Markov Chain's Natural Filtration

Given a continuous time Markov chain $\left(X_t \right)_{t\geq 0} $ with finite or countable state space $S$, transition matrix $P(t)$, what I want to prove is: $$\text{Let} \quad f:S \to \mathbb{R} ...
1
vote
1answer
35 views

Connection between Ergodic Theory and Markov Chains

Could someone suggest a good reference where the connection between Ergodic Theory and (ergodic) Markov Chains is nicely explained ?
0
votes
0answers
24 views

Distribution of states given observations in HMM

Suppose you have an HMM with two states $(S_1, S_2)$ and two observations $(a, b)$. We know the following: $P(S_1|S_1) = 0.5$ $P(S_1|S_2) = 0.25$ $P(a|S_1) = 0.25$ $P(a|S_2) = 0.5$ Initial state at ...
0
votes
1answer
51 views

Markov chain - distribution of probability of state at generic step

Let $S$ be a finite discrete state set. Let $X(i) \in S, i = 1,2, \ldots$ be a random variable sequence. I've built-up a Markov transition matrix from a set of sequences of states. State $s_1 \in S$ ...
0
votes
0answers
19 views

probability a markov chain leaves a transient state in favor of a recurrent state

Let $\{X_n\}$ be a time-homogenuous markov chain, with state space $\mathbb{X}$ and transition matrix $P$. Let $C_1$ be a transient class and $C_2$ be a recurrent class and let also $x\in ...
0
votes
1answer
59 views

Application of CLT to random walks

Let $X_1,X_2,\ldots$ be an iid sequence such that $P\{X_1 = 1\} = p$, $P\{X_1 = -1\} =p$ and $P\{X_1 = 0\} = 1-2p$. We have that $E[X_1] = 0$ and $E[X_1^2] = 2p$. Define $S_n = \sum_{i=1}^nX_i$ and ...
0
votes
0answers
34 views

Finiteness of the hitting time of random walk

Let $X_1,X_2,\ldots$ be an iid sequence such that $P\{X_1 = 1\} = u$, $P\{X_1 = -1\} = d$ and $P\{X_1 = 0\} = 1-(u+d)$. We have that $E[X_1] \neq 0$. Define $S_n = \sum_{i=1}^nX_i$ and $S_0 = 0$ and ...
0
votes
0answers
43 views

Ergodic Theorem for Markov Chain with one closed communicating class and several transient states

It is known, that if a markov chain with a finite state space has only one closed aperiodic communicating class and several transient states, then there is a unique stationary distribution for this ...
2
votes
2answers
89 views

Convergence of mean of an irreducible Markov chain / ergodic theorem

Let $\{X_n\}$ be an irreducible Markov chain on a discrete state space $\mathbb{N}$, that has a stationary distribution $\pi$. Prove or disprove : with probability $1$: $$\lim_{n\rightarrow ...
1
vote
1answer
53 views

Markov Chain Questions

I've been stuck on these problems for a while. I keep banging my head against the wall, but my calculations are incorrect each time. I sum the probabilities together for each possibility (it's a ...
1
vote
0answers
27 views

Transform Markov chain that doesn't have stationary transition probabilities to one that does?

This question concerns Exercise 7.3 in Walsh's Knowing the Odds. A Markov chain is defined as having stationary transition probabilities if for all $i, j, n$ we have $P(X_{n+1} = j \mid X_n=i) = ...