Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
2answers
54 views

In a transition matrix, is the biggest component of the stationary distribution the one that correspond to the column with the biggest entries-sum?

Given a transition matrix, is the biggest component of the stationary distribution the one that correspond to the column whose sum of entries is the biggest among all columns? (By "correspond" I mean ...
0
votes
2answers
3k views

What is the eigenvalue of stochastic matrix?

I have a stochastic matrix $A \in R^{n \times n}$ whose sum of the entries in each row is 1. When I found out the eigenvalues and eigenvectors for this stochastic matrix, it always happens that one of ...
0
votes
2answers
29 views

Soft Question - book recommendation - Stochastic Processes

My mother language book on stochastic processes is pretty much complete(~500 pages) but would like another one in English, to have in my library. I'm looking for a similar book containing the ...
0
votes
1answer
43 views

Why are the $n$-step transition probabilities well defined?

I was reading a proof for the Chapman-Kolmogorov equations and now I understand why it is the case that for a discrete-time homogeneous Markov Chain $X=(X_n) _{n\geq 0}$ (with state space $S$) the ...
0
votes
1answer
29 views

Up-to-date or Behind - [Markov Chain]

Alex is taking a bioinformatics class and in each week he can be either up-to-date or he may have fallen behind. If he is up-to-date in a given week, the probability that he will be up-to-date (or ...
0
votes
1answer
26 views

Random Early Discard Markov Chain

I'm trying to sketch the Markov chain for a Random Early Discard queueing policy where customers arrive to the queue of infinite size according to a Poisson process with rate $\lambda$. Customers that ...
0
votes
2answers
36 views

How to find the limit of a matrix $P^n = UD^nU^{-1}$ where $D$ is a diagonal matrix of eigenvalues and $U$ a matrix of eigenvectors?

If we have a matrix where $P = UDU^{-1}$, where $D$ is a diagonal matrix of real eigenvalues that are less than or equal to 1, and $U$ is the corresponding matrix of eigenvectors, how can we show that ...
0
votes
2answers
54 views

Markov chains steady-state distribution

Ok so we are given a Markov chain $X_n$, $P=P(ij)$ as the transition matrix and the $(\pi_1,\pi_2,\pi_3,...,\pi_n)$ as steady-state distribution of the chain. We are asked to prove that for every $i$: ...
0
votes
1answer
24 views

Detailed balance implies time reversibility, how about the converse?

Given a Markov chain (finite state space) $X_1,X_2,...$ with transition matrix $P$ and initial distribution $\pi$, if they satisfy $\pi(x)P(x,y)=\pi(y)P(y,x)$, we say they satisfy detailed balance. ...
0
votes
1answer
50 views

Markov Chain: memoryless property?

I have a question about the Markov Chain. We were doing derivation of the Chapman-Kolmogorov Equations, the $n+m$ step state transition probability (please see below): where $P_{i,j}$ is the ...
0
votes
1answer
41 views

Has a Markov chain in compact metric space a stationary distribution (possibly non-unique)?

Let $Y_n$ be a Markov chain in a compact metric space. Is it true that it has a stationary distribution?
0
votes
1answer
45 views

Showing that a Markov-Chain has this property.

So I have to show that: Let $\{X_n\}$ be a Markov chain. Show that the property $P(X_{n+1}=i|X_n=j_n,X_0=j_0)=P(X_{n+1}=i|X_n=j_n)$ holds. Hint: use the Markov-property. The Markov Property being: ...
0
votes
1answer
58 views

I think I've found an invariant distribution for a transient discrete Markov chain - Where is my mistake?

Let $E$ be an at most countable set equipped with the discrete topology $\mathcal E$ $X=(X_n)_{n\in\mathbb N_0}$ be a discrete Markov chain with values in $(E,\mathcal E)$, distributions ...
0
votes
1answer
31 views

Meaning of $\bigvee$ for collection of sets

In a book* I found the following: $\mathcal{F} = \bigvee_i \mathfrak{B}(X_i)$ where $\mathcal{F}$ denotes a $\sigma $-algebra on a markov chain and $\mathfrak{B}(X_i)$ is the Borel sigma algebra ...
0
votes
1answer
49 views

Markov Chain Example.

A Markov chain $X_0,X_1,X_2,\ldots$ has the transition probability matrix $$P= \begin{bmatrix} 0.3 & 0.2 & 0.5 \\ 0.5 & 0.1 & 0.4 \\ 0.5 & 0.2 ...
0
votes
1answer
31 views

is the largest number $X_n$ shown up to the nth roll a Markov chain?

I'm trying to understand what is a Markov chain through One Thousand Exercises in Probability by Geoffrey Grimmet and David Stirzaker. I'm on exercise 6.1.2 of this book. I know that a $X_n$ is a ...
0
votes
1answer
24 views

MC whose transition probabilities have a PDF w.r.t. $\mu$ are reversible w.r.t to $\mu$

Let $(Y_n)_{n \in \mathbb{N}}$ be a Markov Chain with transition probability $$p(x, dy) \sim N(x, \epsilon)$$ Show that $Y$ is reversible w.r.t to the lebesgue measure . What I have done is just ...
0
votes
2answers
149 views

When to stop checking if a transition matrix is regular?

The definition that I have of a Transition Matrix for a Markov Chain is: A transition matrix is regular if some power of it is positive. Doesn't this mean though that in theory, you could keep ...
0
votes
1answer
33 views

Model Complexity for higher order markov model

I do not understand why is there an increase in parameters when moving from first to second order markov model For example considering a feature space of (a - z) For first order markov model, the ...
0
votes
1answer
22 views

Expected Number of Visits - why is $E_x[N_x]=\sum_{n \geq 1} p_{x,x}^{(n)}$

Suppose $(X_n)_{n \geq 0}$ is a discrete-time time-homogeneous Markov chain with transition probabilities $$P[X_{n+1}= y \mid X_{1}=x] = p_{x,y}^{(n)}.$$ Let $$N_x:=\sum_{n \geq 1} 1( X_n=x)$$ denote ...
0
votes
1answer
158 views

Is {$X_n,n\geq 0$} a markov chain?

Consider a process {$X_n,n=0,1,\dots$}, which takes on the values $0,1,2$. Suppose $$P(X_{n+1}=j|X_n=i,X_{n-1}=i_{n-1},\dots,X_0=i_0)$$ $$=P_{ij}^I,\text{when n is even}$$ ...
0
votes
2answers
36 views

Computing $\lim \limits_{n\to \infty}Pr(X_n=0)$ for r.v $X_n$ using matrices and Markov Chains.

There are three coins on the table showing "Heads". Every round, Danny comes and turns a coin upside down: the left one with probability of $1\over 2$, the middle with probability of $1\over 3$ and ...
0
votes
1answer
33 views

Transition matrix in Markov's chain

I'm trying to find the probability transition matrix in this Markov's chain problem. Three black and three white balls are distributed between two polls, in a way that each poll contains three balls. ...
0
votes
2answers
56 views

Determing a transition probability matrix

I need some support with this homework exercise: An urn contains at most $N$ balls. Let $X_n$ be the number of balls in the urn after the $n$-th execution of the following procedure: If the urn is not ...
0
votes
1answer
84 views

Beginner's questions about Hidden Markov Models

I have started reading about Hidden Markov Models, and have some (more or less) minor questions about things I am not sure I understood correctly. I hope asking here is fine: 1. Assumption about ...
0
votes
1answer
131 views

Can you help me with this Markov Chain question?

The Problem: Prove that if the number of States in a Markov Chain is M, and that state j can be reached from state i, then it can be reached in M steps or less. The work: I assumed by contradiction ...
0
votes
2answers
210 views

Markov chain steady state existence

Is it possible for a Markov chain to have no steady state solution ? What is an example of such system ?
0
votes
2answers
215 views

Are irreducible, positiv-definite Markov chains aperiodic?

If $M$ is the transition matrix of a discrete Markov chain, and $M$ is both irreducible, symmetric and positiv-definite, is the resulting Markov chain necessarily aperiodic? In my intuition, ...
0
votes
1answer
48 views

Convergence of the number of visits in a Markov Chain

Suppose we have an irreducible and recurrent discrete-time Markov chain with states over the finite set $\mathcal{X}$. Let $N_t (x)$ denote the number of visits to state $x$ up to time $t$. Let ...
0
votes
1answer
97 views

Show that this Markov chain is recurrent

So I have a Markov chain on the nonnegative integers such that, starting from $x$, the chain goes to $x+1$ with probability $p$, $0<p<1$, and goes to state $0$ with probability $1-p$. I'm ...
0
votes
1answer
164 views

M/G/1 queue has embedded Markov chain

I tried to prove that the M/G/1 queue has an embedded discrete-time Markov chain. But I'm not sure if I have done it right and properly. Specially I'm not 100% sure if i calculated right the ...
0
votes
2answers
82 views

Given a Markov-chain, what is the probability of being at a given state?

Given a Markov-chain, what is the probability of being at a given state? I drew the diagram below just as an example, there is nothing special about it but it would be nice if your answer used it as ...
0
votes
1answer
45 views

Why does this hold for the mean hitting time?

Let $X$ be a Markov chain and $T_A$ the hitting time. My text uses this in a proof: $$\mathbb E[T_A \ | \ X_0=k ] = \sum_{l\in S} \mathbb P(X_1=l \ | X_0=k\ )(1+\mathbb E[T_A \ | \ X_0=l ])$$ and I ...
0
votes
1answer
42 views

Integration with respect to conditional measure?

Let $(X_n)$ be a Markov chain. For $i\in S$ my text defines $$N_i:=\sum_{n=0}^\infty \mathbf 1_{\{ X_n=i \}}$$ and then, as a part of a larger proof, claims that $$\mathbb E_i(N_i)=\sum_{n=0}^\infty ...
0
votes
1answer
321 views

Understanding detailed balance equations

I'm trying to understand how the equilibrium distribution satisfy the detailed balance equation. To my understanding, I only understand that a detailed balance equation would only be satisfied if ...
0
votes
3answers
163 views

How fast does this Markov chain converge?

Observe the above a Markov chain and limiting matrix of it. Finding the limiting matrix if it exists is easy but I am curious as to how fast this given matrix converges to its limiting matrix. Is ...
0
votes
1answer
84 views

Continuous time Markov chain. proportion of time spent in state i

If a question asks for the proportion of time spent in a specific state is this the same as the stationary distribution or something else? For continuous time Markov chain with finite state space.
0
votes
2answers
107 views

Weather Transition Matrix

Based on observation, I've gathered some data: Day 1 2 3 4 5 6 7 8 9 10 S R S F S R F F R S (S) = Sunny (R) = Rainy (F) = Foggy How do I construct this into a transition matrix, for markov ...
0
votes
1answer
66 views

Markov chain probability that a state changes

For the Markov chain given below what is the best way to find the probability P{$x_n = x_{n-1}$} and P{$x_n \neq x_{n-1}$} The transition matrix of the chain is \begin{array}{} 1/3 & 2/3 & ...
0
votes
1answer
664 views

Random walk with single absorbing boundary

There is a random walk on a linear lattice of size $\{0,N\}$, where $0$ is the origin and a reflecting boundary and $N$ is the absorbing boundary. It moves forward or backward one step at a time with ...
0
votes
1answer
120 views

Stationary distribution for a Markov chain which is not irreducible

I have a Markov chain with $K$ states $S$: {$s_1,s_2,...,s_K$}. $s_1$ is reachable from any state in $S$; however not all the states can be reached from $s_1$. What does the stationary distribution ...
0
votes
1answer
352 views

Load balance equation of birth-death process

I am not able to understand the reasoning behind the load-balance equation of the birth-death processes (Markov chain) . The equation is $\pi_ib_i = \pi_{i+1}d_{i+1}$ , where the symbol have their ...
0
votes
1answer
174 views

Expected length of generating a pattern (throwing dice)

A fair die is tossed repeatedly. The experiment ends as soon as the last six outcomes form the pattern 131131 What is the expected length (i.e. the number of rolls of the die) of this experiment?
0
votes
1answer
66 views

Kolmogorov backward equation: what is it computing?

The kolmogorov backward equation states: $P_{ij}^{'}(t) = \sum_{k \ne i} q_{ik}P_{kj}(t) - v_iP_{ij}(t)$ Is this computing the rate of transition from i to j?
0
votes
2answers
1k views

Markov chain: closed, finite classes are recurrent?

In Norris: Markov Chains the closed class C is defined as one for which $i\in C$ and $P_i(X_n=j \text{ for some }n\ge0)>0$ implies that $j\in C$. Here's theorem 1.5.6 from the book with proof ...
0
votes
1answer
151 views

Need help in understanding state transition diagram of a convolutional coder. How are the output bits calculated?

Have a look at the above figure. I am confused in how the output bits are calculated. e.g. according to my understanding a state transition from 00 to 10 (with input bit 1) should produce output 10 ...
0
votes
2answers
120 views

Discrete Markov Transition Matrix

Well, I've been reading over the internet but I've been unable to find a straight answer. I've got a transition matrix for a Markov Discrete Chain. I've made the graph and according to my knowledge, ...
0
votes
1answer
106 views

Expected number of shots for a game to end

A basketball player plays a shooting game. He gets +1 point if he scores a basket and -2 points if he misses. He starts with 0 points. The game ends when the player reaches +10 or -10. What is the ...
0
votes
1answer
663 views

Markov Chain Equilibrium Distribution Question

Suppose I have the following equilibrium probability distribution: $\vec π = ({2\over5} , {1\over5} , {3\over20},{1\over4})$, corresponding to states 0,1,2,3, respectively.From my possible states of ...
0
votes
1answer
72 views

A basic doubt on the sojourn time of a CTMC

By using the memoryless property, I find the CDF of sojourn time of a CTMC as follows : $$F_X(t) = 1-e^{-F_X'(0)t}$$ I am slightly confused about the intuitive meaning of the term $F_X'(0)$. How ...