Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
2answers
1k views

Can Markov Chain state space be continuous?

I looked for a formal definition of Markov chain and was confused that all definitions I found restrict chain's state space to be countable. I don't understand purpose of such a restriction and I have ...
0
votes
2answers
141 views

Issue with calculating the cholesky decomposition

I am trying to calculate the cholesky decomposition of the matrix Q= ...
0
votes
3answers
99 views

Capacity of a discrete channel in the telegraphy case

I'm reading Shannon's article A Mathematical Theory of Communication, and I'm stuck at the telegraphy case example, on page 4. Shannon writes a formula involving $N(t)$, the number of sequences of ...
2
votes
2answers
190 views

Markov chain basic positive recurrency question

If a discrete markov chain is stationary (as far as I know: doesn't modify itself with time), irreducible (doesn't have transient states) and aperiodic (no periodic states), is it positive recurrent? ...
1
vote
0answers
72 views

Positive recurrence of a continuous-time jump process, from its jump chain

I am looking at an irreducible, continuous-time jump process $(X_t)_{t\geq0}$ with the following jump times. Let a Poisson process $(T_i)_{i=1}^\infty$ determine the event times. With probability ...
2
votes
0answers
200 views

Computing the stationary distribution of a markov chain

I have a markov chain with transition matrix below, $$\begin{bmatrix} 1-q & q & & & \\ 1-q & 0 & q & & \\ & 1-q & ...
14
votes
1answer
514 views

Eigenvalues for $3\times 3$ stochastic matrices

This is a plot of the non-real eigenvalues of 10000 randomly generated $3\times3$ stochastic matrices. It's pretty clear that they lie in the convex hull of the three cube roots of unity. The ...
1
vote
2answers
177 views

Finding again the stationary distribution of a markov chain

I am asked to compute the stationary distribution of the markov chain with state space $E=\{0\dots,n\}$ and transition matrix below: \begin{bmatrix} 0 & 1 \\ \frac{1}{n} ...
3
votes
1answer
2k views

Finding the stationary distribution of a markov chain

I am asked to compute the stationary distribution of the markov chain with state space $E=\mathbb{N}_0$, $q_n >0$ for all $n \in \mathbb{N}_0$ and transition matrix below: \begin{bmatrix} q_0 ...
3
votes
1answer
1k views

Calculating stationary distribution of markov chain

I am asked to compute the stationary distribution of the markov chain with state space $E=\{0\dots,n\}$ and transition matrix below: \begin{bmatrix} 0 & 1 \\ \frac{1}{n} ...
1
vote
1answer
355 views

Why does this example of Hardy-Weinberg equilibrium not work?

Background Info Hardy-Weinberg equilibrium is a mathematical model of the frequencies of alleles (i.e., versions of a gene) in a population. The model states that the frequency of the 2 alleles in a ...
3
votes
1answer
4k views

Kendall notation's “General distribution”, what does that mean?

The first and second parameters for the Kendall's notation may have a G value, which stands for General distribution, see here. But what does that mean? What is a ...
2
votes
0answers
178 views

Ergodicity and mixing

From MathOverflow, R W said: Unfortunately, the way the term "ergodic" is used in the theory of (finite) Markov chains is completely misleading from the point of view of general ergodic ...
1
vote
1answer
398 views

Markov Process: Have you seen this notation and do you know what it means?

Ok, I've already posted this a minute ago, but my text deleted itself while I was editing it :-( So next try: Can you help me to understand the notation my professor uses to describe Markov ...
6
votes
1answer
1k views

Stationary distribution of random walk

Let $\mathcal{X}$ be a simple random walk with barrier at zero, state space $E = \mathbb{N}_0$ and transition matrix below with $0<q<1$. \begin{bmatrix} 1-q & q & & ...
1
vote
1answer
2k views

Show irreducibility of markov chain

I need to show that the markov chain that has transition matrix written below is irreducible. \begin{bmatrix} 0.2 & 0.5 & 0.1 & 0.1 & 0.1 \\ 0.2 & 0.5 ...
0
votes
1answer
290 views

Why are persistent states of a Markov chain on a finite state space non-null?

i would like to understand the following statement about Markov chains on a finite state space S: "If S is finite, then one state ist persistent and all persistent states are non-null." It is more ...
1
vote
1answer
108 views

Why does this probability equivalence of events hold?

$P(X_0 = j, X_m \ne j, 1 \le m \le n-1) = P(X_m \ne j, 1 \le m \le n-1) - P(X_m \ne j, 0 \le m \le n-1) $ Where $\{X_n\}$ is an irreducible Markov Chain with a finite state space.
2
votes
1answer
216 views

Confused about Markov property

The sample space is $\Omega$ with $\omega = (\omega_0, \omega_1, \ldots) \in \Omega$ an infinite sequence of a set $S$. So the measure space is $(S^{\mathbb{N}}, \mathcal{S}^{\mathbb{N}})$ where ...
2
votes
1answer
361 views

Non-symmetric simple random walk stopping time

Say there is a random walk $\{S_n\}$ with $S_0=0$ and $0<p=P(S_1=1)<\frac{1}{2}$. We know such a random walk would go to $-\infty$ eventually. Define the stopping time $T=\inf\{n: S_n=-\infty\}$, ...
0
votes
0answers
247 views

How to calculate the limit kernel of a non-ergodic Markov Chain?

This question is about finding the limit kernel $P^\infty$ of a non-ergodic Markov Chain. The problem We consider a TDMC (Time Discrete Markov Chain) $(X_t)_{t \geq 0}$ with $X \in \mathcal{X}$ ...
1
vote
0answers
163 views

Continuous-time Markov process - need help with flux and discrete-time equivalent

I have a continuous-time markov process and I need to calculate the following transition frequencies matrix (aka intensity matrix) transition probabilities all parameters which define permanence ...
0
votes
1answer
117 views

Manipulating ergodic Markov chains in order to make them non-ergodic

Consider a Markov chain, for simplicity let us consider time discrete chains. The problem We consider a TDMC (Time Discrete Markov Chain) $(X_t)_{t \geq 0}$ with $X \in \mathcal{X}$ (having ...
2
votes
2answers
2k views

What does the steady state represent to a Markov Chain?

I'm a little confused as to the interpretation of the steady state in the context of a Markov chain. I know Markov chains are memoryless, in that each state only depends on its immediate predecessor, ...
1
vote
1answer
204 views

Finding the distribution function of a markov process

Given a markov process $\mathcal{Y} = (Y_t : t \ge 0)$ with state space $E=\{1,2,3\}$ and with generator matrix $G = \left[ \begin{array}{ccc} -3 & 1 & 2 \\ 1 & -2 & 1 \\ 0 & 0 ...
3
votes
1answer
871 views

Probability distribution of markov chain

I have a Markov chain with state space $E = \{1,2,3,4,5\}$ and transition matrix below: $$ \begin{bmatrix} 1/2 & 0 & 1/2 & 0 & 0 \\ 1/3 & 2/3 & 0 ...
2
votes
2answers
3k views

Calculating conditional probability for markov chain

I have a Markov chain with state space $E = \{1,2,3,4,5\}$ and transition matrix below: $$ \begin{bmatrix} 1/2 & 0 & 1/2 & 0 & 0 \\\ 1/3 & 2/3 & 0 & 0 & 0 \\\ 0 & ...
3
votes
3answers
736 views

Probability of absorption in a discrete Markov chain

Let $\{X_{n}\}$ be a Markov Chain on the state space $S=\{1,...,100\}$ with $X_{0}=30$, and transition probabilities given by $p_{1,1}=p_{100,100}=1$, $p_{99,100}=p_{99,98}=1/2$ and for $2\leq ...
4
votes
1answer
878 views

Strong Markov property - Durrett

I recently had great success with my first question here so I will boldly go on to a second. Here goes: I'm studying Markov Chains in Rick Durrett - Probability: Theory and example and I'm stuck with ...
1
vote
1answer
3k views

Markov Models simple introduction

We're studying Markov models (still at the basis: transitory states, periodic states, etc..) but the professor isn't very good at teaching and I feel I'm getting lost soon. I'd love to have a simple ...
1
vote
1answer
628 views

Use Discrete Markov Chain to predict n steps ahead

I modeled a Markov Chain with M states. Assuming that the process is homogeneous in time. But, each state has a differente resident time. Moreover, each state has a self-loop transition and a ...
4
votes
3answers
151 views

Probability of finding 2012 before any other occurence of 012 in a random infinite sequence of digits 0,1,2

The following problem is from the semifinals of the Federation Francaise des Jeux Mathematiques: One draws randomly an infinite sequence with digits 0, 1 or 2. Afterwards, one reads it in the ...
1
vote
1answer
389 views

Finite State Markov Chain Stationary Distribution

How does one show that any finite-state time homogenous Markov Chain has at least one stationary distribution in the sense of $\pi = \pi Q$ where $Q$ is the transition matrix and $\pi$ is the ...
1
vote
1answer
413 views

Constructing the transition matrix for device failure

There are three machines. Let the probability that an operable machine fails on any given day be $0.1$, independently of the other machines. Only one machine can be repaired on the same day (so it is ...
0
votes
1answer
262 views

Find stationary distribution decomposable Markov chain

Again a probability exercise: Let $X=U \cup V$ be the finite state space of a Markov chain, where $U$ and $V$ are disjoint subsets of $X$ and $p_{ij}=0$ if both $i,j \in U$ or both $i,j \in V$. ...
3
votes
1answer
116 views

How to find the limit of a markov chain

Given a markov chain where the next state is related to the previous state by the following matrix: $$\begin{array}{c|ccc} & A & B & C\\ \hline A & p_1 & q_1 & r_1\\ B & ...
3
votes
1answer
1k views

coin flips and markov chain

Consider the case of an infinite (or finite $n$) string of coin tosses, and let $q$ and $1-q$ be the probabilities that the coin comes up tails and heads, respectively. (For simplicity, we can take ...
2
votes
1answer
464 views

Expected time of mouse's survival (stochastic matrix)

In the following wikipedia page explaining stochastic matrices, there is an example with 5 boxes and a cat and a mouse where they jump to a left or right box at every turn and it explains how to ...
3
votes
3answers
192 views

Markov chain stationary probability simulation

Having a defined markov chain with a known transition matrix, rather than to calculate the steady state probabilities, I would like to simulate and estimate them. Firstly, from my understanding there ...
2
votes
2answers
648 views

Proof of Markov property for Ehrenfest urn

[the question got downvoted on MO with the recommendation to ask here] In many books Ehrenfest Urn is used as an example of a homogeneous Markov chain, where entries in transition probabilities ...
2
votes
1answer
182 views

Some basic questions on Markov chains (Durrett)

If you have a state space $S$, usually I think of a Markov chain $X_n$ on it as $X_n$ takes values in $S$ and satisfies the obvious Markov property and so on. In Durrett's book, he says one should ...
1
vote
1answer
886 views

Proving a process is Markov chain

Could anyone give me an example of a problem where it is requested to prove rather than assume that a stochastic process forms a Markov chain. I can think of something like this: if $X_{n+1} = X_{n} + ...
2
votes
2answers
1k views

What values makes this Markov chain aperiodic?

Let the following transition matrix represent a $4$ state Markov chain $$\begin{pmatrix} 0 & a & 0 & b \\ \frac{1}{2} & 0 & \frac{1}{3}+c & d \\ 0 & a & 0 & ...
1
vote
1answer
342 views

Show that two states in the same communicating class of a Markov chain must have the same period

How would you go about showing that two states in the same communicating class of a Markov chain must have the same period? Any help would be greatly appreciated.
1
vote
1answer
111 views

Difference in markov chain persistense probability

I feel confused! Suppose that we are given a markov chain that has transition rates for each $q_{ij}$. So you can multiply the matrix $[p_1,p_2,\ldots,p_n] Q=0$ and solve the system. $p_1, p_2, ...
1
vote
0answers
95 views

Quasi-stationary distribution of a state in a birth-and-death MC

I need to find an expression for the first state in an MC with transition matrix $P$ with tridiagonal entries. The state space is $U={1,2,..n}$ with the last state being absorbing. Expressions for ...
4
votes
1answer
229 views

Does every continuous time minimal Markov chain have the Feller property?

Consider a Q-matrix on a countable state space. (A Q-matrix is a matrix whose rows sum up to $0$, with nonpositive finite diagonal entries and nonnegative offdiagonal entries). As explained for ...
3
votes
2answers
481 views

Simple proof that stationary birth-death chains are reversible

A Markov chain with state space $\mathbb{Z}$ is a birth-death chain if the transition probabilities satisfy $p(x,y) = 0$ for $|x-y| > 1$. That is, the only possible transitions are to move one ...
1
vote
0answers
55 views

Green kernel of Hunt processes

Let $X$ be a regular Hunt process on $R^+$ with starting at $x$ and $T_y :=\inf\{t>0: X_t=y\}, y\in R^+$. $G_q(\cdot,\cdot), q >0$ denotes Green kernel of the process $X$. We have the following ...
2
votes
1answer
432 views

Eigenvalues of a infinitesimal generator matrix

Consider a Markov process on a finite state space $S$, whose dynamic is determined by a certain infinitesimal generator $Q$ (that is a matrix in this case) and an initial distribution $m$. 1) Is ...