Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
3answers
202 views

The limit of matrices

Consider a square matrix $P$. We call it stochastic if it holds that $$ p_{ij}\geq0\text{ and } \sum\limits_{j=1}^m\,\,\,\,p_{ij} = 1 $$ for all $1\leq i,j\leq m$. I wonder when the following limit ...
1
vote
2answers
203 views

Average run lengths for large numbers of trials: Intuition and proof

This article states that the formula for the average run lengths for large numbers of trials is:$$\frac{1}{1-Pr(event\ in\ one\ trial)}.$$ My questions What is the intuition behind this formula? Do ...
1
vote
1answer
1k views

What does the steady state represent to a Markov Chain?

I'm a little confused as to the interpretation of the steady state in the context of a Markov chain. I know Markov chains are memoryless, in that each state only depends on its immediate predecessor, ...
1
vote
2answers
130 views

markov chain with arbitrary period

Given any positive integer, how can I think of a Markov Chain (states and transition probabilities) to have that integer as the period of two of its states? Thanks.
1
vote
2answers
24 views

Transition Matrix eigenvalues constaints

I have a Transition Matrix, i.e. a matrix whose items are bounded between 0 and 1 and either rows or columns sum to one. I would like to know if it is possible that in any such matrices the ...
1
vote
1answer
54 views

A question regarding Markov Chains

Is it possible that we combine some states of a Markov chain, like in this figure? (All non-zero states combined) 1) If yes, what are the new transition probabilities, i.e. p1 and p2 and p3 in the ...
1
vote
2answers
42 views

Markov chains question?

A country is divided into three geographic regions. It is found that each year 5% of the residents move from region I to region II and 5% move from region I to region III. In region II, 15% move to ...
1
vote
1answer
48 views

Are there open questions in Markov chains?

I would be curious to know if there's still open question about discrete markovian chains
1
vote
1answer
327 views

example of irreductible transient markov chain

Can anyone give me a simple example of an irreductible (all elements communicate) and transient markov chain? I can't think of any such chain, yet it exists (but has to have an infinite number of ...
1
vote
2answers
96 views

Specific question to a Markov chain proof in Durrett

I apologize if this is to specific but i've already talked to two of my professors without much success and I really need to understand this subject. The following theorem is stated in Durrett page ...
1
vote
2answers
1k views

What values makes this Markov chain aperiodic?

Let the following transition matrix represent a $4$ state Markov chain $$\begin{pmatrix} 0 & a & 0 & b \\ \frac{1}{2} & 0 & \frac{1}{3}+c & d \\ 0 & a & 0 & ...
1
vote
2answers
158 views

Is it meaningful to talk about margins-of-error in a Markov Chain?

We're creating a Markov Chain based on an analysis of user's history. We add up and normalise users behaviour, and then normalise to create a two dimensional map of probabilities, but some of these ...
1
vote
4answers
263 views

Figuring out probabilities with Hidden Markov Models

I'm really new to Math so sorry in advance if this question does not make sense. Also I cross posted this on stats.stackexchange.com also. Background: I'm trying to learn about hidden Markov models ...
1
vote
2answers
59 views

Markov property and $A=\bigcup_{k=1}^{\infty} A_k$

I am reading Norris's "Markov Chains" and would appreciate an explanation of the following bit. After stating the Markov property, it is said that (on page 4) In general, any event A determined ...
1
vote
1answer
225 views

Continuous Time Markov Chains

What are some techniques to convert Continuous Time Markov Processes into Discrete time Markov Processes? (for purposes of simulations)
1
vote
1answer
429 views

Markov Chains - 2 clarification questions

I'm just getting started with Markov chains and have a few simple questions: Is it possible to define a period for a reducible Markov chain? If so, how? Can we define balance equations and a ...
1
vote
1answer
44 views

P is transition probability matrix.I is identitiy matrix.A is matrix whose entries are all 1.Then prove I+A-P is invertible

$P$ is the transition probability matrix for a finite irreducible markov chain. $I$ is identitiy matrix. $A$ is the matrix whose entries are all $1$. Prove $I+A-P$ is invertible. I don't have any ...
1
vote
2answers
64 views

Expected number of steps between states in a Markov Chain

Suppose I am given a state space $S=\{0,1,2,3\}$ with transition probability matrix $\mathbf{P}= \begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 \\[0.3em] \frac{2}{3} ...
1
vote
1answer
83 views

Markov Chain Solution Eigenvalue

I am having trouble understanding how to solve for the state vector at time $t$ for a markov chain using matrix algebra. I have the following Markov Transition Intensity Matrix, for the states A, N, ...
1
vote
1answer
55 views

Transition Matrix $P$ of a Linear Birth-Death process

I am working on a problem where I have to prove that $P_{20}(t)=P_{10}^2(t)$, given that I have a linear Birth and death process: i.e. $\lambda_n=n.\lambda$ and $\mu_n=n.\mu$. I think the solution ...
1
vote
1answer
208 views

If $P$ is a regular transition probability matrix then $P^{n^2}$ has no zero element

A transition probability matrix $P\in M_{n\times n}$ is regular if for some $k$ the matrix $P^k$ has all of its elements strictly positive. I read that this can be tested by using the following ...
1
vote
1answer
34 views

Markov Chain Converging in Single Step

I have a markov kernel K. From this I find the invariant probability $\pi$. The question is to design a "dream" matrix K*, that converges in one step. Such that $\lambda_{SLEM}=0$ (second largest ...
1
vote
1answer
69 views

A book on finite state continuous time Markov chain

I want to read in detail about finite state continuous time Markov chain. Can anybody suggest a book which deal this topic in detail?
1
vote
2answers
149 views

What's the probability of a gambler losing \$10 in this dice game? What about making \$5? Is there a third possibility?

Can you please help me with this question: In a gambling game, each turn a player throws 2 fair dice. If the sum of numbers on the dice is 2 or 7, the player wins a dollar. If the sum is 3 or 8, ...
1
vote
2answers
86 views

Using the canonical Markov property to prove an obvious fact about Markov chains

Given a Markov chain $\{X_n: n \geq 1 \}$, such that $$\mathbb{P}(X_{n+1} = x_{n+1} | X_n = x_n) = \mathbb{P}(X_{n+1} = x_{n+1} | X_n = x_n, \ldots X_1= x_1)$$ How can I formally prove that: ...
1
vote
2answers
335 views

Expected number of visits to state $j$ between successive visits to a state $i$ in a Markov chain given conditional information

Say I have a Markov chain $\{X_n: n \geq 1\}$ with state space $E = \{1,2,3,4,5\}$ and transition matrix, $$ P = \begin{bmatrix} 0 & 1/2 & 1/2 & 0 & 0 \\\ 1/2 & 0 & 1/2 & ...
1
vote
1answer
337 views

Markov Process: Have you seen this notation and do you know what it means?

Ok, I've already posted this a minute ago, but my text deleted itself while I was editing it :-( So next try: Can you help me to understand the notation my professor uses to describe Markov ...
1
vote
1answer
80 views

Why does this probability equivalence of events hold?

$P(X_0 = j, X_m \ne j, 1 \le m \le n-1) = P(X_m \ne j, 1 \le m \le n-1) - P(X_m \ne j, 0 \le m \le n-1) $ Where $\{X_n\}$ is an irreducible Markov Chain with a finite state space.
1
vote
1answer
60 views

On continuous Markov chains: statistics of recurrent states

Given a continuous Markov chains (and given the transition rates between the states) I would like to know the following: mean time of permanence for all states. higher order moments (i.e., ...
1
vote
2answers
191 views

Reversibility of a Markov Chain

Rephrased question: Is it ever possible for a reducible Markov chain to be reversible?
1
vote
1answer
290 views

Transition matrix

I have a directed graph $G_1$. I extract its transition matrix $T_1$. Now I also have directed graph $G_2$, which is equal to $G_1$ with inverted edges. If I get its transition matrix $T_2$, what is ...
1
vote
1answer
80 views

Markov chain problem, Help!

I am stuck on this question for a long time Question: Consider 4 balls, labelled from 1 to 4 and distributed amongst two urns (Urn 1 and Urn 2). At each time $n>1$, a number from 1 to 4 is chosen ...
1
vote
2answers
32 views

Looking for good literature on Markov Chains with explicit calculations

I am currently starting my thesis on Markov Chains and am looking for good books and papers that include explicit calculations. I have taken a small course on Markov Chains so the subject is not ...
1
vote
1answer
77 views

First step analysis on random walk

Let us consider random walk on integers {0,1,...,N} where $P(N,N)=1$,$P(0,1)=1$, $P(N,N-1)=0$ and all other connections have probability $\frac{1}{2}$. Using first step analysis, compute $p_{00}$ for ...
1
vote
1answer
50 views

Equivalence Classes of a Markov Chain with Transition Matrix

I have the following transition probability matrix for a markov chain with state space S={0,1,2,3,4,5,6}: $\begin{bmatrix} \frac13 & \frac13 &0 & 0 & \frac16 & 0 & \frac16\\ ...
1
vote
1answer
51 views

Finding non-negative matrices, 0 on the main diagonal for which this positive vector is invariant.

This is a sort of reverse eigenvector problem. Usually, given a matrix, we want to describe its eigenvalues. Here -- given a vector, we'd like to determine matrices (satisfying some conditions) for ...
1
vote
1answer
46 views

Given an invariant distribution is the (finite state) Markov transition matrix unique?

Doeblin's theorem states that for a given transition probability matrix there exists a unique invariant distribution for that chain. Is the converse true as well? Can two (finite state, discrete) ...
1
vote
1answer
60 views

Conditioning a CTMC on the future on a Yule pure birth process

I need to solve a problem where I am asked to calculate $M=P(X(0)=2|X(1)=3,X(2)=4,X(3)=5)$ in a Yule pure birth process where $\lambda=1$, so $\lambda_n=\lambda.n=n$ and $\mu_i=0$ (the death rate is ...
1
vote
1answer
108 views

Markov Chain, closed, recurrent states

There are two urns, with the first one containing three white balls and the second one containing three black balls. At each step, we draw a ball from each urn, and then put the ball drawn from the ...
1
vote
1answer
95 views

Exponentially-distributed lifetimes (death process)

In a pure death process where the individual death rate is fixed at v, because the process is a time-homogeneous Markov process, the wating time till the next "event" (i.e. the wating time till the ...
1
vote
1answer
82 views

How is a simple birth process is time-homogeneous?

Why is it that a simple birth process is time-homogeneous? The incidence of a birth in a small time interval depends on the population size at the time of start of the interval. Doesn't this ...
1
vote
1answer
200 views

Is a Bernoulli process a Markov chain?

For a Bernoulli process, the outcome of a future trial is independent of the outcome of past trials. I.e., the future behaviour of a Bernoulli process is independent of its past, i.e. a Bernoulli ...
1
vote
2answers
239 views

Estimating the transition matrix given the stationary distribution

Let's say we are given a Markov chain for variable $X = [x_1, ..., x_n]$; also we are given a desired stationary distribution for this graph $P_\infty = [p_1, ..., p_n]^\top$. How can we design an ...
1
vote
1answer
105 views

Markov Chain transitional probability query.

Say I have the transitional probability matrix P= $\begin{bmatrix}.8 & .2\\.6 & .4\end{bmatrix}$ And the entry (1,1) denotes the probability that I stay in state 0, (1,2) I move from state 0 ...
1
vote
1answer
173 views

Finding Markov chain transition matrix using mathematical induction

Let the transition matrix of a two-state Markov chain be $$P = \begin{bmatrix}p& 1-p\\ 1-p& p\end{bmatrix}$$ Questions: a. Use mathematical induction to find $P^n$. b. When n goes to ...
1
vote
1answer
55 views

A simple case of random walk

$\forall n \in \mathbb{N}$ we can either move from state $S_n$ to state $S_{n+1}$ with probability $p$ or to state $S_{n-1}$ with probability $q=1-p$. Also we move from state $S_0$ to state $S_{1}$ ...
1
vote
2answers
138 views

How can I calculate the expected number of changes of state of a discrete-time Markov chain?

Assume we have a 2 state Markov chain with the transition matrix: $$ \left[ \begin{array} (p & 1-p\\ 1-q & q \end{array} \right] $$ and we assume that the first state is the starting state. ...
1
vote
1answer
2k views

Steady-state and Equation System

Two questions: Given the transition matrix: $ \begin{vmatrix} \ 0.4 & 0.4 & 0.2 \\ \ 0.5 & 0.3 & 0.2 \\ \ 0.1 & 0.5 & 0.4 \end{vmatrix} $ I would like to know HOW to find ...
1
vote
1answer
38 views

Finding the limiting distribution of a $3 \times 3$ Markov chain

This is a question from a book. Find $\lim_{n\rightarrow \theta}P^n$ where $$P=\begin{pmatrix}0 & 1 & 0\\ \frac{1}{6} & \frac{1}{2} & \frac{1}{3}\\ 0 & \frac{2}{3} & ...
1
vote
1answer
303 views

A few questions about Markov chains

Let $\{X_n\}$, $n \geq 0$ be a Markov chain with the transition matrix $P$ such that $$ \begin{array}{c|ccc} &A &B &C \\ \hline A &0.2 & 0.2 &0.6\\ B &0 & 0.25 ...