Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

2
votes
1answer
29 views

Why can a Markov chain having two states and no self-loop have a stationary distribution?

Why does a Markov chain having two states and no self-loop can have a stationary distribution? Lets consider a markov chain with two nodes = $\{A, B\}$ and the transition matrix: $P = ...
2
votes
1answer
70 views

A Markov Chain Flea Problem

A Flea moves around the vertices of a triangle in the following manner: Whenever it is at vertex i it moves to its clockwise neighbor vertex with probability $p_i$ and to the counterclockwise neighbor ...
2
votes
1answer
76 views

Counterexample for $\| (\Phi^\top \Xi \Phi)^\dagger \Phi^\top \Xi P \Phi \|_2 \leq 1$

All matrices are real. Let $P$ be any stochastic matrix (i.e. square, non-negative, rows summing to one). Let $\Xi$ be a diagonal matrix containing on its diagonal any left eigenvector of $P$ ...
2
votes
1answer
47 views

Find conditions on the distribution on $X$, but what is meant by $X$?

Let $X, X_{n,k}$ for $k,n\in\mathbb{N}$ denote independent random variables with values in $\mathbb{N}_0$. Define $N_0:=1$ and for $n\in\mathbb{N}$ set $$ N_n:=\begin{cases}0, & \text{ ...
2
votes
1answer
68 views

Why left multiplication when it comes to Markov chains?

When working with Markov chains and transition matrices $P$ we multiply from the left, meaning that for example $\mu^{(n)} = \mu^{(0)}P^n$ or that the stationary distribution satisfies $\pi = \pi P$. ...
2
votes
1answer
43 views

probability that a game finishes at $n$th step

A coin is flipped sequentially. The game finishes when the sequence TTH is formed(player X wins) or the sequence HTT is formed(player Y wins). I can find the expected time until absorption by X or Y ...
2
votes
1answer
90 views

Safe small wins vs. risky large wins at roulette

Short statement of problem : Two players play roulette at a casino. They both start with the same initial amount. Each player always plays his favorite bet each time, and stops playing as soon as he ...
2
votes
1answer
86 views

What does a customer see when it begins to be served in $M/M/1$ queue?

In queueing theory, the PASTA (Poisson Arrivals See Time Averages) principle [wiki] justifies $a_n = P_n$ where $$a_n = \text{proportion of customers that find } n \text{ customers in the system when ...
2
votes
1answer
37 views

How to get transition matrix of markov process?

I am monitoring a Markov process with ~21 states. I know all the states, initial state and what states transitions can/cannot be, so that zero elements of transition matrix are known. I know the ...
2
votes
1answer
70 views

continuous time Markov chain, something the book does not explain

I have a problem with something in my book, under the chapter of continuous time Markov chains. I will post a link to what the book does. They do something which they seem to take for granted, but I ...
2
votes
2answers
435 views

Mean recurrence time and stationary distribution of a Markov chain?

In a Markov chain is there a theorem relating the existence of the stationary distribution and the mean recurrence time? E.g. impossible for stationary distribution to exist therefore mean recurrence ...
2
votes
1answer
52 views

Coarser cyclic decomposition of Markov chain

For a irreducible Markov chain with period $d$ there is a standard construction which shows that the state space can be partitioned into $d$ sets $C_1, \ldots, C_d$ such that $P(x,y)>0$ only if $x ...
2
votes
1answer
83 views

Can I invert a single row of a very large sparse matrix?

Problem I research electron behavior in organic solar cells and have found a way to recast this problem in terms of a large (n=~60 million) Absorbing Markov Chain that I represent as a sparse matrix. ...
2
votes
2answers
94 views

Expected value of a series of random variables in a markov chain

I have a Markov Chain such that $X_n = \max(X_{n-1}+\xi _n,0)$ where the $\xi_n$ series is independent and identically distributed. I want to show that if $\mathbb E(\xi_n) > 0$ (where $\mathbb ...
2
votes
1answer
117 views

What is the difference between positive presistent and null persistent state in a Markov Chain?

I'm not looking for the difference in the mathematical definition, but rather for an intuitive explanation of their differences and possible examples, so that I can have them in my head when ...
2
votes
1answer
56 views

Elementary probability question (Random walks)

Given a random walk $X_{t \ge 0}$ on $\mathbb{Z}$ starting at $0$ with probabilities $P(n, n + 1) = p$ and $P(n, n - 1) = 1 - p$, let $Y = \min\{X_0, X_1 \dots \}$. What is the probability that $Y = ...
2
votes
1answer
137 views

If $P$ is an invertible transition probability matrix, does $P^{-1}[i,j]$ have any interesting meaning?

Suppose we have a Markov chain transition probability matrix $P$ that is invertible, i.e., $P^{-1}$ exists. Question: Does there exist a meaningful interpretation of the $(i,j)$ entry in $P^{-1}$? ...
2
votes
1answer
209 views

Markov Chain Ergodic Theorem

Consider a discrete time Markov Chain on countable state space $X_{0},X_{1},\ldots$. Assume that the chain satisfies the Foster Lyapunov criteria, and since it is countable state space chain, we ...
2
votes
1answer
104 views

Confidence intervals on maximum likelihoods of observed data

I observed 400 episodes of nursing care in a hospital. I tracked the movement of the nurses between 5 rooms $A-E$. The maximum likelihood of them moving from room $i\rightarrow j$ is given by: ...
2
votes
1answer
267 views

Two different ways of constructing a continuous time Markov chain from discrete time one

Consider a homogeneous continuous time Markov chain (CTMC) with Markov transition function $p(t)$ and infintesimal generator $G$. Its embeded discrete time Markov chain (DTMC) has its transition ...
2
votes
2answers
235 views

$(X_n)$ an irreducible transient Markov chain. Is $f(x) = \mathbb{P}(X_n = x_0 \text{ for some } n > 0 | X_0=x)$ constant?

Let $(X_n)_{n=0}^{\infty}$ be an irreducible transient Markov chain with countably infinite state space $E$. Let $T_x = \inf\{n > 0 : X_n = x\}$. Let $\mathbb{P}_x$ be probability conditioned on ...
2
votes
1answer
1k views

Finding the steady state Markov chain?

I have drawn a certain Markov chain with a weird transition matrix. Here's the drawing: And here's the transition matrix: My problem is that I don't quite know how to calculate the steady state ...
2
votes
3answers
489 views

DTMC : Example of Irreducible Aperiodic Null Recurrent Chain

Can someone give me an example of a Discrete Time Markov Chain (DTMC) which is Irreducible Aperiodic Null Recurrent I know that a Simple Symmetric Random Walk on Integers is Irreducible ...
2
votes
1answer
2k views

Irreducible, finite Markov chains are positive recurrent

I am under the impression that an irreducible, finite Markov chain is necessarily positive recurrent. How might I show this? Regards, Jon
2
votes
2answers
63 views

Convergence of an Ergodic process

I'm having trouble working through the math of the problem below. I believe the problem as described is an ergodic process. I've written a simple simulation of the problem, that converges to 66.6...6% ...
2
votes
1answer
329 views

Markov Chains - How to calculate prob. a state is visited at least N times? what about Expectation?

In Markov chains, if I was given a transition probability matrix with each of the probabilities specified, then how do I determine the following: 1- Probability that state y is visited at least n ...
2
votes
1answer
73 views

probability equation multiplication understanding

I want to understand this probability equation. I'll be grateful, if someone can help. $$P(\text{Birth})P(\text{Death}) + (1-P(\text{Birth}))(1-P(\text{Death}))$$ The above is the transition ...
2
votes
1answer
254 views

How to tackle markov chains with transition cost?

A simple example would be: I have 3 states, A,B,C The transition matrix for the chain is: ...
2
votes
1answer
229 views

Random walk on finite graph

I know that the stationary distribution of a random walk on the graph is given by, (degree of the node)/($2\times$ total number of links in graph). My question is, how do we get this solution?
2
votes
1answer
369 views

Non-symmetric simple random walk stopping time

Say there is a random walk $\{S_n\}$ with $S_0=0$ and $0<p=P(S_1=1)<\frac{1}{2}$. We know such a random walk would go to $-\infty$ eventually. Define the stopping time $T=\inf\{n: S_n=-\infty\}$, ...
2
votes
2answers
3k views

Calculating conditional probability for markov chain

I have a Markov chain with state space $E = \{1,2,3,4,5\}$ and transition matrix below: $$ \begin{bmatrix} 1/2 & 0 & 1/2 & 0 & 0 \\\ 1/3 & 2/3 & 0 & 0 & 0 \\\ 0 & ...
2
votes
1answer
448 views

Eigenvalues of a infinitesimal generator matrix

Consider a Markov process on a finite state space $S$, whose dynamic is determined by a certain infinitesimal generator $Q$ (that is a matrix in this case) and an initial distribution $m$. 1) Is ...
2
votes
1answer
241 views

Markov Decision Process - Utility Function

Reward R(S) in a Markov Decision Process is a mapping from a State S -> Bounded number. I want to know how a Utility Function is defined for an MDP. I think it has to be a mapping from a sequence of ...
2
votes
1answer
236 views

Comparing two Markov chains

I am interested in the question of the positive recurrence of a Markov chain that (in some sense) converges to another Markov chain known to be positive recurrent. The following is a concrete example ...
2
votes
1answer
169 views

How does the backward/forward algorithm work if there is no end?

I'm using Jason Eisner's spreadsheet to understand HMM more better. There's a box at the top that have a transition matrix. I see the Cold day and Hot day options, but don't understand why there's a ...
2
votes
1answer
966 views

Computing the similarity between two matrices / Monte Carlo analysis

I am studying the article at the following link, http://www-stat.stanford.edu/~cgates/PERSI/papers/MCMCRev.pdf Which applies Monte Carlo analysis to a decryption problem. The math is admittedly over ...
2
votes
1answer
37 views

Eigenvectors of transition matrices in PageRank algorithm

In my probability course, we were discussing applications of Markov Chains to computer science -- in particular, how the PageRank algorithm goes about finding stationary distributions, and thus, ranks ...
2
votes
0answers
27 views

Mixing time for metropolis chain on graph coloring

I'm reading the Markov Chains and Mixing Times by David Levin et al.. In section 5.4 page 71 a proof is given for a bound of mixing time for the Metropolis Chain on graph coloring. In the proof, such ...
2
votes
1answer
40 views

Rational Thief Problem, optimal stopping strategy

A thief goes out stealing every day and has a chance of $p_j$ of stealing a sum $j$ with $0\leq j \leq N$. But there's also a chance $p$ of getting caught, in which case he loses everything he got ...
2
votes
0answers
20 views

Estimating the discrete laplacian to prove recurrence of simple random walk for d=2

Given a function $f : \mathbb Z\times \mathbb Z \rightarrow \mathbb{R}$ we define the discrete laplacian of $f$, $\triangle_df$, by the following rule $\triangle_df(x,y)= \dfrac{f(x + 1, y)+f(x, y + ...
2
votes
0answers
60 views

Induced Markov chain - verify Markov property and another property

First, here is how we defined induced Markov chains: Suppose that $(X,E,P)$ is an irreducible Markov chain, where $X=(X_i)_{i\in\mathbb{N}_0}$, $E$ is the state space and $P=(p_{i,j})_{i,j\in E}$ is ...
2
votes
0answers
41 views

Invariant Probability of Discrete Time MC from Continuous Time Markov Chain

Given rates α of an irreducible continuous-time MC on finite state space and told that π is the invariant probability measure of this chain, we define a discrete time MC as having transition ...
2
votes
0answers
59 views

How to prove a matrix norm inequality?

$P$ is a stochastic matrix i.e. square, non-negative, rows sum to one. Let $\Phi$ be a real matrix of size $n \times k$ with independent columns and $k < n$. Let $\Xi$ be the diagonal matrix with a ...
2
votes
0answers
10 views

Monte Carlo Markov Chain Simulation Issues

The Markov Chain is uniformly distributed across all $50$x$50$ matrices of entries $0$ and $$1 with no neighboring $1's$. I am supposed to run a MC simulation to check the probability that the ...
2
votes
0answers
26 views

The independence of random variables

Here is my question: Consider a homogeneous ergodic Markov chain on a finite state space $X=\{1,\ldots,r\} $. Define the random variables $\tau_n \,,n\ge1$ as the consecutive times when the Markov ...
2
votes
1answer
49 views

Layman perspective of mean time spent in transient state of a Markov chain.

Let $X=\{X_n\}$ be a finite state Markov Chain with the state space $\{0,1,2,\ldots,N\}$ such that $0$ is the single absorbing state and all the rest states are transient. The following is the ...
2
votes
1answer
70 views

Is this a Markov chain? [duplicate]

Let $\{\xi_n \}_{n \geq 1}$ be i.i.d random variables taking values on $\mathbb{Z}$. Let $\xi_0 = 0$. $S_n = \sum\limits_{i=1}^{n} \xi_i,$ where $S_0=0$ $Y_n = \sum\limits_{i=0}^{n} S_i$. My ...
2
votes
0answers
19 views

Proving that an inductively defined function is a Markov chain

Let $X_0$ be a random variable with values in a countable set $I$. Let $Y_1,Y_2,\ldots$ be a sequence of independent random variables, uniformly distributed on $[0,1]$. Suppose we are given a function ...
2
votes
1answer
51 views

Green's function of a Markov Chain, and maybe of a Feller Process?

How are the Green's functions of a Markov chain related to the notion from PDE theory? For instance, if the Markov chain (i.e. discrete state space) is continuous time, then the Green's function I'm ...
2
votes
0answers
42 views

Geometric ergodicity and mixing - stationary case

I have this theorem: The Markov Chain {$X_n$} is stationary and geometrically ergodic if and only if {$X_n$} is stationary and absolutely regular with $\beta_n=O(\gamma^n)$ for some $\gamma\in ...