Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
37 views

Proving that the Markov chain is recurrent - Confusion/Help

Giving the following transition matrix [ 0.9 0.1 ] [ 0.8 .2 ] Classify the states From drawing the graph I know that both stats are recurrent. However I'm really failing to prove mathematically ...
1
vote
0answers
12 views

Is the steady state of a uniform markov chain always a vector of proportions?

Given that all edges in a markov chain are bi-directional (though not necessarily equally weighted), and each edge for a given node has equal probability, does the steady state always converge to a ...
1
vote
1answer
18 views

Definition of limiting distribution in a Markov chain — why do we condition on the initial state?

Given a Markov chain $\{X_n \mid n \in \{0, 1, \ldots\}\}$ with states $\{0, \ldots, N\}$, define the limiting distribution as $$ \pi = (\pi_0, \ldots, \pi_N) $$ where $$ \pi_j = \lim_{n \to +\infty} ...
0
votes
1answer
32 views

Expected time to absorption

I have been trying to solve the following problem for quite a while now, but not with much luck. The Question Let $P$ be the TPM(Transition Probability Matrix) of a DTMC with state space ...
0
votes
0answers
45 views

I am stuck on this Probability Question. Please help.

The Problem: Let a Markov Chain have R states. Show that if j is recurrent, then there exists $0\leq x\leq 1$ such that for $n > r$ the probability that the first return from state j occurs after ...
0
votes
1answer
31 views

Infinite$-$state absorbing Markov chains

Could someone provide a good reference/book about infinite$-$state absorbing Markov chains? Most of what I've found so far deals only with the finite$-$state case.
0
votes
0answers
37 views

Help with this Markov Chain Proof please

Problem: Consider a finite Markov Chain with N states $(1,2,...,N)$. Let $P(n) = [P_{i,j} (n)]$, be an n-step transition matrix. Suppose that $lim_{n\to\infty} P_{i,j} (n) = \pi_{j} $ for any $1 ...
1
vote
0answers
12 views

Conditional independence for a dynamic random field

Let $X = \left\{ {{X^{\left( \alpha \right)}}:\alpha \in {\mathbb{N}_0}} \right\}$ be a dynamic random field with a set of places $V$ and a phase space $\Lambda $ such that $\left| V \right| < ...
0
votes
0answers
58 views

When is $A^{+} P^{\top} A$ non-negative?

$P$ is a $n \times n$ stochastic matrix (non-negative, rows sum to one). $A \in \mathbb{R}^{n \times k}$ with $k < n$ has non-negative entries and independent columns. Denote by $A^+ \in ...
0
votes
1answer
28 views

Understanding the proof of stationary distribution of a markov chain

I am reading the proof of existence of stationary distribution in an irreducible markov chain from the book Markov Chains and Mixing Times by P. D. A. Levin, Y. Peres, E. L. Wilmer, and I have the ...
0
votes
0answers
22 views

Repair Chain (Markov Chain Sample Model)

A machine has $3$ critical parts that are subject to failure, but can function as long as two of these parts are working. When two are broken, they are replaced and the machine is back to working ...
0
votes
0answers
18 views

Show if $p^n(i,j)\rightarrow \pi(j)$ as $n\rightarrow\infty$ then $\pi(j)$ is a stationary measure..

Suppose $p(i,j)$ is a transition kernel on $S$ for a countable state markov chain $X_n$ with $$p^n(i,j)\rightarrow \pi(j)$$ as $n\rightarrow\infty$ for all $i,j\in S$. want to verify that $\pi$ is a ...
0
votes
1answer
29 views

Proof of aperiodic Markov Convergence Theorem for null recurrent case.

Status quo: We consider a irreducible, aperiodic Markov chain $(X_n)_{n\in\mathbb{N}}$ on a countable set $S$ with tranistion function $p(\cdot,\cdot)$. Now we want to examine ...
1
vote
1answer
26 views

The lower bound for the smallest eigenvalue given the condition

In a paper, i saw a statement that the smallest eigenvalue of $P$($P$ is reversible Markov chain with stationary distribution $\pi$) is greater than $2 \beta - 1$ with the condition, $P \geq \beta I$. ...
1
vote
0answers
30 views

Does Markov Chain converge in Variance Norm?

Assume the chain $\{X_n\}_{n\in\mathbb{N}}$ on the statespace $(S,\mathcal{F})$ is aperiodic, irreducible and positive recurrent. We denote with $\pi$ its (unique) stationary distribution. Is it true ...
0
votes
1answer
24 views

How would I solve for long run average profit?

I was looking at a problem, and I was wondering how I would set this up. Any help would be welcome. Thank you! A store stocks a particular item. The demand for the product each day is 1 item with ...
0
votes
1answer
89 views

How do I compute the variance of expected number of fair coin flips for HTH sequence using linear system of equations?

Assuming fair coin flips, I know how to compute the expected number of coin flips to see HTH sequence by writing out the linear system of equations from the state transition diagram below. ...
0
votes
0answers
11 views

Proof of the “Markovian property” for the LERW?

I'm trying to understand this proof by Werner of the Markovian property of the Loop-erased random walk http://arxiv.org/pdf/math/0303354v1.pdf (page 10). The first part I see but the second "again, ...
2
votes
0answers
20 views

Formal argument on independence of consecutive hitting times of a Markov chain.

I'm refering to the question: Differences of consecutive hitting times. I'm interested in the independence of consequtive hitting times of certain values of a Markov chain. And I do "understand" the ...
1
vote
1answer
48 views

Can a reducible Markov chain have an unique stationary distribution? [closed]

I know for irreducible and positive recurrent Markov Chain there exists an unique stationary distribution. For Markov Chain with several communication classes (example C1, C2) there exist stationary ...
1
vote
2answers
66 views

How to compute the variance of number of coin flips to see HTH sequence using linear system of equations.

Assuming fair coin flips, I know how to compute the expected number of coin flips to see HTH sequence by writing out the linear system of equations from the state transition diagram below. Define ...
1
vote
1answer
46 views

What is the probability there will be no failures?

"A machine has 4 components and the machine cannot operate when any one of these components fail. At the beginning of each day, the machine starts running. During any day component $i$ fails with ...
3
votes
0answers
25 views

$ X_n = 2 Y_n + Y_{n+1} $ (non)Markov Chain

Let $Y_1,Y_2,\dots$ be iid random variables with $P(Y_n=0)=1-p,\; P(Y_n=1)=p$ where $p\in(0,1)$. Define $$ X_n = 2 Y_n + Y_{n+1} $$ The question is, whether $\{X_n\}$ is a Markov chain or not. ...
0
votes
1answer
38 views

Calculating the information per symbol of a markov chain source

I have a 4-state 2nd order markov chain source with symbols 0 and 1. I have all the transition probabilities and have worked out the probabilities of each state. How do I go about finding the amount ...
1
vote
0answers
36 views

The value of a stochastic game

I understand why a stochastic game with discounted payoff has a value $v$ and optimal strategies over the set of stationary strategies. But why is $v$ also the game's value over the set of behavioral ...
0
votes
0answers
47 views

Transition probability matrix for $X_1 = \# heads$, *flip heads* $X_2 = \# tails$ * flip tails* $X_3 = \# heads$

Three fair coins are tossed, and we let $X_1$ denote the number of heads that appear. Those coins that were heads on the first trial (there were $X_1$ of them) we pick up and toss again, and now we ...
0
votes
0answers
26 views

Decide whether a class is recurrent or transient (Example)

Consider the Markov chain $(X_n)_{n\geqslant 0}$ with state space $E=\left\{1,2,3,4\right\}$ and transition matrix $$ T=\begin{pmatrix}0 & 1/3 & 1/3 & 1/3\\0 & 0 & 1 ...
0
votes
0answers
18 views

Explicit Probability for Markov Chain on Power Set

A have a Markov chain $F_t$ in discrete time on the power set of a finite totally ordered set $A$. Its probably easiest to explain the transition probabilities in a small example, since they are easy ...
1
vote
1answer
34 views

Transience in a simple Markov chain

Consider the following simple game from a textbook called "Competitive Markov Processes" by Filar & Vrieze (Springer 1996). This is a two player game with two states. In the first state (the ...
1
vote
2answers
51 views

If a Markov chains converges then the limit is a stationary distribution

Let $p$ be a transition function of a Markov Chain on a countable state $S$ and $i \in S$. Assume for every $j \in S$, $$ \lim_{n\to \infty} p^n(i,j) = \pi(j)$$ Show that $\pi$ is a stationary ...
2
votes
0answers
70 views

Why are inter arrival times in the continuous version of discrete-time Markov chains always exponentially distributed?

I am curious whether there exist continuous time Markov processes for which the times between jumping times (which I call inter arrival times) are not exponentially distributed, but have some other ...
3
votes
1answer
58 views

Stochastic matrices with orthogonal eigenvectors

I stumbled across an odd fact while thinking about Markov chains, and I have an odd proof for it. Claim: Let $P$ be the transition matrix of a finite irreducible aperiodic Markov chain, and assume ...
1
vote
1answer
97 views

Expectation and Variance of random walks

Consider random walks of fixed length (e.g. $5$) starting at node $u$ in an undirected and connected graph with $N$ vertices. If a node $k$ has $N_k$ edges, the probability of the walk reaching any of ...
1
vote
1answer
20 views

Markov chain knowing future

I was wondering whether or not P(X1 = S1 | X0 = S0) and P(X1 = S1 | X0 = S0 and X2 = S2) are the same? What I mean is can we get some information from the future states? Thanks!
0
votes
0answers
12 views

Is Markov Chain sampled at stopping times a Markov chain?

Given a Markov hain $\{X_n\}$ and $T_n$ be an increasing sequence of stopping times, is $\{X_{T_n}\}$ Markov chain ?
1
vote
0answers
26 views

Prove that $E_x[T_y]<\infty$ for irreducible positive recurrent Markov chains.

This is an exercise from Durret's "Probability: Theory and Examples". Suppose $p$ is a irreducible and positive recurrent Markov chain. Then $E_x[T_y]<\infty$ for all $x,y$. I had the following ...
2
votes
1answer
74 views

Maximal Principle: Why using the new transition matrix $\tilde{P}$?

First some notation: Let $(X,E,P)$ denote a finite, irreducible Markov chain with finite state space $E$ and transition matrix $P$. Choose and fix a subset $E^°$ of $E$, which will be called ...
5
votes
3answers
129 views

A recursive formula to approximate $e$. Prove or disprove.

Let the sequence $\{x_n, n=1,2,...\}$ be defined as follows: Let $x_2=x_1=1$ and for $n>2$ let $$x_{n+1}=x_n-\frac{1}{n}x_{n-1}.$$ This sequence, generated by the recursion above, tends to zero ...
0
votes
0answers
51 views

emails arriving in a Poisson process

Emails arrive according to a Poisson process with rate $λ=2/hour$. You check your inbox (instantly reading all new emails) at time $t=5$ hours and also at some uniformly distributed random time ...
0
votes
0answers
26 views

Mean and Variance of an offspring

If I have that the number of offspring of an individual in a population is $0$, $1$, or $2$ with respective probabilities $a>0$, $b>0$ and $c>0$, where $a+b+c=1$, how would I express the mean ...
2
votes
1answer
36 views

Finding mean and variance of a population problem

A population beings with a single individual. In each generation, each individual in the population dies with probability $1/2$ or doubles with probability $1/2$. If I let $X_n$ denote the number of ...
4
votes
1answer
88 views

Find the Stationary Distribution of an infinite state Markov chain

A Markov Chain on states 0,1,..... has transition probabilities $P_{ij}=1/(i+2)$ for j=0,1,....,i,i+1. I'm supposed to find the stationary distribution. So do I take the limit as n goes to ...
1
vote
2answers
44 views

Limiting Distribution of a Markov Chain

I'm having trouble understanding how to find a limiting distribution. If I have a Markov Chain whose transition probability matrix is: $$ \mathbf{P} = \matrix{~ & 0 & 1 & 2 & 3 & ...
1
vote
1answer
39 views

Electrical components failing in a Poisson process

A machine has infinitely many identical components. They fail according to a Poisson process with rate $\lambda = 4$/hour. A repairman arrives at time $t$ and instantly repairs all of the broken ...
2
votes
1answer
69 views

Prove that the absolute value of the difference of two invariant distributions on a Markov chain is invariant

If we have $a(x)$, $b(x)$ which are invariant distributions on a Markov chain $X_n$ with state space $S$, how can I prove that $|a(x)-b(x)|$ is also invariant? I know that I must show that: ...
0
votes
0answers
33 views

Multiclass Markov process

There are two car M/M/1 queues Q1 and Q2. Arrival rate of Red car and Green car in Q1 is $\lambda_{1R}$ and $\lambda_{1G}$ respectively. Similarly arrival rate of red car and green car in Q2 is ...
1
vote
0answers
19 views

Converse of Perron Frobenius Theorem: Necessary and Sufficient Conditions for positivity (or non negativity)

As I understand, Perron Frobenius theorem asserts only in one direction, i.e. if Matrix A is positive then there is a perron eigenvalue, eigenvector etc. What I wanted to know is what are the ...
0
votes
1answer
36 views

Measurability of a stopping time in a Markov chain

Suppose you have a finite-state continuous-time inhomogeneous Markov chain with transition rate $Q(t)$. Further, let us suppose that $Q(t)$ is a piecewise continuous function of $t$. Two questions: ...
3
votes
1answer
71 views

Markov Property Confusion

I feel like I'm being very dense/employing some sort of circular reasoning, but I'm having trouble understanding the Markov Property. According to Durrett (ISBN-10:1461436141), $X_n$ is a Markov chain ...
0
votes
1answer
23 views

Strategy for Unbalnaced Gamber Ruin

A gambler plays the following game: A fair coin is tossed until getting three times continuously head. When that happens the Gambler gets 20$\$$. Each round costs the gambler 1$ (even if he won the ...