Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
108 views

Probability of explosion in a Markov chain

I have the following problem: In a chain reaction a particle of a certain kind has probability 4/7 of hitting a nucleus. If that happens then the particle disappears but 3 new particles of the same ...
0
votes
1answer
122 views

Markov chain: relation between hitting time and transition probabilities

Let $X_n$ be a Markov chain on a countable state space $E$. For $x\in E$ let $\tau_x:=\inf\{n\geq 1\vert X_n=x\}$ be the first hitting time. What can be said about the relation between the transition ...
0
votes
1answer
35 views

Recurrence of Markov chain on $\mathbb Z \times \{0,1\}$

Let $Z_n = (X_n,Y_n)$ be the Markov chain with values in $E:=\mathbb Z \times \{0,1\}$ starting in $(0,0)$ and symmetric transition probabilities for every $x\in E$, i.e. $1/3$ for going to left, ...
3
votes
2answers
72 views

What's the probability that A wins finally

Suppose A has \$2 and B has \$3. They play a game, each game gives the winner \$1 from other. A has a probability $\frac{3}{5}$ to win each game. They play this game until one of them is bankrupt. ...
0
votes
1answer
128 views

Example of continuous transient Markov chain in detailed balance?

I have been thinking of such a chain but I've found none. I thought about random walk on $\mathbb{N}$ with probability p to go to right and $q=1-p$ to go back(i.e. this is the transition probabilities ...
4
votes
2answers
419 views

Random walk on lollipop graph

Hi i am trying to prove expected Hitting time on the Lollipop graph. It is a graph on $n$ vertices with clique on $n/2$ vertices and path joined to this. Let vertex $i$ be a vertex on the clique, ...
0
votes
1answer
73 views

Bounding the smallest eigenvalue of an ergodic Markov Chain

I am trying to prove that the smallest eigenvalue of an ergodic Markov chain is greater than -1. Can we do that using proof by contradiction, i.e. assuming the smallest eigenvalue being -1, etc.? The ...
0
votes
1answer
47 views

stationary Markov chain without starting in stationary distribution?

What would be a concrete example (i.e. the transition Matrix $P$) for a discrete time stationary Markov chain, i.e. $(X(t_{1}+t),t_{2}+t),...,t_{n}+t))$ does not depend on $t$, $\forall n\geq 1, ...
0
votes
1answer
100 views

Basic question about Markov Chain stationary distribution using generating functions

This is really basic, but I've been trying hard for a while and didn't get where I made a mistake, so after some consideration decided to ask it here. So I got a basic recurrence equation for ...
1
vote
1answer
81 views

A book on finite state continuous time Markov chain

I want to read in detail about finite state continuous time Markov chain. Can anybody suggest a book which deal this topic in detail?
1
vote
1answer
60 views

Markov chain - recurrence and transience

Does a Markov chain with infinite recurrence states and infinite transience exists? I believe it doesn't exists but I'm not sure how can I prove it. Thanks guys! :D
1
vote
1answer
75 views

Markov chain property

Suppose $\{Y_{n}, n \ge 0\}$ is a Markov chain consisting of $N$ states. Suppose that $i$ and $j$ are states of this Markov chain and that $i \hookrightarrow j$, i.e state $j$ can be reached from ...
0
votes
1answer
82 views

Asymmetric doubly stochastic matrix.

Can a doubly stochastic matrix be asymmetric ? It's a fairly simple question but I cannot find the answer to it anywhere.
2
votes
1answer
72 views

Random Process derived from Markov process

I have a query on a Random process derived from Markov process. I have stuck in this problem for more than 2 weeks. Let $r(t)$ be a finite-state Markov jump process described by ...
1
vote
1answer
742 views

Example of a Markov chain transition matrix that is not diagonalizable?

It is well-known that every detailed-balance Markov chain has a diagonalizable transition matrix. I am looking for an example of a Markov chain whose transition matrix is not diagonalizable. That is: ...
2
votes
0answers
97 views

Markov chain from Poisson

Let $K_t$ be a Poisson process with rate $1$ and $X_n=K_n-n$ $, \ \ \ n\in \mathbb{N}$ am asked to determine whether it is null or positive recurrent, we already know it is recurrent. I ...
1
vote
0answers
61 views

Recurrence criterion for a specific Markov chain

Let $(X_n)$ be a Markov chain on $\mathbb N_0$ defined by $(\alpha \geq 0)$ $$ p(0,1) = 1 \\ p(x,x+1) = 1-\frac{1}{(1+x)^\alpha} \\ p(x,0)= \frac{1}{(1+x)^\alpha}$$ Define the shifted moments for ...
4
votes
1answer
468 views

Return time of a markov chain

I'm having trouble deriving the return time for a Markov chain. The graph has $n$ vertices and is connected by $n - 1$ edges. So we can draw this as a horizontal line of nodes with node $1$ all the ...
2
votes
2answers
194 views

$(X_n)$ an irreducible transient Markov chain. Is $f(x) = \mathbb{P}(X_n = x_0 \text{ for some } n > 0 | X_0=x)$ constant?

Let $(X_n)_{n=0}^{\infty}$ be an irreducible transient Markov chain with countably infinite state space $E$. Let $T_x = \inf\{n > 0 : X_n = x\}$. Let $\mathbb{P}_x$ be probability conditioned on ...
-1
votes
1answer
144 views

Markov Chain Stationary distribution

I constructed a 4*4 state transition matrix from a discrete-time Markov Chain Model as follows: A=[p0 p0 p0 p0; p*p0+(1-p) p0 p0 p0; p0 p*p0+(1-p) p0 p0; p0 p0 p*p0+(1-p) ...
2
votes
0answers
160 views

Boundedness of expected reward Markov chain (may be related to discret $M/M/\infty$ queue)

[EDIT]: I read a bit on $M/M/\infty$ queue and it may not be the right comparison and my notation may be confusing (I'm in discrete time and $\lambda,\mu$ look likes rates when they are probability). ...
0
votes
1answer
129 views

Calculate the determinant of the matrix ad hence prove

By computing the determinant of $\lambda I-L$ where $L$ is the Leslie matrix, derive the Euler Lotka equation. $$L= \begin{bmatrix} b_{1} & b_{2} & \ldots & b_{w-1} & b_{w}\\ s_{1} ...
0
votes
1answer
244 views

How to show that a stochastic process is Markov

How can I prove that a given stochastic process is a Markov chain. Assume the following process: Joey is walking in the woods. at every turn: if at the previous turn Joey turned left then he will ...
4
votes
1answer
129 views

Chance of being able to quit while ahead in a betting game (Markov chain with gambler's ruin)

Suppose a player starts with $N$ chips, and is playing a game with odds $O$, betting 1 chip in each iteration. When the player reaches 0 chips the betting must end. What is the probability that at ...
1
vote
2answers
94 views

Simple Symmetric Random Walk : $P_{00}^{2n}=\binom{2n}{n}\left(\dfrac{1}{2}\right)^{2n}$

I was studying Simple Symmetric Random Walks and my notes state (without proof) that $$P_{00}^{2n}=\binom{2n}{n}\left(\dfrac{1}{2}\right)^{2n}$$ That is the probability of going from $0$ to $0$ in ...
2
votes
1answer
523 views

Finding the steady state Markov chain?

I have drawn a certain Markov chain with a weird transition matrix. Here's the drawing: And here's the transition matrix: My problem is that I don't quite know how to calculate the steady state ...
2
votes
3answers
413 views

DTMC : Example of Irreducible Aperiodic Null Recurrent Chain

Can someone give me an example of a Discrete Time Markov Chain (DTMC) which is Irreducible Aperiodic Null Recurrent I know that a Simple Symmetric Random Walk on Integers is Irreducible ...
1
vote
3answers
119 views

How to prove the existence of the limit of Markov transition matrix?

Does the limit of a Markov transition matrix $M$: $$\lim_{n\to\infty}M^n$$ always exist? And if yes, how to prove it?
6
votes
1answer
151 views

Markov chain stochastic process

Can anyone help me with this question, maybe by giving a hint. Consider a Markov chain with state space $\{0,1,2....\}$. A sequence of positive numbers $p_1,p_2,...$ is given with $\sum p_i=1$. ...
1
vote
2answers
485 views

How to create a transition matrix that will guarantee an outcome after infinite transitions

Let's assume we have the a transition matrix like: 0 0 0 1 2 0 2 4 0 3 6 0 4 7 2 5 9 3 6 6 6 7 7 7 8 8 8 9 9 9 First ...
1
vote
0answers
85 views

Finding probability from a markov chain

If I have a markov chain transition matrix for 2 states. Specifically in my case, it is a transition matrix for a bacterial genome with 4 random variables being A,C,G and T. (The bases) If I want to ...
0
votes
1answer
95 views

The regularity of Markov chains with a threshold

I am studying Paz's "Introduction to Probabilistic Automata", and there is an exercise I cannot solve: Ex. 11, p. 170: Prove that the number of nonregular events of the form $\{x \mid p^A(x) > ...
1
vote
2answers
105 views

Period of Markov Chain when no chance of return

Everywhere I look I see that the period of a state $i$ in a Markov chain is given by $$ \gcd\{n>0 : P_{ii}^n>0 \} $$ but what do we mean if the set $\{n>0 : P_{ii}^n>0 \} = \emptyset$? For ...
2
votes
2answers
152 views

Proof of (Strong) Markov Property using sigma-algebras

I would like to ask if any of you know of a good resource containing rigorous proof (using sigma-algebras) of Markov Property and Strong Markov Property respectively in terms of Discrete Time Markov ...
0
votes
3answers
1k views

Finding the transition probability matrix, two switches either on or off..

Each of two switches is either on or off during a day. On day n, each switch will independently be on with probability [1+number of on switches during day n-1]/4 For instance, if both switches are on ...
1
vote
1answer
413 views

Markov chain with finite positive recurrent states

If I have a Markov chain with finite positive recurrent states $\in S$, then that means starting from a given state $y$, the expected number of steps to return to state $y$ is finite. Now, if I start ...
-1
votes
1answer
135 views

Stationary distribution for different types of graph

This is a follow-up questions to posts: Stationary distribution for directed graph Stationary distribution for different types of graph The definition of stationary distribution in ...
8
votes
1answer
278 views

What happens to a random walk when we increase the probabilities of going right?

Consider a random walk on the integers where the probability of transitioning from $n$ to $n+1$ is $p_n$ (and of course, the probability of transitioning from $n$ to $n-1$ is $1-p_n$); we assume all ...
0
votes
1answer
568 views

Stationary distribution for directed graph

I want to implement the algorithm of graph partitioning of sparse directed graph. In this algorithm after computing the transition matrix ,we should compute the stationary distribution of the random ...
0
votes
0answers
167 views

Infinite Doubly stochastic matrix questions

I have the following question about a Markov chain ${(X_n)}_{n \geq 0}$ with infinite irreducible doubly stochastic matrix $P$. We have the state space $\{1,2,...\}$ . Determine the stationary ...
-1
votes
1answer
25 views

Proof involving different distributions in a discrete time Markov chain

Prove that if the initial distribution $a_0$ equals the stationary distribution $\pi$, then the transient distribution $a_n$ equals $\pi$ for all $n$.
0
votes
1answer
37 views

Near-independence of Markov chain states

Let $X(0), X(1), X(2), \ldots$ be an aperiodic irreducible Markov chain on a finite set. My intuition says that if $m$ is a very large number, then $$X(m), X(2m), X(3m), \ldots$$ should be nearly ...
0
votes
1answer
119 views

Mean Duration of Stochastic/Markov Game

An urn contains five red and three green balls. The balls are chosen at random, one by one, from the urn. If a red ball is chosen, it is removed. Any green ball that is chosen is returned to the urn. ...
0
votes
1answer
214 views

Markov Chains Probability

A Markov chain $X_0$, $X_1$, $X_2$, ... has the transition probability matrix $$ P = \left[ \matrix { 0.3&0.2&0.5 \\ 0.5&0.1&0.4 \\ 0&0&1 } \right] $$ and is known to ...
1
vote
1answer
85 views

Symmetry of hitting times in a Markov chain

Consider an irreducible, aperiodic Markov chain with stationary distribution $\pi$. We will use $E_{\pi} T_j$ to be the hitting time of node $j$ when the initial distribution is $\pi$, and $E_i T_j$ ...
1
vote
0answers
80 views

How to prove ergodic property from aperiodicity and positive recurrence

How to prove that in case of an irreducible, aperiodic and positive recurrent Markov Chain time average along sample paths is equal to the ensemble average ? i.e. $$\lim_{n\to \infty ...
1
vote
1answer
1k views

Irreducible, finite Markov chains are positive recurrent

I am under the impression that an irreducible, finite Markov chain is necessarily positive recurrent. How might I show this? Regards, Jon
1
vote
1answer
390 views

Classify the states of a markov chain

a) P =$\begin{bmatrix} {1-2p} & 2p & {0} \cr {p} & {1-2p} & {p} \cr {0} & 2p & {1-2p} \cr \end{bmatrix}$ b) P = $\begin{bmatrix} 0 & p & 0 & 1-p \cr 1-p & 0 ...
1
vote
0answers
115 views

A question about discrete and continuous-time Markov Chains

I have a test tomorrow about Stochastics Process and I couldn't solve the following questions: A gambler starts with 500\$ and plays till he runs out of money. In each round the probability to win ...
3
votes
0answers
95 views

maximum renewal rate of a Markov chain

Consider a Markov chain $(X_t)$ on a state-space with a countably generated $\sigma$-algebra and assume this Markov chain allows for small sets of order one. This means there exist sets $\mathfrak{S}$ ...