Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
12 views

Simulating a continous time, inhomogenous Markov chain

What algorithms are used to simulate a time-continous, inhomogenous Markov chain? For the homogenous case, I've found (among others) this reference, which contains a few exact and approximative ...
0
votes
0answers
29 views

Problem involving periodic Markov Chains — probability of being in a given state at time $n$

I'm working on the following problem: I believe that the simplest possible irreducible periodic Markov Chain would be one with two states and no self-loops? Does this seem correct? However, I'm ...
0
votes
1answer
40 views

Smallest irreducible periodic Markov chain

What would be the smallest periodic Markov chain? We're studying periodic Markov chains in my probability course. I'm just trying to picture the smallest possible one but I can't seem to come up ...
0
votes
0answers
21 views

finite state space and geometric ergodicity proof

If the state space of is finite, then all irreducible and aperiodic Markov chains are geometrically ergodic. How can one show this fact?
1
vote
1answer
23 views

Discrete-time random process is Markov iif… (Proving a theorem)

First some background: We say that $(X_n)_{n\geq 0}$ is a Markov chain with initial distribution $\lambda$ and transition matrix $P$ if (i) $X_0$ has distribution $\lambda$; (ii) for $n\geq ...
0
votes
0answers
23 views

Markov chains (which book can be recommended?)

This semester I am learning about Markov chains, mainly including basic definitions & properties Recurrence & Transience Perron-Frobenius Theory equilibrium states convergence to ...
0
votes
1answer
27 views

Why is $1/4$ the probability of hitting 6, starting in 0?

We had the following Markov chain: I cannot see the following statement: Starting in 0, the probability of hitting 6 is $1/4$. I do not see because what does this mean "hitting 6"? In ...
0
votes
0answers
22 views

Why is this class recurrent?

In our reading we had the following example for a Markov chain. I cite from the reading: Here we have three communicating classes: $\left\{0\right\}, \left\{1,2,3\right\}$ and ...
1
vote
0answers
25 views

Random Walk and strong law

I want to prove that a Random Walk in 1 dimension is transient when $p\neq\frac{1}{2}$ but i want to prove it by the strong law of large numbers, so i have this: Define a random variable $$X_i = ...
1
vote
1answer
17 views

Construct the Transition matrix for the Markov chain that models this situation?

I'm given this figure and I need to find transition matrix for this. Thep problem says that the robots have been programmed to traverse the maze and at each junction randomly choose which way to go. ...
0
votes
2answers
41 views

About random walk 1D

I just don't understand why is betha expressed in this way. I don't understand the "conditioning on the initial transition" . Hope you help me thanks
1
vote
0answers
67 views

Can we find a correlation between states of a Markov chain?

I have a fair bit of knowledge on Markov chains but I recently wondered if there is a way to find out a correlation between the states of a finite Markov chain. I could not find any material on this. ...
0
votes
1answer
60 views

A way to check the accuracy of a Markov chain?

I am not sure whether I should post this question on MSE or SSE. I will post it here 1st to see if I can get some feedback. Say I have a finite discrete Markov chain constructed maybe using some data ...
1
vote
0answers
45 views

Linear Filtering Problem (Keynman Fac/Particle Model)

$lienar Filtering Problem $$X_n^1 = X_{n-1}^1 + \epsilon_n *W_n $$ $$X_n^2 = (1-\alpha* \delta) X_{n-1}^2 + \beta*\delta X_n^1 $$ $$X_n^3 = X_{n-1}^3 + \delta*X_n^2$$ above is $$\approx$$ $$dX_n^1 ...
0
votes
1answer
25 views

Ergodic behaviour for bounded random dynamical system

Considering an iterated system described by $$ X_n =\gamma_nX_{n-1} , $$ where $\gamma_i$ are non-negative i.i.d. variables. It is easy to show that the expectation will grow unbounded for $\mathbb ...
0
votes
1answer
50 views

Proofs in Stochastic Processes

Let $$X_{n}$$ be an irreducible Markov chain on the state space {1,...,N}. Show that there exists $$C < \infty$$ and $$\rho < 1$$ such that for any states i,j, $$\mathbb{P} [ X_{m}\neq j , m=0 ...
0
votes
0answers
28 views

Stationary VS. limiting probability

I'm just wondering what the difference between stationary probability and limiting probability is. And, if any of you know: What does it mean that some elements exist and some elements doesn't, when ...
1
vote
0answers
42 views

Stochastic process using Markov chain (thief on the run!!)

I'm given an exercise where we are to simulate a thief escaping from an officer. The thief (let's call him/her T for simplicity) and an officer (O) have four cities to be in. Let's call the cities A, ...
0
votes
0answers
54 views

Covariance of states of a finite Markov chain

I know it is possible to construct a covariance matrix for states of a Markov chain but I cannot seem to find a proper way to compute it. I will attach some theories I found from Kemeny and Snell's ...
0
votes
1answer
18 views

Markov chains and boundary theory

In the next semester there is a reading called "Markov chains and boundary theory". I have at least an imagination what a Markov chain is, but what is meant with boundary theory in this context? ...
0
votes
0answers
17 views

A question about Markov chain

Let $(\Omega,\mathscr{F},P)$ be a probability space. Assume $\{X_t\}_{t\geq 0}$ is a Markov chain with finite state space $S$. Assume $u:S\rightarrow\mathbb{R}$. Is it true that the limit ...
0
votes
0answers
34 views

Dealing with conditional OR statements (simple Markov Chain question)

I have a Markov chain with three states, X, Y and Z, and the following transition matrix; $$ P =\pmatrix{0.5 & 0.5 & 0 \\ 0.4 & 0.4 & 0.2 \\ 0 & 0.5 & 0.5 \\}$$ Now, what I ...
1
vote
1answer
46 views

“Almost-Absorbing” Markov Chain with Closed Communicating Classes

I am trying to model the dynamics of a game as an (Absorbing) Markov Chain. There are a bunch of probabilistic transitions between states (as usual) and three "terminal" outcomes: Winning the game ...
1
vote
1answer
83 views

Calculating the expected winner of a Penney's Game using a Markov Chain.

I am trying to calculate the probability that one sequence of coin tosses is more likely to win than the other in a game of Penney's. The sequences are: HTHT and THTT. So far I've come up with the ...
1
vote
1answer
24 views

How to interpret an event of a Markov chain?

Suppose $\{X_n\}_{n=0}^{\infty}$ is a Markov chain with state space $S = \{0,1,2,...,N\}$ with $$ P(X_1=0|X_0=0)=1 \\ P(X_1=N|X_0=N)=1 $$ then why the following result is true? $$ ...
0
votes
0answers
35 views

Strong Markov property of continous time Markov process

In the book "Applied probability and queues" which is available here http://books.google.de/books?id=BeYaTxesKy0C&pg=PA32&hl=de&source=gbs_toc_r&cad=3#v=onepage&q&f=false , ...
0
votes
0answers
46 views

Solving the matrix equation $D P = D P A D$ for stochastic matrices.

Here, I call any real matrix with positive entries with rows summing to one a stochastic matrix (it need not be square). $D,A,P$ are stochastic. $P$ of size $n \times n$ is given. $D$ of size $k ...
3
votes
0answers
45 views

Why a positive recurrent Markov chain implies positive limiting probability?

Let $X=\{X_0,X_1,\ldots\}$ be an irreducible, positive recurrent, and aperiodic Markov chain with the state space $S=\{0,1,2,\ldots\}$ then how do we show that the probability $$ ...
1
vote
2answers
147 views

Thinking about a probability question using Markov chains

The problem is part (b): 1.4.7. A pair of dice is cast until either the sum of seven or eigh appears.  (a) Show that the probability of a seven before an eight is 6/11.  (b) Next, this ...
1
vote
1answer
42 views

Literature request on Markov Chains which state transition probability matrix evolves over time

I want to know is there any literature on markov chains who's state transition probability matrix evolves over time? For instance, I have 2 states, 1 and 2. With $$P = \begin{bmatrix} p_{11} & ...
2
votes
1answer
58 views

Is every PMF on the set of non-negative integers the stationary distribution of some birth-death process?

Let $f(.)$ be a probability mass function on the non-negative integers such that $0<f(n)<1$ and $f(0)+f(1)+...=1$. Then does there exist an irreducible birth-death process with stationary ...
3
votes
1answer
51 views

Gambler's ruin: Distribution of the maximum fortune along the game conditioned to lose

I having troubles with this problem: Let $(X_n)$ a gambler's ruin Markov chain on $\{0,\dots,N\}$ i.e. a Markov chain with state set $E=\{0,\dots,N\}$ and probability transitions $$p(k,k+1)= ...
0
votes
1answer
28 views

Law of Total Probability in Markov Chains

I'm reading about Markov Chains and have come across the following: $ P_x (X_2 = y) = \sum\limits_{z\in \mathbb S} P_x (X_1 = z).P_x(X_2 = y|X_1 = z) $ where $ P_x (X_1 = z) = p(X_1 = z|X_0 = x) $ ...
2
votes
2answers
46 views

finding the generating function $\phi(s) = \mathbb{E}(s^{H_0})$.

i just started the course of markov chains and i'm having a few problems with one of the excercises. Let $Y_1,Y_2, \dots$ be i.i.d random variables with: $\mathbb{P}(Y_1 = 1) = \mathbb{P}(Y_1 = -1) ...
0
votes
0answers
24 views

Memoryless Property of Markov Chains

I'm trying to understand Markov Chains and have across the following in a book: $ \sum\limits_{y=0,1,....m−1}p(x,y)P(T_A<T_B|X_0=x,X_1=y) $ which then becomes the following, under the Markov ...
0
votes
1answer
25 views

Recurrence of states in a function of a Markov chain

Suppose $X$ is a Markov chain (or process, for that matter) and suppose further $f(X)$ is also a Markov chain. Let $s$ be a recurrent state in $X$. Is there a general way to determine the recurrence ...
0
votes
1answer
50 views

Probability transition matrix for maximum of iid random variables

I have a homework problem that goes as follows: Let $\xi_i, \ i=0,1,2,\ldots$ be i.i.d. random variables of discrete type. The distribution of $\xi_0$ is given by: $$\mathbb{P}\{\xi_0=i\} = a_i, \ ...
0
votes
0answers
28 views

“Taking expectation” to yield conditional probability

This argument is taken from Resnicks Adventures in stochastic processes and let $T _{\infty } < \infty $ denote that an infinite number of transitions in a continuous time markov chain has occurd ...
0
votes
1answer
71 views

How do I say that an infinite-state Markov chain is positive recurrent? [closed]

I run into this Markov chain while I'm doing my research, and I can't figure out how to find the condition under which this Markov chain is positive recurrent. This is a brief scenario of my ...
4
votes
1answer
71 views

Are primitive row stochastic matrices diagonalizable?

Let $A$ be an $n \times n$ matrix with real, non-negative entries. Assume $A$ is primitive, meaning there exists an integer $k$ such that $A^k>0$ (here the inequality means all entries in $A$ are ...
0
votes
1answer
35 views

Show that this Markov chain is recurrent

So I have a Markov chain on the nonnegative integers such that, starting from $x$, the chain goes to $x+1$ with probability $p$, $0<p<1$, and goes to state $0$ with probability $1-p$. I'm ...
0
votes
0answers
17 views

Equivalent Formulation of Markov Property for Homogeneous Chains

In Shiryaev's Probability (just above the strong Markov property, p.568), the author says that an equivalent formulation of the usual Markov property for homogeneous chains is $$P[\theta_nX\in B\mid ...
0
votes
1answer
27 views

$n$-step transition probability of a Markov chain

Let $(X_t)_{t\in\mathbb{N}_0}$ be a time-homogenous Markov chain over a finite state space $\left\{1,\dots,m\right\}$, so that we've got $$\Pr(X_{t+1}=j\mid X_t=i_t,\dots,X_0=i_0)=\Pr(X_{t+1}=j\mid ...
0
votes
0answers
21 views

Irreducible Markov chain being recurrent

I've come across the following theorem in Sheldon Ross's book whose converse part I am unable to prove. Theorem: An irrreducible Markov chain with state space 0,1,2,... is recurrent if and only $\ ...
3
votes
2answers
84 views

Interpretation for the determinant of a stochastic matrix?

Is there a probabilistic interpretation for the determinant of a stochastic matrix (i.e. an $n \times n$ matrix whose columns sum to unity)?
2
votes
1answer
70 views

Is this a Markov chain? [duplicate]

Let $\{\xi_n \}_{n \geq 1}$ be i.i.d random variables taking values on $\mathbb{Z}$. Let $\xi_0 = 0$. $S_n = \sum\limits_{i=1}^{n} \xi_i,$ where $S_0=0$ $Y_n = \sum\limits_{i=0}^{n} S_i$. My ...
3
votes
1answer
52 views

Mean exit time / first passage time for a general symmetric Markov chain

Suppose I have a Markov chain as depicted in the following figure: where $N$ is even. State 0 and $N$ are the two sinks of the chain. The transition probabilities have the following property: ...
0
votes
1answer
57 views

How to check if a given Markov chain is positive recurrent.

I'm trying to solve a problem which is related to my research, and I have to check whether this infinite-state Markov chain is positive recurrent or not. Suppose the Markov chain I have has state 0, ...
2
votes
1answer
103 views

Probability of a substring occurring in a string

Consider a random string of length $n<\infty$ where each digit can be between 0-9 with equal probability and a substring of length $k<n$ consisting of only zeros. What is the probability of ...
1
vote
2answers
57 views

$(S_0+\ldots + S_n)_{n\geq 0}$ not a Markov chain

Assume that $Y_0,\ldots , Y_n$ are independent random variables with the following identical distribution: $Y_i=1$ with propability $p$ and $Y_i=0$ with propability $1-p$. Also set $S_0=0$ and ...