Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

2
votes
0answers
57 views

What does a customer see when it begins to be served in $M/M/1$ queue?

In queueing theory, the PASTA (Poisson Arrivals See Time Averages) principle [wiki] justifies $a_n = P_n$ where $$a_n = \text{proportion of customers that find } n \text{ in the system when they ...
2
votes
0answers
36 views

Powers of (large) lower triangular matrix

Consider the following "game" of chance. Each time the player pushes a button he is awarded a random (finite, integer, non-negative) number of points. The probability of receiving any particular score ...
2
votes
0answers
48 views

Markov Chain with Normal Transition Matrix

Consider a (sub)-stochastic matrix $P$, and the associated Markov chain $X$ with \begin{align*} \mathbf P [X_n =y | X_0 = x] = P_{xy}^n. \end{align*} Suppose we have the condition $P^T P = P P^T$, ...
2
votes
0answers
12 views

$\psi$-irreducibility of m-skeletons.

In Proposition 5.4.5 of Meyn and Tweedie's Markov Chains and Stochastic Stability, it is said that if a chain $\Phi$ is $\psi$-irreducible and aperiodic, then every $m$-skeleton of it is also ...
2
votes
0answers
32 views

markov spectral radius independent of states?

Let $\Pi$ be a stochastic matrix of an irreducible markov chain. We define the spectral radius of $\Pi$ as: $\rho(\Pi) := \limsup_{n \to \infty} \left( \pi^{(n)}_{(a,b)} \right)^{\frac{1}{n}}$ Why ...
2
votes
1answer
35 views

On track Prerequisite for Statistics and Probability

I do not really have a solid mathematical background because of the range of courses i had back in high school/university that wasn't really scientific oriented. Presently i am doing an MSc in ...
2
votes
1answer
68 views

Markov Chains- Show state is recurrent

Q's: I suspect this is true: if two states in a markov chain communicate and one is recurrent, then the other is recurrent. My approach is, lets say i and j are two states that communicate and as i ...
2
votes
1answer
51 views

I cannot see what happens to $P_{ij}(s)=p_{ij}^0+\sum^\infty_{n=1}\sum^{n-1}_{k=0}s^{n-k}f_{ij}^{n-k}s^kp_{jj}^k$ to get result

This is self learning and it is stats. $P_{ij}(s)=\sum^\infty_{n=0}p_{ij}^ns^n$ (which you'll probably recognise is a generating function) and $F_{ij}(s)=\sum^\infty_{n=1}f_{ij}^ns^n$ (note n=1 ...
2
votes
1answer
112 views

Dice probability of a winning more than $X\%$ of the time over $Y$ Throws

I have a die with three possible outcomes. The three outcomes are win (+1), draw (0) and lose (-1). $P(w) + P(d) + P(l) = 1$. (1) If I throw the die Y times, what is the probability I will win $X$ ...
2
votes
1answer
82 views

Markov chain and hitting times

I have a Problem about hitting times. That's the following: Let $A\subset E$ and the first passage time $T_A$ and the hitting time $H_A$. Define: $T_A =\inf\{n\geq 0;X_n \in A\}$ and $H_A ...
2
votes
1answer
60 views

A doubt on $d$-dimensional random walk

Consider a $d$-dimensional random walk with equal probabilities in each of the $d$-directions (so, $p(v_i,v_j)=\frac{1}{d(v_i)}=\frac{1}{2d}$ here. Now, suppose the walker takes $2n$ steps. Now I have ...
2
votes
0answers
70 views

simple proof of the $L^2$ weak law for discrete-time ergodic Markov processes

Let $\{X_t\}_{t\in\mathbb{Z}}$ be a stationary and ergodic stochastic process with finite second moment. Von Neuman's ergodic theorem implies that the time average $(1/N)\sum_{j=0}^{N-1} X_j$ ...
2
votes
2answers
103 views

How to understand Markov property?

I'm learning stochastic process in college. How to understand Markov property?I'm curious about what is the power and validity of Markov property ? A stochastic process has the Markov property if ...
2
votes
2answers
350 views

Expected state of a Markov chain

Let's start with a slightly trivial Markov chain defined as follows: the beginning state is called $1$ and the set of states is $\mathbb{N}$. At each step, when the current state is $n$, the ...
2
votes
0answers
60 views

Intuition behind criterion for an irreducible Markov chain to be transient

I have been looking over my notes for Markov chains, and I have come across the following: Theorem: An irreducible Markov chain is transient iff for some state $i$ there exists a nonzero vector $y$ ...
2
votes
1answer
200 views

Question on M/M/2 queue variation

I have the following question: Two workers handle three machines(i.e. we can at most repair two machines at a time). The time until the machine breaks down is exponential distributed with expectation ...
2
votes
1answer
72 views

Random Process derived from Markov process

I have a query on a Random process derived from Markov process. I have stuck in this problem for more than 2 weeks. Let $r(t)$ be a finite-state Markov jump process described by ...
2
votes
0answers
96 views

Markov chain from Poisson

Let $K_t$ be a Poisson process with rate $1$ and $X_n=K_n-n$ $, \ \ \ n\in \mathbb{N}$ am asked to determine whether it is null or positive recurrent, we already know it is recurrent. I ...
2
votes
0answers
160 views

Boundedness of expected reward Markov chain (may be related to discret $M/M/\infty$ queue)

[EDIT]: I read a bit on $M/M/\infty$ queue and it may not be the right comparison and my notation may be confusing (I'm in discrete time and $\lambda,\mu$ look likes rates when they are probability). ...
2
votes
1answer
409 views

Finding the steady state Markov chain?

I have drawn a certain Markov chain with a weird transition matrix. Here's the drawing: And here's the transition matrix: My problem is that I don't quite know how to calculate the steady state ...
2
votes
2answers
148 views

Proof of (Strong) Markov Property using sigma-algebras

I would like to ask if any of you know of a good resource containing rigorous proof (using sigma-algebras) of Markov Property and Strong Markov Property respectively in terms of Discrete Time Markov ...
2
votes
0answers
25 views

Ruin time with a maximum purse size

Imagine I have a gambler's ruin scenario where I start with $m$ dollars and I cannot have more than $N$ dollars. For each of however many rounds, I flip a coin, and with probability $p$ I win a ...
2
votes
0answers
26 views

Broken disk head

Broken disk head: We would like to read 1 byte = sequence of 8 bits from a disk, starting from bit 0. Our disk head reads 1 bit at a time. Disk head can only move forward, but after reaching bit 7 ...
2
votes
0answers
72 views

Is the following interpretation for the stationary distribution of a Markov process correct?

Imagine I have some Markov process with stationary distribution $\pi$ and a mixing time of $\tau$ after which $|Prob[x=s_i] - \pi(s_i)| \leq \epsilon$. Can I assume the following: A state $(x=s_i)$ ...
2
votes
0answers
19 views

Problem with the uniform transience

Let $X$ be a Borel space and let us consider a Markov Chain $(\Phi_n)_{n\geq 0}$ on this space given by the stochastic kernel $$ P(x,\mathrm dy) = p(x,y)\mu(\mathrm dy) $$ where the density $p$ is ...
2
votes
0answers
203 views

In finite-state Markov chain state $i$ is transient

Can you help me please with proof of this question: Prove, that in finite-state Markov chain state $i$ is transient if and only if is exist state $k$ such that $i\rightarrow k$ but k $\nrightarrow ...
2
votes
1answer
728 views

Acceptance probability of Metropolis-Hastings

I am an IT guy writing my masters thesis on MCMC methods for use in predicting the outcome of football(soccer) matches. Right now I am trying to wrap my head around MCMC and Metropolis-Hastings in ...
2
votes
0answers
63 views

What is the difference between a Markov process and a Markov chain? [duplicate]

Possible Duplicate: What is the difference between all types of Markov Chains? I've read a lot about Markov processes and chains but so far I don't understand what the difference is between ...
2
votes
1answer
58 views

Functions of a Markov Chains

can does anybody know if the following expectations are available in closed for... Let $\{ X_t : t = 1, 2, 3 \dots \}$ be a random variable defined on a Markov chain with m -step transition matrix ...
2
votes
1answer
185 views

Solving Discrete Markov Chain with diagonal band matrix.

I am trying to model a certain process as a Discrete Markov Chain. My system has $N+1$ states: $X=0, \ldots N$, and I can assume that the $(N+1)\times (N+1)$ transition matrix $T$ has the general form ...
2
votes
0answers
183 views

Computing the stationary distribution of a markov chain

I have a markov chain with transition matrix below, $$\begin{bmatrix} 1-q & q & & & \\ 1-q & 0 & q & & \\ & 1-q & ...
2
votes
0answers
171 views

Ergodicity and mixing

From MathOverflow, R W said: Unfortunately, the way the term "ergodic" is used in the theory of (finite) Markov chains is completely misleading from the point of view of general ergodic ...
2
votes
1answer
195 views

Confused about Markov property

The sample space is $\Omega$ with $\omega = (\omega_0, \omega_1, \ldots) \in \Omega$ an infinite sequence of a set $S$. So the measure space is $(S^{\mathbb{N}}, \mathcal{S}^{\mathbb{N}})$ where ...
2
votes
2answers
246 views

Looking for an example of a Markov Chain

I am looking for an example of a Markov Chain characterized by, say, 3 by 3 matrix that has more than one eigenvector (say a population distribution of birds, or something). I remember solving a ...
2
votes
0answers
566 views

Classifying a Stochastic Process as transient, null-recurrent or positive-recurrent

For a discrete time Markov chain with state-space the non-negative integers, for $j>0$, $$ p_{j,k} = \begin{cases} p/j & \text{for } k = j+1 \\ 1 - 1/j & \text{for }k=j \\ (1-p)/j & ...
2
votes
2answers
267 views

probability terminology for parameter in a Markov process

Suppose $$P(\text{feature present at time} \ t \ \text{and} \ t+\Delta t) = \beta^{2}+\beta(1-\beta) \exp(\Delta t/\tau)$$ where $\tau = 1/(\pi_{01}+\pi_{10})$. What is $\tau$?
1
vote
2answers
569 views

Perron-Frobenius theorem

In the proof of the Perron-Frobenius theorem why can we take a strictly positive eigenvector corresponding to the eigenvalue $1$? Before that, why can we even take a non-negative eigenvector? Books ...
1
vote
4answers
172 views

Why do these expressions tend to zero?

In a chapter on Markov chains, it is claimed that $\binom {2n}{n} p^n(1-p)^n$ (where $p$ is a probability) tends to $0$ as $n$ tends to $\infty$. But why is this so? It is also claimed that for an ...
1
vote
3answers
206 views

The limit of matrices

Consider a square matrix $P$. We call it stochastic if it holds that $$ p_{ij}\geq0\text{ and } \sum\limits_{j=1}^m\,\,\,\,p_{ij} = 1 $$ for all $1\leq i,j\leq m$. I wonder when the following limit ...
1
vote
2answers
222 views

Average run lengths for large numbers of trials: Intuition and proof

This article states that the formula for the average run lengths for large numbers of trials is:$$\frac{1}{1-Pr(event\ in\ one\ trial)}.$$ My questions What is the intuition behind this formula? Do ...
1
vote
2answers
23 views

Can the ergodic theorem for Markov chains be proved with linear algebra?

This theorem is in my book, let me just say that it is for discrete-time Markov chains, that are time-homogeneous. Ergodic is defined in the book as being positive recurrent and aperiodic. The ...
1
vote
2answers
133 views

markov chain with arbitrary period

Given any positive integer, how can I think of a Markov Chain (states and transition probabilities) to have that integer as the period of two of its states? Thanks.
1
vote
1answer
68 views

A question regarding Markov Chains

Is it possible that we combine some states of a Markov chain, like in this figure? (All non-zero states combined) 1) If yes, what are the new transition probabilities, i.e. p1 and p2 and p3 in the ...
1
vote
2answers
47 views

Markov chains question?

A country is divided into three geographic regions. It is found that each year 5% of the residents move from region I to region II and 5% move from region I to region III. In region II, 15% move to ...
1
vote
1answer
62 views

Are there open questions in Markov chains?

I would be curious to know if there's still open question about discrete markovian chains
1
vote
1answer
356 views

example of irreductible transient markov chain

Can anyone give me a simple example of an irreductible (all elements communicate) and transient markov chain? I can't think of any such chain, yet it exists (but has to have an infinite number of ...
1
vote
2answers
99 views

Specific question to a Markov chain proof in Durrett

I apologize if this is to specific but i've already talked to two of my professors without much success and I really need to understand this subject. The following theorem is stated in Durrett page ...
1
vote
2answers
1k views

What values makes this Markov chain aperiodic?

Let the following transition matrix represent a $4$ state Markov chain $$\begin{pmatrix} 0 & a & 0 & b \\ \frac{1}{2} & 0 & \frac{1}{3}+c & d \\ 0 & a & 0 & ...
1
vote
2answers
168 views

Is it meaningful to talk about margins-of-error in a Markov Chain?

We're creating a Markov Chain based on an analysis of user's history. We add up and normalise users behaviour, and then normalise to create a two dimensional map of probabilities, but some of these ...
1
vote
4answers
285 views

Figuring out probabilities with Hidden Markov Models

I'm really new to Math so sorry in advance if this question does not make sense. Also I cross posted this on stats.stackexchange.com also. Background: I'm trying to learn about hidden Markov models ...