Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
62 views

Construction of pure birth process

I am considering a Markov chain $\lbrace X(t) \rbrace_{t≥0}$ in continunous time on the countable state space $S=\lbrace 0 \rbrace\cup \lbrace (i,j) \mid i \in \mathcal{A} , j \in \mathbb{N} \rbrace, ...
1
vote
1answer
196 views

Variance on the number of steps in an absorbing Markov chain

I've been looking for a proof for the variance on the number of steps before being absorbed in an absorbing Markov chain. The theorem is given on Wikipedia without citation. Following the references ...
1
vote
1answer
82 views

Markov Chains where Time Spent in State Matters

I have done a good bit of research on the subject, and cannot seem to find many materials. I was just wondering if you all knew of a good resource regarding chains which are Markov excepting the fact ...
1
vote
0answers
100 views

Explosion of a Markov chain

I am considering a Markov chain i continunous time on the countable state space $S=\lbrace 0 \rbrace\cup \lbrace (i,j) \mid i \in \lbrace A,B,C,D,E,F \rbrace , j \in \mathbb{N} \rbrace$. The ...
0
votes
0answers
26 views

Transience question for Markov Chains

Let's suppose I have a countable state discrete time MC that is known to be transient, irreducible and reversible with respect to some measure that assigns positive finite mass to each singleton, but ...
1
vote
0answers
57 views

How would I find this constant?

I have this equation, and I'm not sure how to solve for the constant $\nu$, since everything else is known: $$\begin{equation} a + \sqrt{a_i + 4 b_i \nu} + \sum^N_{j=1} (\sqrt{a_j + 4 b_j \nu}) p_{i,...
2
votes
1answer
113 views

Rational Thief Problem, optimal stopping strategy

A thief goes out stealing every day and has a chance of $p_j$ of stealing a sum $j$ with $0\leq j \leq N$. But there's also a chance $p$ of getting caught, in which case he loses everything he got ...
2
votes
2answers
176 views

Equilibrium distribution of Ehrenfest's urn

(I'll post my own answer to this, but others may be of interest, so post your own if you have one.) (PS: In reply to comments posted below: Stackexchange encourages posting an answer to one's own ...
1
vote
0answers
38 views

What is the state space of this markov chain?

Consider a system where two persons sit at a table and share three books. At any point in time both are reading a book, and one book is left on the table. When a person finishes reading his/her ...
2
votes
0answers
62 views

Estimating the discrete laplacian to prove recurrence of simple random walk for d=2

Given a function $f : \mathbb Z\times \mathbb Z \rightarrow \mathbb{R}$ we define the discrete laplacian of $f$, $\triangle_df$, by the following rule $\triangle_df(x,y)= \dfrac{f(x + 1, y)+f(x, y + ...
0
votes
2answers
542 views

What are “Filtering” and “Smoothing” with regards to Hidden Markov Models?

The Wikipedia article about Hidden Markov Models mentions "filtering" and "smoothing" tasks, see here: http://en.wikipedia.org/wiki/Hidden_Markov_model#Filtering. It gives a brief explanation but no ...
0
votes
2answers
94 views

Are these transient or recurrent states in a Markov chain?

I have the following transition matrix for a Markov chain with states $A, B, C, D, E$ $ \left| \begin{array}{ccc} 0 & 0 & \frac{1}{2} & \frac{1}{2} & 0 \\ \frac{1}{2} & 0 & \...
0
votes
1answer
46 views

How to define a transition matrix mathematically?

I'm writing my master thesis. Given the adjacency matrix of a graph, I need to define the transition matrix formally. I'm not able to figure out how to define it in mathematical notation. Can you help ...
1
vote
1answer
58 views

Can a chain with repeated nodes still be considered a Markov chain?

The well-known Markov Property is that $$P(X_n = i | X_{n-1} = k_1, \dots, X_{n-j} = k_n ) = P(X_n = i | X_{n-1} = k_1) $$ Suppose we lay out some stochastic model in the following transition ...
0
votes
2answers
256 views

Markov chain steady state existence

Is it possible for a Markov chain to have no steady state solution ? What is an example of such system ?
5
votes
0answers
164 views

Harmonic functions and null recurrence

Let $(X_n)_{n \geq 0}$ be a irreducible Markov chain defined on a countable state space $S$. It is known some ways to figure out if this chain is recurrent or not looking for superharmonic functions ...
1
vote
1answer
57 views

Understanding the proof of an Ergodic theorem for Markov chains

An ergodic theorem for Markov chains is as follows. If a Markov chain $(X_n)_{n \ge 0}$ is irreducible and has an invariant distribution $\pi$, then $$\frac{1}{n} \sum_{k=0}^{n-1} f(X_k) \to \...
0
votes
1answer
502 views

How to find transition probability matrix $P$ by using transition rate matrix $T$?

Let $$T = \left(\begin{matrix} -2 & 1 & 1&0 \\ 2 & -3 & 1&0 \\ 1 & 2 & -4 & 1\\ 1 & 3 & 1 & -5\end{matrix} \right) $$ be a transition rate matrix of ...
1
vote
1answer
31 views

Robustness of Markov Chains

A Markov Chain on a measurable space $X$ is uniquely determined by a stochastic kernel $P$ on $X$. Let $\mathsf P_x$ denote the probability on paths generated by $P$ and the initial condition $x\in X$....
0
votes
1answer
58 views

Expected number of lines in use in call centre (markov process: queuing theory)

Suppose we have a call centre with infinitely many lines to be able to call to. Calls come in a rate of $\lambda$ and customers are served with rate $\mu$. It is easy to see that the $Q$-matris looks ...
0
votes
0answers
58 views

probability of hitting state $i$ in random walk

We have a random walk on the integers with probability of going to the right is $\lambda$ and to the left is $\mu$. Suppose we start at 0. I want to find the probability of ever hitting a fixed state $...
3
votes
2answers
210 views

Limiting probability that the sum of the values of a die is a multiple of 13

A fair die is thrown repeatedly. Let $X_n$ denote the sum of the $n$ first throws. I have to find $\lim_{n\rightarrow \infty}P(X_n \text{ is multiple of 13})$. Now follows what I tried, which I don't ...
0
votes
0answers
99 views

Markov Chain bonus-malus system

I'm having some troubles with this problem because I don't know how to construct the transition matrix, because they are talking about "more than 1 step". I think that the State space is S={0%,30%,40%,...
4
votes
1answer
102 views

Applying MCMC Metropolis algorithm

I'm interested in all possible paths (on the grid $\mathbb{N}^2 $) that goes from $ (0,0) $ to $ (n, n) $. At each step there are two possibilities: go right or go up. The path is a sequence $ z=(z_0,...
0
votes
1answer
230 views

How to solve a discrete SIR epidemic model?

Let $(S(t), I(t), R(t))$ be a continuous time Markov chain SIR model with discrete space, where $S(t)$ stands for the number of susceptible people at time $t$; $I(t)$ stands for the number of ...
2
votes
0answers
158 views

Induced Markov chain - verify Markov property and another property

First, here is how we defined induced Markov chains: Suppose that $(X,E,P)$ is an irreducible Markov chain, where $X=(X_i)_{i\in\mathbb{N}_0}$, $E$ is the state space and $P=(p_{i,j})_{i,j\in E}$ is ...
1
vote
1answer
68 views

Show that the space of superharmonic functions is not a linear space

Let $(X_n)_{n\in\mathbb{N}_0}$ be a Markov chain with state space $E$ and transition matrix $P=(p_{i,j})_{i,j\in E}$. A real valued function $h$ on $E$ is called superharmonic if $h(x)\geq Ph(x)$ ...
1
vote
1answer
38 views

Irreducible Markov chain and finite sets

Let $(X_n)_{n\geq 0}$ be a irreducible Markov chain defined on a countable state space $S.$ Let $F \subset S$ a finite set and $\tau=inf\{n \geq 1; X_n \notin F\}$. If $x \in F$ how to prove that $\...
0
votes
1answer
1k views

Markov Chain Steady State 3x3

I have been learning markov chains for a while now and understand how to produce the steady state given a 2x2 matrix. For example given the matrix, ...
0
votes
1answer
90 views

Solve Unknown Matrix Variables

I have a markov chain matrix with probabilities as such, on finding the steady state.. ...
0
votes
1answer
80 views

Markov chain modes of convergence

This is continuation of the question stated here. Let $\left( {{X_\alpha }:\alpha \in A} \right)$ be a finite space Markov chain (discrete or continuous), consisting of only transient and absorbing ...
1
vote
1answer
213 views

Estimate the speed of convergence to the stationary distribution for a ergodic Markov process

I have encountered a Markov process with following transition matrix $P= \begin{bmatrix} 0.6 & 0.4 \\ 0.2 &0.8\end{bmatrix} $. This is an ergodic Markov matrix since all the elements are ...
0
votes
2answers
34 views

Show $P(X(t)=0 | X_{0}=2)= P(X(t)=0 | X_{0}=1)^{2}$

Question: Let $X(t)$ be a continuous-time Markov chain on all non-negative integers with generator matrix $Q$ having for all $i\geq 0:$ $$ q_{i,i}=-i(\lambda +\mu ) \qquad q_{i,i+1}=\lambda i \qquad ...
0
votes
1answer
51 views

Limit for a Markov chain

I am considering a Markov chain on S = {1, . . . , 21} with transition matrix given by: $ 1 =p_{1,2} = p_{2,1}= p_{13,14} = p_{18,14} = p_{15,16} = p_{16,3} = p_{17,16}\\ 1/2 =_{p5,5} =p_{5,7}=p_{7,7} ...
1
vote
1answer
70 views

Two definitions of the strong Markov property

In Durrett's textbook, the strong Markov property is defined as follows: For every bounded and measurable $\varphi$ and stopping time $N$: $$\mathbb{E}(\varphi\circ\theta_N|\mathcal{F}_N)=\mathbb{E}_{...
1
vote
0answers
105 views

Compute transition probability in n step in infinte markov chain

I want to calculate the probability of transition in n step from state 0 to state 0 ($p_{00}^{(n)}$) in below Markov-Chain : if self loop in state 0 doesn't exist, probability computed with Catalan ...
0
votes
2answers
228 views

Are irreducible, positiv-definite Markov chains aperiodic?

If $M$ is the transition matrix of a discrete Markov chain, and $M$ is both irreducible, symmetric and positiv-definite, is the resulting Markov chain necessarily aperiodic? In my intuition, ...
2
votes
0answers
89 views

Invariant Probability of Discrete Time MC from Continuous Time Markov Chain

Given rates α of an irreducible continuous-time MC on finite state space and told that π is the invariant probability measure of this chain, we define a discrete time MC as having transition ...
3
votes
1answer
68 views

Prove matrix is positive semi-definite

$P$ is a stochastic matrix (square, non-negative, rows sum to 1). $\Xi$ is a diagonal matrix with a left principal eigenvector of $P$ on the diagonal and zeros elsewhere (stationary distribution if $P$...
1
vote
1answer
252 views

Random Walk on Clock Hands

We do a random walk on a clock. Each step the hour hand moves clockwise or counterclockwise each with probability 1/2 independently of previous steps. If you start at 1 what is the expected number ...
2
votes
1answer
136 views

Why can a Markov chain having two states and no self-loop have a stationary distribution?

Why does a Markov chain having two states and no self-loop can have a stationary distribution? Lets consider a markov chain with two nodes = $\{A, B\}$ and the transition matrix: $P = \begin{array}{...
2
votes
0answers
229 views

How to prove a matrix norm inequality?

$P$ is a stochastic matrix i.e. square, non-negative, rows sum to one. Let $\Phi$ be a real matrix of size $n \times k$ with independent columns and $k < n$. Let $\Xi$ be the diagonal matrix with a ...
1
vote
1answer
46 views

Prove $\Xi (I - P)$ has eigenvalues in the non-negative real half-plane.

Let $P$ be a stochastic matrix (square, non-negative,rows sum to one). Let $\Xi$ be a diagonal matrix with any left principal eigenvector of $P$ on the diagonal and zeros elsewhere (stationary ...
3
votes
0answers
142 views

Prove or disprove: If $h$ is harmonic on $E$, then $h$ is constant on each $C_i$

For a general finite Markov chain $(X_n)_{n\in\mathbb{N}_0}$ with state space $E$ and transition matrix $P=(p_{x,y})_{x,y\in E}$, not necessarily irreducible, we define the linear space of harmonic ...
0
votes
1answer
49 views

Convergence of the number of visits in a Markov Chain

Suppose we have an irreducible and recurrent discrete-time Markov chain with states over the finite set $\mathcal{X}$. Let $N_t (x)$ denote the number of visits to state $x$ up to time $t$. Let $\pi(x)...
2
votes
2answers
142 views

Stationary probability in an M/M/$1$ queue with a lazy server

Customers arrive to a single server queue according to a Poisson process with rate $\lambda$. Each customer requires Exponential($\mu$) service time. In the beginning when there are $0$ customer, ...
5
votes
1answer
154 views

Conditions for birth and death process having only finitely many deaths.

Consider a birth and death process on $\mathbb{N}=\left\{0,1,2,\ldots\right\}$, given by the transition probabilities $p(n,n+1)=\lambda_n$ and $p(n,n-1)=\mu_n$ (those are the birth and death rates, ...
1
vote
1answer
38 views

Determining invariant probability measure and calculating $\lim_{n}p_{ij}^{(n)}$

Consider the Markov chain $(X_n)_{n\in\mathbb{N}_0}$ with state space $E=\left\{1,2,3\right\}$ and transition matrix $$ P=\frac{1}{2}\begin{pmatrix}0 & 1 & 1\\1 & 0 & 1\\1 & 1 &...
1
vote
0answers
25 views

Checking the closeness of probability distributions

Suppose I have a Markov chain that satisfies all the conditions of ergodicity and has a stationary distribution pi. I want to find the time when the probability distribution of the markov chain is ...
0
votes
1answer
75 views

Calculating the probability in $m$ steps of a Homogeneous Markov Chain

I have the next problem: Consider a homogeneous Markov chain $\{X_n: n = 0,1,2, ... \} $ with state space $E = \{0,1,2, ... \} $, with the following transition probabilities where $ 0 <\theta <$...