Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
22 views

Null recurrence to positive recurrence in DTMC

What are some examples of null recurrent DTMC whose jump chain is positive recurrent? Specifically, for this null recurrent DTMC, removing self loops and normalizing the other outgoing edges from each ...
0
votes
0answers
41 views

recurrence/transience on random walk

Let $X_n$ be a markov chain, $p>\frac{1}{2}$ and $E=\{0,1,2,...\}$ its state space. Let $\Pi$ be its transition matrix with $\Pi(0,0)=p$, $\Pi(i+1,i)=p$, $\Pi(i,i+1)=1-p$ , $i\ge0$. ...
0
votes
0answers
23 views

Reference on Discrete Markov Chains

I am essentially looking for reference books on Discrete Markov Chains. You can see our full syllabus here.
0
votes
0answers
22 views

Show recurrence of a class

I am a little bit confused with the definition of recurrence with respect to Markov chains. For example consider the transition matrix $$ P=\frac{1}{2}\begin{pmatrix}0 & 1 & 1 & 0 & ...
0
votes
0answers
57 views

Dynamics of birth-death process with discouraged arrivals (alternatively, M/M/1 queue with balking customers)

Take a continuous-time birth-death process, where $k \in \{0,1,\ldots\}$ is the count and the arrival rate of death is $\mu \geq 0$ for $k = 1, 2, \ldots$ the arrival rate of births is $\alpha_k ...
0
votes
0answers
26 views

Does a state which is passed at least 3 times had to be passed 5 times in Markov chain

Prove of disprove: Let $\{X_n\}_n$ be homogenous Markov chain. if we start from state $i$, there is a positive probability that we pass at least 3 times at state $j$. Does it follows that exists ...
0
votes
0answers
22 views

continuous markov chain generator

I am trying to learn Markov process with my own. I am a little confused about the generator of markov process. I understand that Markov process consists of embbedded Markov chain matrix and the ...
0
votes
0answers
110 views

Proof of “strong law of large numbers” in Markov Chains

I have been given a theorem stating an analogue of the strong law of large numbers for Markov chains. It states that if $X=(X_n)_{n\in\mathbb{N}}$ is a Markov chain with transition matrix $p$ and ...
0
votes
1answer
75 views

Markov chain doesn't sum up to 1

Let $\{X_n\}$ be a Markov chain on $S=\{1,2,3,4,5,6\}$ with the matrix suppose we define a new sequence $\{Y_n\}$ by $$Y_n=\cases{1\quad X_n=1\vee X_n=2\\2\quad X_n=3\vee X_n=4\\3\quad ...
0
votes
0answers
38 views

Proof of theorem on markov chains

If $X=(X_n)_{n\in\mathbb N}$ is a Markov chain on a space $E$, it has an initial distribution $(\lambda_i)_{i\in E}$ such that $\sum\lambda_i=1$ and a transition matrix $(p_{ij})_{i,j\in E}$ such that ...
0
votes
0answers
42 views

non-integer powers of transition matrices with complex eigenvalues and resulting negative probabilities

I am currently working on a Markov Chain model for transition probabilities of a certain set of states. I am trying to figure out how to scale my transition matrix to arbitrary time periods by raising ...
0
votes
0answers
23 views

Memory less property of a Markov chain- Validation methods

Are there any tests to check the memory less property of a discrete time homogeneous Markov chain? I found a chi squared test to verify the time homogeneity of a Markov chain constructed from a set of ...
0
votes
0answers
5 views

Hastings algorithm

Let $Q=\begin{pmatrix} 0 & 1 & 0 & 0 & 0\\0.5 & 0 & 0.5 & 0 & 0\\ 0 & 0.5 & 0 & 0.5 & 0\\ 0 & 0 & 0.5 & 0 & 0.5\\ 0 & 0 & 0 ...
0
votes
0answers
22 views

Markov chain general help

If I have an absorbing state markov chain (with 2 absorbing states, graduate and dropout), and I know how many people I have in each state (say total for all states is 1000), how would I work out what ...
0
votes
1answer
76 views

Proof that there exists a non-negative eigenvector corresponding to eigenvalue 1 of stochastic matrix

Let $P \in [0,1]^{n \times n}$ be a [irreducible or reducible] stochastic matrix where its rows sum to 1 i.e. $$ \forall i \in \{ 1 , \dots n \} \quad \sum_{j=1}^{n} P_{ij} = 1 $$ It is easy to show ...
0
votes
0answers
49 views

Create a Martingale out of a Markov Chain.

Consider a homogeneous finite state Markov chain $\{X_n\}$ with transition matrix $P$ and state set $S$ consisting of real numbers. How to choose the elements of $S$ so that $\{X_n\}$ be a ...
0
votes
0answers
6 views

Condition under which Markov Chain remains in a compact set a.s.

Let $\{Y_n\}$ be a Marov chain. Is it good question to ask under what conditions this chain will take values from a compact set a.s. ?
0
votes
0answers
12 views

Given stochastic matrices D and K, under what conditions can I find stochastic matrices that satisfy a given equality?

Let $D \in \Re^{n \times m}$ and $K \in \Re^{m \times n}$ be two stochastic matrices, with $n > m$. The problem is to determine under which conditions there exist stochastic matrices $P \in \Re^{m ...
0
votes
1answer
46 views

Simple Random Walk; Proof hitting theorem; Ballot Theorem

Suppose that $(X_{n}:n\in\mathbb{N})$ is a $\pm1\mbox{-valued sequence.}$ Let $p\in(0,1)$ and $p=\mathbb{P}(X_{i}=1)\mbox{ and}\mathbb{P}(X_{i}=-1)=1-p=q$ . Define the simple random walk ...
0
votes
0answers
37 views

Stationary solution of a binary Markov chain of order m

Let $X$ be a binary Markov chain of order m. What is the stationary solution of X? In other words, find $\lim_{n\to \infty} P( (X_{n-m+1},X_{n-m},...,X_{n}) =(a_1,a_2,...,a_m))$, for arbitrary values ...
0
votes
0answers
29 views

Steady state of a specific Markov martirx

Let $n = 2^L$ for an arbitrary integer $L>0$ and let $A=(a_{i,j})$ be an $n \times n$ matrix with the following structure: For $1\leq i \leq \frac{n}{2}$, $a_{i,2i-1} = p_i$, $a_{i,2i} = 1- ...
0
votes
0answers
10 views

hidden Markov model with multiple observations

I am wondering if HMM can be used for the case that in a particular state, there are more than one observations. For instance at time t, we can observe the position, velocity and acceleration at the ...
0
votes
0answers
29 views

Problem related to variance of first passage matrix of a absorbing Markov chain

Consider the below computations taken from Kemeny/Snell Finite Markov Chains. Here $N=(I-Q)^{-1}$ calculated from some absorbing MC. $N_2$ is the variance matrix of $N$ and $N_{sq}$ is taken by ...
0
votes
0answers
29 views

How to prove that this stochastic matrix has a limiting distribution

I have the following stochastic matrix with $p_{ij} > 0$ and $\sum_j p_{ij} = 1$ $$ P = \begin{bmatrix} p_{11} & p_{12} & 0 & 0 & 0 & 0 \\ p_{21} & ...
0
votes
0answers
26 views

A problem on markov chain

I have seen in a book the following : Let $X_n$ be an ergodic markov chain taking values in a complete separable metric space $S$. Now consider the function $\mu(t) = \delta_{Y_n(\omega)}$ when $t ...
0
votes
0answers
25 views

Transience question for Markov Chains

Let's suppose I have a countable state discrete time MC that is known to be transient, irreducible and reversible with respect to some measure that assigns positive finite mass to each singleton, but ...
0
votes
0answers
31 views

What is the state space of this markov chain?

Consider a system where two persons sit at a table and share three books. At any point in time both are reading a book, and one book is left on the table. When a person finishes reading his/her ...
0
votes
2answers
88 views

what are “filtering” and “smoothing” mentioned in hidden Markov model wikipedia article?

the article mentions "filtering" and "smoothing" tasks, see here http://en.wikipedia.org/wiki/Hidden_Markov_model#Filtering . It gives brief explanation but no motivating examples and no references to ...
0
votes
2answers
40 views

Are these transient or recurrent states in a Markov chain?

I have the following transition matrix for a Markov chain with states $A, B, C, D, E$ $ \left| \begin{array}{ccc} 0 & 0 & \frac{1}{2} & \frac{1}{2} & 0 \\ \frac{1}{2} & 0 & ...
0
votes
1answer
40 views

How to define a transition matrix mathematically?

I'm writing my master thesis. Given the adjacency matrix of a graph, I need to define the transition matrix formally. I'm not able to figure out how to define it in mathematical notation. Can you help ...
0
votes
1answer
131 views

How to find transition probability matrix $P$ by using transition rate matrix $T$?

Let $$T = \left(\begin{matrix} -2 & 1 & 1&0 \\ 2 & -3 & 1&0 \\ 1 & 2 & -4 & 1\\ 1 & 3 & 1 & -5\end{matrix} \right) $$ be a transition rate matrix of ...
0
votes
0answers
45 views

probability of hitting state $i$ in random walk

We have a random walk on the integers with probability of going to the right is $\lambda$ and to the left is $\mu$. Suppose we start at 0. I want to find the probability of ever hitting a fixed state ...
0
votes
0answers
57 views

Markov Chain bonus-malus system

I'm having some troubles with this problem because I don't know how to construct the transition matrix, because they are talking about "more than 1 step". I think that the State space is ...
0
votes
0answers
54 views

Transition matrix of a double induced Markov chain

Here is how we defined induced Markov chains: Suppose that $(X,E,P)$ is an irreducible Markov chain, where $X=(X_i)_{i\in\mathbb{N}_0}$, $E$ is the state space and $P=(p_{i,j})_{i,j\in E}$ is the ...
0
votes
0answers
20 views

Period ground state 1-dim Ising model

Good morning! I'm at the beginning of my study about the Ising model and it has been proposed to me this problem: Find all periodic ground-state configuration for the following one-dimensional Ising ...
0
votes
1answer
154 views

How to solve a discrete SIR epidemic model?

Let $(S(t), I(t), R(t))$ be a continuous time Markov chain SIR model with discrete space, where $S(t)$ stands for the number of susceptible people at time $t$; $I(t)$ stands for the number of ...
0
votes
1answer
239 views

Markov Chain Steady State 3x3

I have been learning markov chains for a while now and understand how to produce the steady state given a 2x2 matrix. For example given the matrix, ...
0
votes
1answer
49 views

Solve Unknown Matrix Variables

I have a markov chain matrix with probabilities as such, on finding the steady state.. ...
0
votes
0answers
62 views

Finding invariant probability of discrete time Markov Chain

Suppose that $\alpha$ gives a rate for an irreducible cont. time Markov chain on a finite state space. Then suppose the invariant probability measure is $\pi$. Then let $p(x,y)=\alpha(x,y)/\alpha(x)$ ...
0
votes
0answers
24 views

Steady state state distributions.

I am looking for a less "proofy" explanation of how a finite, irreducible, aperiodic Markov chain has a unique steady state $\pi$. No need define terms or include proofs of Bezout's lemma or number ...
0
votes
0answers
38 views

HMM walk through for backward algorithm with given example

This pdf file is a resource that walk through a simple HMM algorithm of two states http://www.indiana.edu/~iulg/moss/hmmcalculations.pdf, I have question in step 4.1 of the algorithm Specifically ...
0
votes
0answers
30 views

Prove or disprove: A statement about generating functions of Markov chains

For a given Markov chain $(X,E,P)$ ($X=(X_n)_{n\in\mathbb{N}_0}$, $E$ is the state space, $P=(p_{x,y})_{x,y\in E}$ the transition matrix), prove or give a counterexample to the following ...
0
votes
2answers
24 views

Does the function of a random variable have the same transition matrix as the variable itself?

If I have a variable X, that follows a Markov Chain with a transition density $\rho(X)$ does a function of that variable f(X) have the same density or is there a one to one mapping to the density of ...
0
votes
0answers
28 views

What is the limiting distribution of this Markov Chain?

Take a Markov Chain with state space $\left\{ 0, 1, \dots, 20 \right\}$. Then we have the rule that given $X_n$: Compute $Z = X_n + 1$ or $Z = X_n - 1$ with probability $\frac{1}{2}$ each (if the ...
0
votes
1answer
155 views

Markov Chain Expected value

Let $(X_n)_{n\in\mathbb{N}}$ be a Markov chain with State space $E=\{1,2,3\}$ and transission matrix $$P=\begin{bmatrix} 0 & 1/3 & 2/3 \\ 1/4 & 3/4 & 0 \\ 2/5 & 0 & ...
0
votes
1answer
33 views

Markov Chain Expected Value notation.

I have question to answer regarding $X_n$ where $X_n$ is a Markov chain, $n$=$0$,$1$,$2$,... I am loking for What I don't understand is what this $3$ on $X_{n+1}$ is! Any ideas?
0
votes
0answers
9 views

Is there a general formula for determining this distribution in a Markov chain?

Let $C$ be an irreducible Markov chain with state set $S$, $\left| S\right| = n$, transition matrix $T$, starting at state $s_0 \in S$, and yielding the states $s_0, s_1, s_2, \ldots$ during a random ...
0
votes
1answer
58 views

Understanding steady state distribution

I need some help verifying that my understanding of steady state distribution is indeed correct. I have a transition diagram (model). With around 100 states and 6 variables. I have used a software ...
0
votes
0answers
36 views

Monte Carlo Markov Chain simulation

I am going to post the python code logic we used however I want someone to look at the number that are printing out. The Markov chain is uniformly distributed across all $50x50$ matrices with entries ...
0
votes
0answers
81 views

Random Walk, Markov Process

I'm stuck on a homework question and am wondering if anyone can offer some hints. Suppose we have some straight line graph G over the set $ V = \{1, 2, 3, ... , n\} $ of vertices, with an edge between ...