Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
1answer
24 views

Discrete-time Markov chain properties

A Markov chain in discrete time is irreducible, has state space $\{0,1,\dots\}$ and starts at $1$. It is both a branching process and a martingale. Determine the probability of hitting $0$.
3
votes
0answers
70 views

Conditional probability and integrating out part of a random walk

Suppose that I have a random walk process defined by $\alpha_{t+1}$ ~ N$(\alpha_t, \omega^2)$. Given $\alpha_t$ and $\alpha_{t+2}$, I understand why the conditional formula for ...
0
votes
1answer
51 views

Markov Chain - Snakes and Ladders

A simple game of snakes and ladders is played on a board of nine squares. At each turn a player tosses a fair coin and advances one or two places according to whether the coin lands heads or tails. If ...
2
votes
0answers
30 views

Intuition behind criterion for an irreducible Markov chain to be transient

I have been looking over my notes for Markov chains, and I have come across the following: Theorem: An irreducible Markov chain is transient iff for some state $i$ there exists a nonzero vector $y$ ...
0
votes
0answers
22 views

Markov proof that a state is either transitive or ergotic - can it be so simple?

This is the chart associated with a Markov matrix The equivalence(communication) classes are: {1,2,3,4} - transitive {5,6,7} - transitive {8} - ergotic My teacher said that "all equivalence ...
1
vote
0answers
25 views

Meaning of $\pi$ in case of irreducible positive recurrent DTMC which is not aperiodic

In case of a irreducible positive recurrent DTMC which is not aperiodic, we know that there exist a positive unique probability mass function $\pi$ satisfying $\pi=\pi p$. The meaning of this can be : ...
0
votes
1answer
22 views

A basic doubt on the sojourn time of a CTMC

By using the memoryless property, I find the CDF of sojourn time of a CTMC as follows : $$F_X(t) = 1-e^{-F_X'(0)t}$$ I am slightly confused about the intuitive meaning of the term $F_X'(0)$. How ...
0
votes
1answer
20 views

Can an absorbing CTMC be reversible?

Can a CTMC with an absorbing state be reversible? I guess not, as the product of rates through any loop cannot be equal when the loop involves the absorbing state (Kolmogorov criterion). Is my ...
1
vote
1answer
22 views

Mean number of particle present in the system: birth-death process, $E(X_t|X_0=i)$, $b_i=\frac{b}{i+1}$, $d_i=d$

Let $\{X_t\}$ be a birth–and–death process with birth rate $$ b_i = \frac{b}{i+1}, $$ when $i$ particle are in the system, and a constant death rate $$ d_i=d. $$ Find the expected number of particle ...
0
votes
1answer
38 views

three-state Markov chain

a male and a female go to a 2-table restaurant on the same day. each day the male sits at one or the other of the 2 tables, starting at the table 1, with a Markov chain transition matrix: ...
1
vote
1answer
43 views

Finding Markov chain transition matrix using mathematical induction

Let the transition matrix of a two-state Markov chain be $$P = \begin{bmatrix}p& 1-p\\ 1-p& p\end{bmatrix}$$ Questions: a. Use mathematical induction to find $P^n$. b. When n goes to ...
-1
votes
0answers
35 views

Why can't finite closed communicating classes be null recurrent?

Why can't finite closed communicating classes be null recurrent ? I want both formal proof and intuitive answer. I know that if state $j$ is null recurrent then $$\lim ...
1
vote
1answer
55 views

Limit theorem of Markov chains applied to higher order Markov chains

I have a second order Markov chain with 4 states {A,T,C,G} (the 4 DNA nucleotides). the transition matrix looks like this: ...
0
votes
1answer
29 views

Markov chain problem in Ross's Introduction to probability models

It is example 4.10 and the problem states that a pensioners receives 2 at the beginning of the each month. The amount of money he needs to spend is independent of the amount he has and is equal to $i$ ...
1
vote
0answers
32 views

Log Moment Generating function of a two-state Markov source

Let's say you have a two-state markovian source whose transition matrix is $P=\begin{pmatrix}1-\sigma & \sigma\\ \tau & 1-\tau\end{pmatrix}$, for the state 0 the data rate is 0 and for the ...
0
votes
1answer
26 views

Question on Markov chains of expected number of states

I am confused with an statement from my probability book that has to do with Markov chains. I hope someone could clarify that, if possible....Consider a Markov chain for which $P_{11}=1$ and ...
1
vote
1answer
37 views

Hidden Markov Model Coin Toss Problem

Given two coins and transition matrix between them given by: $\begin{bmatrix} 1-\alpha&\alpha \\ \beta&1-\beta \end{bmatrix}$ Where the first coin has probability of heads $p$ and tails ...
0
votes
0answers
11 views

Importance Sampling: Binomial Distribution

I'm trying to estimate the probability $P(X\geq80)$, X~bin(100,p), with importance sampling. So I came up with a Markov Chain and defined a random variable $Y$ on that Markov chain so that ...
0
votes
1answer
26 views

Convergence of probabilistic process

Take a random vector $V$ of length $n$ where $V_i \in [m]$. Define a random process as follows. Repeatedly pick two indices $i, j \in [n]$ uniformly at random and let $V_i := V_j$ (that is change the ...
0
votes
0answers
28 views

Expected number of generations

Consider the following simplistic model of transitions between social classes as defined by Sociologists. Only males are considered, and by assumption every male has exactly one son. Let Xn denote the ...
1
vote
0answers
19 views

Estimate on Galton-Watson process distribution

Let $(Zn)_{n\in \mathbb N_0}$ be a Galton-Watson process, i.e. $$ Z_{n+1} = \sum_{k=1}^{Z_n}\xi_{n,k},\qquad (\xi_{n,k})_{n\in \mathbb N_0,k \in \mathbb N} \quad \text{i.i.d } \mathbb N_0 \text{ ...
0
votes
0answers
69 views

Modified M/M/1/2 with 2 possible arrival rates and M/M/1/5 queue

I've been stuck on this question for hours, and could use some help :) "An M/M/1/2 queue has service rate $\mu$ and arrival rate of either $\lambda_1$ or $\lambda_2$. The rate can change only when ...
0
votes
1answer
63 views

Random walk with absorbing barriers

Consider a random walk with absorbing barriers at $0$ and $3$. $\mathbb P(S_{n+1}-S_n=1)=0.6$ and $\mathbb P(S_{n+1}-S_n=-1)=0.4$. What is the probability of eventual absorption at $0$, given that the ...
4
votes
2answers
79 views

Prove that a random walk on $\mathbb{Z}_+\cup \{0\}$ is transient

Prove that a random walk on $\mathbb{Z}_+ \cup \{0\}$ is transient with $p_{i,i+1}=\frac{i^2+2i+1}{2i^2+2i+1}$ and $p_{i,i-1}=\frac{i^2}{2i^2+2i+1}$. So since this Markov chain has only a single ...
0
votes
1answer
31 views

General State Space Markov Chain

I am having some difficulty understanding some early results of Markov Chain theory on a general state space. We have a function (Kernel) $K:E \times E \rightarrow \mathbb{R}$, and a distribution ...
0
votes
0answers
27 views

simulating a Markov Chain from the transitional probability matrix of a second order Markov Model [closed]

any idea how can I simulate a second order Markov chain given the transitional probability matrix in R
0
votes
1answer
23 views

Time Periodic Homogeneous Markov Chain

I want to find a textbook or survey article reference with a treatment of discrete-time, inhomogeneous, yet time periodic, markov chains on finite state spaces. Elaboration: I have an inhomogeneous ...
0
votes
1answer
42 views

Identity in Markov Processes

I want to know if my reasoning here is correct, it seems simple enough but I just want clarification (I am considering the proof that if a Markov process satisfies the detailed balance condition, then ...
0
votes
1answer
26 views

recurrence for 2nd order Markov Chain

Given that $X_n$ given $X_{n-1},...,X_0$ is Poisson distributed with mean $a+bX_{n-1}+cX_{n-2}$, for $n\geq 2, a>0,b,c\geq 0$. Define $Y_n=\begin{pmatrix}X_n\\X_{n-1} \end{pmatrix}$. Prove that ...
2
votes
0answers
54 views

Question on M/M/2 queue variation

I have the following question: Two workers handle three machines(i.e. we can at most repair two machines at a time). The time until the machine breaks down is exponential distributed with expectation ...
0
votes
0answers
39 views

Question about Infinite Markov chains

Do 2 Markov chains $\left\{X_n\right\}^\inf_{n=0} $ and $\left\{Y_n\right\}^\inf_{n=0} $ with all of these properties exist so that the probability for infinite n values to maintain $X_n=Y_n$ is 0? ...
4
votes
1answer
181 views

Lower bound for multivariate recurrence

I have a recurrence that looks like $$p(i,j,k) = \frac{j}{n}p(i-1,j-1,k-1) + \frac{i-j}{n}p(i-1,j,k-1)$$ $$p(i,0,k) = 1$$ $$p(i,j,0) = 0$$ $$p(0,j,k) = 0$$ The base cases are to be considered in ...
0
votes
0answers
59 views

Markov Chains Worked Example (Stirzaker)

I have a Markov Chain with state space the non-negative integers. The rules of the M.C. are that when it is in state $i \neq 0$, it moves to one of {${0,1,2,\ldots,i+1}$} with probability $1/(i+2)$ ...
3
votes
0answers
53 views

Prove the 2 definitions of the periodicity of Markov Chain are equivalent.

In many textbooks, there are basically 2 ways of defining the periodicity of Markov Chain. One is by partitioning the graph in to subgraph such that transition in one group of state leads to the other ...
0
votes
1answer
56 views

Probability, Markov chain

A teacher leaves out a box of N stickers for children to take home as treats. Children form a queue and look at the box one by one. When a child finds $k \geqslant 1 $ stickers in the box, he or she ...
0
votes
0answers
12 views

Some questions about a “relaxed” invariant probability problem $|\mu(P-I)|\leq \epsilon$

Let's consider the set $\mathcal{M}=\{\mu:|\mu(P-I)|_1\leq \epsilon\}$ where $\mu$ is a probability vector, $P$ is the transition matrix of a discrete homogeneous Markov chain, $I$ is the identity ...
1
vote
2answers
63 views

Question about Markov chain derived from a Poisson process

Let $(N_t)$ be a Poisson process of rate $\lambda$. Define $$ X_n = N_n − n,\quad\text{for }\; n = 0, 1, 2, \ldots $$ Explain why $(X_n)$ is a Markov chain and give its transition probabilities. Using ...
0
votes
0answers
22 views

Transition matrix after lumping Markov Chain?

I managed to partition the set of states $P=\{A_1,A_2,...\}$ so that they satisfy the lumpability criterion (for all states in $A_i$ the sum of the outgoing rates to the states in the target partition ...
1
vote
0answers
35 views

Kolmogorov backward and forward equations for a discrete-time Markov chain?

I found Kolmogorov backward equations and forward equations for diffusion processes, and for continuous time Markov chains in Wikipdia. I was wondering what Kolmogorov backward and forward equations ...
1
vote
0answers
16 views

What is the matrix norm in defining the generator of a continuous time Markov chain?

For a continuous time Markov chain with finite state space and Markov transition function $p(t)$, its generator $G$ can be defined entry-wise as $$ G_{i,j}:=\lim_{t\to 0^+} \frac{p_{i,j}(t) - ...
2
votes
1answer
49 views

Two different ways of constructing a continuous time Markov chain from discrete time one

Consider a homogeneous continuous time Markov chain (CTMC) with Markov transition function $p(t)$ and infintesimal generator $G$. Its embeded discrete time Markov chain (DTMC) has its transition ...
0
votes
2answers
22 views

A state $i$ is recurrent, if and only there exists $n \geq 1$ s.t. $p_{ii}^{(n)} =1$?

For a homogeneous discrete time Markov chain with transition matrix $p$, a state $i$ is recurrent, if and only there exists $n \geq 1$ s.t. $p_{ii}^{(n)} =1$? I have it copied from somewhere in my ...
0
votes
0answers
24 views

Markov Chain: Turn a problem into a markov chain

We have 2 machines, which are working in 0.75 precent if it wasn't out o order the day before. When a machine goes out of order, it takes 2 days to fix her up. let Xn be the num of working machines in ...
1
vote
1answer
46 views

A simple case of random walk

$\forall n \in \mathbb{N}$ we can either move from state $S_n$ to state $S_{n+1}$ with probability $p$ or to state $S_{n-1}$ with probability $q=1-p$. Also we move from state $S_0$ to state $S_{1}$ ...
2
votes
1answer
62 views

Construction of positive recurrent Markov chain

Let $\{X_i\}_{i\geq 1}$ be i.i.d. with values in $\mathbb N_0$. Define a Markov chain via the following transition matrix: $$p(0,n) = \mathbb P(X_1 = n-1) \qquad p(m,n) = \mathbb P\left(\sum_{k=1}^m ...
0
votes
1answer
38 views

Calculating probabilities in genetic sequences

I am working with certain recurring sequences in genetics and try to calculate certain probabilities: Let for instance $$\langle g_i\rangle :=\{1,1,1,6,1,1,1,6,...,1,1,1,6\}$$ and $$\langle ...
0
votes
2answers
69 views

Prove Markov Chain by definition

I came across this problem in homework: $X_n$ are i.i.d random variables with $\mathbb{P}[X_n=1]=\mathbb{P}[X_n=-1]=\frac{1}{2}$ and we we also have $S_n=X_1+...+X_n.$ Show that $S_n'=|S_n|$ is a ...
0
votes
0answers
13 views

Invariant sets of modulo shift map

Let $a>0$ and define the modulo shift map as $$ S:[0,1) \rightarrow [0,1), S(x) = (x+a) \text{mod }1 $$ How can one explicitly characterize the invariant sets of this map, i.e. $A\in \mathfrak ...
0
votes
0answers
97 views

Random walk, Cat and mouse

Here is the problem. In graph G, on different vertices there is cat and mouse. Cat and mouse do independent random walk, but time is synchronous, in one unit of time both cat and mouse do one step. ...
1
vote
2answers
55 views

How can I calculate the expected number of changes of state of a discrete-time Markov chain?

Assume we have a 2 state Markov chain with the transition matrix: $$ \left[ \begin{array} (p & 1-p\\ 1-q & q \end{array} \right] $$ and we assume that the first state is the starting state. ...

1 2 3 4 5 9