Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-chains]

Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

0
votes
0answers
8 views

Cramér-Lundberg model for on-demand insurance

I am looking for inspiration and perhaps guidance on the following as I’ve been stuck for a while now: Context: I am working on a practically oriented project to adjust the Cramér-Lundberg model ...
0
votes
0answers
13 views

How to prove that for a Markov Chain is $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$

I am new to Markov Chains and using this concept in statistics. For a Markov Chain, may I say that $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$ How can I prove this statement?
-1
votes
0answers
10 views

Find a vector λ which is in detailed balance with the transition matrix and with λ 1 = 1.

Markov chain $(X_n)n≥0$ has state space $\{1, …, N\}$ and transition probabilities $p_{ij} = 0$ if $ j ≤ N – i$ and $ p_{ij} = 1/i$ if $j > N – i $. (a) Find a vector λ which is ...
0
votes
0answers
14 views

Are we allowed to modify the order of the states within a Markov chain?

For instance, if I were to switch the states such that the rows are in the order (0,1,2,3) but the columns are (1,2,0,3), can we still do the relevant analysis? By observation, I see that the row sum ...
1
vote
1answer
16 views

1D Random walk with reflective barriers — average time to return to origin

I have very limited probability background, but I came across a problem in an engineering application: Is there a formula that computes the average number of steps taken for a particle beginning at ...
1
vote
0answers
19 views

Advantage of using Hidden Markov Model over Markov Chain

There are many problems that can be modeled using both Markov chain and Hidden Markov model (HMM). Can anyone please explain mathematically, why HMM should be preferred over Markov chain? Thank you.
2
votes
2answers
20 views

How to find the probability that a process never enters a particular state in a Markov Chain?

I am given the following Markov Chain: Assume that the Markov Chain is in state 3 immediately before the first trial. I am asked to find the probability that the process never enters state 1. I ...
0
votes
2answers
15 views

How do you express the probability of outcomes for the next state in Markov chain (given some state)?

How do you express the probability of outcomes for the next state in Markov chain (given some state)? This is an expected value, i.e. suppose 5 states, then if one starts at state 4, then the "...
0
votes
0answers
7 views

Optimization over transition probabilities

I am interested in the following question: Fix a natural number $n\ge 2$. Let $\Omega=\{\omega_1, \omega_2, \dots, \omega_n\}$. A Markov process with state space $\Omega$ and transition matrix $P$ ...
0
votes
0answers
16 views

What is known about optimizing a cost function $C$ over a Markov Chain steady state distribution?

Suppose I have $N$ (very large integer) states $S=\{s_1, ..., s_{N}\}$. Suppose we also have a cost for each state $C:S\rightarrow \mathbb{R}$. For every pair of states $(s_i, s_j)$, the pair form ...
0
votes
1answer
23 views

Boundary of the sum of an Infinite horizon discounted model

The sum of an infinite horizon discounted model is given as follows: $$R_t = r_{t+1} + \gamma r_{t+2} + \gamma^2r_{t+3} + ... = \sum_{k=0}^\infty \gamma^kr_{t+k+1}.$$ As can be seen, the sum is ...
0
votes
0answers
8 views

to show how a relation containing summation of products leads to a compact relation using generating function

I have faced with the relation $$I_k(x,t)=\sum_{n=0}^{\infty}x^n\sum_{n_1+\dots+n_k=n}\prod_{j=1}^{k}p(n_j,t)\tag{1}\label{eq1}$$ in some books. In these books, it is said that the above relation ...
1
vote
0answers
22 views

Linking Markov Chain with Renewal Process

GIVEN: $X_0,X_1,...$ irreducible, recurrent Markov chain with transition matrix $P$ Starting state $X_0=x$ $g(m)=P\{X_m=y\}$ for some fixed state $y$ I know that the renewal process is $g(m)=b(m)+\...
0
votes
0answers
19 views

Do multivariate markov chains have to be IID? [closed]

Everything is in the title : I have a set of random variables verifying the Markov property. However they are neither independent nor identically distributed. Are they still considered to form a ...
0
votes
0answers
30 views

estimate Markov chain mean transition time

Assume a continuous time Markov chain which is run through in one direction and finally absorbed at the last state $1 \rightarrow 2 \rightarrow 3 \rightarrow ... \rightarrow n $ The transition ...
-2
votes
0answers
49 views
+50

Prove periodicity is a class property

Prove that if state $i$ in a class has period $p$ then all states in that class have period $p$. The proof is given on this answer is this: One way to define the period of state $i$ is as the ...
6
votes
1answer
40 views

Markov Chain upper bound on the probability of hitting time

I encountered the following problem. $\{x_t\}$: Markov chain in discrete time; $\Omega$: a finite state space s.t. $|\Omega|=n<\infty$; $\tau_w\equiv\min\{t\ge 0\,|\,x_t=w\}$, $w\in\Omega$ (first ...
0
votes
1answer
10 views

Explicit formula for the one-dimensional distributions of a time-homogeneous Markov chain subordinated by a Poisson process

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space, $(E,\mathcal E)$ be a measurable space, $(X_n)_{n\in\mathbb N_0}$ be a $(E,\mathcal E)$-valued time-homogeneous Markov chain on $(\...
0
votes
0answers
36 views

expectation of a linear operator

Could anyone help me with this? We define $T: C[0,1]\to C[0,1]\ni T(f(x))= \sum_{k=1}^{m} p_k (f\circ f_k)(x):=\mathbb E( f(X_{n+1}|X_n=x)$ for a system $X_{n+1}=f_{\omega_n}(X_n), n=0,1,2\dots,$ $\...
0
votes
0answers
15 views

Positive recurrence of finite state space continuous Markov chain

Given an irreducible continuous Markov chain X on a finite state space with a Q-matrix Q, how can we show that it is positive recurrent? Could only find this for DMCs. I know finite state space ...
0
votes
0answers
39 views

Definition of transient state

Consider the following definition Transient States It is often useful to talk about whether a process entering a state will ever return to this state. Here is one possibility. A state ...
0
votes
0answers
15 views

help with understanding Wright-Fisher model Markov Chain

Markov chain was just introduced, and on one page of our lecture notes "Markov Chain examples" it had wright-fisher model and I cannot understand it at all. Help please. Thank you! Wright-Fisher ...
0
votes
1answer
56 views

Understanding definition of Periodicity of Markov chain

Consider the following example that is used to understand the definition of periodicity property. Why does it says that: starting in state $1$, it is possible for the process to enter ...
0
votes
0answers
17 views

Reference for simple Markov chain construction

Let $M_n$ , $n\in\mathbb{N}_0$ be a Markov chain on a general state space $X$. Fix $m\in \mathbb{N}$. ¨ My question is if there's a name / reference for this trivial Markov chain on $X^m$ defined by ...
2
votes
1answer
45 views

What is the intermediate step in this negative binomial proof?

On Slide 47 of these slides, there is a formula stating $$ s_k(t) = - \sum_{j=0}^{k} s_j(0) \sum_{n=j}^{k} \frac{e^{-\lambda_n t} \prod_{m=j}^{k-1} \lambda_m }{\prod^{k}_{m=j, m \neq n} (\lambda_m - \...
0
votes
1answer
65 views

Markov chain with known duration

Let's say there are 2 states (A & B), where the probability of going from state A to B in interval i is $P_{ab}$. State B always lasts for n intervals, then always goes to back to state A. I ...
0
votes
0answers
15 views

Same transition matrix implies markov chain.

Let $X_i$ , $i \geq 0 $ be discrete time and state stochastic process. Suppose that $P(X_n = j | X_{n-1} = i) $ does not depend on $n$. that is $$P(X_n = j| X_{n-1} = i) = P(X_1 = j | X_0 = i) $$ for ...
0
votes
1answer
26 views

Exponential Distribution and Markov Chains [closed]

A submarine has three navigational devices but can remain at sea if at least two are working. Suppose that the failure times are exponential with means 1,1.5 and 3 years. Formulate a ...
0
votes
1answer
20 views

How can I compute the Limiting Distribution in the following problem?

Consider the transition matrix $ P = \begin{bmatrix} 1-p&p\\ q&1-q \end{bmatrix} $ for general $2$-state Markov Chain $(0 \le p, q\le 1)$. Find the limiting distribution ...
0
votes
0answers
30 views

Transition probability matrix, weather changes

I am trying to solve the following problem: and the probability that there will be three sunny days in a row specifically. I know that the answer is $1/5$ but I am trying to figure out how to get ...
1
vote
0answers
11 views

Intuition behind invariant measure for a stochastic matrix Norris Theorem 1.7.6

I'm trying to figure out the meaning of the following theorem from J. Norris - Markov Chains. Let $$\gamma_i^k=E_k\sum_{n=0}^{T_k-1}1_{\{X_n=i\}}$$ Let $P$ be irreducible and let $\lambda$ be an ...
0
votes
2answers
39 views

Understanding the meaning of transition rates in a CTMC

I am reading the queueing theory volume 1 by Kleinrock. In the chapter on Continuous Time Markov Chain(CTMC), the author defines the infinitesimal generator, $Q(t)$ as having the following elements: \...
0
votes
1answer
25 views

Markov Chain involving gerrymandering in Pennsylvania.

I am currently working on a project that involves the use of analyzing the 18 districts of Pennsylvania and using the results of the 2018 house of reps congressional election. I understand that ...
0
votes
1answer
17 views

How do we derive the spectral projector associated with a simple eigenvalue?

Result 7.2.12 of Meyer's Matrix Analysis and Applied Linear Algebra gives the following: If $x$ and $y^*$ are respective right and left eigenvectors of a matrix $A$ associated with a simple ...
0
votes
1answer
38 views

Finding limiting distribution.

Consider the transition matrix $ P = \begin{bmatrix} 1-p&p\\ q&1-q \end{bmatrix} $ for general $2$-state Markov Chain $(0 \le p, q\le 1)$. (a) Find the limiting ...
0
votes
0answers
15 views

What does this expression say in plain English in case of a Markov chain?

$P(X_1 = 3, X_2 = 1) = 0.14$ What does this expression say in plain English w.r.t. a Markov chain? According to me, The probability of the system is at state-$3$ in the first step AND at the ...
1
vote
1answer
27 views

Transition matrix, Expected value and Covariance

Let $\lbrace X_{n} \rbrace _{n}$ a Markov Chain and let $P$ be the associated transition matrix $P$= $\begin{pmatrix}.2&.3&.5\\.1&.7&.2\\.5&.3&.2\end{pmatrix}$ Given $ P( ...
2
votes
0answers
51 views

Maximum size of a subcritical birth-death process

Beginning with a population of $n_0$ individuals, let each individual have a probability $p$ to survive until it replicates into two independent and identical individuals, where $p<\frac12$. It ...
2
votes
1answer
60 views

Gambler's Ruin variant: Play until the player earns target money

I have a question that is a variant of Gambler's Ruin problem. Setting: A wins a bet with probability $p\neq \frac{1}{2}$ and loses a bet with probability $q=1-p$. If A wins, A gets 1 dollar. ...
0
votes
1answer
13 views

Irreducible aperiodic Markov chain $P$ with invariant distribution implies $p_{ij}^{(n)} \rightarrow \pi_j$

I am reading Norris' book on Markov chains, and there is a theorem that sais : Let $P$ be irreducible and aperiodic, and suppose that $P$ has an invariant distribution $\pi$. Let $\lambda$ be any ...
1
vote
0answers
12 views

Expected Growth of General Pure Birth Process

Assume we have a non-explosive Markov Chain $(X_t)_t$ in continuous time with Q matrix $$ Q(k,l) = \begin{cases} \lambda_k &\text{if } l=k+1\\ -\lambda_k &\text{if } l=k\\ 0 &\text{...
4
votes
2answers
76 views

Given probability vectors $x$ and $y$, how can I compute probability vector of $z = x + y$ using linear algebra operations

I have probability vector $t_1$ (hours to get '$A$' done), $p({t^{a}_{1})} = [0.25, 0.25, 0.25, 0.25]$ (here: $0.25$ probability for $A$ done in $1, 2, 3$ or $4$ hours). Another probability vector $...
1
vote
0answers
14 views

If detailed balance does not hold, then it does not hold at any time

If we have a Markov chain represented by the matrix $W_{ij}$ and we know it satisfies the detailed balance condition $W_{ij} P_j = W_{ji} P_i$ $\forall i,j$ , then we know that for larger time $n$ the ...
0
votes
0answers
12 views

Absorption probabilities of a Markov chain

So I was reading on Markov chains and came across this problem: $P$ is an $N\times N$ transition matrix such that $\sum_{j=0}^{N} jp_{ij}=i$ for all $1\leq i \leq N$. a) Prove that states 1 ...
1
vote
1answer
47 views

Simplifying summation of binomials

I am working on a proof and think that the following equality holds but am unable to prove it: $$ \sum_{k_1=0}^{m} \sum_{k_2=0}^{m-k_1} \dots \sum_{k_{d^2/2}=m-(k_1+k_2+\dots+k_{d^2/2-1})}^{m-(k_1+k_2+...
0
votes
1answer
24 views

Calculating return probability to $0$ for SRW on $\mathbb{Z}$

I know that $P^{2n}(0,0) = C^{2n}_{n}2^{-2n}$ where $C^{2n}_{n}$ is $2n$ choose $n$. Using Sterling's approximation, $n! \sim \sqrt{2\pi}n^{n+\frac{1}{2}}e^{-n}$, we get $$P^{2n}(0,0) = C^{2n}_{n}2^{-...
1
vote
0answers
25 views

Reference request: Markov chain

I am in a situation to know whether a discrete time Markov chain evolving on Banach space $\mathbb R^n$ whose evolution (of states) equation we know and then I am interested to project( Hopefully an ...
0
votes
1answer
23 views

Independence in Markov's chains.

Consider a $\xi_n$ be a Markov chain. With $\mathbb{P}(\xi_{n+1} = k+1| \xi_n = k) = p$ and $\mathbb{P}(\xi_n = k| \xi_{n-1} = k) = 1 - p$. Let $\tau_{k} = \min\{n : \xi_n = k\}$. We need to prove ...
-2
votes
1answer
14 views

Markov Chain, Irreducible, Stationary distribution

Please help with this question, I have no clue where to even start. Please please please help!!
0
votes
1answer
16 views

Markov chain: infinite number of stationary distributions

Is it possible to construct a Markov chain having an infinite number of stationary distributions $\pi_i$? Maybe also with a finite set of states $S$? Maybe someone can explain why the following ...