Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-chains]

Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

0
votes
0answers
5 views

dual formulation of a Markov chain

I have a discrete time Markov chain on some state space $S$ which is specified by an integer $m$ and collection of maps $f_j, j=1,2,\dots,m$ and the probability function $p_j:S\to [0,1]$, given the ...
0
votes
0answers
3 views

Non-unique stationary distribution but all states inter-communicate?

I have a Continuous Time Markov Chain with the following probability transition matrix: $$P_t= \begin{bmatrix} 1-\lambda t e^{-\lambda t} & \lambda t e^{-\lambda t} \\ \mu t e^{-\mu t} &...
0
votes
0answers
10 views

Derive the Conditional Distribution of a Brownian Motion Process

Let $\ X(t),t \ge 0$ be a Brownian motion process. That is, $\ X(t)$ is a process with independent increments such that: $$\ X(t) - X(s) \sim N(0,t-s), 0\le s \lt t $$ and $\ X(0)= 0$. ...
0
votes
0answers
12 views

When is it necessary to solve Kolmogorov forward equations (KFE) for a Markov Chain?

Say I have a continuous time markov chain, time homogeneous $X$ with a few states (say, 2). I want to know the distribution of where $X$ is at time $t$, call it $\mu_t$, which will be a vector of 2 ...
0
votes
1answer
10 views

Why does $P(Q_t = q | X_{0:L} = i_{0:L}) = P(Q_t = q, X_{0:L} = i_{0:L})$?

This is a derivation of an equation used to maximize the posterior probability that $Q_m = i_m$ given a model and a sequence of observations. $Q_m$ is a RV which maps to some $q \in S$, the state ...
3
votes
0answers
20 views

joint distribution of number of visits of a Markov chain

Let $\{X_{n}\}_{n \geq 0}$ be a Markov chain on the state space $S = \{1,\dots,r\}$, $r \geq 2$. Let $\alpha = (\alpha_{1},\dots,\alpha_{r})$ be the initial distribution and $P$ the transition matrix. ...
0
votes
0answers
10 views

Is there some probability measure associated to a markov chain?

Clearly there are some probabilities involved with the markov chains, but I cannot see how to extract a sigma algebra from it. Is it that the probability of getting from state $i$ to state $j$ is ...
0
votes
0answers
10 views

Probability of a machine working at certain time

This question arises while I am learning Continuous Time Markov Chain : A machine is working for an exponential time with rate $\mu$ before breaking down. The repair time of the machine is ...
1
vote
0answers
24 views

If $Y$ is a Markov chain and $h>0$, why is $(Y_{\lfloor t/h\rfloor})_{t\ge0}$ not a Markov process?

Let $\left(Y^{(n)}_k\right)_{k\in\mathbb N_0}$ be a Markov chain for $n\in\mathbb N$, $(h_n)_{n\in\mathbb N}\subseteq(0,\infty)$ with $h_n\xrightarrow{n\to\infty}\infty$ and $$X^{(n)}_t:=Y^{(n)}_{\...
3
votes
0answers
34 views

How to find probability transition matrix for continuous time markov chain?

In Grimmet and Stirzaker, on page 258 it explains how to find transition probabilities, given a generator matrix: (a) nothing happens during $(t,t+h)$ with probability $1+g_{ii}*h+o(h)$ (b) ...
1
vote
0answers
21 views

Reference Markov martingale Harmonic function

I've just finished a course of stochastic process (discret martingale and markov chain). I would like to go further, I heard it exists a link between martingale markov process and harmonic functions. ...
2
votes
0answers
40 views

Random walk on $\{0,1,…,k\}$, find the average gain in 10 000 steps

I have the following problem which I can't seem to figure out. The problem is as follows. Consider simple random walk on {0, 1, ... , k} with reflecting boundaries at 0 and k, that is, random ...
0
votes
0answers
16 views

Markov's chain limiting probability theorem question [on hold]

enter image description here i wonder the answers to this question. please let me know solution.
1
vote
1answer
23 views

Period of each state

I am trying to determine the period of each state $ j = 0, 1, 2$ for this irreducible Markov Chain with transition probability matrix $$P=\begin{bmatrix}0&0&1\\1&0&0\\\frac{1}{2}&\...
0
votes
0answers
15 views

every stochastic matrix defines a markov chain?

Maybe the Question is obvious, but if $P$ is a $( n \times n)$ stochastic matrix, can I associate it with a discrete time finite markov chain?
0
votes
0answers
22 views

Markov Chain whose state space $S=\{0,1,2,\dots,m\}$

I'm studying Markov chains and I came across the following exercise and I do not know how I should approach it. We note $(P_{x,y})_{0\le x,y\le m}$ the transition of the chain. Show that there is $0&...
1
vote
0answers
22 views

General techniques for coupling a set of random variables with mutual dependence

Disclaimer: First, the usage of "coupling" in the title is not of the usual definition in probability theory. Second, cross-posting from stats.stackexchange.com. Suppose I have a set of random ...
0
votes
0answers
24 views

Entropy Rate of Markov Chains

I'm trying to compute the entropy rate of a given Markov chain using the following formula: $$H = -\sum\limits_{i,j}\pi_iP_{i,j}\log( P_{i,j.})$$ Below the pseudo-code I'm using in order to ...
0
votes
0answers
17 views

Fundamental Matrix of a Reducible Recurrent Markov Chain

I have a Markov chain with states $\{A,B,C,D,E\}$ and the following transition matrix $P$: $$ \begin{bmatrix} 0 & 0 & 0 & 0 & 1 \\ 0 & 1 & 0 & 0 &...
0
votes
0answers
54 views

Show that $Y0, Y1, Y2, \cdots$ is not a Markov chain.

Working through some problems from Introduction to Probability, Blitzstein A Markov chain has two states, A and B, with transitions as follows: Suppose we do not get to observe this Markov ...
0
votes
0answers
15 views

Markov property for mixed joint densities

Let $X \rightarrow Y \rightarrow Z$ be a Markov chain in that order, $X$ and $Y$ be jointly Gaussian, and $Z$ be a discrete random variable with finite alphabet $\mathcal{Z}$. Denote their mixed ...
0
votes
0answers
24 views

Simulation of Markov chain

I wrote this algorithm on matlab to simulate a markov chain : ...
0
votes
0answers
17 views

How to show that $X_n$ and $Y_n$ are Markov chain and how to find their transition probabilities.

The Sujata store across our campus stocks Lyril soap. He follows $(1/5)$ inventory schedule. This means, if he has $\le 1$ soaps by the closing time today, he will get some more from his godown to ...
4
votes
5answers
812 views

Simple proof of a theorem on convergence of series

Let $p_j\ge0,\ j=1,2,3,\dots,$ and suppose $\sum_j p_j=1.$ Is there a simple proof that $$\sum_{j=1}^\infty{jp_j}\tag{1}$$ converges? My question arises from the answer to this question. Consider a ...
0
votes
1answer
26 views

can someone explain me in simpler way of what this paper has conveyed

I do not have a thorough knowledge of measure-theoretic probability and Markov chain but I would start to learn by myself soon, but for few research-related works, I have to understand the theme of ...
0
votes
1answer
130 views

What is the stationary distribution of the following Markov chain?

Consider a chain with state space $\{1,2, \cdots \}.$ If you are at $1$ go to state $j$ with probability $p_j$ $($$ j=1,2,\cdots$ $) ,$ where these are non-negative numbers adding to $1$. If you are ...
2
votes
0answers
38 views

What is the difference between ergodicity and the law of large numbers?

I want to begin by saying that I know absolutely no measure theory. To my knowledge, roughly speaking a stochastic process is ergodic if its time average converges to the expectation (space average) ...
1
vote
1answer
29 views

Find the limiting probability…

I have the following problem: Two men are shooting at a target. We can call them $X$ and $Y$. $X$ shoots after each hit and $Y$ after each miss. Their respective probabilities of hitting the target ...
0
votes
1answer
16 views

Confusion about given proof of the compensated Poisson process being a Martingale?

Given the following proof of the compensated Poisson process being a Martingale Why does the proof start with $E[N(t)-\lambda t|N(s)]$ when the question asks to prove that $X(t)$ is a Martingale? ...
1
vote
1answer
51 views
+50

Steps of a Markov chain subordinated to a Poisson process

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $\left(Y^{(n)}_k\right)_{k\in\mathbb N_0}$ be a time-homogeneous Markov chain on $(\Omega,\mathcal A,\operatorname P)$ $D([0,1]):=\...
1
vote
1answer
32 views

Markov chain with expected values and time optimization

So maybe I'm approaching this completely in the wrong way, but i can't seem to grasp how to do this easily and I hope y'all can help. The problem itself, I think, is quite complex. Therefore I will ...
0
votes
0answers
14 views

Showing that simple random walks on two graphs have the same type

Question Let $G$ be a connected infinite graph of bounded degree (which means that there exists $K>0$ such that $\text{deg}(v)\leq K$ for all vertices $v$ in G). Let $G_k$ be the graph obtained ...
0
votes
1answer
14 views

What exactly does it mean to multiply a vector by the transition matrix of a Markov process?

I know that given a stationary distribution and 2 state transition matrix that $\begin{pmatrix} \Pi _{1} & \Pi _{2} \end{pmatrix}\begin{pmatrix} P_{00} & P_{01}\\ P_{10}& P_{11} \end{...
2
votes
0answers
26 views

Probabilistic argument to show that power series of submatrix of connected graph converges

Suppose I have a connected graph $G = (V,E)$, and a subset of vertices $U$, such that $W$ is the subgraph of $G$ induced by the vertices $V \setminus U$, and that $W$ is also connected. Now, let $...
0
votes
1answer
23 views

Radius of convergence of power series of sub-stochastic matrix

Suppose I have a connected graph $G = (V,E)$, and a subset of vertices $U$, such that $W$ is the subgraph of $G$ induced by the vertices $V \setminus U$, and that $W$ is also connected. Now, let $...
1
vote
1answer
38 views

Distance from root of random walk on regular tree

Let $G$ be the $k$-regular tree (a tree where every vertex has degree $k$), and let $X_n$ be a random walk on $G$ that follows the transition probabilities induced by the edge weights of $G$, which ...
0
votes
1answer
33 views

Random walk on infinite graph with zero probability of leaving subgraph

Let $G = (V,E)$ be an graph which is locally finite (every vertex has only finitely many edges - but there may be infinitely many vertices), and connected. Let $X_n$ be a random walk on $G$ that ...
2
votes
1answer
45 views

Every finite state Markov chain has a stationary probability distribution

I am trying to understand the following proof that every finite-state Markov chain has a stationary distribution. The proof is from here. Let $P$ be the $k \times k$ (stochastic) transition ...
0
votes
0answers
11 views

Limit distribution of the Unbounded Gambling with House Edge

I was presented the following example: where can I find a proof of the ergodicity of this chain and what is the value of the limiting distribution (in case it exists)?
1
vote
1answer
24 views

Probability that a random walk does not leave a subgraph after $k$ steps

Suppose I have a connected, finite, graph $G = (V,E)$, and I have some vertex set $U$ such that the subgraph of $G$ induced by the vertices $Y = V \setminus U$ is still connected. Now, suppose I have ...
0
votes
0answers
18 views

Markov Chain Example Problem: [closed]

Ted only eats dinner at Boloco or Molly’s. However, he refuses to eat at Boloco two days in a row. If he eats at Molly’s, there is an equal chance that he’ll eat at Boloco or that he’ll eat at Molly’s ...
0
votes
0answers
14 views

In a finite aperiodic Markov chain, the limiting distribution exists.

In a finite aperiodic Markov chain, the limiting distribution exists. I'm beggining with Markov chains in a course that does not show the proofs of the lemmas. Does anyone know how can I prove this ...
0
votes
1answer
5 views

Is uniqueness of steady-state vector sufficient to regular transition matrix?

If P is a transition matrix, then a steady-state vector for is a probability vector q such that $P\mathrm{q}=\mathrm{q}$. A transition matrix P is regular if some power $P^k$ contain only ...
0
votes
1answer
24 views

How to calculate probablity for approaching destination for choosing path in directed graph

[Scene:] This is the problem...here A,B,C, and D represent path connected to each other Thsi represent the directed path shown by the arrow.. [problem:] Here i have to find probablity that if ...
2
votes
0answers
34 views

Is it possible to give an intuitive idea of the notation at the base of the transition Kernel for a Markov Chain on a general state space?

Given $A \in \sigma(\mathcal{S})$, with $\mathcal{S}$ is the state space, the transition kernel is a function $K(\cdot , \cdot): \mathcal{S} \times \mathcal{B}(\mathcal{S}) \to [0,1]$ $\forall x \in ...
0
votes
0answers
70 views

Multiclass M/M/1 queue which can simultaneously have one customer per class

Suppose there is a queue with exponential service time $\frac{1}{\mu}$ which accepts customers from K classes with Poisson distribution and rate $\lambda_k$ but if a customer from any class arrives at ...
2
votes
0answers
105 views

Convergence of discrete-time Markov chain to Feller processes

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(X_t)_{t\ge0}$ be a Feller process on $(\Omega,\mathcal A,\operatorname P)$ $(h_d)_{d\in\mathbb N}\subseteq(0,\infty)$ with $$h_d\...
-1
votes
1answer
29 views

understanding iterated function system by Markov chain [closed]

The Iterated Function System at node $i$ is a discrete time Markov chain on the state space ${\cal S}_i=\mathbb{R}^d$. The chain is specified by an integer $m$ and a collection of maps $f_j^{(i)}: ...
0
votes
0answers
11 views

Choosing between non-determinism and probabilistic models

I have a stochastic system such that there are discrete states. At each discrete state, one or more probabilistic transition rules apply. For example, when Sam is in house he can go to school with a ...
1
vote
1answer
44 views

Probability it rains on random day

Problem State of the weather in city can be modeled with simple probability. After rainy day it will rain the next day with probability of $0.5$ and after sunny day it will be sunny next day with ...