Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-chains]

Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

1
vote
0answers
9 views

can a Markov chain have a periodic and transient state?

I want to say that in a Markov chain it is not possible for there to exist a state that is both transient and periodic. Here are the definitions I am working with. Let $P$ denote the transition ...
0
votes
0answers
6 views

Markov Chain generated by iterations of functions

$X_n$ is a Markov Chain on (..,-2, -1, 0, 1,..) obtained by random iterations with functions $f_1(x)=x+2$, $f_2(x)=x−1$, $f_3(x)=0$. In each iteration step we choose function to iterate with equal ...
1
vote
1answer
19 views

Sufficient condition for a Markov chain to be Aperiodic

If I want to prove that a Markov chain is aperiodic, then if I can show that $P(X_{n+1}=i\mid X_n=i)\gt 0$ $ \forall i$. Then can I say that the chain is aperiodic?
0
votes
0answers
11 views

Existence of recurrent and transient classes in general state Markov chains

For Markov chains defined over countably finite-state spaces, it has been proven that recurrent and transient classes exist [Durrett, Richard, "Essentials of stochastic processes",Springer,1999]. I ...
1
vote
1answer
34 views

Proof of property of Markov chain

Given markov chain $X$, how to prove following property by markov property: $$P(X_{n+1} = s | X_{n_1} = x_{n_1}, X_{n_2} = x_{n_2}, ...,X_{n_k} = x_{n_k} ) = P(X_{n+1} = s | X_{n_k} = x_{n_k}) , \quad ...
0
votes
0answers
11 views

Aperiodic but not irreductible Markov Chain

If I understood well, a Markov Chain with state space $E$ is said to be irreductible if for all $x,y\in E$ there is $n$ such that $$P^n(x,y)>0,$$ where $P$ is the transition matrix. Also, I know ...
0
votes
1answer
27 views

Calculate $EX_{\tau}$ where $\tau=[inf\space n: \space X_n=2 \space or\space X_n=3]$

I came up with this task myself so it might be blurry, actually I changed a bit another exercise which was easy, but I'd like to know the way of coming up to an answer if it was like that. Let $X_n$ ...
1
vote
0answers
12 views

Markov process intensity matrix

$X$ is a Markov process with state space $(1,2,3)$. How can I find the matrices of transition probabilities $P(t)$ if the generator is \begin{bmatrix}-2&2&0\\2&-4&2\\0&2&-2\end{...
0
votes
0answers
23 views

Showing $f_n(f(e^{-C_{n+1}^{-1}})^z) = f_{n+1}(e^{-zC_{n+1}^{-1} })$ where $f_n$ is the pgf of a Galton-Watson process $Z_n$

I want to show $$f_n(f(e^{-C_{n+1}^{-1}})^z) = f_{n+1}(e^{-zC_{n+1}^{-1} })$$ In this case $Re(z)\ge 0$, $(C_n)$ is a sequence of constants and $f_n$ is the probability generating function of $Z_n$. ...
0
votes
1answer
31 views

What is the probability that the pawn will be at the origin after 2n moves

Let's say we are on two-dimensional lattice. The pawn can move in every direction by length one. (it can move up, down, right, left or diagonally with equal probabilities) What is the probability that ...
1
vote
0answers
27 views

Computing a Diffusion Limit of a Markov Chain

Fix $\alpha >0$, and for $h > 0$, consider the Markov kernel $K_h$ derived by composing the following two `moves': From $x^t$, move to $x^{t+1/2} = x^t + y^t$, where $y^t \sim \text{Gamma}(h, 1)...
3
votes
1answer
53 views

Is $Y_n=f(X_n)$ a Markov chain, when $X_n$ is?

Let $X_n$ be an independent Markov chain which has values (states) $X_n={0,1,2}$ with its transition matrix. $$\\p= \begin{pmatrix} 0 & \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & \...
4
votes
2answers
548 views

King on reduced chessboard $2\times 2$ moving randomly, what is the probability that it ends up in one of the corners after $1000$ moves?

As mentioned in the title, we have a chessboard $2\times2$, the king moves with equal probability to each square on the chessboard. King begins from the left upper corner. What is the approximate ...
1
vote
1answer
64 views

Typo in Ross' Introduction to Probability Models 11th ed p. 205?

possible typo I have problems understanding the text marked in red. Is this supposed to say something else? The syntax just seems off to me.
1
vote
1answer
31 views

How many steps are required for the pattern 0-0-1-1..

So I'm doing this exercise where it is asked to find the number of steps in order to get the pattern 0-0-1-1 A sequence of 0s and 1s is generated by a Markov chain with transition matrix: $$ P= \...
0
votes
2answers
21 views

Discrete-time Markov Chain; $n$-step transitions

Let $\{X_{n}\}_{n\geq0}$ be a discrete-time Markow chain on the state space $S=\{1,2,3\}$ with transition matrix \begin{pmatrix} 1/3 & 1/3 & 1/3 \\ 0 & 2/3 & 1/3 \\ 2/3 & 1/...
0
votes
1answer
12 views

On distribution of the time between two consecutive events (either arrival or departure) in an M/M/1 queueing system

For an M/M/1 queueing system, distribution of the time ($t_e$) between two consecutive events (either arrival or departure) can be derived as follows with the independent assumption, $$F(t_e\ge t)=F(...
-1
votes
2answers
32 views

How to prove a state is recurrent in a Markov Chain?

Examine the nature of the states of the Markov chain having transition probability matrix link Here is my approach so far. link What I did was follow the technique mentioned here. link However, I ...
0
votes
0answers
16 views

Stronger convergence of Markov chains and non-dissipative property

Let $P$ be the transition probability matrix of a Markov chain on $\mathbb{N}$. Then $P$ acts as a bounded operator on $\ell^1$ by $\mu \mapsto \mu P$ for $\mu \in \ell^1$. Identify $\ell^1$ with the ...
1
vote
0answers
16 views

Linear ODEs system with Markov-switching coefficients

I have the following system of ODEs $$d\mathbf{x}\left(t\right)=A\left(s\left(t\right)\right)\mathbf{x}\left(t\right)dt$$ where $s\left(t\right)\in\left\{ 1,2\right\}$ is the state of the system which ...
1
vote
0answers
18 views

Third Moment of Hitting Time

We recently went through expected hitting times of markov chains in my class and were asked about computing the various moments of hitting times. As such, I'm wondering if my thinking is correct as ...
0
votes
0answers
5 views

forward sampling for Bayesian network with continuous variables and equation based causal relationship

I have a physical system which can be represented by the following Bayesian network. It has the following characteristics The encoded variables are continuous variables. The causal relationships ...
0
votes
0answers
14 views

Counting measure for Markov chains

My question concerns whether there exists an expression for a counting measure on the "path space" of Markov chains. As an analogous example, consider that the probability mass function for a ...
0
votes
1answer
77 views

A finite state space Markov chain has no null-recurrent states

I'm fairly new to Markov chains. At the moment, I'm trying to understand why a finite state space Markov chain cannot have any null-recurrent states. Searching on math.SE, I found this answer, which ...
0
votes
0answers
13 views

Converse to Spectral/Sobolev Mixing Time Bounds?

There are classic upper bounds on mixing times in terms of the log-sobolev constant or the inverse spectral gap so I am wondering: Are there any results in the converse direction, e.g. that fast ...
0
votes
0answers
19 views

Notation for Markov chain

I am reading this paper. However, I don't understand the Markov chain notations(my go to book (Mathematical Notation: A Guide for Engineers and Scientists) sadly doesn't contain anything about Markov ...
0
votes
2answers
63 views

Markov Chain: Optimal stopping to determine the price at which stock is traded

The stock price starts at 100\$. At any given time, there is 50% probability that stock price increases further by 1 and 50% probability that stock price goes back to 100\$. You are paying 1\$ to ...
1
vote
0answers
37 views

What is the hitting probability $f_{ij}$ where $i=j$?

E.g. What is $f_{22}$? Intuitively I just think this is simply equal to $1$. Surely the probability of hitting $2$ is certain if I'm already at $2$. I'm not sure how to justify this rigorously though,...
0
votes
0answers
18 views

Monte Carlo Markov Chain Line fitting

I am working on creating a Metro hastings MCMC simulation to fit a line so that I can learn more about MCMC by building one and learn more about statistics. My confusion: I am having trouble wrapping ...
0
votes
1answer
27 views
0
votes
0answers
14 views

Interpretation of the invariant measure for an irreducible recurrent Markov chain

In An introduction to stochastic modeling, 4th edition; Pinsky and Karlin, on page 172 the authors give two interpretations of the limiting distribution. My question is about the second one, which is ...
0
votes
0answers
15 views

Does this linear system have a unique solution?

$x \in R^n$, $P \in R^{n \times n}$, $P$ is a known stochastic matrix (each row sums to 1), $b \in R^{n}$ is a known non-zero vector. $e = [1, \dots, 1]^T \in R^n$, we have the linear system: $x = P^...
-1
votes
0answers
24 views

Finite Markov Chain: Distribution and Expected Value

The complete graph on {1,...,N} is the simple graph with these vertices such that any pair of distinct points is adjacent. Let Xn denote simple random walk on this graph and let T be the first time ...
2
votes
0answers
34 views

Mean time to absorption for finite general birth death process with one absorbing state

I have a continuous birth-death chain $\{X(t)\}$ on state space $\{0, 1, \dots, n-1, n\}$, with birth rates $\lambda_i$ and death rates $\mu_i$, where state 0 is absorbing ($\mu_0 = \lambda_0 = 0$), ...
0
votes
0answers
16 views

Has the Gaussian Process the Markov Property only when is variance is a diagonal matrix?

I'm studying stochastic process and Markov Chain. I was wondering if a Gaussian Process has the Markov Property (that is the conditional probability distribution (given the present states) of future ...
0
votes
0answers
9 views

what is the role here of $\mu$

consider a Markov chain moving around in $(S,\rho)$ a complete, separable metric space, according to the following rule: Starting from $x\in S$, the chain chooses $f\in X$ at random from $\mu$ and ...
0
votes
0answers
28 views

quadratic variation of the solution to the martingale problem

Solution to the martingale problem for continuous-time Markov process $(X_t)$ with generator $\mathcal{L}$ is $$M_t=f(X_t)-f(X_0)-\int_0^t\mathcal{L}f(X_s)ds,$$ given any $f$. What I want to show ...
0
votes
1answer
15 views

If a transition matrix $A$ is regular, prove that $A^∞$ has all the same columns and that the columns are the steady state vector.

I know that this is quite an elementary theorem, but I have yet to see a proof of this except for quoting that $A^∞$ is going to be of rank 1, so that all the columns must be the same. Any help is ...
0
votes
0answers
17 views

Constructing Markov chains

Can anyone explain what is going on in the below proposition, please? *Uniform Representation of a Random Variable. Let $\alpha$ be a prob- ability measure on $S = \{0,1,...\}$, and let $U$ be a ...
1
vote
0answers
21 views

Markov Chain — Transient or Recurrent class?

Specify the classes of the Markov chain, and state whether they are recurrent or transient. \begin{align*} P_4= \begin{bmatrix} \frac{1}{4} & \frac{3}{4} & 0 & 0 & 0\\ \...
1
vote
1answer
51 views

Definition of Markov chain

I'm getting confused about how to correctly state the definition of Markov chain verbally. The formal definition of Markov chain is that $P(X_n|X_{n-1}, ..., X_1) = P(X_n|X_{n-1})$ According to ...
1
vote
0answers
37 views

How to find the probability for Markov chain discrete time parameter?

SO I just study the Markov chain which seems to be very powerful tools of what of have learned, So I got the Idea Markov chain only require the repetitive event like if today sunny will tomorrow sunny ...
0
votes
1answer
19 views

Markov chain. Is steady state a scaled eigenvector of transition probability matrix

So suppose we have transition matrix P for a Markov chain and suppose it satisfies the relevant criteria so that $$ \lim_{n\rightarrow \infty} P^{(n)} = \pi $$ is well behaved and is some steady ...
0
votes
1answer
16 views

How to formulate a queueing model where riders and drivers arrive randomly and are matched at every specific interval?

I am going to develop a queueing model in which riders and drivers arrive with inter-arrival time exponentially distributed. All the riders and drivers arriving in the system will wait for some ...
-1
votes
1answer
23 views

Why the following set of eigenvalues will not work for any allowed values of a and b? [closed]

Following markov chain has transition matrix: $ \begin{pmatrix} 0.7&a&0.3-a\\b&0.5-b&0.5\\1&0&0\end{pmatrix} $ Explain why the following set of eigenvalues is impossible for ...
2
votes
0answers
25 views

Ordering of Markov chain transition matrices

Consider a finite-state, discrete time Markov chain $X$ with transition matrix $M$. If $M$ is the identify matrix then $X$ stays forever in the same state. Is there a metric for how far a general ...
-1
votes
0answers
40 views

Markov Chain transition probability matrix

A Markov Chain has the following transition matrix: My problem is to calculate $P(X_n=j$ for some $n\geq1|X_0=i)$ for any $i, j\in S$, where $S$ is the state space $S=(1,2,3,4,5,6)$. What do they ...
0
votes
0answers
17 views

Two views on continuous Markov chains

The following is a quotation from a computer science course I'm taking which uses continuous markov chains. In the lecture, we switched between two views on CTMCs. First, waiting in a state for ...
0
votes
1answer
33 views

Calculating return time / showing null recurrence of discrete Markov chain

Recently I have been attempting a question on continuous time Markov chains, and one of the parts prompts me to verify the null recurrence of a CTMC's jump chain, which is a discrete Markov chain. ...
-1
votes
0answers
18 views

What exactly is a stationary distribution of a Markov Chain

I understand the mathematical definition of $ \pi P=\pi $, but exactly does this distribution signify? Let's say I have a 3x3 block, and I can only progress to the left, right, top and bottom, with ...