# Questions tagged [markov-chains]

Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

5,776 questions
Filter by
Sorted by
Tagged with
30 views

### Is it true that $\lim_{n}p_{ij}^{(n)}=0$ for transient $j$?

For a discrete-time Markov chain that is not necessarily irreducible or aperiodic, I am attempting to show that for transient $j$ \begin{equation*} \lim_{n\to\infty}p_{ij}^{(n)} = 0. \end{equation*} ...
1 vote
80 views

16 views

### How many times does a random walk starting at vertex v return to v before first visiting vertex to u

Suppose we have a simple random walk on a graph, or possibly more generally a reversible Markov chain, that begins at vertex v and continues until vertex u is first reached. What is the expected ...
29 views

### Petite Set for Deterministic Markov Process

I am reading the book Markov Chain and Stochastic Stability by Meyn and Tweedie, and I came across the definition of petite sets that was used later in chapter 15 to derive some ergodicity result of ...
1 vote
60 views

### How does this Markov chain behave?

I encountered a specific kind of Markov chain with two parameters $\alpha,\beta\in(0,1)$. It works as follows: the variables $X_0,X_1,\dots$ live in the real interval $[0,1]$. When we have $X_n$, the ...
36 views

### Stopping Time of a Sequence of Digits

Now I have a program which will generate a sequence of digits. Each digit will output a number uniformly randomly in $\{0,1,2,3,4,5,6,7,8,9\}$. However it will never print the same digit twice in a ...
36 views

### Number of heads streaks across $16$ coin tosses with unfair coin

Here is a question I am trying to solve: An unfair coin with probability $p$ for head is thrown $16$ times in a row. Find the expectation of the number of streaks of heads. My thoughts: The result ...
43 views

15 views

48 views

### Intuition behind Penney's game

Penney's game: Player A selects a sequence of heads and tails (of length 3 or larger), and shows this sequence to player B. Player B then selects another sequence of heads and tails of the same length....
44 views

### Hitting time for integer random walk

Suppose there is a random walk on the integer line, where the value cannot go below $0$, and the process stops when it hits a barrier at value $k$. The goal is to compute the expected number of steps ...
261 views

### Markov chains: Hitting time of random walk on Sierpinski triangle

Given a Sierpinski triangle $G_n$ and a random walk on $G_n$ denoted as $(X_i)_{i\in\mathbb{N}}$, I'm attempting to prove that the hitting time $T_n$ to go from one corner to any of the other two ...
1 vote
20 views

### Separating the product of dependent samples from a Markov Chain

Suppose $$x_1, x_2,...,x_N$$ are samples from a Markov Chain $P(x_t|x_{t-1})$ with all the nice properties (unique steady state $\pi$ or any other reasonable conditions you need) Is there a nice way ...
### when transmission matrix $P(t)$ of markov process is differentiable, is stationary distribution differentiable?
"Markov chain {Xn} is inreducible and positive reccurent, the transimission function $P_{ij}(t)$ is differentiable about t for any state i,j. prove the stationary distribution $\pi$ of {Xn} is ...
Statement Let $A\in\mathbb{R}^{m\times m}$ be a (row) stochastic matrix. It is known that the eigenvalues of such matrix lies in the complex unit disk. Now I am only interested in the eigenvalues ...