# Tagged Questions

Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

9 views

### Why does all vectors achieve steady state after multiplying by Markov matrices for infinite number of times?

what is so special about Markov matrices and how to prove this statement?
22 views

### Example of Markov property

I am reading Durrett's Probability: Theory and Examples, and trying to understand its context about Markov Chains. It is not hard to understand the theorem and proofs but when it comes to concrete ...
34 views

### proof of Markov chain Monte Carlo

This is the first step of proof of MCMC in my notes I have a question, how come $\pi(x)\pi(x_p\mid x)=\pi(x_p)\pi(x\mid x_p)$? Is it true for any markov chains which are ergodic and aperiodic? The ...
26 views

82 views

### Modified gambler's ruin problem: quit when going bankruptcy or losing $k$ dollars in all

In each round, the gambler either wins and earns 1 dollar, or loses 1 dollar. The winning probability in each round is $p<1/2$. The gambler initially has $a$ dollars. He quits the game when he has ...
35 views

### Distribution of Markov Chain at a Stopping Time

Suppose $(X_t)_{t \geq 0}$ is a Markov chain on the state space $S$ with transition probability $p$, and that $\pi$ is a stationary distribution for $p$. If $X_0 \sim \pi$, then we know $X_t \sim \pi$ ...
29 views

26 views

### Basic Limit Theorem for Markov Chain (Knowing the odds)

In the book "Knowing the Odds", Basic Limit Theorem for Markov Chain is stated as follows. Theorem 7.41 (Basic Limit Theorem). Suppose $j$ is a recurrent aperiodic state in an irreducible Markov ...
30 views

### What is the difference between a reversible markov chain and a reversible in equilibrium markov chain?

In the text I'm using it says: Let X = {$X_n : 0 \leq n \leq N$} be an irreducible Markov chain such that $X_n$ has the stationary distribution $\pi$ for all $n$. The chain is called reversible if ...
18 views

### stopping criteria for power-iteration to find rank-1 matrix

I start with B=I, A positive matrix, and compute B=(BA)/norm(B) by iterating until B is sufficiently close to rank-1 matrix. What is a good stopping criterion for this algorithm? There's Birkhoff ...
57 views

Suppose $(S_n)_{n\geq1}$ is a Markov chain on the two dimensional lattice of the integers. Then define the stopping time $\tau_A'=\inf\{n\geq1:S_n\notin A'\}$ and consider the following for $A\subset ... 1answer 26 views ### Show that it is a Markov chain, determine the transition-probability matrix and reversibility Certain machine has three possible states:$0=working,\,1=broken\,and\, awaiting\,repair,\,2=broken\,and\,being\,repaired$. The permanence times (in minutes) in each state have independent geometric ... 2answers 34 views ### Distribution of throws of die rigged to never produce twice in a row the same result A die is “fixed” so that each time it is rolled the score cannot be the same as the preceding score, all other scores having probablity 1/5. If the first score is 6, what is the probability that the ... 1answer 19 views ### Random walk mean number of visits to state before absorption This is from Stirzaker's book Random Processes. Suppose we have a simple random walk with probability going "up" p, "down" q. At time 0 it stats at 0, so $$S_0 = 0$$ Now let$u_b $be the mean ... 1answer 29 views ### Extension of ergodic theorem with WLLN Suppose you have a ergodic (or irreducible) Markov chain$(A_t)_{t\geq0}$in continuous time. Denote by$\pi$the invariant distribution of$A$. If$f$is a function of$A_s$which is integrable w.r.t.... 2answers 76 views ### Expected steps to eliminate a character? [closed] This came up in a game theory crafting exercise. Imagine a character has 195 hit points. You can shoot at them and there are three results: Critical - 100 damage - 40% of shots Body - 50 damage -... 0answers 17 views ### First time of passage, discrete random walk with disjoint absorbing regions I have a sum$T^i$of zero/one Bern$(p)$random variables$T_i$and multiple disjoint absorbing regions, i.e. the absorbing region is a union of disjoint, closed sets:$$T^i \in \bigcup_{u \in \... 0answers 12 views ### Filtering/MCMC methods for this HMM I have a Discrete HMM with hidden Markovian signals of the form$\{X_t\}_{t \in [0, \infty)} \in \{ 1,2,3\}$and observed outputs of the form$\{Y_n\}_{n \in \mathbb{N}} \in \{ 1,2\}$. Each ... 0answers 31 views ### Application Strong Markov Property I am considering a random walk$S_n$on a state space$\mathbb{Z}^d$. I want to show that$E_x\left[\sum_{n=0}^{\tau_A-1}{1_{\{S_n=y\}}}\right]=\frac{1}{P_x[\tau_A<\tau_y]}$, where$\tau_A=\inf\{n\...
I have a chain $C_t$. At every instant $t$ an exponential random variable $X_t$ with parameter $\lambda$ is added to the chain or if the chain has a value greater than $Q$ then a value $Q$ is ...