# Tagged Questions

Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

29 views

### Stationary distribution vs invariant distribution of a Markov chain

Lets $p$ be a distribution on a finite sample space with $n$ points. I wish to find a transition matrix that is invariant with respect to $p$, that is $$p^T T = p^T$$. The problem is clearly ...
42 views

### Markov dynamic programming recursion

I'm learning Markov dynamic programming problem and it is said that we must use backward recursion to solve MDP problems. My thought is that since in a Markov process, the only existing dependence is ...
40 views

27 views

### M/M/1 queue derivation: how to “recursively solve in dependence on $p_0$”

I want to sketch out the derivation of the equations for an M/M/1 queue for a presentation I'm giving. I can understand most of the derivation from Willig but I don't understand this section from p10 ...
92 views

### What is the difference between “state-transition-matrix” and a transition matrix?

What's the difference between a state-transition-matrix and a transition matrix (say, for an ergodic Markov Chain) that is typically taught in a basic probability theory course? This is the first ...
Consider a Markov chain $X$ on the positive integers where for each $n$: $$n\longrightarrow 1,\;2,\;3\;\dots \;n,\;n+1$$ with equal probability, and $n\longrightarrow m$ with zero probability if $m>... 1answer 17 views ### Random Walk Markov Chain Long run distribution In the question above do I have to calculate the stationary distribution? I've been learning about the ergodic theorem but I'm not sure if it's applicable here. I know that the probability that Xn = ... 2answers 37 views ### Expected time between successive visits in a Markov Chain? This is a pretty basic question and I know the answer is probably really obvious, but I am having trouble reasoning as to why the following is true: (From my lecture notes): """ Expected time ... 1answer 30 views ### Transition matrix and communicating classes Firstly I wanted to check if they have a mistake in the solution. So for the transition matrix, the element p11 should be 0 and p12 should be 1, but they have it the other way round so I just wanted ... 1answer 36 views ### Constructing transition graph from transition matrix Ok so for this question I'm having trouble understanding how the transition graph has been drawn from the given transition matrix. This is what I understand and hopefully someone can correct the flaws ... 1answer 32 views ### Computation of n-step transition matrix : method of matching coefficients For the third step I don't understand how they have worked out$C_1^1=0? where did they get the value of 0 from? p11(n)=(c01+c11n)lamda 1^n. Using p11(1)=1-a and c01=1, n=1,lamda 1=1 and lamba 1^1 =... 2answers 41 views ### Soft Question - book recommendation - Stochastic Processes My mother language book on stochastic processes is pretty much complete(~500 pages) but would like another one in English, to have in my library. I'm looking for a similar book containing the ... 0answers 14 views ### How to determine the transition probability in Sequential Importance Sampling (SIS) for Particle Filter Given a state-space model \begin{align} x_k &= f_k(x_{k-1}, v_{k-1}),\\ z_k &= h_k(x_k, w_k), \end{align} wherex_k \in {\mathbb R}^{n}$and$y_k \in {\mathbb R}^{m}$are the system state ... 1answer 24 views ### Markov's matrix into stationary Distribution [closed] How do I know if this Markov's Transition Matrix converges into a stationary distribution? $$P= \begin{bmatrix} .8 & .2 & 0 \\ .3 & .4 & .3 \\ .2 & .1 & .7 \\ \end{bmatrix}... 1answer 50 views ### Construction of continuous-time markov chain and finding stationary distribution There are 15 lily pads and 6 frogs. Each frog, with rate 1, jumps to one of the other 9 unoccupied pads chosen uniformly at random. What is the stationary distribution for the set of occupied lily ... 1answer 65 views ### Prove that Y_n=X_{n-1}X_n is a markov chain Let \{X_n\}_{n=0}^\infty a sequence of discrete random variables independent identically distributed. Let Y_n such that Y_n=X_{n-1}X_n for all n\ge 1 Is \{Y_n\}_{n=0}^\infty a markov chain?... 2answers 45 views ### Find the general expression for the values of a steady state vector of an n\times n transition matrix I have a question that is asking to find the values of the elements in the steady state vector for a regular transition matrix P of size n \times n. All I'm given is that the the elements in each ... 1answer 24 views ### What is the (practical) difference between a stationary distribution and an equilibrium distribution of a MC? I know that, for a Markov Chain, a stationary distribution is the (row) vector \pi such that \pi \cdot P = \pi, where P is the one-step transition matrix for the MC. Intuitively, I assume that ... 1answer 68 views ### prove homogeneous markov chain Y_0, Y_1,Y_2,\dots are independent and identically distributed random non-negative integer outcomes. Let X_0 = Y_0 Let X_0 = Y_0 and X_n = X_{n-1} - Y_n if X_{n-1}>0, else X_n = X_{n-1}... 0answers 21 views ### Exercises on the following topics on Markov Chains We are being taught the following topics in Markov Chains: 1) Markov Chain Monte Carlo: Hard Core model, Counting random q-colourings of a graph 2) Total variation distance for a Simple Symmetric ... 1answer 40 views ### Transience, recurrence and null recurrence of markov chain I am trying to get an intuition for transience, recurrence and null recurrence. I constructed an example MC for myself represented by this graph below: I'm thinking that all of the states are ... 0answers 37 views ### Defective Markov transition matrix and relation to its limiting distributions Im trying to come to grips with what the physical interpretation of a non diagonalisable Markov Matrix means in terms of what we can deduce about it having a limiting distribution/ what potential ... 1answer 49 views ### Leaving time of a set I want to prove the following result. Let S_n be a symmetric irreducible random walk on the integers (d=dimension). Claim: If x\in A and P_x(T_A=\infty)>0 then \forall \epsilon>0\exists ... 0answers 34 views ### Variation on the classic ABRACADABRA problem Suppose letters are chosen randomly from the alphabet, one at a time. It is well known that the expected time until ABRACADABRA is spelled out is 26^{11}+26^{4}+26. The proof uses discrete time ... 2answers 66 views ### Renewal process problem, where X_i's are i.i.d. with exponential distribution. A room is lit by 2 bulbs. Bulbs are replaced only when both bulbs burn out. Lifetimes of bulb's are i.i.d exponentially distributed with parameter λ=1. What fraction of the time is the room only ... 2answers 20 views ### Attitude true for Markov Chains(Maybe duplicate) Suppose, that we have a Markov-chain with finite domain. For all i,j elements P_{i,j}>0, where P is our matrix. Show, that reversible(sorry, I forgot that out) stationary distribution exists ... 2answers 60 views ### Prove that i is an accesible state in a markov chain Let i be a recurrent state of an homogeneous markov chain such that the state j is accesible from i (that is \exists k\ge 1 such that p_{ij}(k)>0) Prove that i is accesible from j ... 0answers 43 views ### Computing hitting times from the stationary distribution My question is whether it is possible to compute hitting times from the previously calculated stationary distribution \pi of a continuous-time Markov process (X_t)_{t \geq 0}. I know that, from ... 2answers 39 views ### Mean number of tosses of a fair dice to get a sum of outcomes being a multiple of 5 Let S_n denote the sum of the outcomes of the n tosses of a fair dice. Let T=\inf\{n>0: S_n is a multiple of 5\}. Compute E(T) (by means of markov chains). Attempt. Instead of ... 0answers 23 views ### How is the following attitude true?(Markov chains) Let us have an X_n Markov-chain with finite S set as domain. A \subset S is given, so that P_x(T_A < \infty) > 0 for all x \in S. T_A=\inf \{ n \geq 1: X_n \in A\}. Then we take a ... 1answer 24 views ### Mean number of steps needed to reach a recurrent state in a finite irreducible Markov chain Let \mathbb{X} be a finite state space of an irreducible markov chain \{X_n\} and let T_x=\inf\{k\geq 0\mid X_k=x\} be the number of steps until \{X_n\} reaches state x\in \mathbb{X}. True ... 1answer 49 views ### Markov chains and queues I do not understand how may I use the Markov Chain Y and and describe the system X using the states that the exercise suggest. I was searching queue's examples and -i understand this is a M/M/1 ... 0answers 26 views ### Binary Hidden Markov Model Consider a binary HMM with 2 observed variables O_n \in \{0,1\} \; \forall n \in \mathbb{N}. Suppose that the hidden Markov process X_n is characterised by a known transition probability matrix ... 2answers 40 views ### Continuous time Markov Chain's Natural Filtration Given a continuous time Markov chain \left(X_t \right)_{t\geq 0} with finite or countable state space S, transition matrix P(t), what I want to prove is:$$\text{Let} \quad f:S \to \mathbb{R} ... 1answer 39 views ### Connection between Ergodic Theory and Markov Chains Could someone suggest a good reference where the connection between Ergodic Theory and (ergodic) Markov Chains is nicely explained ? 0answers 25 views ### Distribution of states given observations in HMM Suppose you have an HMM with two states$(S_1, S_2)$and two observations$(a, b)$. We know the following:$P(S_1|S_1) = 0.5P(S_1|S_2) = 0.25P(a|S_1) = 0.25P(a|S_2) = 0.5$Initial state at ... 1answer 54 views ### Markov chain - distribution of probability of state at generic step Let$S$be a finite discrete state set. Let$X(i) \in S, i = 1,2, \ldots$be a random variable sequence. I've built-up a Markov transition matrix from a set of sequences of states. State$s_1 \in S$... 0answers 21 views ### probability a markov chain leaves a transient state in favor of a recurrent state Let$\{X_n\}$be a time-homogenuous markov chain, with state space$\mathbb{X}$and transition matrix$P$. Let$C_1$be a transient class and$C_2$be a recurrent class and let also$x\in C_1,~y\...
Let $X_1,X_2,\ldots$ be an iid sequence such that $P\{X_1 = 1\} = p$, $P\{X_1 = -1\} =p$ and $P\{X_1 = 0\} = 1-2p$. We have that $E[X_1] = 0$ and $E[X_1^2] = 2p$. Define $S_n = \sum_{i=1}^nX_i$ and \$...