# Questions tagged [markov-chains]

Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

5,280 questions
Filter by
Sorted by
Tagged with
1 vote
17 views

### Markov umbrellas

"I have $4$ umbrellas randomly distributed between my house and my office. Each day I go from my house to the office and back. If it's raining, i will take an umbrella in my way to the other ...
10 views

### Cover time of lollipop graphs

A lollipop graph is a graph with a complete graph $K_{n/2}$ connected to a path on $\frac{n}{2}$ vertices, like a shape of a lollipop. Let $u$ be the point where the complete graph and the path meets, ...
16 views

### Proof of Chapman-Kolmogorov equations for the general case

I've started studying Markov Chains, but I'm fairly new to the topic and I don't follow some of the reasonings done in the books I'm reading, especially those related with integral manipulation, For ...
1 vote
18 views

### Finding stationary distribution of random process

Suppose we are given $x_t, \bar{x_t}, t\in \mathbb{Z_+}$ independent 2-states $\{0, 1\}$ Markov chains with positive transition probabilities. Initial states are $x_0 = 0; \bar{x}_0 = 1$. For which ...
14 views

### Properties of Continuous Time Markov Chains

In school, I only learned about about "Discrete Time Markov Chains" - in Discrete Time Markov Chains, transitions can only happen at fixed time points. For example, suppose there are three ...
14 views

### Deriving Transition Probabilities of a Markov Process

I was reading this article about Markov Processes (https://www.jstatsoft.org/article/view/v066i06) and saw the following equation (4): I am trying to understand why equation (4) is true. I can ...
1 vote
18 views

### Trouble understanding identity $\mathcal{L}(X_n) = \lambda P^n$ in Nummelin's Markov Chains

I've recently started studying Markov chains and I've started reading Nummelin's General Irreducible Markov Chains and non-negative operators. I don't follow the following reasoning made in page 5. ...
1 vote
6 views

### Can we construct an equivalent markov process with newer states, some of which are combination of previous states?

Let there be a continuous time Markov chain with three possible states $C_1, C_2, C_3$, and the rate of going from configuration $C_i$ to $C_j$ be $r_{ij}$. A very simple markov chain could be such ...
1 vote
24 views

### Best way to estimate probability of heads of a biased coin

What is the best way of estimating the probability p of getting heads in a biased coin? An intuitive idea is to flip it over an over again, and look at the empirical frequency etc, MLE type ...
21 views

### Perron-Frobenius theorem for reducible non-negative matrices

Let $M$ be a non-negative matrix ($M_{ij}\geq 0$ for all $i,j$). If $M$ is irreducible, then we know that there exists an eigenvalue $\lambda$ of $M$ that equals the spectral radius $\rho(M)$ and has ...
25 views

### The expectation of event occurrence (two-state model in CTMC)

Below problem is excerpted from Stochastic Processes (2e, Ross). The solution for 5.12(b) can be found here. 5.12 Suppose that the “state” of the system can be modeled as a two-state continuous-time ...
21 views

1 vote
38 views

### Expected time in markov mouse problem

Here is the problem A mouse trap is placed in room 1 of the house with the pictured floor plan. Each time the mouse comes into room 1, he is trapped with probability p = 0.1. If he is not trapped, he ...
1 vote
20 views

### How to reorder matrix to its canonical form (Markov chain)?

Is it possible to rearrange this matrix in its canonical form (link below)? I have searched numerous websites and videos and have only found answers for small matrices where you automatically get the ...
108 views

1 vote
32 views

### Durrett's Probability: Example 5.5.14 (M/G/1 queue)

I am reading over the stationary distribution section Durrett's textbook Probability: Theory and Example, and I got stuck by a statement in an example he gave about the M/G/1 queue. Here's the problem:...
13 views

### Markov chain converges to the same steady state for different initial probability vectors.

I was asked to write a code to simulate the following Markov chain, and find the PMF of the random variable $X$: The code I've written for simulating the given Markov chain: ...
11 views

### Is Little's Law applicable to all Continous Time Markov Chain Models?

I was reading about Little's Law which is in general(infinite capacity system) form L = R*W ( R: throughput rate ,W : expected waiting time of a customer). I know it is applicable on M/M/S type ...
39 views

### Monte Carlo computation of probability of a subset of samples

I would like to compute the probability for some subset $\omega \subset \Omega$ of events to occur, i.e. $P(\omega) = \sum_{x \in \omega} P(x)$ where I know $P(x)$ for all $x \in \Omega$, which are ...