# Tagged Questions

14 views

### Queue system with queue-triggered input process

I have a queue system, a classic system with an input generator, a queue and a servant. The servant is a $M$-servant with a certain serving rate $\mu$. The queue can contain an infinite number of ...
56 views

### Markov Chain: classify states of finite Markov chain

I can easily see the states of this MC, recurrent and transient if I graph them, but how do I prove that a state is recurrent or transient. My book refers to probability to ever returning to state $j$ ...
24 views

35 views

### Sub-Markov Chain

In Raymond Yeung's book "Information theory and network coding", Proposition 2.10 states that any subchain of Markov chain $X_1\rightarrow X_2\rightarrow \cdots \rightarrow X_n$ also forms a Markov ...
24 views

### period of product markov chain

Consider $Z_n := (X_n,Y_n)$ where $(X_n)_{n\in \mathbb{N}}$ and $(Y_n)_{n\in \mathbb{N}}$ are irreducible markov chains with periods $\lambda$ and $\mu$. We know that $(Z_n)_{n\in \mathbb{N}}$ is a ...
34 views

### Is this two dimensional Markov chain correct for this queueing system?

The problem that I have two single server station with no queuing space a customer goes to station 1 if it is available else it goes to station 2 if it is available or it will be lost output from ...
48 views

### transition matrix for Markov chain

Can any one help me to solve this home work please? The city of Sacramento recently completed a new light rail system to bring commuters and shoppers into the downtown area and relieve freeway ...
40 views

### Estimate the probability using Markov chains

please consider this question: A study using Markov chains to estimate a patient's prognosis for improving under various treatment plans gives the following transition matrix as an example ...
46 views

### Factor graphs: HMM

A friend of mine and me are struggling for a while now on how to start with this example that we have to work out. There is a 10x10 map which an agent is randomly placed on. The single tiles of ...
37 views

### Markov chain with an absorbing state to which all other states lead

The question is Let $X$ be a Markov chain containing an absorbing state $s$ to which all other states lead (i.e. $j \rightarrow s$ for all $j$). Show that all states other than $s$ are transient. ...
26 views

### Finding Steady state using markov chains. Am I right?

Suppose that there are two doctors in a country town, Dr Black and Dr White. Each year, 13% of patients move from Dr Black to Dr White, while 19% of patients move from Dr White to Dr Black. Suppose ...
29 views

### Discrete Time Markov Chain Probability Question

I just wanted clarification for the probability in a DTMC. I know this conditional probability with 3 variables if S = {a,b,c}: $$P(X_1=a,X_2=b|X_3=c) = P(X_1=a|X_2=b,X_3=c)P(X_2=b|X_3=c)$$ but ...
77 views

### Markov Chain discarding balls from urn

The following question has me stumped. Any ideas on how to get started? An urn contains $n$ green balls and $n+2$ red balls. A ball is picked at random: if it is green then a red balls is also ...
88 views

### expected life absorbing Markov Chain

No idea on how to start this question. Any help would be much appreciated. A ﬂea lives on a polyhedron with N vertices, labelled $1, . . . , N$. It hops from vertex to vertex in the following manner: ...
34 views

### Markov Chain Converging in Single Step

I have a markov kernel K. From this I find the invariant probability $\pi$. The question is to design a "dream" matrix K*, that converges in one step. Such that $\lambda_{SLEM}=0$ (second largest ...
47 views

### Discrete Hidden Markov Model Steady State

Excuse me if 'steady state' is the incorrect term - it probably is because Google does not give me a clear answer. For context - this is an assignment that asks that we write a HMM sampler that ...
43 views

### Question about Infinite Markov chains

Do 2 Markov chains $\left\{X_n\right\}^\inf_{n=0}$ and $\left\{Y_n\right\}^\inf_{n=0}$ with all of these properties exist so that the probability for infinite n values to maintain $X_n=Y_n$ is 0? ...
201 views

### Question about Markov chain derived from a Poisson process

Let $(N_t)$ be a Poisson process of rate $\lambda$. Deﬁne $$X_n = N_n − n,\quad\text{for }\; n = 0, 1, 2, \ldots$$ Explain why $(X_n)$ is a Markov chain and give its transition probabilities. Using ...
801 views

### Prove Markov Chain by definition

I came across this problem in homework: $X_n$ are i.i.d random variables with $\mathbb{P}[X_n=1]=\mathbb{P}[X_n=-1]=\frac{1}{2}$ and we we also have $S_n=X_1+...+X_n.$ Show that $S_n'=|S_n|$ is a ...
397 views

### Transition Probability Matrix and Stationary Distribution

Suppose that a communications network transmits binary digits, $0$ or $1$. A message may pass through several links on its way from source to destination, with the possibility of a transmission error ...
73 views

### Markov chain property

Suppose $\{Y_{n}, n \ge 0\}$ is a Markov chain consisting of $N$ states. Suppose that $i$ and $j$ are states of this Markov chain and that $i \hookrightarrow j$, i.e state $j$ can be reached from ...
93 views

### Markov chain from Poisson

Let $K_t$ be a Poisson process with rate $1$ and $X_n=K_n-n$ $, \ \ \ n\in \mathbb{N}$ am asked to determine whether it is null or positive recurrent, we already know it is recurrent. I ...
272 views

### Return time of a markov chain

I'm having trouble deriving the return time for a Markov chain. The graph has $n$ vertices and is connected by $n - 1$ edges. So we can draw this as a horizontal line of nodes with node $1$ all the ...
230 views

### Finding the steady state Markov chain?

I have drawn a certain Markov chain with a weird transition matrix. Here's the drawing: And here's the transition matrix: My problem is that I don't quite know how to calculate the steady state ...
104 views

### How to prove the existence of the limit of Markov transition matrix?

Does the limit of a Markov transition matrix $M$: $$\lim_{n\to\infty}M^n$$ always exist? And if yes, how to prove it?
105 views

### Mean Duration of Stochastic/Markov Game

An urn contains five red and three green balls. The balls are chosen at random, one by one, from the urn. If a red ball is chosen, it is removed. Any green ball that is chosen is returned to the urn. ...
180 views

### Markov Chains Probability

A Markov chain $X_0$, $X_1$, $X_2$, ... has the transition probability matrix $$P = \left[ \matrix { 0.3&0.2&0.5 \\ 0.5&0.1&0.4 \\ 0&0&1 } \right]$$ and is known to ...
57 views

### Convergence of an Ergodic process

I'm having trouble working through the math of the problem below. I believe the problem as described is an ergodic process. I've written a simple simulation of the problem, that converges to 66.6...6% ...
117 views

### how to calculate limit of P(Xn = j | X0 = i) in markov chain?

1. in http://robotics.eecs.berkeley.edu/~wlr/126/w12.htm lim N ® ¥ [1{X1 = j} + 1{X1 = j} + … + 1{XN = j}]/N = 0 What is N? how it limit to zero? and what do 1 in 1{X1 = j} represent? /N must be ...
23 views

### how to be sure that from one state to another state

http://robotics.eecs.berkeley.edu/~wlr/126/w12.htm when you draw this graph how can you sure that state go from 1 to 2 is 100%? look at first example, there is a p and q is it probability from 1 to ...
39 views

### What is number coming from in $\gcd$ of Markov chain topic?

For example for [6], $d(1) = \gcd\{3, 5, 6,\, ...\} = 1$. What do $3,5,6$ calculated from?
58 views

### Markov Chain: Solidarity theorems

When a chain is irreducible (so each state can be reached from every other state, eventually), we quote that all states have the same character: all aperiodic / periodic with the same period, all ...
224 views

### Renewal Processes for Uniform and exponential Distributions

Suppose the lifetime of a component Ti in hours is uniformly distributed on [100, 200]. Components are replaced as soon as one fails and assume that this process has been going on long enough to reach ...
399 views

### Branching Process Extinction Probability

I'm doing a branching process problem and am not sure I did it correctly. Suppose $X_0 = 1$ and $p_0 = .5, p_1 = .1,$ and $p_3 = .4$ represent the probabilities that zero, one, and three individuals ...
479 views

245 views

### Find stationary distribution decomposable Markov chain

Again a probability exercise: Let $X=U \cup V$ be the finite state space of a Markov chain, where $U$ and $V$ are disjoint subsets of $X$ and $p_{ij}=0$ if both $i,j \in U$ or both $i,j \in V$. ...
67 views

### Demonstrate that is a Markov Chain

I've got a box with 10 balls inside, 5 reds 5 blacks. Every step i take a ball. If it is black i hold it out, if it is red i put all the blacks ball that are out and the red one inside the box. Called ...
204 views

### Markov chain $(X_n)$ has $X_n \rightarrow \infty$ a.s

I have the following homework problem: Let $(X_n)_{n \geq 0}$ be a Markov chain on the state space $\lbrace0,1,...\rbrace$. Writing $p_i := p_{i,i+1}$ and $q_i := p_{i,i-1}$, the transition ...
490 views

### Gambler's ruin (calculating probabilities--hitting time)

Im meant to produce the transition matrix which I've already done (in the picture) and list the communication classes. But Im not sure how to find the probability regarding the hitting times (see ...
Let $\mathcal{X}=(X_n:n\in\mathbb{N}_0)$ denote a Markov chain with state space $E=\{1,\dots,5\}$ and transition matrix ...