1
vote
1answer
27 views

Continuous-time Markov Question

I have a question about a continuous-time Markov process on the discrete space. I am given the generator and asked for find the expected time the Markov process needs to get back to state 3, given ...
0
votes
2answers
42 views

Find the generator of Markov Process

Homework question: Consider the Markov process $X_t=B_t-t^2+t$ where $B_t$ is the Brownian motion. Find the generator $Q$ of this process. I am completely confused how to find the generator for ...
0
votes
1answer
20 views

Queue system with queue-triggered input process

I have a queue system, a classic system with an input generator, a queue and a servant. The servant is a $M$-servant with a certain serving rate $\mu$. The queue can contain an infinite number of ...
0
votes
1answer
65 views

Exercise on Markov chain

Prove, or give an explicit counterexample to refute, the following assertion: if $\{X_n\}$ is a Markov chain, then $\{X_n^2\}$ is also a Markov chain. It's easy to show that ...
2
votes
1answer
95 views

Markov chain problem with finite states

Consider any Markov chain on a state space with exactly $r$ states. We want to find the largest $N>0$ such that there exists states $i,j$ for every $n < N$ we have $p_{ij}^{(N)} > 0$ and ...
1
vote
0answers
32 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
0
votes
1answer
40 views

Poisson Process (Easy)

I'm stuck at the following question: Customers with items to repair arrive at a repair facility according to a Poisson process with rate λ. The repair time of an item has a uniform distribution on ...
0
votes
0answers
37 views

Is this two dimensional Markov chain correct for this queueing system?

The problem that I have two single server station with no queuing space a customer goes to station 1 if it is available else it goes to station 2 if it is available or it will be lost output from ...
0
votes
0answers
58 views

transition matrix for Markov chain

Can any one help me to solve this home work please? The city of Sacramento recently completed a new light rail system to bring commuters and shoppers into the downtown area and relieve freeway ...
1
vote
1answer
44 views

Estimate the probability using Markov chains

please consider this question: A study using Markov chains to estimate a patient's prognosis for improving under various treatment plans gives the following transition matrix as an example ...
0
votes
1answer
82 views

Markov Chain discarding balls from urn

The following question has me stumped. Any ideas on how to get started? An urn contains $n$ green balls and $n+2$ red balls. A ball is picked at random: if it is green then a red balls is also ...
0
votes
1answer
46 views

Showing a certain random process is a Markov Process

I have the following example of a random process: A person has two houses, house A and house B in which he can stay, we denote by $X_{i}\in\left\{ A,B\right\}$ the house he stayed in on the i-th day ...
1
vote
3answers
111 views

How to prove the existence of the limit of Markov transition matrix?

Does the limit of a Markov transition matrix $M$: $$\lim_{n\to\infty}M^n$$ always exist? And if yes, how to prove it?
4
votes
1answer
334 views

Recursively Solving a Bellman Equation

Problem Overview I am to figure out $v_\pi$ of a certain Markov state. Given Information A set of actions, $a$ containing ${up, down, left, right}$ $v_\pi(12), v_\pi(13), v_\pi(14)$ (I am given ...