0
votes
0answers
7 views

Computing the PMF for N(t) in a renewal process

I'm in a stochastic processes course, and we just started on renewal theory. Unfortunately, we skipped the section on queuing theory, and nearly example in my textbook for renewal processes uses ...
0
votes
1answer
7 views

First property of discrete time homogenous markov chain

I'm trying to understand the properties of a DTHMC. I am having trouble understanding with the first one. My textbook says - "$X_t$ takes values in $X$ for all $t$ (i.e. $X_t$ is a random variable ...
0
votes
1answer
13 views

M/M/1 queue with probability of new client leaving

I'm looking at a M/M/1 queue system and trying to show that $\{M_t\}_{t\geq}0$, the number of clients in the system, is a birth-death process. In the simplest of cases this is true if $\lambda_i = ...
3
votes
0answers
28 views

Invariant measure of Euler-Maruyama Discretisation of an Ito diffusion

Let $(X_t)_{t \geq 0}$ be a diffusion process with dynamics governed by the stochastic differential equation \begin{equation} dX_t = b(X_t)dt + \sigma(X_t)dW_t, ~~ X_0 = x_0, \end{equation} where ...
0
votes
1answer
26 views

Mean time spent in transient states/Markov chain

I dont get this in my book: For transient states $i$ and $j$ , let $s_{ij}$ denote the expected number of time periods that the markov chain is in state $j$ , given that it starts in state $i$. Let ...
3
votes
0answers
35 views

Markov chains and natural filtration

I have the following problem Consider a homogeneous Markov chain $(X_n)$ with countable state set $E$. Suppose that $A$ is a proper subset of $E$ and consider the stopping times $\tau^0=0 $ and ...
0
votes
1answer
30 views

Markov chains for group decision making

I am new to Markov chains since I am doing my own studying on it recently. I was doing some questions and came across this one that got me stuck. Suppose there are four employees and they need to ...
2
votes
1answer
51 views

continuous time Markov chain, something the book does not explain

I have a problem with something in my book, under the chapter of continuous time Markov chains. I will post a link to what the book does. They do something which they seem to take for granted, but I ...
0
votes
0answers
22 views

Transition matrix, stationary distribution and expected number

A company wants to operate s identical machines, but they are subject to failure according to a given probability law. To replace them, the company orders new machines at the beginning of each week to ...
0
votes
0answers
41 views

Expected time to reach 6th return time of Markov Chain

I'm having a hard time figuring out this problem from Resnick's Adventures in Stochastic Processes: Harry is negotiating a new tv show and the negotiations follow a discretely indexed Markov chain. ...
2
votes
2answers
71 views

Professor has 4 umbrellas, Markov chain and Probability

OK this problem is making me tear my hair out. I need someone to walk me through this in baby-steps method like 1 + 1 = 2. I am trying to figure out what I don't understand. I know this is going to be ...
2
votes
1answer
36 views

Probability distribution of Poisson process

Let $X_t$ and $Y_t$ be two independent Poisson process with rate parameter $\lambda_1$ and $\lambda_2$, respectively, measuring the number of customers arriving in stores $1$ and $2$, respectively. ...
1
vote
1answer
80 views

Markov chain problem, Help!

I am stuck on this question for a long time Question: Consider 4 balls, labelled from 1 to 4 and distributed amongst two urns (Urn 1 and Urn 2). At each time $n>1$, a number from 1 to 4 is chosen ...
0
votes
0answers
4 views

How can I calculate distribution of minima of sections of a continuous path (from a stochastic process)?

I have a long slab whose width is defined by a stochastic process, whose complete statistics I am aware of, say. I now cut it into smaller sections of uniform length, and calculate the minimum width ...
0
votes
1answer
26 views

Show $P(S_{2n}=x|S_0=x) \ge \frac{1}{N}$

Let $X_n$ be an aperiodic, discrete-time Markov chain so $S=\{1,...,N\}$ whose transition probability is symmetric. How can I show that for all $x \in S$ and all integers $n$, $P(S_{2n}=x|S_0=x) \ge ...
0
votes
0answers
107 views

understanding submartingale proof with discrete state space

I am reading a text about branching markov chains: My question is about the first half of page 8 where $Q(t)$ is proven to be a submartingale. Briefly the used notation: $t$ is discrete time, $n(t)$ ...
2
votes
0answers
31 views

markov spectral radius independent of states?

Let $\Pi$ be a stochastic matrix of an irreducible markov chain. We define the spectral radius of $\Pi$ as: $\rho(\Pi) := \limsup_{n \to \infty} \left( \pi^{(n)}_{(a,b)} \right)^{\frac{1}{n}}$ Why ...
0
votes
1answer
46 views

Proving a chain is aperiodic, and finding a stationary distribution.

We have an irreducible Markov chain with a not necessarily finite state space. It has a transition matrix $P$ such that $P^2=P$. Prove (1) the chain is aperiodic, and (2) prove $p_{ij}=p_{jj}$ ...
0
votes
1answer
17 views

Question involving an invariant measure on a Markov chain

Suppose $\mu$ is an invariant measure for a Markov chain with state space $S$ with $\mu(i)p_{ij}=\mu(j)p_{ji}$ $\forall i,j \in S$. Describe a Markov chain with this property. Also, show that $\mu$ is ...
0
votes
1answer
19 views

Transition rate matrix from transition probability matrix

If I have a $2 \times 2$ continuous time Markov chain transition probability matrix (generated from a financial time series data), is it possible to get the transition rate matrix from this and if ...
5
votes
0answers
75 views

Generating a stochastic matrix with a given second dominant eigenvalue

I need a procedure (iterative or otherwise) that, given a positive integer $N$ and a (possibly complex) number $\lambda$ such that $0 < \vert \lambda \vert < 1$, will be able to generate an $N ...
0
votes
0answers
28 views

Continuous time Markov chains, how would this definition be expanded from time-homogeneous to time-inhomogeneous.

Below I have a picture of how we can view a continuous time Markov chain that is time-homogeneous. Now, I am wondering what happens when we have a inhomogeneous continuous Markov chain. I have ...
0
votes
1answer
30 views

Markov Chain probabilities

I'm having trouble with this problem from Resnick's Adventures in Stochastic Processes: Consider a Markov Chain on states {0,1,2} with transition matrix $ \left( \begin{array}{ccc} 0.3 & 0.3 ...
0
votes
0answers
26 views

Markov renewal process vs Markov Jump process

The venerable wikipedia for "Markov renewal process" says that: "a Markov renewal process is a random process that generalizes the notion of Markov jump processes". So what's the definition of a ...
3
votes
1answer
56 views

When are stable continuous time Markov chains Feller? Always?

This is a question is similar to this 2 year-old one that never got answered (truthfully it's pretty much the same question except that I'm adding a bit more detail and the assumption that the $Q$ ...
3
votes
1answer
58 views

Sum of two Markov processes another Markov process?

Let $dX_{t} = m_1(l_1-X_{t})dt+\sigma_1 dW_{t}$ and $dY_{t} = m_2(l_2-Y_{t})dt+\sigma_2(\rho dW_{t}+\sqrt{1-\rho^2}dW_{t}^{1})$ where the $m_i$'s, $l_i$'s and $\sigma_i$'s are constants, $\rho \in ...
1
vote
1answer
87 views

Elementary proof of geometric / negative binomial distribution in birth-death processes

The birth-death process concerns a population of $n_0$ individuals, each of which reproduce and die at a constant rate as time $t$ increases from $t=0$. Each individual splits into two individuals ...
1
vote
0answers
18 views

Finite state Markov chains and expectations

Let $X_t$ be a finite state Markov chain with generator matrix $Q$. For a give function $f(x)>0$ define: $$ u(t, i) = \mathbb{E}\left[\int_t^T f(X_s) \mathrm{d} s| X_t = i\right] $$ Are there any ...
1
vote
2answers
38 views

Counterintuitive Markov chain problem

In class, my professor said that given a Markov chain $\{X_k\}$ it intuitively should be true that $P(X_{k+1} = x_{k+1} \, \mid \, X_0 = a_0, \dots, X_{k-1}= a_{k-1}) = P(X_{k+1} = x_{k+1}\, \mid ...
3
votes
1answer
51 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is ...
1
vote
1answer
16 views

Question regarding Markov chains

I have this problem: Random variables $U_1,U_2,...$ are i.i.d with the distribution, $P(0)=0.1, P(1)=0.3,P(2)=0.2,P(3)=0.4.$ Consider a new sequence $X=(X_n=X(n))$ defined as $X(0)=0, ...
1
vote
1answer
45 views

The “maximum” of a simple random walk

Suppose $S_n$ is a simple random walk started from $S_0=0$. Denote $M_n$ to be the maximum of the walk in the first $n$ steps, i.e. $M_n=\max_{k\leq n}S_k$. Show that $M_n$ is not a Markov chain, but ...
1
vote
0answers
32 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
0
votes
0answers
34 views

Probability distribution after n-steps with different initiation state in Markov chain

The transition matrix at n-th time step for a discrete time Markov chain with $ S = \{1, 2, 3, 4\} $is given as below: $$ P(n) = \pmatrix{0 & 0.6 & 0.4 & 0 \\ 0.8 & 0 & 0 & ...
0
votes
0answers
22 views

Calculation of the Gallavotti-Cohen fluctuation theorem made by Lebowitz

I have a problem understanding a calculation in this paper (another form of the theorem an be found here at equation 11). For those who want to read the paper, I have difficulties with formula 2.14 in ...
1
vote
1answer
57 views

Library chain stationary distribution

This is an exercise 1.47 from Richard Durrett's Essentials of Stochastic Processes p.85 (doi: 10.1007/978-1-4614-3615-7_1 or Google Books). On each request the ith of the $n$ possible books is the ...
0
votes
1answer
82 views

Linear Algebra Stochastic Matrix and Markov Chains

I have a few true and false questions I need help with. Can someone please check my work? The product of two stochastic matrices is a stochastic matrix. This is false I found a counterexample. 2 ...
0
votes
1answer
84 views

“Multivariate” Markov Chains

I am interested in estimating regime-switching VAR models to a regime setup I don't know the name of. I am hoping that someone can help me out with some references, or if there exists a name for it ...
1
vote
0answers
27 views

period of product markov chain

Consider $Z_n := (X_n,Y_n)$ where $(X_n)_{n\in \mathbb{N}}$ and $(Y_n)_{n\in \mathbb{N}}$ are irreducible markov chains with periods $\lambda$ and $\mu$. We know that $(Z_n)_{n\in \mathbb{N}}$ is a ...
1
vote
1answer
50 views

Equivalence Classes of a Markov Chain with Transition Matrix

I have the following transition probability matrix for a markov chain with state space S={0,1,2,3,4,5,6}: $\begin{bmatrix} \frac13 & \frac13 &0 & 0 & \frac16 & 0 & \frac16\\ ...
1
vote
0answers
22 views

Prove equilibrium theorem without irreducibility and aperiodicity

I have to solve the following question: Consider a random walk Markov chain on $S = \{1, 2, \ldots, 100\}$. If the chain is between 2 and 99, it selects one of the adjacent states with equal ...
2
votes
2answers
89 views

Wald's equation example controversy

I'm trying to get a grip of Wald's equation, applying it to the following example. Suppose, we have a simple sequence of fair coin flips, where heads wins us a dollar, while tails means loss of a ...
0
votes
1answer
97 views

Markov chains: is “aperiodic + irreducible” equivalent to “regular”?

I have two books on stochastic processes. In one book, it says that the limiting matrix is possible to find if the matrix is regular, that is, if for some $n$ $P^n$ has only positive values. The ...
0
votes
0answers
27 views

References for Hidden Markov Chains

I'm looking for some nice introductions to Hidden Markov Chains. Preferably some that begin from the basic definitions. I would like some of these references to be papers published in journals. Any ...
3
votes
1answer
32 views

Coupling between two CTMCs

Suppose I have two random processes $X(t)$ and $Y(t)$ starting at time $t=0$ and $X(0)=Y(0)=0$. The processes obey the following transition rates: $$ X(t):\begin{cases} 0\to 1,\text{at rate } A\\ ...
-1
votes
1answer
54 views

Proof that the square of a stochastic matrix is stochastic

We know that the square of a stochastic matrix is also stochastic, because the two-step transition matrix of a Markov chain is necessarily stochastic. However, in there another way to independently ...
1
vote
1answer
54 views

A question regarding Markov Chains

Is it possible that we combine some states of a Markov chain, like in this figure? (All non-zero states combined) 1) If yes, what are the new transition probabilities, i.e. p1 and p2 and p3 in the ...
0
votes
1answer
38 views

how to reformulate general markov property in discrete case

I read the wiki article on the markov property http://en.wikipedia.org/wiki/Markov_property#Definition and wondered how to work out this reformulation. It seems intuitively but I can not work it out. ...
1
vote
0answers
39 views

Two independent renewal processes

We have two urns (blue and red) that are connected, and two particles, $p_1$ and $p_2$, are traveling between these urns independently. The amount of time $Z_1$ that $p_1$ spends in blue urn is iid ...
0
votes
0answers
37 views

Is this two dimensional Markov chain correct for this queueing system?

The problem that I have two single server station with no queuing space a customer goes to station 1 if it is available else it goes to station 2 if it is available or it will be lost output from ...