1
vote
1answer
22 views

Continuous-time Markov Question

I have a question about a continuous-time Markov process on the discrete space. I am given the generator and asked for find the expected time the Markov process needs to get back to state 3, given ...
0
votes
2answers
39 views

Find the generator of Markov Process

Homework question: Consider the Markov process $X_t=B_t-t^2+t$ where $B_t$ is the Brownian motion. Find the generator $Q$ of this process. I am completely confused how to find the generator for ...
0
votes
1answer
20 views

Steady state of a $4 \times 4$ transition matrix

Normally I just take $q(M_{m\times n} - I_{m\times n})$ to workout the steady state, but here I have: $$\left(\begin{array}{rrrr} 0 & 0 & .8 & .2 \\ .4 & .6 & 0 & 0 \\ .2 ...
0
votes
0answers
28 views

Stationary distribution of a “birth-death model” that does not have Markov property

A typical birth-death process is defined such as the probability of going from any state $j$ to any state $i$ is given by: $$ p_{ij}= \begin{cases} b_i & \text {if $j = i+1$} \\ ...
3
votes
1answer
61 views

What is the value of this game?

We have 3 black and 2 red balls in an urn. If we pick a black ball, we lose 1 USD. If we pick a red ball we win 1 USD. We can chose to start the game or not. If we start the game we can stop after ...
1
vote
1answer
84 views

Elementary proof of geometric / negative binomial distribution in birth-death processes

The birth-death process concerns a population of $n_0$ individuals, each of which reproduce and die at a constant rate as time $t$ increases from $t=0$. Each individual splits into two individuals ...
1
vote
2answers
63 views

Expected number of steps between states in a Markov Chain

Suppose I am given a state space $S=\{0,1,2,3\}$ with transition probability matrix $\mathbf{P}= \begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 \\[0.3em] \frac{2}{3} ...
0
votes
1answer
62 views

Exercise on Markov chain

Prove, or give an explicit counterexample to refute, the following assertion: if $\{X_n\}$ is a Markov chain, then $\{X_n^2\}$ is also a Markov chain. It's easy to show that ...
3
votes
1answer
50 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is ...
3
votes
1answer
59 views

Strong Markov property of Brownian motion

I was able to understand Brownian Motion $\{B(t):t\geq0\}$ has Strong Markov Property i.e. For any stopping time $\tau$, $P(B(t+\tau)\leq y | \mathcal{F}_{\tau})=P(B(t+\tau)\leq y|B(\tau))$ a.s. , $y ...
3
votes
1answer
58 views

Random walk where increment depend on current position

Consider the following stochastic process, $$b(i+1) = b(i) + \xi_i (b_i),$$ where $\xi_i(b_i) \in \{-1, k \}$ are the independent increments having the following distribution: $$\begin{align} P (\xi ...
1
vote
1answer
57 views

Library chain stationary distribution

This is an exercise 1.47 from Richard Durrett's Essentials of Stochastic Processes p.85 (doi: 10.1007/978-1-4614-3615-7_1 or Google Books). On each request the ith of the $n$ possible books is the ...
1
vote
1answer
47 views

Equivalence Classes of a Markov Chain with Transition Matrix

I have the following transition probability matrix for a markov chain with state space S={0,1,2,3,4,5,6}: $\begin{bmatrix} \frac13 & \frac13 &0 & 0 & \frac16 & 0 & \frac16\\ ...
4
votes
1answer
76 views

random walk in a certain environment

Consider the following random walk in one dimension, starting from $r(0)=0$. $$ r(i+1) = r(i) + \xi, $$ where $\xi(i, r(i))$ is an increment with distribution $P(\xi=1) = \frac{c^{r(i)}}{i-r(i)+1}$ ...
3
votes
1answer
41 views

Probability of a sequence of events in a Poisson process.

I am starting to study Poisson processes and I came up with this question: Let there be two Poisson processes with rates $\lambda$ and $\mu$ respectively, monitoring the occurrence of events (e.g. ...
0
votes
1answer
94 views

Markov chains: is “aperiodic + irreducible” equivalent to “regular”?

I have two books on stochastic processes. In one book, it says that the limiting matrix is possible to find if the matrix is regular, that is, if for some $n$ $P^n$ has only positive values. The ...
0
votes
0answers
61 views

MDP problem - How is the expected cost calculated?

I have been stuck with a problem for a while regarding Markov Decision Processes for a Policy improvement algorithm. Assume that I have probabilities for certain states to evolve the system into, ...
0
votes
0answers
57 views

transition matrix for Markov chain

Can any one help me to solve this home work please? The city of Sacramento recently completed a new light rail system to bring commuters and shoppers into the downtown area and relieve freeway ...
1
vote
1answer
42 views

Estimate the probability using Markov chains

please consider this question: A study using Markov chains to estimate a patient's prognosis for improving under various treatment plans gives the following transition matrix as an example ...
0
votes
0answers
44 views

Continuous-Time Markov Chain conditioned on not visiting a part of the state space

Let $(X_t)_{t\geq 0}$ be a homogeneous continuous-time Markov chain with state space $\Psi =\{1,\dots,N\}$. Consider $S=\{1,\dots,M\} \subset \Psi$. Define $T = \inf \{ t \geq 0 : (X_t \not \in S ...
1
vote
0answers
68 views

First and second moments of recurrence time in a finite two-dimensional Markov chain

I have a two dimensional finite Markov chain with $(m+1)^2$ states, and with transition rates: $q_x((x,y)\to (x+1,y))=(m-x)\lambda,\quad 0\leq x< m, 0\leq y \leq m$, $q_x((x,y)\to ...
1
vote
1answer
47 views

Calculating cumulative Markov Chain outcomes

I have a Markov process, with 2 possible states (1 or 0) and a transition matrix P. State at time t=n is determined by x0*Pn. As n goes to infinity, xn goes to the steady state vector, q = [q1 q2]. ...
0
votes
1answer
29 views

Does a Markov Blanket allow connections between Parents of a Node?

In a Markov Blanket, we can connect the childredn of a node between them, as a child can be parent (or spouse) of another child. Does this rule apply as well for Parents of a node? In advance, Thank ...
0
votes
0answers
63 views

Markov Chains Proof using Statistics

Source: This came from "Introduction to probability" by Charles Miller Grinstead, and James Aurie Snell. It was located on page 407 and is Theorem 11.1 in the section 11.1 Introduction. Theorem: The ...
3
votes
0answers
50 views

A Continuous-Time Markov Process Taking All Possible Values

Let $\mathbb{N}$ be the set of positive integers. For each $n \in \mathbb{N}$, let $X^{(n)}=\{ X^{(n)}(t): t \geq 0 \}$ be a Markov chain with state-space the two point set $\{0,1\}$ and $Q$-matrix ...
14
votes
1answer
467 views

Hilbert's Barber Shop

Hilbert opens a barber shop with an infinite number of chairs and an infinite number of barbers. Customers arrive via a Poisson random process with an expected 1 person every 10 minutes. Upon arrival, ...
2
votes
2answers
258 views

Expected state of a markov chain

Let's start with a slightly trivial Markov chain defined as follows: the beginning state is called $1$ and the set of states is $\mathbb{N}$. At each step, when the current state is $n$, the ...
0
votes
0answers
62 views

Formal Theory Regarding M/M/s Queue

I have some difficulties with formally deducing the Q-matrix or infinitesimal generator for M/M/s Queues. Although I undestrand the intuitive idea I would like to know the real formal definition of ...
1
vote
2answers
238 views

Estimating the transition matrix given the stationary distribution

Let's say we are given a Markov chain for variable $X = [x_1, ..., x_n]$; also we are given a desired stationary distribution for this graph $P_\infty = [p_1, ..., p_n]^\top$. How can we design an ...
1
vote
1answer
61 views

Absorbing state for a collection of random walks

Further to this question; having learned some stuff since I posed it. Consider a collection of random walks $X_i$ which take finite integer values. These evolve as time-inhomogeneous Markov Chains. ...
3
votes
1answer
66 views

A equivalent definition of the Feller Process.

I saw this on Liggett's Book (P.95). Let $S=% %TCIMACRO{\U{2115} }% %BeginExpansion \mathbb{N} %EndExpansion ,$ and suppose $\left( X_{t}\right) _{t\geq 0}$ is a continuous-time Markov process with ...
1
vote
1answer
68 views

A question about Infinitesimal generator of Feller Process

Let $S=% %TCIMACRO{\U{211d} }% %BeginExpansion \mathbb{R} %EndExpansion $, and consider the Feller process $\left( X_{t}\right) _{t\geq 0}$ with state space $S$ such that $X_{t}=t+X_{0}$ for all ...
1
vote
1answer
96 views

Discrete-time Markov chain properties

A Markov chain in discrete time is irreducible, has state space $\{0,1,\dots\}$ and starts at $1$. It is both a branching process and a martingale. Determine the probability of hitting $0$.
3
votes
2answers
378 views

Probability of Extinction in a simple Birth and Death Process

We are asked to show that the probability of extinction $\zeta=\lim_{t\to \infty} P\left(X(t)=0\right)$ given by: $$\zeta=\begin{cases}1&\text{if }\lambda\le \mu,\\ \left(\frac \mu\lambda ...
1
vote
0answers
28 views

Single evaluation for using exponential sampling until past a point

I am trying to improve an algorithm that looks like the following (and am getting stumped): I am provided with a starting time, rate, and a target time. I then use an exponential distribution to ...
1
vote
1answer
103 views

Hitting times of Markov chain/process have always finite moments?

Consider an irreducible ergodic Markov chain on a finite state space $\Omega$. Then any state is positive recurrent and this should suffice to conclude that the mean hitting time of state $s \in ...
0
votes
2answers
355 views

Expected value of stochastic process

I have the following problem: $X_1,X_2,...$ are positive identically distributed random variables with the distribution function $F(x) :=P(X_n \leq x)$ and we assume that $F(0)<1$ for all $n$. Let ...
1
vote
0answers
71 views

Conditional distributions of (higher-order) autoregressive Markov processes

If we specify an $p$-th order autoregressive process in discrete time by its transition distribution $F_{t|t-1,\ldots,t-p}$, what can be said about lower order conditional distribution where we ...
0
votes
1answer
46 views

Showing a certain random process is a Markov Process

I have the following example of a random process: A person has two houses, house A and house B in which he can stay, we denote by $X_{i}\in\left\{ A,B\right\}$ the house he stayed in on the i-th day ...
0
votes
3answers
981 views

Finding the transition probability matrix, two switches either on or off..

Each of two switches is either on or off during a day. On day n, each switch will independently be on with probability [1+number of on switches during day n-1]/4 For instance, if both switches are on ...
1
vote
2answers
410 views

Markov Chain Transition Intensity Conversion

I have a question about converting a 3-state discrete state, continuous-time, markov chain to a 2-state. My 3-state model has states: Well (state 1), Ill (state 2) and Dead (state 3). ...
5
votes
0answers
146 views

Question on Conditional expectation

Let $X_1$ and $X_2$ be two random variables on $(\Omega,\mathcal{B},P)$. Suppose there is a function $g:\mathcal{B}\times\mathbb{R}\rightarrow[0,1]$ such that for any $x$, $g(\cdot,x)$ is a ...
1
vote
1answer
162 views

Is this Markov?

Consider a process $\{X_n, n\geq 0\}$ with state space $S=\{0,1,2\}$ s.t. $$ P(X_{n+1}=j | X_n=i, X_{n-1}=i_{n-1}, \dots, X_0=i_0)=\begin{cases} P_{ij}^I \ \ \ n \ \mbox{ even},\\ P_{ij}^{II} \ \ \ ...
0
votes
1answer
36 views

Characterizing the Dependence Structure of a Rewards for a Finite State Homogenous Markov Chain

Let $\{X_n, n\geq 1\}$ be a finite state homogenous Markov chain with states $i = 1, \ldots, N$ . Let $g$ denote a function which returns out a reward for any given state of the Markov chain. Let ...
1
vote
2answers
86 views

Using the canonical Markov property to prove an obvious fact about Markov chains

Given a Markov chain $\{X_n: n \geq 1 \}$, such that $$\mathbb{P}(X_{n+1} = x_{n+1} | X_n = x_n) = \mathbb{P}(X_{n+1} = x_{n+1} | X_n = x_n, \ldots X_1= x_1)$$ How can I formally prove that: ...
1
vote
2answers
333 views

Expected number of visits to state $j$ between successive visits to a state $i$ in a Markov chain given conditional information

Say I have a Markov chain $\{X_n: n \geq 1\}$ with state space $E = \{1,2,3,4,5\}$ and transition matrix, $$ P = \begin{bmatrix} 0 & 1/2 & 1/2 & 0 & 0 \\\ 1/2 & 0 & 1/2 & ...
4
votes
0answers
161 views

estimation of transition probabilities from aggregate data

Please, O mathematicians, help me understand the approach to the problem of estimating transition probabilities given only aggregate data in Kalbfleisch & Lawless' 1984 paper "Least-Squares ...
3
votes
1answer
101 views

Limit of a probability regarding a random walk

Consider a random walk on the integers starting at 0, where in each step you move either 1 or 2 meters (back or forth alike). As soon as you reach either the $N$ or $N+1$ meter mark, you stop. What is ...
2
votes
0answers
62 views

What is the difference between a Markov process and a Markov chain? [duplicate]

Possible Duplicate: What is the difference between all types of Markov Chains? I've read a lot about Markov processes and chains but so far I don't understand what the difference is between ...
1
vote
1answer
135 views

Explanation of Parsimony

Can someone explain what Parsimony is in the context of probability, more specifically in Parsimonious Markov models? I have been trying to search around a simple explanation of this but I only seem ...