0
votes
0answers
20 views

Showing the square of a Markov process is or isn't Markov

Hi I am trying to show that if $X_n$ is a markov process, whether or not $X_n^2$ is a markov process. $X_n$ is a markov process if $P\{X_k = a_k|X_{k-1} = a_{k-1}, X_{k-2} = a_{k-2}, ..., X_k = a_1 ...
-1
votes
0answers
38 views

Markovian birth-death process [on hold]

A linear Markovian death process, initialized at five members, experiences an average daily death rate $u=0.1$. Determine the probability of having fewer than three members in the population after a ...
0
votes
0answers
20 views

Error in thinking: Poisson Process is a Markov Process

I am a bit confused on proving the Markov property for Poisson processes. That is, we want to prove, if $X = (X_t: t \in \mathbb{R})$ is a Poisson process with rate $\lambda$: $P(X_{t_n} = a_n | ...
2
votes
1answer
50 views

continuous time Markov chain, something the book does not explain

I have a problem with something in my book, under the chapter of continuous time Markov chains. I will post a link to what the book does. They do something which they seem to take for granted, but I ...
0
votes
1answer
38 views

Representation of Markov process adapted to given Filtration

Let $X$ be continuous Markov process adapted to a filtration generated by Brownian motion $B$. Does there exist a function $f$ such that $X_t = f(t,B_t)$? My guess is that it should have such ...
0
votes
1answer
32 views

Ornstein-Uhlenbeck processs: Markov, but not martingale?

I'm puzzled about properties of the Ornstein-Uhlenbeck process, given by the Itō integral $$ X_t = x e^{-\lambda t} + \sigma \int_0^t e^{-\lambda(t-s)} d W_s \,. $$ I compute that $\{X_t\}$ is not ...
0
votes
1answer
26 views

Show $P(S_{2n}=x|S_0=x) \ge \frac{1}{N}$

Let $X_n$ be an aperiodic, discrete-time Markov chain so $S=\{1,...,N\}$ whose transition probability is symmetric. How can I show that for all $x \in S$ and all integers $n$, $P(S_{2n}=x|S_0=x) \ge ...
0
votes
1answer
19 views

Transition rate matrix from transition probability matrix

If I have a $2 \times 2$ continuous time Markov chain transition probability matrix (generated from a financial time series data), is it possible to get the transition rate matrix from this and if ...
0
votes
0answers
26 views

Continuous time Markov chains, how would this definition be expanded from time-homogeneous to time-inhomogeneous.

Below I have a picture of how we can view a continuous time Markov chain that is time-homogeneous. Now, I am wondering what happens when we have a inhomogeneous continuous Markov chain. I have ...
0
votes
1answer
33 views

Markov processes and semimartingales

Semimartingales and Markov processes are two fundamental families in probability theory. There are many specific processes that belongs to the intersection of those two families, e.g. Levy processes. ...
0
votes
0answers
25 views

Markov renewal process vs Markov Jump process

The venerable wikipedia for "Markov renewal process" says that: "a Markov renewal process is a random process that generalizes the notion of Markov jump processes". So what's the definition of a ...
0
votes
0answers
44 views

The difference between Dynamic Optimization, Stochastic Programming, Optimal control and Markov Decision Processes

I've seen the following terms thrown around somewhat interchangeably, and I'm confused. What are the distinctions between them, and what are some representative problems that each deals with? ...
1
vote
1answer
84 views

Elementary proof of geometric / negative binomial distribution in birth-death processes

The birth-death process concerns a population of $n_0$ individuals, each of which reproduce and die at a constant rate as time $t$ increases from $t=0$. Each individual splits into two individuals ...
6
votes
2answers
118 views

What is the importance of the infinitesimal generator of Brownian motion?

I have read that the infinitesimal generator of Brownian motion is $\frac{1}{2}\small\triangle$. Unfortunately, I have no background in semigroup theory, and the expositions of semigroup theory I have ...
1
vote
0answers
18 views

Finite state Markov chains and expectations

Let $X_t$ be a finite state Markov chain with generator matrix $Q$. For a give function $f(x)>0$ define: $$ u(t, i) = \mathbb{E}\left[\int_t^T f(X_s) \mathrm{d} s| X_t = i\right] $$ Are there any ...
3
votes
1answer
50 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is ...
1
vote
1answer
27 views

Stochastic processes with independent increments

If $\{X_{t}:t\geq 0\}$ is a real-valued stochastic process with independent increments then $\{X_{t}:t\geq0\}$ is a Markov process? Let $\{ \mathcal{F}_{t} \}_{t\geq0} $ be a natural filtration of ...
1
vote
0answers
32 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
3
votes
1answer
59 views

Strong Markov property of Brownian motion

I was able to understand Brownian Motion $\{B(t):t\geq0\}$ has Strong Markov Property i.e. For any stopping time $\tau$, $P(B(t+\tau)\leq y | \mathcal{F}_{\tau})=P(B(t+\tau)\leq y|B(\tau))$ a.s. , $y ...
0
votes
0answers
22 views

Calculation of the Gallavotti-Cohen fluctuation theorem made by Lebowitz

I have a problem understanding a calculation in this paper (another form of the theorem an be found here at equation 11). For those who want to read the paper, I have difficulties with formula 2.14 in ...
1
vote
1answer
57 views

Library chain stationary distribution

This is an exercise 1.47 from Richard Durrett's Essentials of Stochastic Processes p.85 (doi: 10.1007/978-1-4614-3615-7_1 or Google Books). On each request the ith of the $n$ possible books is the ...
0
votes
1answer
39 views

Poisson Process (Easy)

I'm stuck at the following question: Customers with items to repair arrive at a repair facility according to a Poisson process with rate λ. The repair time of an item has a uniform distribution on ...
1
vote
1answer
48 views

Equivalence Classes of a Markov Chain with Transition Matrix

I have the following transition probability matrix for a markov chain with state space S={0,1,2,3,4,5,6}: $\begin{bmatrix} \frac13 & \frac13 &0 & 0 & \frac16 & 0 & \frac16\\ ...
1
vote
0answers
22 views

Prove equilibrium theorem without irreducibility and aperiodicity

I have to solve the following question: Consider a random walk Markov chain on $S = \{1, 2, \ldots, 100\}$. If the chain is between 2 and 99, it selects one of the adjacent states with equal ...
0
votes
1answer
94 views

Markov chains: is “aperiodic + irreducible” equivalent to “regular”?

I have two books on stochastic processes. In one book, it says that the limiting matrix is possible to find if the matrix is regular, that is, if for some $n$ $P^n$ has only positive values. The ...
2
votes
1answer
103 views

Equivalent defining Markov property

Consider the stochastic process $(X_t)_{t \in \mathbb{R}}$ and show the equivalence of the following two Markov properties: (a) $P(X_t \in A \mid X_u, u \leq s) = P(X_t \in A\mid X_s) \qquad ...
0
votes
0answers
37 views

Is this two dimensional Markov chain correct for this queueing system?

The problem that I have two single server station with no queuing space a customer goes to station 1 if it is available else it goes to station 2 if it is available or it will be lost output from ...
1
vote
1answer
48 views

Strong Markov property given transition functions

Suppose we are given family of transition functions satisfying Chapman-Kolmogorov equation, what conditions will ensure that there exists a continuous or cadlag Markov process with given transition ...
0
votes
1answer
41 views

Stationary distribution for a Markov chain which is not irreducible

I have a Markov chain with $K$ states $S$: {$s_1,s_2,...,s_K$}. $s_1$ is reachable from any state in $S$; however not all the states can be reached from $s_1$. What does the stationary distribution ...
0
votes
0answers
31 views

Does time homogeneity imply strong Markov property in a Markovian process

Does a time homogeneous Markovian process necessarily have strong Markovian property? Does continuity in state space, time, or path make a difference? What are the examples if it does not?
0
votes
0answers
30 views

An example of a simple ergodic diffusion process?

I'm looking for a simple example (ideally two-dimensional) of an ergodic diffusion process with polynomial drift vector and diffusion matrix for which there a no known explicit expressions for the ...
2
votes
0answers
38 views

Examples of decreasing-in-some-time-interval variance of a time homogeneous Markovian process

Let $x_t$ be a zero mean, time homogeneous Markovian process over time $t$ starting from $x_0=0$. What are the examples of $x_t$ where the variance at $t$ decrease over some interval of $t$? The ...
1
vote
0answers
68 views

First and second moments of recurrence time in a finite two-dimensional Markov chain

I have a two dimensional finite Markov chain with $(m+1)^2$ states, and with transition rates: $q_x((x,y)\to (x+1,y))=(m-x)\lambda,\quad 0\leq x< m, 0\leq y \leq m$, $q_x((x,y)\to ...
0
votes
0answers
24 views

Why must a stochastic process be at least second order in terms of differential equations?

A first order differential equation in $q(t)$ has a unique path through each possible value of $q(0)$. This is opposed to a stochastic process (e.g. random walk), where any place might be "hopped ...
2
votes
1answer
53 views

transition kernel

I've got some trouble with transition kernels. We look at Markov process with statspace $(S,\mathcal{S})$ and initial distribution $\mu^0$. We have a transition kernel $P:S\times ...
0
votes
0answers
32 views

Under what circumstance should I use a Continuous time Markov Chain instead of a discrete time Markov Chain?

Why should I use one over the other, if I can basically reduce the small time-interval $h$ to be small enough that it simulates continuity? I guess this question is somewhat analogous to control ...
1
vote
1answer
58 views

Transition function is a Markov semigroup?

How does the transition function in a Markov process become a Markov semigroup in time homogeneous Markov processes? Thanks a lot.
1
vote
2answers
104 views

Question on the proof of the simple Markov property of a Brownian motion

Today we proofed the (simple) Markov property for the Brownian motion. But I really don't get a crucial step in the proof. The theorem states in particular that for $s\geq0$ fixed, the process ...
1
vote
1answer
38 views

makov chains and property

let $X_t$ for $t \in \{1,2,...,10\}$ be a sequence of independent tosses of a fair coin. We denote heads with 1 and tails with 0. Define the random variable $S_n=\sum_{t=1}^n X_t$ for $n \in ...
0
votes
1answer
76 views

Markov time $ T= \min\{n : X[n] = 1\}$

Let $T$ is a Markov time such that $T= \min \{ n : X[n] = 1\}$ , $X[n]$ is the number of $h$ (heads) in coin tossing for $n$ times. Let's say I will toss the coin 3 times, so the event collection is ...
1
vote
0answers
57 views

Proof for a Markov process example (using measure theory)

Consider the probability space $(\mathbb{R},B(\mathbb{R}),\delta_x)$ for a given $x\in\mathbb{R}$ (where $\delta_x$ is the Dirac measure) and define the process $X_t(\omega)=\omega - t$, for $t\geq ...
0
votes
0answers
62 views

Show that all Gauss-Markov (GM) Processes can be Written in Recurrent Form

The book shows that the following is always GM. $$x_{k+1} = A_k x_k + w_k$$ where $$x_0, w_o, w_1,...$$ are independent and Gaussian. It then leaves the converse as "an exercise." Given: $$x_k$$ is ...
1
vote
1answer
82 views

How is a simple birth process is time-homogeneous?

Why is it that a simple birth process is time-homogeneous? The incidence of a birth in a small time interval depends on the population size at the time of start of the interval. Doesn't this ...
1
vote
1answer
109 views

Joint density of Markov process as product of conditional densities.

Let $(X(t))_{t\geq0}$ be a Markov proces such that s $X(0)=x_0$. Consider the random vector $(X(t_1),\dots,X(t_n))$ with corresponding joint density $g(x_1,\dots ,x_n)$. Is it then true that ...
1
vote
1answer
220 views

Why is this infinite-state-space Markov chain positive recurrent?

Given the following transition matrix for a Markov chain, how can I see that the chain is positive recurrent? I want to convince myself that the chain has a limiting distribution, and the chain is ...
2
votes
2answers
91 views

How to understand Markov property?

I'm learning stochastic process in college. How to understand Markov property?I'm curious about what is the power and validity of Markov property ? A stochastic process has the Markov property if ...
14
votes
1answer
467 views

Hilbert's Barber Shop

Hilbert opens a barber shop with an infinite number of chairs and an infinite number of barbers. Customers arrive via a Poisson random process with an expected 1 person every 10 minutes. Upon arrival, ...
0
votes
1answer
49 views

question about the transformation of a Markov process

I have a question about Markov Process: Let $X_t=(X_t^1, X_t^2,..., X_t^n)$ be a Markov process with regard to the filtration $\mathcal{F}_t$, let $Y_t:=\max_{1\leq k\leq n}X_t^k$, then is $Y_t$ a ...
2
votes
0answers
63 views

References for basics of Piecewise-Deterministic Markov Processes

I am looking for introductory/pedagogical material to Piecewise-Deterministic Markov Processes (see http://en.wikipedia.org/wiki/Piecewise-deterministic_Markov_process) (For the moment I am interested ...
0
votes
0answers
69 views

Stopping time inequality Markov process

Let X be a right-continuous Feller-Dynkin process and define the stopping time $$\nu_{r}=\inf\{t\geq 0\mid ||X_{t}-X_{0}||\geq r\}$$ Let $B_{x}(\epsilon)=\{y\mid ||y-x|| \leq \epsilon\}$, for $x$ not ...