A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
0answers
42 views

Under what circumstance should I use a Continuous time Markov Chain instead of a discrete time Markov Chain?

Why should I use one over the other, if I can basically reduce the small time-interval $h$ to be small enough that it simulates continuity? I guess this question is somewhat analogous to control ...
1
vote
0answers
67 views

Find Markov policy that minimizes(maximizes) the expected discounted cost(reward)

It's an exam problem I found online.Here's a link to the pastpaper. The problem is stated as follows. A repairman who services Q facilities moves between location s and location j according to the ...
2
votes
1answer
93 views

What is the difference between positive presistent and null persistent state in a Markov Chain?

I'm not looking for the difference in the mathematical definition, but rather for an intuitive explanation of their differences and possible examples, so that I can have them in my head when ...
0
votes
0answers
83 views

Markov Chains Proof using Statistics

Source: This came from "Introduction to probability" by Charles Miller Grinstead, and James Aurie Snell. It was located on page 407 and is Theorem 11.1 in the section 11.1 Introduction. Theorem: The ...
1
vote
1answer
71 views

Transition function is a Markov semigroup?

How does the transition function in a Markov process become a Markov semigroup in time homogeneous Markov processes? Thanks a lot.
2
votes
2answers
199 views

Question on the proof of the simple Markov property of a Brownian motion

Today we proofed the (simple) Markov property for the Brownian motion. But I really don't get a crucial step in the proof. The theorem states in particular that for $s\geq0$ fixed, the process ...
1
vote
1answer
45 views

makov chains and property

let $X_t$ for $t \in \{1,2,...,10\}$ be a sequence of independent tosses of a fair coin. We denote heads with 1 and tails with 0. Define the random variable $S_n=\sum_{t=1}^n X_t$ for $n \in ...
0
votes
1answer
88 views

Markov time $ T= \min\{n : X[n] = 1\}$

Let $T$ is a Markov time such that $T= \min \{ n : X[n] = 1\}$ , $X[n]$ is the number of $h$ (heads) in coin tossing for $n$ times. Let's say I will toss the coin 3 times, so the event collection is ...
1
vote
0answers
64 views

Proof for a Markov process example (using measure theory)

Consider the probability space $(\mathbb{R},B(\mathbb{R}),\delta_x)$ for a given $x\in\mathbb{R}$ (where $\delta_x$ is the Dirac measure) and define the process $X_t(\omega)=\omega - t$, for $t\geq ...
1
vote
1answer
79 views

How to Prove by definition, the given process is a Markov Process?

Define the process Xt by X0 = 1, and for t = 1, 2, . . . Xt = { uXt-1, with probability p, { vXt-1, with probability 1-p where 0 < v < 1 ...
1
vote
1answer
95 views

How is a simple birth process is time-homogeneous?

Why is it that a simple birth process is time-homogeneous? The incidence of a birth in a small time interval depends on the population size at the time of start of the interval. Doesn't this ...
2
votes
0answers
70 views

Interpretation of potential kernels for Markov processes

One can associate a strongly continuous contraction semi group (SCCSG) to a Markov process with state space $S$ through its transition function, say $P_t$. Now one can interpret $P_t$ as a linear ...
1
vote
1answer
143 views

Joint density of Markov process as product of conditional densities.

Let $(X(t))_{t\geq0}$ be a Markov proces such that s $X(0)=x_0$. Consider the random vector $(X(t_1),\dots,X(t_n))$ with corresponding joint density $g(x_1,\dots ,x_n)$. Is it then true that ...
1
vote
0answers
28 views

Advanced reference in Markov processes

I am interested in a book which covers the more in depth stuff on continuous time Markov processes e.g. semi-groups, generators... Preferably such book would also contain a list of analysis results ...
1
vote
1answer
330 views

Why is this infinite-state-space Markov chain positive recurrent?

Given the following transition matrix for a Markov chain, how can I see that the chain is positive recurrent? I want to convince myself that the chain has a limiting distribution, and the chain is ...
1
vote
4answers
175 views

Why do these expressions tend to zero?

In a chapter on Markov chains, it is claimed that $\binom {2n}{n} p^n(1-p)^n$ (where $p$ is a probability) tends to $0$ as $n$ tends to $\infty$. But why is this so? It is also claimed that for an ...
2
votes
2answers
112 views

How to understand Markov property?

I'm learning stochastic process in college. How to understand Markov property?I'm curious about what is the power and validity of Markov property ? A stochastic process has the Markov property if ...
4
votes
0answers
76 views

A Continuous-Time Markov Process Taking All Possible Values

Let $\mathbb{N}$ be the set of positive integers. For each $n \in \mathbb{N}$, let $X^{(n)}=\{ X^{(n)}(t): t \geq 0 \}$ be a Markov chain with state-space the two point set $\{0,1\}$ and $Q$-matrix ...
14
votes
1answer
497 views

Hilbert's Barber Shop

Hilbert opens a barber shop with an infinite number of chairs and an infinite number of barbers. Customers arrive via a Poisson random process with an expected 1 person every 10 minutes. Upon arrival, ...
0
votes
1answer
95 views

Markov Chain discarding balls from urn

The following question has me stumped. Any ideas on how to get started? An urn contains $n$ green balls and $n+2$ red balls. A ball is picked at random: if it is green then a red balls is also ...
3
votes
1answer
63 views

For a generator $G$ of a Markov process in continuous time and finite state space, how would one prove that the entries of $e^{tG}$ are non-negative?

I have a generator matrix G for a Markov chain in continuous time and finite state space and I am looking to prove that the entries of $e^{tG} \geq 0 $ By definition $G = P'(0)$ with entries $g_{ij} ...
2
votes
2answers
416 views

Expected state of a Markov chain

Let's start with a slightly trivial Markov chain defined as follows: the beginning state is called $1$ and the set of states is $\mathbb{N}$. At each step, when the current state is $n$, the ...
0
votes
1answer
128 views

Inner product space and Reversible Markov chain

I am trying to clarify the following: Suppose P is the $N\times N$ transition matrix of a finite-dimensional Markov Chain, with invariant distribution given by N-vector $\mu$ (i.e. $\mu^T=\mu^T P$). ...
0
votes
0answers
49 views

Propagation of standard deviation for random variable with Markov Property

I have a discrete random variable, $X \in \{0,1,2,3\}$. Define the indicator function: $$ 1_{k}\left(x\right) = \begin{cases} 1, & \text{if $x=k$} \\ 0, & \text{otherwise} \\ \end{cases}$$ ...
0
votes
1answer
59 views

question about the transformation of a Markov process

I have a question about Markov Process: Let $X_t=(X_t^1, X_t^2,..., X_t^n)$ be a Markov process with regard to the filtration $\mathcal{F}_t$, let $Y_t:=\max_{1\leq k\leq n}X_t^k$, then is $Y_t$ a ...
-1
votes
1answer
411 views

What is the Expectations of all 3 ants meeting at same point?

Say we have 3 ants in three corner's of triangle. What is the expectations that all 3 ants meeting together given that the ant moves in any direction. So by just seeing it I figured out that in 2 ...
1
vote
2answers
390 views

Estimating the transition matrix given the stationary distribution

Let's say we are given a Markov chain for variable $X = [x_1, ..., x_n]$; also we are given a desired stationary distribution for this graph $P_\infty = [p_1, ..., p_n]^\top$. How can we design an ...
0
votes
1answer
28 views

Is it possible to reverse probabilistic automaton?

Is it possible to reverse probabilistic automaton (PA), i.e. calculate the probability of previous state given current state? Will reversed automaton be a PA (Markov?), i.e. will next probability ...
1
vote
1answer
66 views

Absorbing state for a collection of random walks

Further to this question; having learned some stuff since I posed it. Consider a collection of random walks $X_i$ which take finite integer values. These evolve as time-inhomogeneous Markov Chains. ...
2
votes
1answer
60 views

Markov property of a random process (a solution of piece-wise deterministic equations)

Consider a piece-wise deterministic (Markov!) process \begin{eqnarray} \dot{x}(t) & = & A_{\theta(t,x(t))}x(t)\\ x(0) & = & x_0 \in \mathbb{R}^n \notag \end{eqnarray} where ...
2
votes
0answers
74 views

References for basics of Piecewise-Deterministic Markov Processes

I am looking for introductory/pedagogical material to Piecewise-Deterministic Markov Processes (see http://en.wikipedia.org/wiki/Piecewise-deterministic_Markov_process) (For the moment I am interested ...
0
votes
0answers
75 views

Stopping time inequality Markov process

Let X be a right-continuous Feller-Dynkin process and define the stopping time $$\nu_{r}=\inf\{t\geq 0\mid ||X_{t}-X_{0}||\geq r\}$$ Let $B_{x}(\epsilon)=\{y\mid ||y-x|| \leq \epsilon\}$, for $x$ not ...
0
votes
1answer
43 views

Max. reachability in infinite-state MDP

Following [1], the maximum probability to reach a set of states $B\subseteq S$ from state $s\in S$ in a Markov decision process with finite state space $S$ can be expressed as the unique solution to ...
3
votes
1answer
73 views

A equivalent definition of the Feller Process.

I saw this on Liggett's Book (P.95). Let $S=% %TCIMACRO{\U{2115} }% %BeginExpansion \mathbb{N} %EndExpansion ,$ and suppose $\left( X_{t}\right) _{t\geq 0}$ is a continuous-time Markov process with ...
1
vote
1answer
79 views

A question about Infinitesimal generator of Feller Process

Let $S=% %TCIMACRO{\U{211d} }% %BeginExpansion \mathbb{R} %EndExpansion $, and consider the Feller process $\left( X_{t}\right) _{t\geq 0}$ with state space $S$ such that $X_{t}=t+X_{0}$ for all ...
1
vote
1answer
106 views

Discrete-time Markov chain properties

A Markov chain in discrete time is irreducible, has state space $\{0,1,\dots\}$ and starts at $1$. It is both a branching process and a martingale. Determine the probability of hitting $0$.
4
votes
0answers
133 views

Conditional probability and integrating out part of a random walk

Suppose that I have a random walk process defined by $\alpha_{t+1}$ ~ N$(\alpha_t, \omega^2)$. Given $\alpha_t$ and $\alpha_{t+2}$, I understand why the conditional formula for ...
0
votes
1answer
118 views

Canonical Markov Process

Let $X$ be a canonical, right-continuous Markov process with values in a Polish state space $E$, equipped with Borel-$\sigma$-algebra $\mathcal{E}$ and we assume that $t\rightarrow E_{X_{t}}f(X_{s})$ ...
0
votes
1answer
133 views

three-state Markov chain

a male and a female go to a 2-table restaurant on the same day. each day the male sits at one or the other of the 2 tables, starting at the table 1, with a Markov chain transition matrix: ...
0
votes
1answer
70 views

Rewriting Markov process

Let $X$ be a Markov proces with state space $(E,\mathcal{E})$with initial distribution $\nu$ and transition function $P_{t}$, so $$E_{\nu}(f(X_{t+s})\mid\mathcal{F}_{s})=P_{t}f(X_{s})$$ Suppose that ...
0
votes
1answer
64 views

Question on Markov chains of expected number of states

I am confused with an statement from my probability book that has to do with Markov chains. I hope someone could clarify that, if possible....Consider a Markov chain for which $P_{11}=1$ and ...
1
vote
2answers
311 views

Diffusion process. Distribution vs transition probability.

I need confirmation on the following problem: Take a SDE of the form: \begin{equation} dX_t=a(X_t,t)dt+b(X_t,t)dW_t \end{equation} where all the conditions, such that the solution $X_t$ is defined ...
0
votes
1answer
68 views

General State Space Markov Chain

I am having some difficulty understanding some early results of Markov Chain theory on a general state space. We have a function (Kernel) $K:E \times E \rightarrow \mathbb{R}$, and a distribution ...
1
vote
1answer
72 views

Identity in Markov Processes

I want to know if my reasoning here is correct, it seems simple enough but I just want clarification (I am considering the proof that if a Markov process satisfies the detailed balance condition, then ...
0
votes
0answers
89 views

Markov Chains Worked Example (Stirzaker)

I have a Markov Chain with state space the non-negative integers. The rules of the M.C. are that when it is in state $i \neq 0$, it moves to one of {${0,1,2,\ldots,i+1}$} with probability $1/(i+2)$ ...
1
vote
1answer
203 views

Have there some discrete-time continuous-state Markov processes been studied?

I have seen discrete-time discrete-state Markov processes (such as random walks), continuous-time discrete-state Markov processes (such as Poisson processes), and continuous-time continuous-state ...
0
votes
1answer
56 views

On discrete-time stochastic attractivity

Let $m$ be a probability measure on $Y \subseteq \mathbb{R}^p$, so that $m(Y)=1$. Consider a function $f: \mathbb{R}^n \times Y \rightarrow \mathbb{R}^n$, continuous on the first arguments, ...
3
votes
2answers
483 views

Probability of Extinction in a simple Birth and Death Process

We are asked to show that the probability of extinction $\zeta=\lim_{t\to \infty} P\left(X(t)=0\right)$ given by: $$\zeta=\begin{cases}1&\text{if }\lambda\le \mu,\\ \left(\frac \mu\lambda ...
0
votes
1answer
101 views

Metropolis Hastings definition - Proving $\pi(x)$ is the invariant density of our transition matrix

I'm currently working through the proof of the Metropolis-Hastings algorithm, and using two sources: page 328, section 3 page 1704-1705 I have a good understanding of most of the proof until ...
1
vote
0answers
29 views

Single evaluation for using exponential sampling until past a point

I am trying to improve an algorithm that looks like the following (and am getting stumped): I am provided with a starting time, rate, and a target time. I then use an exponential distribution to ...