A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
2answers
109 views

Question on the proof of the simple Markov property of a Brownian motion

Today we proofed the (simple) Markov property for the Brownian motion. But I really don't get a crucial step in the proof. The theorem states in particular that for $s\geq0$ fixed, the process ...
1
vote
1answer
39 views

makov chains and property

let $X_t$ for $t \in \{1,2,...,10\}$ be a sequence of independent tosses of a fair coin. We denote heads with 1 and tails with 0. Define the random variable $S_n=\sum_{t=1}^n X_t$ for $n \in ...
0
votes
1answer
76 views

Markov time $ T= \min\{n : X[n] = 1\}$

Let $T$ is a Markov time such that $T= \min \{ n : X[n] = 1\}$ , $X[n]$ is the number of $h$ (heads) in coin tossing for $n$ times. Let's say I will toss the coin 3 times, so the event collection is ...
1
vote
0answers
57 views

Proof for a Markov process example (using measure theory)

Consider the probability space $(\mathbb{R},B(\mathbb{R}),\delta_x)$ for a given $x\in\mathbb{R}$ (where $\delta_x$ is the Dirac measure) and define the process $X_t(\omega)=\omega - t$, for $t\geq ...
0
votes
0answers
66 views

Chapman-Kolmogorov equation for Borel sets.

I have been reading a paper which does not give explicit explanation. Here is the question: Let $(X_t)_{t\geq 0}$ be a family of random variables on finite dimensional $R^d$. The author of the paper ...
0
votes
1answer
74 views

How to Prove by definition, the given process is a Markov Process?

Define the process Xt by X0 = 1, and for t = 1, 2, . . . Xt = { uXt-1, with probability p, { vXt-1, with probability 1-p where 0 < v < 1 ...
0
votes
0answers
67 views

Show that all Gauss-Markov (GM) Processes can be Written in Recurrent Form

The book shows that the following is always GM. $$x_{k+1} = A_k x_k + w_k$$ where $$x_0, w_o, w_1,...$$ are independent and Gaussian. It then leaves the converse as "an exercise." Given: $$x_k$$ is ...
1
vote
1answer
83 views

How is a simple birth process is time-homogeneous?

Why is it that a simple birth process is time-homogeneous? The incidence of a birth in a small time interval depends on the population size at the time of start of the interval. Doesn't this ...
2
votes
0answers
59 views

Interpretation of potential kernels for Markov processes

One can associate a strongly continuous contraction semi group (SCCSG) to a Markov process with state space $S$ through its transition function, say $P_t$. Now one can interpret $P_t$ as a linear ...
1
vote
1answer
111 views

Joint density of Markov process as product of conditional densities.

Let $(X(t))_{t\geq0}$ be a Markov proces such that s $X(0)=x_0$. Consider the random vector $(X(t_1),\dots,X(t_n))$ with corresponding joint density $g(x_1,\dots ,x_n)$. Is it then true that ...
1
vote
0answers
22 views

Advanced reference in Markov processes

I am interested in a book which covers the more in depth stuff on continuous time Markov processes e.g. semi-groups, generators... Preferably such book would also contain a list of analysis results ...
1
vote
1answer
230 views

Why is this infinite-state-space Markov chain positive recurrent?

Given the following transition matrix for a Markov chain, how can I see that the chain is positive recurrent? I want to convince myself that the chain has a limiting distribution, and the chain is ...
1
vote
4answers
168 views

Why do these expressions tend to zero?

In a chapter on Markov chains, it is claimed that $\binom {2n}{n} p^n(1-p)^n$ (where $p$ is a probability) tends to $0$ as $n$ tends to $\infty$. But why is this so? It is also claimed that for an ...
2
votes
2answers
91 views

How to understand Markov property?

I'm learning stochastic process in college. How to understand Markov property?I'm curious about what is the power and validity of Markov property ? A stochastic process has the Markov property if ...
3
votes
0answers
50 views

A Continuous-Time Markov Process Taking All Possible Values

Let $\mathbb{N}$ be the set of positive integers. For each $n \in \mathbb{N}$, let $X^{(n)}=\{ X^{(n)}(t): t \geq 0 \}$ be a Markov chain with state-space the two point set $\{0,1\}$ and $Q$-matrix ...
14
votes
1answer
467 views

Hilbert's Barber Shop

Hilbert opens a barber shop with an infinite number of chairs and an infinite number of barbers. Customers arrive via a Poisson random process with an expected 1 person every 10 minutes. Upon arrival, ...
0
votes
1answer
82 views

Markov Chain discarding balls from urn

The following question has me stumped. Any ideas on how to get started? An urn contains $n$ green balls and $n+2$ red balls. A ball is picked at random: if it is green then a red balls is also ...
0
votes
0answers
40 views

Similarities and differences between Memoryless property and Markovian property

I want to ask about similarities and differences between Memoryless property (MLP) and Markovian property (MKP), which I've been curious since long time ago. According to what I have searched and ...
3
votes
1answer
57 views

For a generator $G$ of a Markov process in continuous time and finite state space, how would one prove that the entries of $e^{tG}$ are non-negative?

I have a generator matrix G for a Markov chain in continuous time and finite state space and I am looking to prove that the entries of $e^{tG} \geq 0 $ By definition $G = P'(0)$ with entries $g_{ij} ...
2
votes
2answers
267 views

Expected state of a markov chain

Let's start with a slightly trivial Markov chain defined as follows: the beginning state is called $1$ and the set of states is $\mathbb{N}$. At each step, when the current state is $n$, the ...
0
votes
1answer
111 views

Inner product space and Reversible Markov chain

I am trying to clarify the following: Suppose P is the $N\times N$ transition matrix of a finite-dimensional Markov Chain, with invariant distribution given by N-vector $\mu$ (i.e. $\mu^T=\mu^T P$). ...
0
votes
0answers
45 views

Propagation of standard deviation for random variable with Markov Property

I have a discrete random variable, $X \in \{0,1,2,3\}$. Define the indicator function: $$ 1_{k}\left(x\right) = \begin{cases} 1, & \text{if $x=k$} \\ 0, & \text{otherwise} \\ \end{cases}$$ ...
0
votes
0answers
63 views

Formal Theory Regarding M/M/s Queue

I have some difficulties with formally deducing the Q-matrix or infinitesimal generator for M/M/s Queues. Although I undestrand the intuitive idea I would like to know the real formal definition of ...
0
votes
1answer
50 views

question about the transformation of a Markov process

I have a question about Markov Process: Let $X_t=(X_t^1, X_t^2,..., X_t^n)$ be a Markov process with regard to the filtration $\mathcal{F}_t$, let $Y_t:=\max_{1\leq k\leq n}X_t^k$, then is $Y_t$ a ...
-1
votes
1answer
262 views

What is the Expectations of all 3 ants meeting at same point?

Say we have 3 ants in three corner's of triangle. What is the expectations that all 3 ants meeting together given that the ant moves in any direction. So by just seeing it I figured out that in 2 ...
1
vote
2answers
243 views

Estimating the transition matrix given the stationary distribution

Let's say we are given a Markov chain for variable $X = [x_1, ..., x_n]$; also we are given a desired stationary distribution for this graph $P_\infty = [p_1, ..., p_n]^\top$. How can we design an ...
0
votes
1answer
25 views

Is it possible to reverse probabilistic automaton?

Is it possible to reverse probabilistic automaton (PA), i.e. calculate the probability of previous state given current state? Will reversed automaton be a PA (Markov?), i.e. will next probability ...
1
vote
1answer
61 views

Absorbing state for a collection of random walks

Further to this question; having learned some stuff since I posed it. Consider a collection of random walks $X_i$ which take finite integer values. These evolve as time-inhomogeneous Markov Chains. ...
2
votes
1answer
47 views

Markov property of a random process (a solution of piece-wise deterministic equations)

Consider a piece-wise deterministic (Markov!) process
0
votes
0answers
55 views

Stability condition for queueing model with 2 servers, non-identical hypoexponetial-2 and FCFS

We consider a queueing model with 2 servers and a shared queue, let us label the servers as $A$ and $B$. Jobs arrive in batches of size $N$ with rate $\lambda$. Each server has a hypoexponential ...
2
votes
0answers
63 views

References for basics of Piecewise-Deterministic Markov Processes

I am looking for introductory/pedagogical material to Piecewise-Deterministic Markov Processes (see http://en.wikipedia.org/wiki/Piecewise-deterministic_Markov_process) (For the moment I am interested ...
0
votes
0answers
69 views

Stopping time inequality Markov process

Let X be a right-continuous Feller-Dynkin process and define the stopping time $$\nu_{r}=\inf\{t\geq 0\mid ||X_{t}-X_{0}||\geq r\}$$ Let $B_{x}(\epsilon)=\{y\mid ||y-x|| \leq \epsilon\}$, for $x$ not ...
0
votes
1answer
39 views

Max. reachability in infinite-state MDP

Following [1], the maximum probability to reach a set of states $B\subseteq S$ from state $s\in S$ in a Markov decision process with finite state space $S$ can be expressed as the unique solution to ...
0
votes
0answers
27 views

Question on MRF with Efficient Approximations [Boykov98]

My question is about this paper. Although most of the explanations are perfectly clear to me, there is this one part I feel unsure about.. If you could enlighten me, I would really appreciate it. ...
3
votes
1answer
67 views

A equivalent definition of the Feller Process.

I saw this on Liggett's Book (P.95). Let $S=% %TCIMACRO{\U{2115} }% %BeginExpansion \mathbb{N} %EndExpansion ,$ and suppose $\left( X_{t}\right) _{t\geq 0}$ is a continuous-time Markov process with ...
1
vote
1answer
69 views

A question about Infinitesimal generator of Feller Process

Let $S=% %TCIMACRO{\U{211d} }% %BeginExpansion \mathbb{R} %EndExpansion $, and consider the Feller process $\left( X_{t}\right) _{t\geq 0}$ with state space $S$ such that $X_{t}=t+X_{0}$ for all ...
1
vote
1answer
97 views

Discrete-time Markov chain properties

A Markov chain in discrete time is irreducible, has state space $\{0,1,\dots\}$ and starts at $1$. It is both a branching process and a martingale. Determine the probability of hitting $0$.
4
votes
0answers
122 views

Conditional probability and integrating out part of a random walk

Suppose that I have a random walk process defined by $\alpha_{t+1}$ ~ N$(\alpha_t, \omega^2)$. Given $\alpha_t$ and $\alpha_{t+2}$, I understand why the conditional formula for ...
0
votes
1answer
98 views

Canonical Markov Process

Let $X$ be a canonical, right-continuous Markov process with values in a Polish state space $E$, equipped with Borel-$\sigma$-algebra $\mathcal{E}$ and we assume that $t\rightarrow E_{X_{t}}f(X_{s})$ ...
0
votes
1answer
112 views

three-state Markov chain

a male and a female go to a 2-table restaurant on the same day. each day the male sits at one or the other of the 2 tables, starting at the table 1, with a Markov chain transition matrix: ...
0
votes
1answer
68 views

Rewriting Markov process

Let $X$ be a Markov proces with state space $(E,\mathcal{E})$with initial distribution $\nu$ and transition function $P_{t}$, so $$E_{\nu}(f(X_{t+s})\mid\mathcal{F}_{s})=P_{t}f(X_{s})$$ Suppose that ...
0
votes
1answer
58 views

Question on Markov chains of expected number of states

I am confused with an statement from my probability book that has to do with Markov chains. I hope someone could clarify that, if possible....Consider a Markov chain for which $P_{11}=1$ and ...
1
vote
2answers
187 views

Diffusion process. Distribution vs transition probability.

I need confirmation on the following problem: Take a SDE of the form: \begin{equation} dX_t=a(X_t,t)dt+b(X_t,t)dW_t \end{equation} where all the conditions, such that the solution $X_t$ is defined ...
0
votes
1answer
61 views

General State Space Markov Chain

I am having some difficulty understanding some early results of Markov Chain theory on a general state space. We have a function (Kernel) $K:E \times E \rightarrow \mathbb{R}$, and a distribution ...
0
votes
0answers
50 views

Variability in estimations over a non-ergodic/non-regular Markov process

Imagine we have a non-ergodic/non-regular Markov Process with with $n$ states. Among these $n$ states, there are $k$ absorbing states. For each of the $n-k$ non-absorbing states, it is not possible ...
1
vote
1answer
61 views

Identity in Markov Processes

I want to know if my reasoning here is correct, it seems simple enough but I just want clarification (I am considering the proof that if a Markov process satisfies the detailed balance condition, then ...
0
votes
0answers
84 views

Markov Chains Worked Example (Stirzaker)

I have a Markov Chain with state space the non-negative integers. The rules of the M.C. are that when it is in state $i \neq 0$, it moves to one of {${0,1,2,\ldots,i+1}$} with probability $1/(i+2)$ ...
1
vote
1answer
158 views

Have there some discrete-time continuous-state Markov processes been studied?

I have seen discrete-time discrete-state Markov processes (such as random walks), continuous-time discrete-state Markov processes (such as Poisson processes), and continuous-time continuous-state ...
0
votes
1answer
52 views

On discrete-time stochastic attractivity

Let $m$ be a probability measure on $Y \subseteq \mathbb{R}^p$, so that $m(Y)=1$. Consider a function $f: \mathbb{R}^n \times Y \rightarrow \mathbb{R}^n$, continuous on the first arguments, ...
3
votes
2answers
382 views

Probability of Extinction in a simple Birth and Death Process

We are asked to show that the probability of extinction $\zeta=\lim_{t\to \infty} P\left(X(t)=0\right)$ given by: $$\zeta=\begin{cases}1&\text{if }\lambda\le \mu,\\ \left(\frac \mu\lambda ...