A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...
0
votes
1answer
55 views
Canonical Markov Process
Let $X$ be a canonical, right-continuous Markov process with values in a Polish state space $E$, equipped with Borel-$\sigma$-algebra $\mathcal{E}$ and we assume that $t\rightarrow E_{X_{t}}f(X_{s})$ ...
0
votes
1answer
37 views
three-state Markov chain
a male and a female go to a 2-table restaurant on the same day.
each day the male sits at one or the other of the 2 tables, starting at the table 1, with a Markov chain transition matrix:
...
0
votes
1answer
45 views
Rewriting Markov process
Let $X$ be a Markov proces with state space $(E,\mathcal{E})$with initial distribution $\nu$ and transition function $P_{t}$, so $$E_{\nu}(f(X_{t+s})\mid\mathcal{F}_{s})=P_{t}f(X_{s})$$
Suppose that ...
0
votes
1answer
25 views
Question on Markov chains of expected number of states
I am confused with an statement from my probability book that has to do with Markov chains. I hope someone could clarify that, if possible....Consider a Markov chain for which $P_{11}=1$ and ...
1
vote
2answers
37 views
Diffusion process. Distribution vs transition probability.
I need confirmation on the following problem: Take a SDE of the form:
\begin{equation}
dX_t=a(X_t,t)dt+b(X_t,t)dW_t
\end{equation}
where all the conditions, such that the solution $X_t$ is defined ...
0
votes
1answer
30 views
General State Space Markov Chain
I am having some difficulty understanding some early results of Markov Chain theory on a general state space.
We have a function (Kernel) $K:E \times E \rightarrow \mathbb{R}$, and a distribution ...
0
votes
0answers
20 views
Variability in estimations over a non-ergodic/non-regular Markov process
Imagine we have a non-ergodic/non-regular Markov Process with with $n$ states.
Among these $n$ states, there are $k$ absorbing states.
For each of the $n-k$ non-absorbing states, it is not possible ...
0
votes
1answer
42 views
Identity in Markov Processes
I want to know if my reasoning here is correct, it seems simple enough but I just want clarification (I am considering the proof that if a Markov process satisfies the detailed balance condition, then ...
0
votes
0answers
26 views
Probability - How to calculate Xt by time t
A large group of cars are waiting for a car wash, and the line is infinitely long, always a car in line waiting to be washed, and that the staff can only tend to 1 car at a time. The business has 1 ...
0
votes
0answers
58 views
Markov Chains Worked Example (Stirzaker)
I have a Markov Chain with state space the non-negative integers. The rules of the M.C. are that when it is in state $i \neq 0$, it moves to one of {${0,1,2,\ldots,i+1}$} with probability $1/(i+2)$ ...
0
votes
0answers
21 views
On discrete-time stochastic attractivity of linear systems
Let $m$ be a probability measure on $Y \subseteq \mathbb{R}^p$, so that $m(Y)=1$.
Consider a continuous function $f: \mathbb{R}^n \rightarrow \mathbb{R}^p$. Assume that $f(0) = 0$, and that there ...
1
vote
1answer
32 views
Have there some discrete-time continuous-state Markov processes been studied?
I have seen discrete-time discrete-state Markov processes (such as random walks), continuous-time discrete-state Markov processes (such as Poisson processes), and continuous-time continuous-state ...
0
votes
0answers
36 views
Continuous time markov chain
Jobs arrive at a central computer according to a $PP(\lambda)$. The job processing times are i.i.d. $\exp(\mu)$. The computer processes them one at a time in the order of arrival. The computer is ...
0
votes
1answer
46 views
On discrete-time stochastic attractivity
Let $m$ be a probability measure on $Y \subseteq \mathbb{R}^p$, so that $m(Y)=1$.
Consider a function $f: \mathbb{R}^n \times Y \rightarrow \mathbb{R}^n$, continuous on the first arguments, ...
3
votes
2answers
75 views
Probability of Extinction in a simple Birth and Death Process
We are asked to show that the probability of extinction $\zeta=\lim_{t\to \infty} P\left(X(t)=0\right)$ given by:
$$\zeta=\begin{cases}1&\text{if }\lambda\le \mu,\\
\left(\frac \mu\lambda ...
0
votes
1answer
41 views
Metropolis Hastings definition - Proving $\pi(x)$ is the invariant density of our transition matrix
I'm currently working through the proof of the Metropolis-Hastings algorithm, and using two sources:
page 328, section 3
page 1704-1705
I have a good understanding of most of the proof until ...
1
vote
0answers
23 views
Single evaluation for using exponential sampling until past a point
I am trying to improve an algorithm that looks like the following (and am getting stumped): I am provided with a starting time, rate, and a target time. I then use an exponential distribution to ...
1
vote
0answers
83 views
Why Markov matrices always have 1 as an eigenvalue
Also called stochastic matrix. Let
$A=[a_{ij}]$ - matrix over $\mathbb{R}$
$0\le a_{ij} \le 1 \forall i,j$
$\sum_{j}a_{ij}=1 \forall i$
i.e the sum along each column of $A$ is 1. I ...
1
vote
1answer
38 views
Hitting times of Markov chain/process have always finite moments?
Consider an irreducible ergodic Markov chain on a finite state space $\Omega$. Then any state is positive recurrent and this should suffice to conclude that the mean hitting time of state $s \in ...
0
votes
2answers
42 views
Specifying differential equation that describes a particular set of dynamics.
There are $S$ individuals who are susceptible to infection, and $I$ who are infectious. $S + I = N$, where $N$ is the total size of the population.
Each infectious transmit the disease to a ...
4
votes
1answer
51 views
Showing a process is not markov
I keep searching but I can't find any place that gives a good method of showing a process is NOT Markov. The definition I am using is that for every $s<t$ and $g$ bounded borel there is $f$ borel ...
2
votes
1answer
32 views
Amount of information a hidden state can convey (HMM)
In this paper (Products of Hidden Markov Models, http://www.cs.toronto.edu/~hinton/absps/aistats_2001.pdf), the authors say that:
The hidden state of a single HMM can only convey log K bits of ...
0
votes
2answers
35 views
How do you explain $f(x_4|x_3)f(x_3|x_2)f(x_2|x_1)f(x_1) = f(x_4,x_3,x_2,x_1)$?
Let $x_1=x(n_1)$, $x_2=x(n_2)$, $x_3=x(n_3)$ and $x_4=x(n_4)$ be random Markov processes $(n_1 < n_2 < n_3 < n_4)$.
I don't understand the identity given below on their probability density ...
-2
votes
2answers
83 views
Is first order moving average a Markov process?
Given first order moving average
$$
x(n) = e(n) + ce(n-1)
$$
where $e(n)$ is a sequence of Gaussian random variables with zero mean and unit variance which are independent of each other, and $c$ is ...
3
votes
0answers
105 views
Is there monotone class theorem used in one of these steps?
IN Rogers & Williams "Diffusions, Markov Process and Martingales" they introduce the resolvent as:
$$R_\lambda f(x):=\int_{[0,\infty)}e^{-\lambda t}P_tf(x)dt=\int_ER_\lambda(x,dy)f(y)$$
where ...
0
votes
1answer
27 views
A book on finite state continuous time Markov chain
I want to read in detail about finite state continuous time Markov chain. Can anybody suggest a book which deal this topic in detail?
2
votes
1answer
49 views
Random Process derived from Markov process
I have a query on a Random process derived from Markov process. I have stuck in this problem for more than 2 weeks.
Let $r(t)$ be a finite-state Markov jump process described by
...
0
votes
2answers
94 views
Expected value of stochastic process
I have the following problem:
$X_1,X_2,...$ are positive identically distributed random variables with the distribution function $F(x) :=P(X_n \leq x)$ and we assume that $F(0)<1$ for all $n$. Let ...
1
vote
0answers
47 views
Conditional distributions of (higher-order) autoregressive Markov processes
If we specify an $p$-th order autoregressive process in discrete time by its transition distribution $F_{t|t-1,\ldots,t-p}$, what can be said about lower order conditional distribution where we ...
2
votes
2answers
37 views
Question on MIT Markov Matrices video
Markov matrices are pretty new to me and I'm a little rusty with my linear algebra. My question stems from watching this video from YouTube on Markov matrices. For those who wish to skip the video, ...
0
votes
1answer
33 views
Showing a certain random process is a Markov Process
I have the following example of a random process: A person has two houses, house A and house B in which he can stay, we denote by $X_{i}\in\left\{ A,B\right\}$
the house he stayed in on the i-th day ...
1
vote
2answers
39 views
How to prove the existence of the limit of Markov transition matrix?
Does the limit of a Markov transition matrix $M$:
$$\lim_{n\to\infty}M^n$$
always exist? And if yes, how to prove it?
10
votes
1answer
192 views
Motivation of Feynman-Kac formula and its relation to Kolmogorov backward/forward equations?
Kolmogorov backward/forward equations are pdes, derived for the semigroups constructed from the Markov transition kernels.
Feynman-Kac formula is also a pde corresponding to a stochastic process ...
0
votes
3answers
229 views
Finding the transition probability matrix, two switches either on or off..
Each of two switches is either on or off during a day. On day n, each switch will independently be on with probability
[1+number of on switches during day n-1]/4
For instance, if both switches are on ...
0
votes
1answer
74 views
Markov Process with Stationary Distribution
I have the following problem:
If I have a markov process with stationary distribution. The state space for the MP is integers. I also know that $P_{i,j}>0$ for all i and j. It is also given that ...
0
votes
0answers
35 views
Markov Model Brainteaser
An orangutan and a chimpanzee each sit at a computer typing 1 character per second. The orangutan chooses each character independently from $S$ with probability $\frac{1}{27}$. The chimpanzee follows ...
0
votes
0answers
53 views
A different Markov property definition
In Shreve's Stochastic Calculus in Finance, the Markov property is defined as
Definition 2.3.6. Let $(\Omega,\mathcal F,P)$ be a probability space, let $T$ be a fixed positive number, and let ...
0
votes
0answers
82 views
What are the definitions of a diffusion process and a jump process?
I have seen following different definitions of a diffusion process and of a jump process. I was wondering how they are actually defined?
Also are diffusion processes and jump processes necessarily ...
0
votes
0answers
32 views
Is HMM discriminative or generative?
Wikipedia
"An HMM can be considered as the simplest dynamic Bayesian network." Here.
"In probability and statistics, a generative model is a model for randomly generating observable data, ...
1
vote
0answers
54 views
How to prove ergodic property from aperiodicity and positive recurrence
How to prove that in case of an irreducible, aperiodic and positive recurrent Markov Chain time average along sample paths is equal to the ensemble average ? i.e.
$$\lim_{n\to \infty ...
3
votes
1answer
125 views
Multidimensional infinitesimal generator of a jump-diffusion
Let $X=\{X_t\}_{t\geq0}$ be an $n$-dimensional Markov process, defined by the SDE
$$dX_t = \mu(t, X_t) \, dt + \sigma(t,X_t) \, dB_t+\beta(t-,X_{t-}) \, dN_t,$$
where $\mu, \sigma$ and $\beta$ are ...
1
vote
1answer
113 views
Markov Chain Transition Intensity Conversion
I have a question about converting a 3-state discrete state, continuous-time, markov chain to a 2-state.
My 3-state model has states: Well (state 1), Ill (state 2) and Dead (state 3).
...
5
votes
0answers
76 views
Question on Conditional expectation
Let $X_1$ and $X_2$ be two random variables on $(\Omega,\mathcal{B},P)$. Suppose there is a function $g:\mathcal{B}\times\mathbb{R}\rightarrow[0,1]$ such that for any $x$, $g(\cdot,x)$ is a ...
0
votes
2answers
34 views
Different limiting distributions but they both satisfy same equations
I needed to find the limiting distribution of the matrix
$$\pmatrix{ 0 & 0 & 1 \\ 1 & 0 & 0 \\ \frac{1}{2} & \frac{1}{2} & 0}$$
Instead of $\pi$ I'll use $A, B$ and $C$ ...
0
votes
0answers
37 views
HMM - how to calculate p(x[t]=i)?
Suppose we have an HMM, where $y_t$ are the observations and $x_t$ are the latent states:
$p(y_1,\ldots,y_T,x_1,\ldots,x_T) = p(x_1)\prod_{t=1}^Tp(x_{t+1}|x_t)p(y_t|x_t)$
Suppose we already used the ...
1
vote
1answer
46 views
HMM as special case of MRF
I have learned that any Hidden Markov Model (HMM) can be described as a special case of a Markov Random Field (MRF) model.
However, AFAIK, the dependencies in a HMM are directed, while the ...
0
votes
1answer
63 views
Markov Process: Show that the minimum time taken to get back to state $1$ is $(0.5)^{k-1}$
Suppose that the chain is intitially in state $1$, i.e $P(X_0 = 1) = 1$. Let $\tau$ denote the time of first returen to state $1$, i.e
$$\tau = \min\{n > 0: X_N = 1\}.$$
Show that
...
0
votes
1answer
51 views
geometric sum - weighted random walk
I am trying to model the following sum:
$\sum_{i=0}^{n}{W_i \alpha^{i}}$
where $\alpha \in[0, 1) $ and $W_n$ takes values 0 or 1 and may be modeled as a markow chain or for simplicity as a binary ...
1
vote
1answer
66 views
Is this Markov?
Consider a process $\{X_n, n\geq 0\}$ with state space $S=\{0,1,2\}$ s.t.
$$
P(X_{n+1}=j | X_n=i, X_{n-1}=i_{n-1}, \dots, X_0=i_0)=\begin{cases}
P_{ij}^I \ \ \ n \ \mbox{ even},\\
P_{ij}^{II} \ \ \ ...
4
votes
1answer
58 views
Fixed point of transition kernel generates martingale
Let $P^{h}, h \geqslant 0$ be a transition kernel for some homogenous Markov process $X_t$, $\mathbb{E}|X_t|<\infty$:
$$
P_{X_{t+h},X_t}(A,B) = \int\limits_{A}P^h(x,B)P_{X_t}(dx)
$$
where ...
