A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

4
votes
3answers
3k views

Why Markov matrices always have 1 as an eigenvalue

Also called stochastic matrix. Let $A=[a_{ij}]$ - matrix over $\mathbb{R}$ $0\le a_{ij} \le 1 \forall i,j$ $\sum_{j}a_{ij}=1 \forall i$ i.e the sum along each column of $A$ is 1. I ...
1
vote
1answer
152 views

Hitting times of Markov chain/process have always finite moments?

Consider an irreducible ergodic Markov chain on a finite state space $\Omega$. Then any state is positive recurrent and this should suffice to conclude that the mean hitting time of state $s \in ...
0
votes
2answers
57 views

Specifying differential equation that describes a particular set of dynamics.

There are $S$ individuals who are susceptible to infection, and $I$ who are infectious. $S + I = N$, where $N$ is the total size of the population. Each infectious transmit the disease to a ...
4
votes
1answer
91 views

Showing a process is not markov

I keep searching but I can't find any place that gives a good method of showing a process is NOT Markov. The definition I am using is that for every $s<t$ and $g$ bounded borel there is $f$ borel ...
2
votes
1answer
61 views

Amount of information a hidden state can convey (HMM)

In this paper (Products of Hidden Markov Models, http://www.cs.toronto.edu/~hinton/absps/aistats_2001.pdf), the authors say that: The hidden state of a single HMM can only convey log K bits of ...
0
votes
2answers
49 views

How do you explain $f(x_4|x_3)f(x_3|x_2)f(x_2|x_1)f(x_1) = f(x_4,x_3,x_2,x_1)$?

Let $x_1=x(n_1)$, $x_2=x(n_2)$, $x_3=x(n_3)$ and $x_4=x(n_4)$ be random Markov processes $(n_1 < n_2 < n_3 < n_4)$. I don't understand the identity given below on their probability density ...
-2
votes
2answers
263 views

Is first order moving average a Markov process?

Given first order moving average $$ x(n) = e(n) + ce(n-1) $$ where $e(n)$ is a sequence of Gaussian random variables with zero mean and unit variance which are independent of each other, and $c$ is ...
1
vote
0answers
165 views

Is there monotone class theorem used in one of these steps?

IN Rogers & Williams "Diffusions, Markov Process and Martingales" they introduce the resolvent as: $$R_\lambda f(x):=\int_{[0,\infty)}e^{-\lambda t}P_tf(x)dt=\int_ER_\lambda(x,dy)f(y)$$ where ...
1
vote
1answer
81 views

A book on finite state continuous time Markov chain

I want to read in detail about finite state continuous time Markov chain. Can anybody suggest a book which deal this topic in detail?
2
votes
1answer
72 views

Random Process derived from Markov process

I have a query on a Random process derived from Markov process. I have stuck in this problem for more than 2 weeks. Let $r(t)$ be a finite-state Markov jump process described by ...
0
votes
2answers
451 views

Expected value of stochastic process

I have the following problem: $X_1,X_2,...$ are positive identically distributed random variables with the distribution function $F(x) :=P(X_n \leq x)$ and we assume that $F(0)<1$ for all $n$. Let ...
1
vote
0answers
79 views

Conditional distributions of (higher-order) autoregressive Markov processes

If we specify an $p$-th order autoregressive process in discrete time by its transition distribution $F_{t|t-1,\ldots,t-p}$, what can be said about lower order conditional distribution where we ...
2
votes
2answers
55 views

Question on MIT Markov Matrices video

Markov matrices are pretty new to me and I'm a little rusty with my linear algebra. My question stems from watching this video from YouTube on Markov matrices. For those who wish to skip the video, ...
0
votes
1answer
48 views

Showing a certain random process is a Markov Process

I have the following example of a random process: A person has two houses, house A and house B in which he can stay, we denote by $X_{i}\in\left\{ A,B\right\}$ the house he stayed in on the i-th day ...
1
vote
3answers
124 views

How to prove the existence of the limit of Markov transition matrix?

Does the limit of a Markov transition matrix $M$: $$\lim_{n\to\infty}M^n$$ always exist? And if yes, how to prove it?
11
votes
1answer
413 views

Motivation of Feynman-Kac formula and its relation to Kolmogorov backward/forward equations?

Kolmogorov backward/forward equations are pdes, derived for the semigroups constructed from the Markov transition kernels. Feynman-Kac formula is also a pde corresponding to a stochastic process ...
0
votes
3answers
1k views

Finding the transition probability matrix, two switches either on or off..

Each of two switches is either on or off during a day. On day n, each switch will independently be on with probability [1+number of on switches during day n-1]/4 For instance, if both switches are on ...
0
votes
1answer
121 views

Markov Process with Stationary Distribution

I have the following problem: If I have a markov process with stationary distribution. The state space for the MP is integers. I also know that $P_{i,j}>0$ for all i and j. It is also given that ...
0
votes
0answers
104 views

A different Markov property definition

In Shreve's Stochastic Calculus in Finance, the Markov property is defined as Definition 2.3.6. Let $(\Omega,\mathcal F,P)$ be a probability space, let $T$ be a fixed positive number, and let ...
1
vote
0answers
80 views

How to prove ergodic property from aperiodicity and positive recurrence

How to prove that in case of an irreducible, aperiodic and positive recurrent Markov Chain time average along sample paths is equal to the ensemble average ? i.e. $$\lim_{n\to \infty ...
3
votes
1answer
509 views

Multidimensional infinitesimal generator of a jump-diffusion

Let $X=\{X_t\}_{t\geq0}$ be an $n$-dimensional Markov process, defined by the SDE $$dX_t = \mu(t, X_t) \, dt + \sigma(t,X_t) \, dB_t+\beta(t-,X_{t-}) \, dN_t,$$ where $\mu, \sigma$ and $\beta$ are ...
1
vote
2answers
493 views

Markov Chain Transition Intensity Conversion

I have a question about converting a 3-state discrete state, continuous-time, markov chain to a 2-state. My 3-state model has states: Well (state 1), Ill (state 2) and Dead (state 3). ...
6
votes
0answers
171 views

Question on Conditional expectation

Let $X_1$ and $X_2$ be two random variables on $(\Omega,\mathcal{B},P)$. Suppose there is a function $g:\mathcal{B}\times\mathbb{R}\rightarrow[0,1]$ such that for any $x$, $g(\cdot,x)$ is a ...
0
votes
2answers
39 views

Different limiting distributions but they both satisfy same equations

I needed to find the limiting distribution of the matrix $$\pmatrix{ 0 & 0 & 1 \\ 1 & 0 & 0 \\ \frac{1}{2} & \frac{1}{2} & 0}$$ Instead of $\pi$ I'll use $A, B$ and $C$ ...
1
vote
1answer
190 views

HMM as special case of MRF

I have learned that any Hidden Markov Model (HMM) can be described as a special case of a Markov Random Field (MRF) model. However, AFAIK, the dependencies in a HMM are directed, while the ...
0
votes
1answer
85 views

Markov Process: Show that the minimum time taken to get back to state $1$ is $(0.5)^{k-1}$

Suppose that the chain is intitially in state $1$, i.e $P(X_0 = 1) = 1$. Let $\tau$ denote the time of first returen to state $1$, i.e $$\tau = \min\{n > 0: X_N = 1\}.$$ Show that ...
0
votes
1answer
146 views

geometric sum - weighted random walk

I am trying to model the following sum: $\sum_{i=0}^{n}{W_i \alpha^{i}}$ where $\alpha \in[0, 1) $ and $W_n$ takes values 0 or 1 and may be modeled as a markow chain or for simplicity as a binary ...
1
vote
1answer
203 views

Is this Markov?

Consider a process $\{X_n, n\geq 0\}$ with state space $S=\{0,1,2\}$ s.t. $$ P(X_{n+1}=j | X_n=i, X_{n-1}=i_{n-1}, \dots, X_0=i_0)=\begin{cases} P_{ij}^I \ \ \ n \ \mbox{ even},\\ P_{ij}^{II} \ \ \ ...
4
votes
1answer
89 views

Fixed point of transition kernel generates martingale

Let $P^{h}, h \geqslant 0$ be a transition kernel for some homogenous Markov process $X_t$, $\mathbb{E}|X_t|<\infty$: $$ P_{X_{t+h},X_t}(A,B) = \int\limits_{A}P^h(x,B)P_{X_t}(dx) $$ where ...
1
vote
0answers
51 views

Efficient random number generation for sojourn times in semi-Markov processes

I'm doing a self-study of semi-Markov processes and was wondering if there are efficient methods for generating random numbers for sojourn times. For example, generating a bunch of random numbers from ...
1
vote
0answers
31 views

Reference on Generators of Markov Processes

I came across these regime switching geometric Brownian Motions where the drift and volatility switches a number of states which is driven by a Markov Chain. Can someone please point me to a ...
3
votes
1answer
2k views

Markov process vs. markov chain vs. random process vs. stochastic process vs. collection of random variables

I'm trying to understand each of the above terms, and I'm having a lot of trouble deciphering the difference between them (note, my mathematics training isn't very strong - so please go easy on the ...
4
votes
1answer
374 views

Covariance of Brownian-motion-like processes

We know that $\operatorname{Cov}(B_s,B_t)=\min(s,t)$ if $B_t$ is Brownian motion. What is $\operatorname{Cov}(B_{f(s)},B_{f(t)})$ for some injective $f$? How can I write $B_{f(t)}$ in an Ito ...
3
votes
3answers
113 views

Introduction to Markov Random Fields

I'm looking for a gentle introduction to this topic. The material I've found so far is substantially related to physics, and requires some background in such field. Is there anything more general and ...
0
votes
1answer
41 views

Characterizing the Dependence Structure of a Rewards for a Finite State Homogenous Markov Chain

Let $\{X_n, n\geq 1\}$ be a finite state homogenous Markov chain with states $i = 1, \ldots, N$ . Let $g$ denote a function which returns out a reward for any given state of the Markov chain. Let ...
1
vote
2answers
97 views

Using the canonical Markov property to prove an obvious fact about Markov chains

Given a Markov chain $\{X_n: n \geq 1 \}$, such that $$\mathbb{P}(X_{n+1} = x_{n+1} | X_n = x_n) = \mathbb{P}(X_{n+1} = x_{n+1} | X_n = x_n, \ldots X_1= x_1)$$ How can I formally prove that: ...
1
vote
2answers
428 views

Expected number of visits to state $j$ between successive visits to a state $i$ in a Markov chain given conditional information

Say I have a Markov chain $\{X_n: n \geq 1\}$ with state space $E = \{1,2,3,4,5\}$ and transition matrix, $$ P = \begin{bmatrix} 0 & 1/2 & 1/2 & 0 & 0 \\\ 1/2 & 0 & 1/2 & ...
4
votes
0answers
187 views

estimation of transition probabilities from aggregate data

Please, O mathematicians, help me understand the approach to the problem of estimating transition probabilities given only aggregate data in Kalbfleisch & Lawless' 1984 paper "Least-Squares ...
1
vote
1answer
126 views

Why is the following example of a Markov process not strong Markov

$X(t) := 0 \;\; (t \leq \tau),\;\; t - \tau\;\;(t \geq \tau)$ with $\tau$ exponentially distributed. Then X has the Markov property but not Strong Markov Property. But why ???? Can someone kindly ...
3
votes
1answer
105 views

Limit of a probability regarding a random walk

Consider a random walk on the integers starting at 0, where in each step you move either 1 or 2 meters (back or forth alike). As soon as you reach either the $N$ or $N+1$ meter mark, you stop. What is ...
2
votes
0answers
64 views

What is the difference between a Markov process and a Markov chain? [duplicate]

Possible Duplicate: What is the difference between all types of Markov Chains? I've read a lot about Markov processes and chains but so far I don't understand what the difference is between ...
0
votes
0answers
79 views

$X_n/(1+r)^n$ martingale implies $X_n$ not Markov?

If $X_n/(1+r)^n$ is a martingale, I can conclude that $X_n$ is not a martingale. But can I also conclude that $X_n$ is not Markov?
1
vote
1answer
252 views

Explanation of Parsimony

Can someone explain what Parsimony is in the context of probability, more specifically in Parsimonious Markov models? I have been trying to search around a simple explanation of this but I only seem ...
3
votes
0answers
133 views

Generalization of Dobrushin's Ergodic Decomposition for continuous Markov Chains

Let $T$ be the shift transformation. Let $P$ be invariant for $T$ and also define a discrete state space Markov Chain. Let $C_{1},\ldots,C_{n}$ be the connected components of the Markov Chain. It ...
1
vote
1answer
38 views

is this the correct way to impose an additional condition on a hidden markov model?

Suppose I have an observation $Y_t$ that is conditionally dependent on $X_t$. (More specifically, Y is a series of observations emitted by underlying hidden markov state sequence X.) I can describe ...
0
votes
1answer
76 views

Linear algebra calculation using Markov process

I have a question about computing the following linear algebraic operation Suppose $u = (1, 0, 0)$ $v = \begin{pmatrix} 1\\ 0\\ 0\end{pmatrix}$ $A = \begin{pmatrix} 0.5 &0.2 &0.3\\0.2 ...
4
votes
1answer
396 views

Recursively Solving a Bellman Equation

Problem Overview I am to figure out $v_\pi$ of a certain Markov state. Given Information A set of actions, $a$ containing ${up, down, left, right}$ $v_\pi(12), v_\pi(13), v_\pi(14)$ (I am given ...
5
votes
3answers
179 views

Does an n-order Markov chain still represent a Markov process?

I am trying to understand Markov processes but am still confused by their definition. In particular, the Wikipedia page gives this example of a non-Markov process. The example is of pulling different ...
2
votes
1answer
194 views

Solving Discrete Markov Chain with diagonal band matrix.

I am trying to model a certain process as a Discrete Markov Chain. My system has $N+1$ states: $X=0, \ldots N$, and I can assume that the $(N+1)\times (N+1)$ transition matrix $T$ has the general form ...
7
votes
0answers
140 views

Uniqueness of an infinite system of linear ODEs

How to prove that $\dot{x}=ax,\space x(0)=1$ has a unique solution if $a,x$ are infinite dimensional matrices? More specifically, let $Q$ be a bounded infinitesimal generator, i.e. ...