A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
1answer
20 views

Markov Decision Process - Optimal policy invariance to scaling in the Utility Function

The title says it all. If i use a discounted Utility Function, why is the optimal policy invariant with respect tot the scaling of the Utility Function by a positive Factor?
2
votes
0answers
48 views

A theorem in the paper “Noncommuting Random Products” by Furstenberg

I have a question concerning the proof of theorem 2.5 at page 395 of the paper Noncommuting Random Products, by H. Furstenberg, Trans. Amer. Math. Soc., 1963. The statement is as follows: Let $\mu$ ...
1
vote
0answers
35 views

What does stationarity of the point process entail in a Markovian setting?

Let $(\Omega,\mathcal{F},P)$ be a probability space where $\Omega$ is the set of cadlag trajectories from $\mathbb{R}$ to a countable state space $S$. Let $X$ be the coordinate process $X_t(\omega) = ...
5
votes
1answer
114 views

Log-likelihood function

I'm not sure if this could be asked here, or in math overflow... In the following paper Cho, Jin Seo, and Halbert White. "Testing for regime switching." Econometrica 75.6 (2007): 1671-1720. doi: ...
0
votes
0answers
71 views

MDP problem - How is the expected cost calculated?

I have been stuck with a problem for a while regarding Markov Decision Processes for a Policy improvement algorithm. Assume that I have probabilities for certain states to evolve the system into, ...
2
votes
1answer
148 views

Equivalent defining Markov property

Consider the stochastic process $(X_t)_{t \in \mathbb{R}}$ and show the equivalence of the following two Markov properties: (a) $P(X_t \in A \mid X_u, u \leq s) = P(X_t \in A\mid X_s) \qquad ...
0
votes
0answers
48 views

Is this two dimensional Markov chain correct for this queueing system?

The problem that I have two single server station with no queuing space a customer goes to station 1 if it is available else it goes to station 2 if it is available or it will be lost output from ...
1
vote
1answer
53 views

Strong Markov property given transition functions

Suppose we are given family of transition functions satisfying Chapman-Kolmogorov equation, what conditions will ensure that there exists a continuous or cadlag Markov process with given transition ...
1
vote
0answers
43 views

State Space Difference Linear Dynamic System

I am interested in finding the DIFFERENCE in the state space distributions for two linear dynamical systems (System A and System B). I am able to solve for this using the matrix exponential. But the ...
0
votes
0answers
63 views

transition matrix for Markov chain

Can any one help me to solve this home work please? The city of Sacramento recently completed a new light rail system to bring commuters and shoppers into the downtown area and relieve freeway ...
1
vote
1answer
51 views

Estimate the probability using Markov chains

please consider this question: A study using Markov chains to estimate a patient's prognosis for improving under various treatment plans gives the following transition matrix as an example ...
0
votes
1answer
58 views

Stationary distribution for a Markov chain which is not irreducible

I have a Markov chain with $K$ states $S$: {$s_1,s_2,...,s_K$}. $s_1$ is reachable from any state in $S$; however not all the states can be reached from $s_1$. What does the stationary distribution ...
0
votes
0answers
38 views

Does time homogeneity imply strong Markov property in a Markovian process

Does a time homogeneous Markovian process necessarily have strong Markovian property? Does continuity in state space, time, or path make a difference? What are the examples if it does not?
0
votes
0answers
37 views

An example of a simple ergodic diffusion process?

I'm looking for a simple example (ideally two-dimensional) of an ergodic diffusion process with polynomial drift vector and diffusion matrix for which there a no known explicit expressions for the ...
2
votes
0answers
48 views

Examples of decreasing-in-some-time-interval variance of a time homogeneous Markovian process

Let $x_t$ be a zero mean, time homogeneous Markovian process over time $t$ starting from $x_0=0$. What are the examples of $x_t$ where the variance at $t$ decrease over some interval of $t$? The ...
0
votes
0answers
72 views

A matrix of size $n\times n$ with several properties like Markov matrices

Could you find a square matrix $A=[a_{ij}]$ of size $n$ such that satisfies to following properties 1) For all $1\le i\le n$, $\sum_{j=1}^n a_{ij}=0$ 2) For all $i$, $a_{ii}<0$ and for $1\le i\ne ...
2
votes
1answer
126 views

Proof that Markov Chains converges to the stationary distribution

Let $P$ is a transition matrix of a Markov Chain, which is irreducible, aperiodic and lets assume $\pi$ is its stationary distribution: $\pi = \pi P$. Does anyone knows the proof for the following ...
1
vote
0answers
19 views

Probabilistic event triggered on a Markov Process transition

I would like to assess a system disponibility using a Markov Process. This system has two states : a functionning state 0 and a failure state 1, with a fault rate $\lambda$ and a mean time to repair ...
0
votes
0answers
58 views

Continuous-Time Markov Chain conditioned on not visiting a part of the state space

Let $(X_t)_{t\geq 0}$ be a homogeneous continuous-time Markov chain with state space $\Psi =\{1,\dots,N\}$. Consider $S=\{1,\dots,M\} \subset \Psi$. Define $T = \inf \{ t \geq 0 : (X_t \not \in S ...
2
votes
0answers
27 views

Exsistence and uniqueness of stationary density for Markov Chain

Suppose we're given a function $f:\mathbb{R}^2\to\mathbb{R}$. We define a Markov Chain $(X_n)$ by \begin{align} X_0&\sim f_X, \\ X_n&=f(X_{n-1},Y_{n-1}), \end{align} where $(Y_n)$ is a ...
0
votes
1answer
27 views

Asymmetric Markov process

The limiting distribution $\xi(x)$ of a Markov process $$x_0=1\text{ and }x_{i+1}=x_i+\Delta x_i,\tag1$$ where $\Delta x_i=-ax_i$ and $\Delta x_i=a$ occur with equal probability for every $i$, and ...
0
votes
0answers
13 views

Complementary of a conditio al probability

I'm trying to get this equality: $P(X\geq 2|X\geq 1) = 1-P(X\geq 1)$ Where X = "number of visits to a state x of a Markov process (simple symmetric random walk) before returning to 0" I tried one way ...
1
vote
0answers
18 views

Markov operators

Transition probability functions can always be used to generate Markov operators, correct? So is it correct to say that a Markov process is a collection of Markov operators? On the other hand, are ...
0
votes
0answers
36 views

How to prove theorem about consistency of Markov edge process?

How to prove such theorem: Markov edge process $p_E(y_E)$ with respect to DAG $G=(V,E)$ defined as $p_E(y_E) = \prod_{v \in V} p_E\left(y_{E_{\rm out}(v)} \,\big|\, y_{E_{\rm in}(v) } \right) = ...
0
votes
1answer
53 views

Help With Eigenvectors and Dynamical Systems

I have the following system of differential equations: $ \frac{d}{dt} \left[ \begin{array}{c} A(t)\\ N(t)\\\end{array} \right] = \left[ \begin{array}{c c} -(a+b) & 0\\ a & -(a+b)\\ ...
1
vote
0answers
94 views

First and second moments of recurrence time in a finite two-dimensional Markov chain

I have a two dimensional finite Markov chain with $(m+1)^2$ states, and with transition rates: $q_x((x,y)\to (x+1,y))=(m-x)\lambda,\quad 0\leq x< m, 0\leq y \leq m$, $q_x((x,y)\to ...
2
votes
1answer
83 views

Markov chain and hitting times

I have a Problem about hitting times. That's the following: Let $A\subset E$ and the first passage time $T_A$ and the hitting time $H_A$. Define: $T_A =\inf\{n\geq 0;X_n \in A\}$ and $H_A ...
0
votes
1answer
42 views

Steady state distribution for Markov pure jump process

Assuming an irreducible, positive recurrent Markov Pure jump markov process with state space, $S={0,1}$ The embedded Markov Chain which is doubly stochastic (i.e) columns and rows of the transition ...
1
vote
1answer
58 views

Calculating cumulative Markov Chain outcomes

I have a Markov process, with 2 possible states (1 or 0) and a transition matrix P. State at time t=n is determined by x0*Pn. As n goes to infinity, xn goes to the steady state vector, q = [q1 q2]. ...
0
votes
0answers
26 views

Why must a stochastic process be at least second order in terms of differential equations?

A first order differential equation in $q(t)$ has a unique path through each possible value of $q(0)$. This is opposed to a stochastic process (e.g. random walk), where any place might be "hopped ...
2
votes
1answer
73 views

transition kernel

I've got some trouble with transition kernels. We look at Markov process with statspace $(S,\mathcal{S})$ and initial distribution $\mu^0$. We have a transition kernel $P:S\times ...
0
votes
0answers
64 views

Financial mathematic with Feynman-Kac

I have a really big task in financial mathematics and a small part of it (to set up the problem), I need to write a PIDE (the Feynman-kac) where we estimate options with jumps. It is derived from the ...
0
votes
1answer
34 views

Does a Markov Blanket allow connections between Parents of a Node?

In a Markov Blanket, we can connect the childredn of a node between them, as a child can be parent (or spouse) of another child. Does this rule apply as well for Parents of a node? In advance, Thank ...
0
votes
0answers
40 views

Under what circumstance should I use a Continuous time Markov Chain instead of a discrete time Markov Chain?

Why should I use one over the other, if I can basically reduce the small time-interval $h$ to be small enough that it simulates continuity? I guess this question is somewhat analogous to control ...
1
vote
0answers
63 views

Find Markov policy that minimizes(maximizes) the expected discounted cost(reward)

It's an exam problem I found online.Here's a link to the pastpaper. The problem is stated as follows. A repairman who services Q facilities moves between location s and location j according to the ...
2
votes
1answer
90 views

What is the difference between positive presistent and null persistent state in a Markov Chain?

I'm not looking for the difference in the mathematical definition, but rather for an intuitive explanation of their differences and possible examples, so that I can have them in my head when ...
0
votes
0answers
83 views

Markov Chains Proof using Statistics

Source: This came from "Introduction to probability" by Charles Miller Grinstead, and James Aurie Snell. It was located on page 407 and is Theorem 11.1 in the section 11.1 Introduction. Theorem: The ...
1
vote
1answer
71 views

Transition function is a Markov semigroup?

How does the transition function in a Markov process become a Markov semigroup in time homogeneous Markov processes? Thanks a lot.
2
votes
2answers
178 views

Question on the proof of the simple Markov property of a Brownian motion

Today we proofed the (simple) Markov property for the Brownian motion. But I really don't get a crucial step in the proof. The theorem states in particular that for $s\geq0$ fixed, the process ...
1
vote
1answer
45 views

makov chains and property

let $X_t$ for $t \in \{1,2,...,10\}$ be a sequence of independent tosses of a fair coin. We denote heads with 1 and tails with 0. Define the random variable $S_n=\sum_{t=1}^n X_t$ for $n \in ...
0
votes
1answer
85 views

Markov time $ T= \min\{n : X[n] = 1\}$

Let $T$ is a Markov time such that $T= \min \{ n : X[n] = 1\}$ , $X[n]$ is the number of $h$ (heads) in coin tossing for $n$ times. Let's say I will toss the coin 3 times, so the event collection is ...
1
vote
0answers
64 views

Proof for a Markov process example (using measure theory)

Consider the probability space $(\mathbb{R},B(\mathbb{R}),\delta_x)$ for a given $x\in\mathbb{R}$ (where $\delta_x$ is the Dirac measure) and define the process $X_t(\omega)=\omega - t$, for $t\geq ...
1
vote
1answer
79 views

How to Prove by definition, the given process is a Markov Process?

Define the process Xt by X0 = 1, and for t = 1, 2, . . . Xt = { uXt-1, with probability p, { vXt-1, with probability 1-p where 0 < v < 1 ...
1
vote
1answer
94 views

How is a simple birth process is time-homogeneous?

Why is it that a simple birth process is time-homogeneous? The incidence of a birth in a small time interval depends on the population size at the time of start of the interval. Doesn't this ...
2
votes
0answers
70 views

Interpretation of potential kernels for Markov processes

One can associate a strongly continuous contraction semi group (SCCSG) to a Markov process with state space $S$ through its transition function, say $P_t$. Now one can interpret $P_t$ as a linear ...
1
vote
1answer
137 views

Joint density of Markov process as product of conditional densities.

Let $(X(t))_{t\geq0}$ be a Markov proces such that s $X(0)=x_0$. Consider the random vector $(X(t_1),\dots,X(t_n))$ with corresponding joint density $g(x_1,\dots ,x_n)$. Is it then true that ...
1
vote
0answers
28 views

Advanced reference in Markov processes

I am interested in a book which covers the more in depth stuff on continuous time Markov processes e.g. semi-groups, generators... Preferably such book would also contain a list of analysis results ...
1
vote
1answer
299 views

Why is this infinite-state-space Markov chain positive recurrent?

Given the following transition matrix for a Markov chain, how can I see that the chain is positive recurrent? I want to convince myself that the chain has a limiting distribution, and the chain is ...
1
vote
4answers
173 views

Why do these expressions tend to zero?

In a chapter on Markov chains, it is claimed that $\binom {2n}{n} p^n(1-p)^n$ (where $p$ is a probability) tends to $0$ as $n$ tends to $\infty$. But why is this so? It is also claimed that for an ...
2
votes
2answers
105 views

How to understand Markov property?

I'm learning stochastic process in college. How to understand Markov property?I'm curious about what is the power and validity of Markov property ? A stochastic process has the Markov property if ...