A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
0answers
58 views

Ornstein-Uhlenbeck a Markov process

Consider the Ornstein-Uhlenbeck process defined by $$ X_t = e^{- \alpha t} X_0 + \sigma \int_0^t e^{ \alpha (s-t)} d W_s$$ with $\sigma,\alpha>0$. In many literature I have found they considered ...
0
votes
0answers
5 views

Long time statistics of random functions

I'd like to understand if an average over random functions can be approximated with a Markov process in the long-time limit. Let $$ X_t = \sum_k a_k \cos(\omega_k t + \phi_k) $$ a random function, ...
0
votes
0answers
23 views

Example of Markov process not strong Markov

This is again a question about this example (also see here). It seems we can write this process as $$X_t := \bigl(t - \text{Exp}(1) \mathbf{1}_{\{X_0 = 0\}}\bigr)^+ + \mathbf{1}_{\{X_0 \neq 0 ...
1
vote
1answer
46 views

Markov property when conditioning on future event

I am reading through an introduction to h-transform (available on https://linbaba.wordpress.com/2010/06/02/doob-h-transforms/), and came upon the following equality: $$P\left(X_{t+s}=y;X_T\in ...
1
vote
0answers
78 views

distribution of the length for a random walk on an infinite 2D grid

In connection with the flatland paradox, consider a 2D-random walk $(X_n)$ on $\mathbb{Z}^2$: the four moves of length one to W,E,N, and S are equaly likely at each time. For a fixed number of moves ...
0
votes
0answers
46 views

Generator of Wiener process and its running maximum

If we let $W$ be a standard linear Wiener process issued from zero and $M$ its running maximum $$ M_t := \sup \{ W_u: u \leq t \}, $$ then we could show that $(X,Y):=(M,M-W)$ is a Markov process on ...
2
votes
0answers
65 views

Absorbed brownian motion is a Markov process

I have been asked to prove that the Brownian motion absorbed at the origin is a Markov process. Formally, let $B_t^x$ be a Brownian motion originating from $x>0$ and let $\tau^x_0 = \inf\{t>0 : ...
0
votes
0answers
8 views

Simple Hidden Markov Model with Autoregressive Structure - Estimation?

I observe a two series over time $Y_{1:T}=\{ Y_{1}, \dots, Y_{T}\}$ and $X_{1:T}=\{ X_{1}, \dots, X_{T}\}$ where the $X$ series supposed to be exogenous (I do not define any stochastic proecess for ...
1
vote
0answers
41 views

Transition function for absorbed Brownian motion

I need an help with the following exercise. I've already seen this question Prove that Brownian Motion absorbed at the origin is Markov but I don't understand the answer. Also I would like to prove ...
1
vote
0answers
27 views

Infinitesimal generator of a semigroup

I know that if $\{T_t, t>0 \}$ is a conservative Markov semigroup on E, and $f \in D(A)$ has an absolute maximum in x then $Af(x) \le 0$. Where $D(A)$ is the infinitesimal generator of $T_t$. I ...
0
votes
0answers
75 views

Limit of a sequence of Marlov processes

Let $(X_n)$ be a sequence of Markov processes on, say, $[0,1]$, that converges in finite dimensional distributions to a process $X$. Is it true that $X$ must also be a Markov process ?
0
votes
0answers
44 views

markov process and markov chains

I have learned that Markov processes are stochastic processes possessing certain mathematical properties (memoryless, etc). My question is, if you say that a process is Markov, is it automatic (as a ...
4
votes
0answers
34 views

Martingale converges to the boundary

I asked an almost same question before and it is solved by considering adjacent $Z_n$ can not be far away and obtain a contradiction. However, if the setting is altered a bit, I wonder whether it is ...
5
votes
1answer
203 views

Convergence of Ornstein-Uhlenbeck process as a scaled Brownian Motion

Let $W$ be a standard Brownian motion. Let $\alpha,\sigma^2 >0$, and let $X_0$ be a $\mathbb{R}$-valued random variable with distibution $\nu$ that is independent of $\sigma(W_t,t\geq 0)$. Now ...
0
votes
0answers
16 views

Diagonalization of Markov Matrices

If we're given a $ \displaystyle 2 \times 2 $ Markov Matrix (so all entries are non-negative and columns add to 1) M$(a,b)$ such that $$M = M(a,b) := \begin{bmatrix}1 - a & b\\a & 1 - b ...
0
votes
0answers
11 views

Example of Semi Markov Process, that isn't a Markov Chain in Continuous Time?

Question says it all I hope. I have an exam in Stochastic Processes tomorrow and one question that may be asked is to give an example of a Semi-Markov Process that isn't a Markov Chain in Continuous ...
1
vote
1answer
52 views

Showing that a Markov jump process is a Feller-Dynkin process

Let $E$ be a countable state space with $\sigma$-algebra $2^E$ and $X_t$ a Markov jump process with transition function $$P_t(x,y) = \sum_{n=0}^\infty e^{-\lambda t}\frac{(\lambda ...
2
votes
2answers
214 views

Is the reflected Brownian Motion a Markov process

Let $W$ be a Brownian Motion (BM). The reflected BM is defined by $X=|X_0+W|$. We need to show that this process is a Markov process w.r.t. its natural filtration and we need to compute its ...
0
votes
1answer
18 views

Eigenvector / eigenvalue pairs for a Markov Matrix

If we're given a $ \displaystyle 2 \times 2 $ Markov Matrix (so all entries are non-negative and columns add to 1) M$(a,b)$ such that $$M = M(a,b) := \begin{bmatrix}1 - a & b\\a & 1 - b ...
0
votes
1answer
15 views

Finding eigenpairs for Markov Matrices

If we're given a $ \displaystyle 2 \times 2 $ Markov Matrix (so all entries are non-negative and columns add to 1) M$(a,b)$ such that $$M = M(a,b) := \begin{bmatrix}1 - a & b\\a & 1 - b ...
1
vote
1answer
43 views

Existence of steady state distribution for finite state Markov chains

Let's assume a Markov chain has 2 recurrent classes and a transient state from which we can go to either of the recurrent classes. If one of those recurrent classes is periodic, would it effect the ...
1
vote
1answer
26 views

Birth-death process: What is the distribution of reached states before reaching an absorbing state?

Intro I am working on a birth-death process. For a given choice of parameter ($n=6$, $Wa=1$, $Wb=0.95$, see below), the transition matrix is $$\left( \begin{array}{ccccccc} 1. & 0.144928 & ...
1
vote
0answers
24 views

Merging rates on a CTMC model

first time question here. I'm having a rough time trying to represent the following CTMC. Any help would be gladly appreciated. We consider a server with a infinite buffer connected to a network. ...
0
votes
0answers
21 views

Recurrent Markov chain: probability of visiting state i precisely k times in N steps

I'm studying this Markov process with transition matrix $P$, given by \begin{equation} P=\left(\begin{array}{cccc} \mu & 1-\mu & 0 & 0\\ 0 & 0 & \mu & 1-\mu\\ \mu & 1-\mu ...
0
votes
0answers
28 views

How do I compute the expected value of a function of two correlated random variables?

I'm trying to figure out how to properly compute the expected value of a function of two random variables and constants. The two random variables determine the state transitions in an MDP: The states ...
0
votes
1answer
22 views

Limiting distribution of a Markov chain?

I have the problem below. There are n identical machines. They are all operational at time 0. The lifetime of each one is an exponential random variable with rate L. There are r repairmen (1 ≤ r ≤ ...
1
vote
1answer
68 views

Is it worth playing this game of St. Petersburg paradox?

A gambler offers you the following deal. You have to keep tossing a fair coin until you get a heads, at which point you stop and collect your winnings: if it happens after n throws, the gambler will ...
0
votes
1answer
31 views

Stationary distribution of a birth and death process

I'm supposed to determine the stationary distribution, when it exists, for a birth and death process having constant parameters $\lambda_n=\lambda$ for $n=0,1,2,...$ and $\mu_n=\mu$ for $n=1,2,...$ ...
1
vote
1answer
59 views

Adjustment coefficient problem

Claims arrive at an insurance company as a Poisson process {$N(t) : t \ge 0$} at rate $\lambda > 0$ and $X_i$ is the claim size of the $ith$ claim. I assume that {$X_i, i=1,2,...$} is iid ...
1
vote
0answers
22 views

Examples of Non-Markov process with continuous time and finite set of states.

What is the best real world examples of non-Markov process with continuous time, but with finite set of states?
1
vote
1answer
31 views

Pure death poisson process

I have a pure death process $X=\{X(t) : 0 \leq t < \infty\}$ with parameters $\lambda_n=0$ and $\mu_n=\mu$ and if $X(0)=N$ and I'm supposed to determine $P_n(t)=P\{X(t)=n\}$ for $n=0,1,2,\ldots,N$ ...
4
votes
1answer
72 views

Expectation and limit of a stop-and-go traveler markov chain

The velocity $V(t)$ of a stop and go traveler is a two-state Markov chain whose generator is given by $$ \begin{array}{cc} &\begin{matrix}0&1\end{matrix}\\ \ \begin{matrix}0\\ 1\end{matrix} ...
1
vote
1answer
26 views

Poisson Expectation of a price asset

If I have that the price of some asset is given by: $S(t)=s \times exp{(\alpha-\lambda \sigma)t} (\sigma + 1)^{N(t)}, t \ge 0$ where $s=S(0)>0, \alpha>0, \sigma > -1$ and $\lambda > 0$ ...
1
vote
1answer
17 views

Poisson conditional probability

If I assume that $\Pr\{N(t) : t \ge 0\}$ is a Poisson process with intensity $\lambda$, I'm supposed to evaluate $\Pr\{N(s)=3\mid N(t)=8\}$ for $0 < s < t$. Since $N(t)$ is poisson, would I ...
0
votes
1answer
78 views

probability, random walk, Markov chain question

Let $P$ be a transition matrix for a regular Markov chain and let $w$ be it’s equilibrium vector. Show that $w$ has no zero entries.
0
votes
1answer
75 views

Mean return time in Markov chain

Given the following Markov chain: $p_{0,1}=1$ (if we are in state 0, we must go to state 1) $p_{i,i+1}=p_{i,i-1}=0.5$ There are infinitely (countably) many states. I assume that $X_0=0$ and define ...
1
vote
2answers
23 views

Prove that markov chain is recurrent

I have the following markov chain : $S=\{0,1,2,3\}$ $p_{i,0} = q$ (if we are in one of the states $0,1,2,3$ we can return to $0$ with probability $q$) $p_{i,i+1} = 1-q , i\in\{0,1,2\}$ (if we are ...
1
vote
0answers
38 views

Markov factorization of the density of an AR(1) process

Suppose we have a causal, stationary AR$(1)$ process with i.i.d. innovations $Z_t$. Then we know that it is a Markov as future value $X_{t+1} = \phi X_t + Z_{t+1}$ given the past $X_1,\ldots X_t$ ...
0
votes
0answers
9 views

Markov Process under Binomial model

I have the following definition of a markov process: Consider the Binomial asset-pricing model. Let $X_0$, $X_1$.., $X_n$ be an adapte process. If for every $n$ between $0$ and $N-1$ and for ...
0
votes
0answers
20 views

How to find the number of transitions, after which the stationary distribution could be found in Markov chain?

Say I have the initial state space vector S = [1 0 0]. and I know both the transition matrix, P and final stationary distribution, S' = [0.3 0.5 0.2]. If I was asked to calculate after how many ...
0
votes
0answers
21 views

“Simple” proof about expected number of visits

Let $X_n$ be a markov chain with state space $\Omega$. Let $G(x,A)$ denote the expected number of visits to $x \in A$ before exiting a subset $A \subset \Omega$. Prove that for all $x,y$ and A, ...
-2
votes
1answer
46 views

Markov processes Hitting times

I'm having trouble understanding what hitting times are in Markov chain processes and how they are calculated. An example follows: A Markov process on $E = \{1, 2, 3\}$ has the following generator ...
0
votes
0answers
14 views

Finding a One Step Transition Matrix for a Markov Process? (Gambling Application)

I need help finding what a one step transition matrix would look like for the following gambling scenario: Using the bold strategy, say you have a certain amount of money x at any time and you're ...
2
votes
1answer
63 views

Show that a Markov Chain is ergodic

Let $Y_n$ be iid random variables with values 1,2,3..n so that $P[Y_i=j]=p_j>0$, where $i\leq1$ and $1\leq j\leq n$. I think I managed to show that $Y_n$ is a Markov chain using the definition, ...
1
vote
1answer
38 views

Stronger version of Markov Chain

I have just started looking into the concept of Markov chains and I was wondering if anyone could help me with this problem. Let $X_1, X_2, ...$ be a Markov chain with the state space $S$. I need ...
0
votes
0answers
31 views

Two-state Markov Chains

If I have a two-state Markov chian $V(t)$ with transition probabilities: $P_{00}(t)=(1-\pi) + \pi e^{-\tau t}$ $P_{01}(t)= \pi - \pi e^{-\tau t}$ $P_{10}(t)=(1-\pi) - (1-\pi)e^{-\tau t}$ ...
0
votes
0answers
15 views

Computing smoothed state distribution in HMM

Suppose we have an HMM with two states: $s_1$ and $s_2$. The transitional model is as follows: $P(s_1|s_1) = 0.5$, and $P(s1|s2) = 0.25$. There are two observations: $P(a|s_1) = 0.25$ and $P(a|s_2) = ...
2
votes
1answer
60 views

Random walks : Hitting and recurrence Times relation

I have trouble understanding that how $$E\left[T_0|X_{0} = 0\right] = 1 + E[H_0|X_0=1] $$ where $T_0 = \inf\{n \geq 1:X_n = 0 \}$ and $H_A =\inf\{ n\geq 0: X_n \in A \}$. In other words $T_0$ is the ...
1
vote
1answer
49 views

Question about HMM

I have this HMM model that I need to solve. Unfortunately, my textbook isn't the best and only describes general cases which I have difficulty working with. Consider an HMM with two states: s1 and ...
1
vote
0answers
32 views

capacity of biased random walk in $\mathbb{Z}^2$

Let $P_{x,y}$ the probability that a random walk starting from $x$ will ever visit $y$. Consider a biased random walk in $\mathbb{Z}^2$. Let $A_k$ be the set of vertices having a distance less than ...