A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

3
votes
0answers
12 views

Martingale converges to the boundary

I asked an almost same question before and it is solved by considering adjacent $Z_n$ can not be far away and obtain a contradiction. However, if the setting is altered a bit, I wonder whether it is ...
1
vote
0answers
18 views

Convergence of Ornstein-Uhlenbeck process

Let $W$ be a standard Brownian motion. Let $\alpha,\sigma^2 >0$, and let $X_0$ be a $\mathbb{R}$-valued random variable with distibution $\nu$ that is independent of $\sigma(W_t,t\geq 0)$. Now ...
0
votes
0answers
15 views

Diagonalization of Markov Matrices

If we're given a $ \displaystyle 2 \times 2 $ Markov Matrix (so all entries are non-negative and columns add to 1) M$(a,b)$ such that $$M = M(a,b) := \begin{bmatrix}1 - a & b\\a & 1 - b ...
0
votes
0answers
9 views

Example of Semi Markov Process, that isn't a Markov Chain in Continuous Time?

Question says it all I hope. I have an exam in Stochastic Processes tomorrow and one question that may be asked is to give an example of a Semi-Markov Process that isn't a Markov Chain in Continuous ...
1
vote
1answer
17 views

Showing that a Markov jump process is a Feller-Dynkin process

Let $E$ be a countable state space with $\sigma$-algebra $2^E$ and $X_t$ a Markov jump process with transition function $$P_t(x,y) = \sum_{n=0}^\infty e^{-\lambda t}\frac{(\lambda ...
0
votes
3answers
58 views

Is the reflected Brownian Motion a Markov process

Let $W$ be a Brownian Motion (BM). The reflected BM is defined by $X=|X_0+W|$. We need to show that this process is a Markov process w.r.t. its natural filtration and we need to compute its ...
0
votes
1answer
15 views

Eigenvector / eigenvalue pairs for a Markov Matrix

If we're given a $ \displaystyle 2 \times 2 $ Markov Matrix (so all entries are non-negative and columns add to 1) M$(a,b)$ such that $$M = M(a,b) := \begin{bmatrix}1 - a & b\\a & 1 - b ...
0
votes
1answer
14 views

Finding eigenpairs for Markov Matrices

If we're given a $ \displaystyle 2 \times 2 $ Markov Matrix (so all entries are non-negative and columns add to 1) M$(a,b)$ such that $$M = M(a,b) := \begin{bmatrix}1 - a & b\\a & 1 - b ...
0
votes
1answer
28 views

Existence of steady state distribution for finite state Markov chains

Let's assume a Markov chain has 2 recurrent classes and a transient state from which we can go to either of the recurrent classes. If one of those recurrent classes is periodic, would it effect the ...
1
vote
1answer
19 views

Birth-death process: What is the distribution of reached states before reaching an absorbing state?

Intro I am working on a birth-death process. For a given choice of parameter ($n=6$, $Wa=1$, $Wb=0.95$, see below), the transition matrix is $$\left( \begin{array}{ccccccc} 1. & 0.144928 & ...
1
vote
0answers
20 views

Merging rates on a CTMC model

first time question here. I'm having a rough time trying to represent the following CTMC. Any help would be gladly appreciated. We consider a server with a infinite buffer connected to a network. ...
0
votes
0answers
12 views

A question about the translation property Markov kernel

Given that ${X_n}$ is a Markov chain, and a Markov kernel with translation propert$p(y+x,E+x)=p(y,E)$. Question:How to show $Y_n=X_n-X_{n-1}$ are i.i.d? I'm trying to use Markov Property and ...
0
votes
0answers
12 views

Recurrent Markov chain: probability of visiting state i precisely k times in N steps

I'm studying this Markov process with transition matrix $P$, given by \begin{equation} P=\left(\begin{array}{cccc} \mu & 1-\mu & 0 & 0\\ 0 & 0 & \mu & 1-\mu\\ \mu & 1-\mu ...
0
votes
0answers
23 views

How do I compute the expected value of a function of two correlated random variables?

I'm trying to figure out how to properly compute the expected value of a function of two random variables and constants. The two random variables determine the state transitions in an MDP: The states ...
0
votes
1answer
18 views

Limiting distribution of a Markov chain?

I have the problem below. There are n identical machines. They are all operational at time 0. The lifetime of each one is an exponential random variable with rate L. There are r repairmen (1 ≤ r ≤ ...
1
vote
1answer
36 views

Is it worth playing this game of St. Petersburg paradox?

A gambler offers you the following deal. You have to keep tossing a fair coin until you get a heads, at which point you stop and collect your winnings: if it happens after n throws, the gambler will ...
0
votes
1answer
20 views

Stationary distribution of a birth and death process

I'm supposed to determine the stationary distribution, when it exists, for a birth and death process having constant parameters $\lambda_n=\lambda$ for $n=0,1,2,...$ and $\mu_n=\mu$ for $n=1,2,...$ ...
2
votes
1answer
40 views
+50

Adjustment coefficient problem

Claims arrive at an insurance company as a Poisson process {$N(t) : t \ge 0$} at rate $\lambda > 0$ and $X_i$ is the claim size of the $ith$ claim. I assume that {$X_i, i=1,2,...$} is iid ...
1
vote
0answers
20 views

Examples of Non-Markov process with continuous time and finite set of states.

What is the best real world examples of non-Markov process with continuous time, but with finite set of states?
1
vote
1answer
26 views

Pure death poisson process

I have a pure death process $X=\{X(t) : 0 \leq t < \infty\}$ with parameters $\lambda_n=0$ and $\mu_n=\mu$ and if $X(0)=N$ and I'm supposed to determine $P_n(t)=P\{X(t)=n\}$ for $n=0,1,2,\ldots,N$ ...
0
votes
0answers
39 views

Markov Chain problem application [on hold]

Let $P$ be the transition matrix for a regular Markov chain and $v$ be its equilibruim vector. Show that $v$ has zero entries. How would you prove this? I am struggling in this class. Any help is ...
4
votes
1answer
58 views
+50

Expectation and limit of a stop-and-go traveler markov chain

The velocity $V(t)$ of a stop and go traveler is a two-state Markov chain whose generator is given by $$ \begin{array}{cc} &\begin{matrix}0&1\end{matrix}\\ \ \begin{matrix}0\\ 1\end{matrix} ...
1
vote
1answer
26 views

Poisson Expectation of a price asset

If I have that the price of some asset is given by: $S(t)=s \times exp{(\alpha-\lambda \sigma)t} (\sigma + 1)^{N(t)}, t \ge 0$ where $s=S(0)>0, \alpha>0, \sigma > -1$ and $\lambda > 0$ ...
1
vote
1answer
17 views

Poisson conditional probability

If I assume that $\Pr\{N(t) : t \ge 0\}$ is a Poisson process with intensity $\lambda$, I'm supposed to evaluate $\Pr\{N(s)=3\mid N(t)=8\}$ for $0 < s < t$. Since $N(t)$ is poisson, would I ...
0
votes
1answer
66 views

probability, random walk, Markov chain question

Let $P$ be a transition matrix for a regular Markov chain and let $w$ be it’s equilibrium vector. Show that $w$ has no zero entries.
0
votes
1answer
65 views

Mean return time in Markov chain

Given the following Markov chain: $p_{0,1}=1$ (if we are in state 0, we must go to state 1) $p_{i,i+1}=p_{i,i-1}=0.5$ There are infinitely (countably) many states. I assume that $X_0=0$ and define ...
1
vote
2answers
21 views

Prove that markov chain is recurrent

I have the following markov chain : $S=\{0,1,2,3\}$ $p_{i,0} = q$ (if we are in one of the states $0,1,2,3$ we can return to $0$ with probability $q$) $p_{i,i+1} = 1-q , i\in\{0,1,2\}$ (if we are ...
1
vote
0answers
33 views

Markov factorization of the density of an AR(1) process

Suppose we have a causal, stationary AR$(1)$ process with i.i.d. innovations $Z_t$. Then we know that it is a Markov as future value $X_{t+1} = \phi X_t + Z_{t+1}$ given the past $X_1,\ldots X_t$ ...
0
votes
0answers
8 views

Markov Process under Binomial model

I have the following definition of a markov process: Consider the Binomial asset-pricing model. Let $X_0$, $X_1$.., $X_n$ be an adapte process. If for every $n$ between $0$ and $N-1$ and for ...
0
votes
0answers
19 views

How to find the number of transitions, after which the stationary distribution could be found in Markov chain?

Say I have the initial state space vector S = [1 0 0]. and I know both the transition matrix, P and final stationary distribution, S' = [0.3 0.5 0.2]. If I was asked to calculate after how many ...
0
votes
0answers
20 views

“Simple” proof about expected number of visits

Let $X_n$ be a markov chain with state space $\Omega$. Let $G(x,A)$ denote the expected number of visits to $x \in A$ before exiting a subset $A \subset \Omega$. Prove that for all $x,y$ and A, ...
-2
votes
1answer
40 views

Markov processes Hitting times

I'm having trouble understanding what hitting times are in Markov chain processes and how they are calculated. An example follows: A Markov process on $E = \{1, 2, 3\}$ has the following generator ...
-4
votes
0answers
45 views

variance of time until the process jumps - Markov Chains

A Markov process on $E = \{1, 2\}$ is constructed according to holding time parameters $λ_1 = 2$ and $λ_2 = 4$; the defining Markov chain has transition probabilities $$p_{11} = p_{12} = 0.5 \quad ...
0
votes
0answers
9 views

Finding a One Step Transition Matrix for a Markov Process? (Gambling Application)

I need help finding what a one step transition matrix would look like for the following gambling scenario: Using the bold strategy, say you have a certain amount of money x at any time and you're ...
2
votes
1answer
57 views

Show that a Markov Chain is ergodic

Let $Y_n$ be iid random variables with values 1,2,3..n so that $P[Y_i=j]=p_j>0$, where $i\leq1$ and $1\leq j\leq n$. I think I managed to show that $Y_n$ is a Markov chain using the definition, ...
1
vote
1answer
37 views

Stronger version of Markov Chain

I have just started looking into the concept of Markov chains and I was wondering if anyone could help me with this problem. Let $X_1, X_2, ...$ be a Markov chain with the state space $S$. I need ...
-2
votes
0answers
10 views

Solution manual of MDP: Discrete Stochastic Dynamic Programming?

Do you know where can I get the solution to the problem sets of the book: Martin Puterman, "Markov decision processes: discrete stochastic dynamic programming". The solution manual can be very ...
0
votes
0answers
30 views

Two-state Markov Chains

If I have a two-state Markov chian $V(t)$ with transition probabilities: $P_{00}(t)=(1-\pi) + \pi e^{-\tau t}$ $P_{01}(t)= \pi - \pi e^{-\tau t}$ $P_{10}(t)=(1-\pi) - (1-\pi)e^{-\tau t}$ ...
0
votes
0answers
13 views

Computing smoothed state distribution in HMM

Suppose we have an HMM with two states: $s_1$ and $s_2$. The transitional model is as follows: $P(s_1|s_1) = 0.5$, and $P(s1|s2) = 0.25$. There are two observations: $P(a|s_1) = 0.25$ and $P(a|s_2) = ...
2
votes
1answer
57 views

Random walks : Hitting and recurrence Times relation

I have trouble understanding that how $$E\left[T_0|X_{0} = 0\right] = 1 + E[H_0|X_0=1] $$ where $T_0 = \inf\{n \geq 1:X_n = 0 \}$ and $H_A =\inf\{ n\geq 0: X_n \in A \}$. In other words $T_0$ is the ...
1
vote
1answer
42 views

Question about HMM

I have this HMM model that I need to solve. Unfortunately, my textbook isn't the best and only describes general cases which I have difficulty working with. Consider an HMM with two states: s1 and ...
-1
votes
0answers
26 views

Conditions for a Markov process to have independent increments [duplicate]

I consistently see "Let $\{X(t)\}$ be a stochastic process with independent increments..." in various texts, though I have yet to find any conditions under which we can guarantee a process to have ...
1
vote
0answers
29 views

capacity of biased random walk in $\mathbb{Z}^2$

Let $P_{x,y}$ the probability that a random walk starting from $x$ will ever visit $y$. Consider a biased random walk in $\mathbb{Z}^2$. Let $A_k$ be the set of vertices having a distance less than ...
0
votes
1answer
29 views

Expected success of trial with conditions

Assume that $n$ people want to achieve a task T. One person can try, and is successful with probability $p$. But when a person try all the other have to do an other trial to have the right to ...
2
votes
1answer
33 views

Metropolis Hastings

So I have seen the Metropolis Hastings algorithm written 2 ways, and I don't quite understand how they can be equivalent: The first way is by defining the 'acceptance probability' as: ...
0
votes
0answers
24 views

Canonical Construction of a Markov Chain: Intuition

Let $P=(p_{xy})_{x,y \in E}$ be a transition probability matrix over a discrete state space $E$ and $\mu_0$ any distribution over $E$. We proved in the lecture that there is a unique ...
2
votes
1answer
36 views

Time sampling an ordinary poisson process

My questions will be given at the end, let me just give some definitions first. The counting process $\{ N(t), t \geq 0 \} $ is said to be a non homogenous Poisson process with intensity function ...
0
votes
1answer
30 views

Markov Process - formulate a Markov chain model for this system ( what is q(i,j)?)

Potential customers arrive at a full-service, two-pump gas station according to a Poisson process at a rate of 40 cars per hour. There are two service attendants to help customers, one for each pump. ...
3
votes
1answer
63 views

Joint density function Poisson Process

We did an example in class that I'm not sure how we came up with the answer. The problem is: If I let X(t) be a Poisson process of rate $\lambda$. I'm supposed to validate the identity ...
0
votes
0answers
38 views

Derivation of Kolmogorov Backward Equation for Inhomogeneous CTMC

I'm trying to clear something up regarding inhomogeneous CTMCs, and I just can't seem to get a proof working. So, I'm hoping that someone here could maybe give me some pointers :) I'm considering a ...