A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
0answers
5 views

How to find the number of transitions, after which the stationary distribution could be found in Markov chain?

Say I have the initial state space vector S = [1 0 0]. and I know both the transition matrix, P and final stationary distribution, S' = [0.3 0.5 0.2]. If I was asked to calculate after how many ...
0
votes
0answers
14 views

Stationary distribution of a CTMC

We are talking about Continuous time Markov chains in class and we are supposed to show that for all s,t > 0: $\mu (t+s)= \mu (t)*P(s)$ and $\mu_i (t+s) = $$ \sum\limits_{j \in E}\mu_j(t)P_{ji}(s)$ ...
0
votes
0answers
20 views

“Simple” proof about expected number of visits

Let $X_n$ be a markov chain with state space $\Omega$. Let $G(x,A)$ denote the expected number of visits to $x \in A$ before exiting a subset $A \subset \Omega$. Prove that for all $x,y$ and A, ...
-2
votes
1answer
35 views

Markov processes Hitting times

I'm having trouble understanding what hitting times are in Markov chain processes and how they are calculated. An example follows: A Markov process on $E = \{1, 2, 3\}$ has the following generator ...
-4
votes
0answers
36 views

variance of time until the process jumps - Markov Chains

A Markov process on $E = \{1, 2\}$ is constructed according to holding time parameters $λ_1 = 2$ and $λ_2 = 4$; the defining Markov chain has transition probabilities $$p_{11} = p_{12} = 0.5 \quad ...
0
votes
0answers
5 views

Finding a One Step Transition Matrix for a Markov Process? (Gambling Application)

I need help finding what a one step transition matrix would look like for the following gambling scenario: Using the bold strategy, say you have a certain amount of money x at any time and you're ...
2
votes
1answer
52 views

Show that a Markov Chain is ergodic

Let $Y_n$ be iid random variables with values 1,2,3..n so that $P[Y_i=j]=p_j>0$, where $i\leq1$ and $1\leq j\leq n$. I think I managed to show that $Y_n$ is a Markov chain using the definition, ...
1
vote
1answer
35 views

Stronger version of Markov Chain

I have just started looking into the concept of Markov chains and I was wondering if anyone could help me with this problem. Let $X_1, X_2, ...$ be a Markov chain with the state space $S$. I need ...
-2
votes
0answers
8 views

Solution manual of MDP: Discrete Stochastic Dynamic Programming?

Do you know where can I get the solution to the problem sets of the book: Martin Puterman, "Markov decision processes: discrete stochastic dynamic programming". The solution manual can be very ...
-1
votes
0answers
28 views

Markov Process with conditional pdfs [closed]

Suppose that Y1,Y2,Y3,Y4 comprise a Markov process. Working with conditional probability density functions, show (Y3,Y4 | Y1,Y2) = (Y3,Y4 | Y2)
0
votes
0answers
26 views

Two-state Markov Chains

If I have a two-state Markov chian $V(t)$ with transition probabilities: $P_{00}(t)=(1-\pi) + \pi e^{-\tau t}$ $P_{01}(t)= \pi - \pi e^{-\tau t}$ $P_{10}(t)=(1-\pi) - (1-\pi)e^{-\tau t}$ ...
0
votes
0answers
11 views

Computing smoothed state distribution in HMM

Suppose we have an HMM with two states: $s_1$ and $s_2$. The transitional model is as follows: $P(s_1|s_1) = 0.5$, and $P(s1|s2) = 0.25$. There are two observations: $P(a|s_1) = 0.25$ and $P(a|s_2) = ...
2
votes
1answer
55 views

Random walks : Hitting and recurrence Times relation

I have trouble understanding that how $$E\left[T_0|X_{0} = 0\right] = 1 + E[H_0|X_0=1] $$ where $T_0 = \inf\{n \geq 1:X_n = 0 \}$ and $H_A =\inf\{ n\geq 0: X_n \in A \}$. In other words $T_0$ is the ...
1
vote
1answer
41 views

Question about HMM

I have this HMM model that I need to solve. Unfortunately, my textbook isn't the best and only describes general cases which I have difficulty working with. Consider an HMM with two states: s1 and ...
-1
votes
0answers
25 views

Conditions for a Markov process to have independent increments [duplicate]

I consistently see "Let $\{X(t)\}$ be a stochastic process with independent increments..." in various texts, though I have yet to find any conditions under which we can guarantee a process to have ...
1
vote
0answers
29 views

capacity of biased random walk in $\mathbb{Z}^2$

Let $P_{x,y}$ the probability that a random walk starting from $x$ will ever visit $y$. Consider a biased random walk in $\mathbb{Z}^2$. Let $A_k$ be the set of vertices having a distance less than ...
0
votes
1answer
25 views

Expected success of trial with conditions

Assume that $n$ people want to achieve a task T. One person can try, and is successful with probability $p$. But when a person try all the other have to do an other trial to have the right to ...
2
votes
1answer
29 views

Metropolis Hastings

So I have seen the Metropolis Hastings algorithm written 2 ways, and I don't quite understand how they can be equivalent: The first way is by defining the 'acceptance probability' as: ...
-1
votes
0answers
44 views

writing down markov chain transition matrix

Question: An experimental animal can stay in room-A until 1 minute,and it can stay in room-B until 2 minutes. There exist deadly gases in room-C. One room among these three rooms is being randomly ...
0
votes
0answers
22 views

Canonical Construction of a Markov Chain: Intuition

Let $P=(p_{xy})_{x,y \in E}$ be a transition probability matrix over a discrete state space $E$ and $\mu_0$ any distribution over $E$. We proved in the lecture that there is a unique ...
2
votes
1answer
36 views

Time sampling an ordinary poisson process

My questions will be given at the end, let me just give some definitions first. The counting process $\{ N(t), t \geq 0 \} $ is said to be a non homogenous Poisson process with intensity function ...
0
votes
1answer
26 views

Markov Process - formulate a Markov chain model for this system ( what is q(i,j)?)

Potential customers arrive at a full-service, two-pump gas station according to a Poisson process at a rate of 40 cars per hour. There are two service attendants to help customers, one for each pump. ...
3
votes
1answer
61 views

Joint density function Poisson Process

We did an example in class that I'm not sure how we came up with the answer. The problem is: If I let X(t) be a Poisson process of rate $\lambda$. I'm supposed to validate the identity ...
0
votes
0answers
34 views

Derivation of Kolmogorov Backward Equation for Inhomogeneous CTMC

I'm trying to clear something up regarding inhomogeneous CTMCs, and I just can't seem to get a proof working. So, I'm hoping that someone here could maybe give me some pointers :) I'm considering a ...
1
vote
1answer
25 views

Conditional Probabilities Poisson Process

If I let ${X(t); t>=0}$ be a Poisson process having rate parameter $\lambda = 2$. I'm supposed to determine the probability: Pr{${X(1)>=2 | X(1) >=1}$} My solution: I looked at this as ...
1
vote
1answer
22 views

Conditional Distribution Poisson Process

In class, our professor told us to verify this solution on our own time. The problem is: Let $\left\{X(t),t \geq 0 \right\}$ be a Poisson process of rate $\lambda$. For $s,t >0$, determine the ...
0
votes
1answer
32 views

Proving that the Markov chain is recurrent - Confusion/Help

Giving the following transition matrix [ 0.9 0.1 ] [ 0.8 .2 ] Classify the states From drawing the graph I know that both stats are recurrent. However I'm really failing to prove mathematically ...
1
vote
0answers
8 views

Is the steady state of a uniform markov chain always a vector of proportions?

Given that all edges in a markov chain are bi-directional (though not necessarily equally weighted), and each edge for a given node has equal probability, does the steady state always converge to a ...
0
votes
1answer
51 views

Finding the Kolmogorov's Backward equation

SO this question is probably really easy, I am just struggling in understanding how to do it It goes like this: we have a system with 3 components, at time $t=0$, component 1 is active and the other ...
1
vote
1answer
45 views

Strong Markov property of Bessel processes

I am thinking about the following: If $(B_t)_{t \geq 0}$ is a Brownian motion in $\mathbb{R}^3$, how can we show that the Bessel process (of order $3$) $(|B_t|)_{t \geq 0}$ has the strong Markov ...
0
votes
1answer
25 views

Understanding the proof of stationary distribution of a markov chain

I am reading the proof of existence of stationary distribution in an irreducible markov chain from the book Markov Chains and Mixing Times by P. D. A. Levin, Y. Peres, E. L. Wilmer, and I have the ...
1
vote
0answers
24 views

Preservation of the Markov Property for SDEs

Let $X$ be a continuous Markov process on $\mathbb{R}^d$ that is also a semimartingale. Let $V=(V_1,...,V_d)$ be a collection of suitably nice vector fields on $\mathbb{R}^d$ such that there exists a ...
0
votes
0answers
9 views

Proof of the “Markovian property” for the LERW?

I'm trying to understand this proof by Werner of the Markovian property of the Loop-erased random walk http://arxiv.org/pdf/math/0303354v1.pdf (page 10). The first part I see but the second "again, ...
1
vote
0answers
33 views

Is {Yt} a Brownian motion?

Suppose {B(t)} and {B˜(t)} are two independent standard Brownian motions and ρ is a constant, −1 < ρ < 1. The process Y(t) = ρB(t) + sqrt(1- ρ^2)*B˜(t) is distributed as a normal random variable ...
3
votes
1answer
50 views

Looking for an example of a process that holds the Markov property but doesn't hold the strong Markov property

I am desperately looking for a Markov process which does only hold the Markov property but doesn't hold the strong Markov property. All examples I can think of hold the Markov property, as well as the ...
1
vote
2answers
65 views

How to compute the variance of number of coin flips to see HTH sequence using linear system of equations.

Assuming fair coin flips, I know how to compute the expected number of coin flips to see HTH sequence by writing out the linear system of equations from the state transition diagram below. Define ...
1
vote
1answer
28 views

Can I think of both arrival times and service times in a Markov chain as Poisson processes?

According to the Wikipedia article about M/M/1 queues, the rate at which new jobs arrive is a Poisson process with parameter $\lambda$, and the rate at which the jobs are finished is an exponential ...
0
votes
1answer
30 views

Calculating the information per symbol of a markov chain source

I have a 4-state 2nd order markov chain source with symbols 0 and 1. I have all the transition probabilities and have worked out the probabilities of each state. How do I go about finding the amount ...
2
votes
0answers
49 views

Why are inter arrival times in the continuous version of discrete-time Markov chains always exponentially distributed?

I am curious whether there exist continuous time Markov processes for which the times between jumping times (which I call inter arrival times) are not exponentially distributed, but have some other ...
0
votes
0answers
23 views

Mean and Variance of an offspring

If I have that the number of offspring of an individual in a population is $0$, $1$, or $2$ with respective probabilities $a>0$, $b>0$ and $c>0$, where $a+b+c=1$, how would I express the mean ...
1
vote
1answer
32 views

Finding mean and variance of a population problem

A population beings with a single individual. In each generation, each individual in the population dies with probability $1/2$ or doubles with probability $1/2$. If I let $X_n$ denote the number of ...
4
votes
1answer
66 views

Find the Stationary Distribution of an infinite state Markov chain

A Markov Chain on states 0,1,..... has transition probabilities $P_{ij}=1/(i+2)$ for j=0,1,....,i,i+1. I'm supposed to find the stationary distribution. So do I take the limit as n goes to ...
1
vote
2answers
40 views

Limiting Distribution of a Markov Chain

I'm having trouble understanding how to find a limiting distribution. If I have a Markov Chain whose transition probability matrix is: $$ \mathbf{P} = \matrix{~ & 0 & 1 & 2 & 3 & ...
0
votes
1answer
31 views

Measurability of a stopping time in a Markov chain

Suppose you have a finite-state continuous-time inhomogeneous Markov chain with transition rate $Q(t)$. Further, let us suppose that $Q(t)$ is a piecewise continuous function of $t$. Two questions: ...
3
votes
1answer
70 views

Markov Property Confusion

I feel like I'm being very dense/employing some sort of circular reasoning, but I'm having trouble understanding the Markov Property. According to Durrett (ISBN-10:1461436141), $X_n$ is a Markov chain ...
2
votes
2answers
26 views

why are the recurrent classes closed?

i've recently started studying about markov chain, we call a communication class a recurrent one in a markov chain if by starting from that class we infinitely return to it with probability 1,with ...
2
votes
1answer
46 views

Resolvent of a Markov process

I have a question about theory of Markov processes. Let $(\Omega,\mathcal{F},P)$ be a probability space. Let $E$ be a Hausdorff topological space and $\mathcal{B}(E)$ be its Borel $\sigma$-algebra ...
2
votes
0answers
17 views

Showing which classes are recurrent and which are transient

If I have a Markov chain on states {0,1,2,3,4,5} $$ \mathbf{a} = \matrix{~ & 0 & 1 & 2 & 3 & 4 & 5 \\ 0 & 1/3 & 0 & 2/3 & 0 & 0 & 0 \\ ...
0
votes
1answer
25 views

Invertibility of operators related to Markov processes in Ethier-Kurtz

Lemma 2.3 of the book by Ethier and Kurtz (first edition, I believe) defines $$ g_n := (\lambda - A)(\lambda_n - A)^{-1}g $$ for some fixed $ g $ but I see no guarantee that $(\lambda_n - A)^{-1} g ...
1
vote
1answer
50 views

Markov chain - Can anyone explain me why this is the solution?

Customers arrive according to a Poisson process at a rate of four customers per hour. A customer who finds four other customers in already waiting gives up and leaves. Some clients in the 3rd ...