A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
0answers
38 views

Derivation of Kolmogorov Backward Equation for Inhomogeneous CTMC

I'm trying to clear something up regarding inhomogeneous CTMCs, and I just can't seem to get a proof working. So, I'm hoping that someone here could maybe give me some pointers :) I'm considering a ...
1
vote
1answer
27 views

Conditional Probabilities Poisson Process

If I let ${X(t); t>=0}$ be a Poisson process having rate parameter $\lambda = 2$. I'm supposed to determine the probability: Pr{${X(1)>=2 | X(1) >=1}$} My solution: I looked at this as ...
1
vote
1answer
23 views

Conditional Distribution Poisson Process

In class, our professor told us to verify this solution on our own time. The problem is: Let $\left\{X(t),t \geq 0 \right\}$ be a Poisson process of rate $\lambda$. For $s,t >0$, determine the ...
0
votes
1answer
33 views

Proving that the Markov chain is recurrent - Confusion/Help

Giving the following transition matrix [ 0.9 0.1 ] [ 0.8 .2 ] Classify the states From drawing the graph I know that both stats are recurrent. However I'm really failing to prove mathematically ...
1
vote
0answers
9 views

Is the steady state of a uniform markov chain always a vector of proportions?

Given that all edges in a markov chain are bi-directional (though not necessarily equally weighted), and each edge for a given node has equal probability, does the steady state always converge to a ...
0
votes
1answer
56 views

Finding the Kolmogorov's Backward equation

SO this question is probably really easy, I am just struggling in understanding how to do it It goes like this: we have a system with 3 components, at time $t=0$, component 1 is active and the other ...
1
vote
1answer
75 views

Strong Markov property of Bessel processes

I am thinking about the following: If $(B_t)_{t \geq 0}$ is a Brownian motion in $\mathbb{R}^3$, how can we show that the Bessel process (of order $3$) $(|B_t|)_{t \geq 0}$ has the strong Markov ...
0
votes
1answer
27 views

Understanding the proof of stationary distribution of a markov chain

I am reading the proof of existence of stationary distribution in an irreducible markov chain from the book Markov Chains and Mixing Times by P. D. A. Levin, Y. Peres, E. L. Wilmer, and I have the ...
1
vote
0answers
25 views

Preservation of the Markov Property for SDEs

Let $X$ be a continuous Markov process on $\mathbb{R}^d$ that is also a semimartingale. Let $V=(V_1,...,V_d)$ be a collection of suitably nice vector fields on $\mathbb{R}^d$ such that there exists a ...
0
votes
0answers
9 views

Proof of the “Markovian property” for the LERW?

I'm trying to understand this proof by Werner of the Markovian property of the Loop-erased random walk http://arxiv.org/pdf/math/0303354v1.pdf (page 10). The first part I see but the second "again, ...
1
vote
0answers
33 views

Is {Yt} a Brownian motion?

Suppose {B(t)} and {B˜(t)} are two independent standard Brownian motions and ρ is a constant, −1 < ρ < 1. The process Y(t) = ρB(t) + sqrt(1- ρ^2)*B˜(t) is distributed as a normal random variable ...
3
votes
1answer
53 views

Looking for an example of a process that holds the Markov property but doesn't hold the strong Markov property

I am desperately looking for a Markov process which does only hold the Markov property but doesn't hold the strong Markov property. All examples I can think of hold the Markov property, as well as the ...
1
vote
2answers
65 views

How to compute the variance of number of coin flips to see HTH sequence using linear system of equations.

Assuming fair coin flips, I know how to compute the expected number of coin flips to see HTH sequence by writing out the linear system of equations from the state transition diagram below. Define ...
1
vote
1answer
31 views

Can I think of both arrival times and service times in a Markov chain as Poisson processes?

According to the Wikipedia article about M/M/1 queues, the rate at which new jobs arrive is a Poisson process with parameter $\lambda$, and the rate at which the jobs are finished is an exponential ...
0
votes
1answer
33 views

Calculating the information per symbol of a markov chain source

I have a 4-state 2nd order markov chain source with symbols 0 and 1. I have all the transition probabilities and have worked out the probabilities of each state. How do I go about finding the amount ...
2
votes
0answers
54 views

Why are inter arrival times in the continuous version of discrete-time Markov chains always exponentially distributed?

I am curious whether there exist continuous time Markov processes for which the times between jumping times (which I call inter arrival times) are not exponentially distributed, but have some other ...
0
votes
0answers
24 views

Mean and Variance of an offspring

If I have that the number of offspring of an individual in a population is $0$, $1$, or $2$ with respective probabilities $a>0$, $b>0$ and $c>0$, where $a+b+c=1$, how would I express the mean ...
2
votes
1answer
36 views

Finding mean and variance of a population problem

A population beings with a single individual. In each generation, each individual in the population dies with probability $1/2$ or doubles with probability $1/2$. If I let $X_n$ denote the number of ...
4
votes
1answer
78 views

Find the Stationary Distribution of an infinite state Markov chain

A Markov Chain on states 0,1,..... has transition probabilities $P_{ij}=1/(i+2)$ for j=0,1,....,i,i+1. I'm supposed to find the stationary distribution. So do I take the limit as n goes to ...
1
vote
2answers
44 views

Limiting Distribution of a Markov Chain

I'm having trouble understanding how to find a limiting distribution. If I have a Markov Chain whose transition probability matrix is: $$ \mathbf{P} = \matrix{~ & 0 & 1 & 2 & 3 & ...
0
votes
1answer
31 views

Measurability of a stopping time in a Markov chain

Suppose you have a finite-state continuous-time inhomogeneous Markov chain with transition rate $Q(t)$. Further, let us suppose that $Q(t)$ is a piecewise continuous function of $t$. Two questions: ...
3
votes
1answer
71 views

Markov Property Confusion

I feel like I'm being very dense/employing some sort of circular reasoning, but I'm having trouble understanding the Markov Property. According to Durrett (ISBN-10:1461436141), $X_n$ is a Markov chain ...
2
votes
2answers
28 views

why are the recurrent classes closed?

i've recently started studying about markov chain, we call a communication class a recurrent one in a markov chain if by starting from that class we infinitely return to it with probability 1,with ...
2
votes
1answer
47 views

Resolvent of a Markov process

I have a question about theory of Markov processes. Let $(\Omega,\mathcal{F},P)$ be a probability space. Let $E$ be a Hausdorff topological space and $\mathcal{B}(E)$ be its Borel $\sigma$-algebra ...
2
votes
0answers
18 views

Showing which classes are recurrent and which are transient

If I have a Markov chain on states {0,1,2,3,4,5} $$ \mathbf{a} = \matrix{~ & 0 & 1 & 2 & 3 & 4 & 5 \\ 0 & 1/3 & 0 & 2/3 & 0 & 0 & 0 \\ ...
0
votes
1answer
25 views

Invertibility of operators related to Markov processes in Ethier-Kurtz

Lemma 2.3 of the book by Ethier and Kurtz (first edition, I believe) defines $$ g_n := (\lambda - A)(\lambda_n - A)^{-1}g $$ for some fixed $ g $ but I see no guarantee that $(\lambda_n - A)^{-1} g ...
1
vote
1answer
52 views

Markov chain - Can anyone explain me why this is the solution?

Customers arrive according to a Poisson process at a rate of four customers per hour. A customer who finds four other customers in already waiting gives up and leaves. Some clients in the 3rd ...
1
vote
1answer
26 views

Show that the process created from taking kth steps of a markov chain is markov.

Suppose $(X_n)_{n\geq0}$ is a Markov chain with transition probability matrix $P$ and initial distribution $\lambda$. Show that the process $Y_n = (X_{kn})_{n\geq0}$ with $k$ fixed is Markov with ...
6
votes
3answers
183 views

Probability of going into an absorbing state

If I have a random walk Markov chain whose transition probability matrix is given by $$ \mathbf{P} = \matrix{~ & 0 & 1 & 2 & 3 \\ 0 & 1 & 0 & 0 & 0 \\ ...
2
votes
1answer
25 views

Showing Stronger result of Weak Law of Large Numbers

So, Khintchine's form of the Weak Law of Large Numbers asserts that $i) E(X_1)=0 \Rightarrow (S_n/n) \rightarrow 0$ The stronger result is: $ii) E(X_1)=0 \Rightarrow E(\|S_n\|)=o(n)$ Now ii) is ...
0
votes
1answer
34 views

Expected number of transition events to complete multiple synchronized Markov chains

Assume the expected number of transitions (events) it takes until a Markov chain with $G+1$ states ranging from $s=0$ to $s=G$ is completed is $M$. Suppose we have $K$ independent instances of this ...
1
vote
1answer
50 views

Markov Chain with heterogeneous transitions

I have a Markov chain as follows: $G+1$ finite states, it begins from $s=G$ and completes at $s=0$ A transition ($s\to s-1$) occurs in case if event $A$ happens. No other form of transition is ...
0
votes
0answers
42 views

Specifying transition probabilities for a Markov Chain

If I have a queueing model and I suppose at most a single customer arrives during a single period, but that the service time of a customer is a random variable Z with geometric probability ...
2
votes
1answer
153 views

The expected time until reaching a specified set in a Markov chain

I am reading an article in which they discuss a specific Markov chain in an example, and it turns out I need to sharpen up my Markov knowledge. First the setup. I have a continuous time Markov chain ...
0
votes
1answer
54 views

Urn Problem-Determining the Transition Probability Matrix

I have two urns A and B containing a total of N balls. An experiment is performed where a ball is selected at random (all selections equally likely) at time t(t=1,2,...) from the totality of N balls. ...
0
votes
0answers
48 views

Define a maximization problem as an optimal stopping problem

We work over $\mathbb{R}_+^L$. Let $V$ be the set of vectors whose coordinates take values $0$ or $1$. Let $\mathbf{w}(t)$ (in $\mathbb{R}_+^L$) a vector that changes each time slot. To each vector ...
1
vote
1answer
45 views

Finding the transition probably matrix

If I have an urn that contains six tags, three are red and three are green. Two tags are selected from the urn. If one tag is red and the other is green, then the selected tags are discarded and two ...
2
votes
0answers
31 views

Define Markov chain and rewrite to recursively solve

Customers arrive at a server with rate $\lambda$ and are served at rate $\mu$. The server breaks down with rate $\gamma$, which causes all customers to leave. New customers can only arrive once the ...
2
votes
0answers
57 views

Model as a continuous time Markov Chain

A system consists of two machines, of which one works and the other is standby. Only the working machine can break down (with rate $\lambda$). If it breaks down the other machine takes over (if it ...
1
vote
0answers
44 views

Discrete Laplacian

I have the following question and I can't figure out how to do the proof. Could you give me some hints in both directions of the equivalence? Suppose $A$ is a bounded subset of $\mathbb{Z}^d$. Then ...
1
vote
0answers
19 views

Opposite of Absorbing State

This should be fairly standard, but I fail to google it, and nothing on the matter is on Math.SE. How do we call the opposite of an absorbing state? If we think about Markov chains/systems, that ...
1
vote
0answers
32 views

How to Simplify a Markov chain in order to estimate the average number of transitions to reach to a final state?

Is there any approach to approximate the expected number of transitions to complete a Markov chain without knowing the exact transition probabilities? The reason I ask this is because I want to ...
2
votes
0answers
23 views

Probability of going from a set $S$ to its complement on a Markov chain

I need to show that if $\pi$ is the stationary distribution of a Markov chain $M$, then for every set of vertices $S$, the probability to choose a random node in $S$ according to $\pi$ and then going ...
0
votes
4answers
51 views

Expected number of steps

I play a game as follows. A bucket contains four red balls and three green balls. At each step, a ball is chosen at random from the bucket, with each of the balls there being equally likely to be ...
0
votes
0answers
13 views

How to estimate a hidden model for an unstationary Markov process?

I have a problem that is very similar to the one solved by the Baum–Welch algorithm. I have a process that is very similar to a hidden Markov process. The only difference is that I have an observable ...
1
vote
1answer
33 views

Determining probabilities Markov Chain

If I have a Markov Chain $X_0, X_1, X_2 \dots$ that has a transition probability matrix $ \textbf{P} = \matrix{~ & 0 & 1 & 2 \cr 0 & 0.3 & 0.2 & 0.5 \cr ...
1
vote
1answer
67 views

Determining a transition probability matrix

If I have that $X_n$ is a two-state Markov chain whose transition probability matrix is: $P = \left( \begin{smallmatrix} \alpha & 1-\alpha\\ 1-\beta & \beta \\\end{smallmatrix} \right)$ ...
-1
votes
1answer
48 views

Calculate expected value for a lazy Random Walk

Calculate the mean of time needed for a lazy random walk on $[0,n]$ which starts on $0<k<n$ to hit $0$ or $n$ if in each step the walk stays in probability $\frac 1 3$, goes to the right in ...
0
votes
1answer
138 views

Random Surfer as a Markov Chain

Consider a random surfer who begins at a web page (a node of the web graph) and executes a random walk on the Web as follows. At each time step, the surfer proceeds from his current page A to a ...
0
votes
2answers
88 views

In M/M/1 Markov process, why must entering and leaving the zero state be equal?

According to the image below, which I snipped from this article, the rate of leaving State 0 and the rate of arriving into ...