A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
1answer
10 views

why are the recurrent classes closed?

i've recently started studying about markov chain, we call a communication class a recurrent one in a markov chain if by starting from that class we infinitely return to it with probability 1,with ...
2
votes
1answer
31 views

Resolvent of a Markov process

I have a question about theory of Markov processes. Let $(\Omega,\mathcal{F},P)$ be a probability space. Let $E$ be a Hausdorff topological space and $\mathcal{B}(E)$ be its Borel $\sigma$-algebra ...
2
votes
0answers
12 views

Showing which classes are recurrent and which are transient

If I have a Markov chain on states {0,1,2,3,4,5} $$ \mathbf{a} = \matrix{~ & 0 & 1 & 2 & 3 & 4 & 5 \\ 0 & 1/3 & 0 & 2/3 & 0 & 0 & 0 \\ ...
0
votes
1answer
23 views

Invertibility of operators related to Markov processes in Ethier-Kurtz

Lemma 2.3 of the book by Ethier and Kurtz (first edition, I believe) defines $$ g_n := (\lambda - A)(\lambda_n - A)^{-1}g $$ for some fixed $ g $ but I see no guarantee that $(\lambda_n - A)^{-1} g ...
1
vote
1answer
32 views

Markov chain - Can anyone explain me why this is the solution?

Customers arrive according to a Poisson process at a rate of four customers per hour. A customer who finds four other customers in already waiting gives up and leaves. Some clients in the 3rd ...
1
vote
1answer
24 views

Show that the process created from taking kth steps of a markov chain is markov.

Suppose $(X_n)_{n\geq0}$ is a Markov chain with transition probability matrix $P$ and initial distribution $\lambda$. Show that the process $Y_n = (X_{kn})_{n\geq0}$ with $k$ fixed is Markov with ...
6
votes
3answers
142 views

Probability of going into an absorbing state

If I have a random walk Markov chain whose transition probability matrix is given by $$ \mathbf{P} = \matrix{~ & 0 & 1 & 2 & 3 \\ 0 & 1 & 0 & 0 & 0 \\ ...
2
votes
1answer
21 views

Showing Stronger result of Weak Law of Large Numbers

So, Khintchine's form of the Weak Law of Large Numbers asserts that $i) E(X_1)=0 \Rightarrow (S_n/n) \rightarrow 0$ The stronger result is: $ii) E(X_1)=0 \Rightarrow E(\|S_n\|)=o(n)$ Now ii) is ...
0
votes
1answer
28 views

Expected number of transition events to complete multiple synchronized Markov chains

Assume the expected number of transitions (events) it takes until a Markov chain with $G+1$ states ranging from $s=0$ to $s=G$ is completed is $M$. Suppose we have $K$ independent instances of this ...
1
vote
1answer
44 views

Markov Chain with heterogeneous transitions

I have a Markov chain as follows: $G+1$ finite states, it begins from $s=G$ and completes at $s=0$ A transition ($s\to s-1$) occurs in case if event $A$ happens. No other form of transition is ...
0
votes
0answers
25 views

Specifying transition probabilities for a Markov Chain

If I have a queueing model and I suppose at most a single customer arrives during a single period, but that the service time of a customer is a random variable Z with geometric probability ...
2
votes
1answer
141 views

The expected time until reaching a specified set in a Markov chain

I am reading an article in which they discuss a specific Markov chain in an example, and it turns out I need to sharpen up my Markov knowledge. First the setup. I have a continuous time Markov chain ...
0
votes
1answer
30 views

Urn Problem-Determining the Transition Probability Matrix

I have two urns A and B containing a total of N balls. An experiment is performed where a ball is selected at random (all selections equally likely) at time t(t=1,2,...) from the totality of N balls. ...
0
votes
0answers
41 views

Define a maximization problem as an optimal stopping problem

We work over $\mathbb{R}_+^L$. Let $V$ be the set of vectors whose coordinates take values $0$ or $1$. Let $\mathbf{w}(t)$ (in $\mathbb{R}_+^L$) a vector that changes each time slot. To each vector ...
1
vote
1answer
33 views

Finding the transition probably matrix

If I have an urn that contains six tags, three are red and three are green. Two tags are selected from the urn. If one tag is red and the other is green, then the selected tags are discarded and two ...
2
votes
0answers
29 views

Define Markov chain and rewrite to recursively solve

Customers arrive at a server with rate $\lambda$ and are served at rate $\mu$. The server breaks down with rate $\gamma$, which causes all customers to leave. New customers can only arrive once the ...
2
votes
0answers
29 views

Model as a continuous time Markov Chain

A system consists of two machines, of which one works and the other is standby. Only the working machine can break down (with rate $\lambda$). If it breaks down the other machine takes over (if it ...
1
vote
0answers
42 views

Discrete Laplacian

I have the following question and I can't figure out how to do the proof. Could you give me some hints in both directions of the equivalence? Suppose $A$ is a bounded subset of $\mathbb{Z}^d$. Then ...
1
vote
0answers
14 views

Opposite of Absorbing State

This should be fairly standard, but I fail to google it, and nothing on the matter is on Math.SE. How do we call the opposite of an absorbing state? If we think about Markov chains/systems, that ...
1
vote
0answers
25 views

How to Simplify a Markov chain in order to estimate the average number of transitions to reach to a final state?

Is there any approach to approximate the expected number of transitions to complete a Markov chain without knowing the exact transition probabilities? The reason I ask this is because I want to ...
2
votes
0answers
23 views

Probability of going from a set $S$ to its complement on a Markov chain

I need to show that if $\pi$ is the stationary distribution of a Markov chain $M$, then for every set of vertices $S$, the probability to choose a random node in $S$ according to $\pi$ and then going ...
1
vote
4answers
35 views

Expected number of steps

I play a game as follows. A bucket contains four red balls and three green balls. At each step, a ball is chosen at random from the bucket, with each of the balls there being equally likely to be ...
0
votes
0answers
8 views

How to estimate a hidden model for an unstationary Markov process?

I have a problem that is very similar to the one solved by the Baum–Welch algorithm. I have a process that is very similar to a hidden Markov process. The only difference is that I have an observable ...
0
votes
1answer
25 views

How to work out the probability of starting and finishing at the same state in a Markov Chain

If I start transitioning from state i, how do I calculate the probability that we are back at state i after t number of transitions? Any help is appreciated Thanks
1
vote
1answer
27 views

Determining probabilities Markov Chain

If I have a Markov Chain $X_0, X_1, X_2 \dots$ that has a transition probability matrix $ \textbf{P} = \matrix{~ & 0 & 1 & 2 \cr 0 & 0.3 & 0.2 & 0.5 \cr ...
0
votes
1answer
37 views

Determining a transition probability matrix

If I have that $X_n$ is a two-state Markov chain whose transition probability matrix is: $P = \left( \begin{smallmatrix} \alpha & 1-\alpha\\ 1-\beta & \beta \\\end{smallmatrix} \right)$ ...
-1
votes
1answer
35 views

Calculate expected value for a lazy Random Walk

Calculate the mean of time needed for a lazy random walk on $[0,n]$ which starts on $0<k<n$ to hit $0$ or $n$ if in each step the walk stays in probability $\frac 1 3$, goes to the right in ...
0
votes
1answer
84 views

Random Surfer as a Markov Chain

Consider a random surfer who begins at a web page (a node of the web graph) and executes a random walk on the Web as follows. At each time step, the surfer proceeds from his current page A to a ...
0
votes
2answers
75 views

In M/M/1 Markov process, why must entering and leaving the zero state be equal?

According to the image below, which I snipped from this article, the rate of leaving State 0 and the rate of arriving into ...
2
votes
0answers
23 views

An irreducible Markov chain is a martingale

Let $\{X_n\}$ be an irreducible Markov chain. Does exist example of such $\{X_n\}$ which is also a martingale given that: a. $\{X_n\}$ is recurrent with finite number of states (but bigger ...
1
vote
1answer
30 views

How long does it take two identical hidden Markov models run on same observations to forget their initial distributions (if ever)?

Let $H_1$ and $H_2$ be two instances of a finite Hidden Markov Model (HMM) $H$. That is, $H_1$ and $H_2$ have identical state spaces $Q$ as well as identical transition $A$ and emission probabilities ...
0
votes
0answers
64 views

Deducing results about continuous time random walks from the corresponding discrete time result

Is there any standard way to prove results about continuous time random walks from the corresponding results for discrete time random walks? Specifically, my problem is that I was reading Lawler and ...
1
vote
1answer
40 views

Intuition behind Stopping Times

I'm attending a stocahstic processes course. I have some trouble with the intuition behind a stopping time. I will consider the discrete case to make it simpler. a stopping time is given by ...
2
votes
1answer
33 views

Can two nodes in a Markov chain have transitions that don't total 1?

In all the Markov diagrams I see, the transitions from state A to B always total to one. Just one of many examples, this image ...
1
vote
1answer
74 views

Using Markov Property in solving PDE/SDE

I am solving the PDE I used Feynman-Kac and eventually arrived at $F(t,x)$ $ = E[X_T^2|X_t = x]$ $ = E[(X_t \pm \sigma (W_T -W_t))^2|X_t = x]$ (iirc) So, I try to evaluate $E[(X_t \pm \sigma ...
0
votes
0answers
23 views

Does a state which is passed at least 3 times had to be passed 5 times in Markov chain

Prove of disprove: Let $\{X_n\}_n$ be homogenous Markov chain. if we start from state $i$, there is a positive probability that we pass at least 3 times at state $j$. Does it follows that exists ...
0
votes
1answer
63 views

Markov chain doesn't sum up to 1

Let $\{X_n\}$ be a Markov chain on $S=\{1,2,3,4,5,6\}$ with the matrix suppose we define a new sequence $\{Y_n\}$ by $$Y_n=\cases{1\quad X_n=1\vee X_n=2\\2\quad X_n=3\vee X_n=4\\3\quad ...
0
votes
0answers
29 views

Operator semigroups (lecture notes)

Can you recommend good lecture notes (or a book) about this topic? Basically I would like something which covers more or less the first chapter of the book "Markov Processes" by Ethier and Kurtz, but ...
1
vote
0answers
46 views

I have to show that the following stochastic process is a Markov process

I don't understand how to show that some stochastic processes have the Markov property. For example, if I have the following process: $$(\Omega, \mathcal{F}, (X_t)_{t \geq 0}, P^y)$$ where $\Omega = ...
0
votes
0answers
15 views

Is a Markov Decision Process finite horizon or infinite when the length of horizon depends on actions taken?

I want to model a problem as a Markov decision process but I'm not able to classify it as finite or infinite horizon. In my problem, the time for which the process will continue is a function of ...
1
vote
1answer
29 views

Uniform convergence of $A^n/n!$

In a proof regarding finite space Markov Jump Processes in which the function $P(t)=e^{tG}$ is a solution to both the backward and forward Chapman-Kolmogrov equations, one of the steps assumes that ...
1
vote
1answer
51 views

Probability of a time-dependent set of states in Markov chain

Consider a Markov matrix $P$ defining $m$ states. For each time $n$, define a set of states $S_n$ that contains a predefined subset of the states $\left\{ {1,...,m} \right\}$. For time $n=k$, I would ...
1
vote
0answers
29 views

Construction of pure birth process

I am considering a Markov chain $\lbrace X(t) \rbrace_{t≥0}$ in continunous time on the countable state space $S=\lbrace 0 \rbrace\cup \lbrace (i,j) \mid i \in \mathcal{A} , j \in \mathbb{N} \rbrace, ...
1
vote
0answers
25 views

Invariant Distribution of Two Dependent $\cdot/M/\infty$ Queues Running in Parallel

This is in preparation for an exam I have coming up. We have two $\cdot / M / \infty$ queues with external arrivals occurring according to a Poisson Process of rate $\lambda$. Service occurs with ...
1
vote
1answer
32 views

First time markov process spends tau units in certain state

Consider a continuous time Markov process $\{X(t)\}_{t≥0}$ on the state space $\{0, 1, 2, . . .\}$ with stationary probabilities $\{π_0, π_1, π_2, . . .\}$. Suppose that, when currently in state $i$, ...
3
votes
2answers
91 views

Solving recursive integral equation from Markov transition probability

How do I solve something like: $$f(x) = \frac{1}{\sqrt{2 \pi}} \int_{-\infty}^\infty e^{\frac{-(y - x/2)^2}{2}}f(y)\:\mathrm{d}y$$ for $f(x)$? Is there also a general formula that this falls under? ...
0
votes
0answers
22 views

How to prove that the rational thief problem model is monotone.

A thief goes out stealing every day and has a chance of $p_k$ of stealing a sum $k$ with $0\leq k \leq N$. But there's also a chance $q$ of getting caught, in which case he loses everything he got ...
2
votes
2answers
72 views

Equilibrium distribution of Ehrenfest's urn

(I'll post my own answer to this, but others may be of interest, so post your own if you have one.) (PS: In reply to comments posted below: Stackexchange encourages posting an answer to one's own ...
0
votes
0answers
26 views

What is the state space of this markov chain?

Consider a system where two persons sit at a table and share three books. At any point in time both are reading a book, and one book is left on the table. When a person finishes reading his/her ...
1
vote
0answers
57 views

Semigroup of operators: weak continuity at 0+ implies weak continuity at any t > 0

Let ($E$, $d$) be a metric space. Consider the semigroup $\{P(t)\}_{t\geq 0}$ of bounded linear operators on the Banach space $\hat{C}(E)$ of continuous real functions on ($E$, $d$) vanishing at ...