Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

0
votes
0answers
6 views

M/M/1/$\infty$ queue: long term distribution of the chain

Consider a (possibly infinite) queue of people and one person serving them one by one. Say that arrival time of a new customer follows $Exp(\lambda)$ and the time of serving a customer follows $Exp(\...
0
votes
0answers
10 views

Solving MDP using TD learning and k-step estimator

I am trying to solve the following MDP using TD learning for $\lambda$. I am trying to find a value of $\lambda$ such that the TD estimate for $\lambda$ equals that of the TD(1) estimate (i.e after ...
1
vote
1answer
9 views

A aperiodic state of a Markov chain has $N\geq 1$ such that $\forall n\geq N:p_{i,i}(n)>0$

The question I get asked is the following, I'm completely stuck on the problem: Let $i$ be an aperiodic state of a Markov Chain. Show that there exists $N\geq 1$ such that $p_{i,i}(n)>0$ for all ...
0
votes
0answers
7 views

Markov property for Simple Birth Process at random time

Let $(X_t)_{t \geq 0}$ be a simple birth process with rates $\lambda_n$, $n\geq 0$ starting from $k$. The Markov property states that the two processes $(X_t)_{0 \leq t \leq r}$ and $(X_{s+r})_{r \...
0
votes
1answer
20 views

understanding Krylov-Bogolubov Theorem

Could anyone tell me what is $P(x,dy)$ and $P^{n+1}(x,dy)$ means here in 1p- 30, in the proof of thm 4.17? And, why $\phi$ was taken bounded by $1$? Are all $P^k(x,A)$, $P$, $Q^n$ probability measure ...
2
votes
0answers
21 views

Given a Markov process, are we able to construct another Markov process with the same transition semigroup but different inital law?

Let $E$ be a locally compact separable metric space $(T(t))_{t\ge0}$ be a strongly continuous contraction semigroups on $C_0(E)$ with generator $(\mathcal D(A),A)$ $(\Omega,\mathcal A,\operatorname P)...
0
votes
0answers
3 views

Lower bounds on discrete time finite Markov chains hitting probabilities.

I am interested in some general theorems related to lower bounds on discrete time finite Markov chains hitting probabilities (preferably ergodic chains , but not necessarily ), with references . ...
1
vote
0answers
24 views

Does this proof in Puterman's MDP book make sense?

I'm reading Chapter 6 of Puterman's MDP :Discrete Stocastic Dynamic Prorgamming. In the book, Bellman operator $\mathscr{L}$ is given as \begin{equation} (\mathscr{L}v)(s) = \sup_{d}\{r_d(s) + \lambda ...
0
votes
1answer
23 views

Markov property definition, for (continuous) markov processes

We have defined markov processes (in continuous time) as a collection of random variables $(X(t))_{t\in \mathbb{R_+}}$ such that in particular we have the property : $P(X(t+s)=j|X(u), 0\leq u \leq t)=...
2
votes
0answers
41 views
+50

Convergence of Feller processes implies convergence of the resolvent operators of their generators

Let $E$ be a locally compact separable metric space $(T_n(t))_{t\ge0}$ and $(T(t))_{t\ge0}$ be strongly continuous contraction semigroups on $C_0(E)$ with generator $(\mathcal D(A_n),A_n)$ and $(\...
1
vote
0answers
24 views

Spectral Theory of Markov Jump Processes (continuous-time)

I stumbled over the eigenvalue problem while analysing an infinite-dimensional jump process. The state space I am working with is $$ \mathbb{N}^{<\infty}=\bigcup_{k=1}^{\infty}\mathbb{N}^k $$ i.e. ...
1
vote
1answer
20 views

simulating a discrete markov process from a reducible transition rate matrix

I'm trying to model an irreversible, discrete Markov process. I have a set of states $S$ arranged in a tree-like structure (it is only possible to move from parent vertex to child vertex). I compute ...
1
vote
0answers
91 views
+50

How can we proof this implication in Corollary 4.8.7 in the book of Ethier and Kurtz?

I'm trying to understand the proof of the implication "(g) $\Rightarrow$ (f)" in Corollary 8.7 of Chapter 4 in the book Markov Processes: Characterization and Convergence by Stewart N. Ethier, Thomas ...
2
votes
0answers
20 views

Strong Markov property and another stopping time

I'm trying to prove that given a regular continuous time Markov chain $X_t$ (pure jump process), its embedded chain given by $Y_n=X_{T_n}$ is a homogeneous Markov chain, where $T_n$ is the time of the ...
0
votes
0answers
19 views

Prove infinite markov reward process converges

The following question is obtained from Stanford CS234 Lecture 2 notes, Excercise 3.7 Let $r_i$ denote the reward obtained from transition $s_i\rightarrow s_{i+1}$. Furthermore, the return $G_t$ of a ...
0
votes
0answers
8 views

Expected population size in Markov probability question [closed]

In a branching process, an individual has 0, 1 or 2 descendants with probability 1/4, 1/4, 1/2, respectively. Start the process at generation 0 with a single ancestor. Compute the expected population ...
0
votes
0answers
41 views

A person often finds that she is up to 1 hour late for work. A decision problem

A person often finds that she is up to 1 hour late for work. If she is from $1$ to $30$ minutes late, $\$4 $ is deducted from her paycheck; if she is from $31$ to $60$ minutes late for work, $\$8$ is ...
0
votes
1answer
42 views

Let $(Y_n)_{n \ge 1}$ be the number of fair coin tosses until 'heads' appears $n$ times. Is $(Y_n)_{n \ge 1}$ a (stationary) Markov chain?

Let $X$ be the random variable counting the number of fair coin tosses until 'heads' appears for the first time. Let $(Y_n)_{n \ge 1}$ be the number of fair coin tosses until 'heads' appears $n$ times....
0
votes
0answers
17 views

Partial Sums of Markov Chain.

Let $X_n$ be a independent identically distributed sequence of integer valued random variables. Suppose $S_n = \sum_{k=1}^n X_k$ with $S_0=0$, and $Z_n = \sum_{j=1}^n S_j$. Does $(Z_n)$ form a ...
0
votes
0answers
17 views

Expected extinction time for birth and death process

Suppose we have a birth and death process, with the states describing the number of lives in the population. Suppose the rate of going from state i to i+1 is $\lambda$ and the rate of going from i to ...
0
votes
0answers
5 views

Can we reparameterize most processes to be Markov decision processes?

In a finite MDP, the transition probability is schematically written as $$ P(s',r' | s,a). $$ This notation reflects the assumption that the environment's evolution and the yielded reward are ...
0
votes
1answer
17 views

Calculating the distribution of the number of people in a shop

This is an extract from a question that I'm thinking about. Suppose that we have a shop and clients arrive (one at a time) according to a Poisson process with some intensity, say 4 per hour. Each ...
0
votes
0answers
13 views

Is the stationary distribution the unique solution to this linear system?

We consider an MDP with a transition matrix $P \in \mathbb{R}^{n \times n}$ and assume it's ergodic. Let $\pi$ be the unique stationary distribution of this MDP, so $\pi$ satisfies $\pi^\top P = \pi^\...
0
votes
1answer
17 views

When doesn't a Markov Chain have a stationary distribution?

Is it possible for a Markov Chain not to have a stationary distribution? When doesn't a Markov Chain have a stationary distribution?
0
votes
2answers
16 views

test a Markov Matrix for a stationary distribution.

Consider a Markov chain with transition matrix $ P = \begin{bmatrix} 1/2&1/2&0\\ 1/5&4/5&0\\ 0&0&1 \end{bmatrix} $ How many stationary distributions does this chain ...
0
votes
0answers
11 views

Testing the Stationary distribution of a Markov Chain.

Consider a Markov chain $(X_n)_n$ on $S=\{1, 2\}$ with initial distribution $α$ and the transition matrix $P = \begin{bmatrix} 2/3 & 1/3 \\ 2/3 & 1/3 \\ \end{bmatrix}$ If $α = (2/...
1
vote
0answers
24 views

Are the terms Limiting distribution and Stationary distribution properly perceived?

Consider a Markov chain $(X_n)_n$ on $S=\{1, 2\}$ with initial distribution $α$ and the transition matrix $P = \begin{bmatrix} 2/3 & 1/3 \\ 2/3 & 1/3 \\ \end{bmatrix}$ Limiting ...
0
votes
0answers
17 views

Finding the probability of a Markov Chain.

If $(X_n)_{n≥0}$ is a Markov chain on $S = {1, 2, 3}$ with initial distribution $α = (1/2, 1/2, 0)$ and transition matrix $ \begin{bmatrix} 1/2&0&1/2\\ 0&1/2&1/2\\ 1/2&1/...
0
votes
0answers
38 views

Where do I find exercises very similar to this one?

In my book Operational Research by author Hamdy A. Taha is this following exercise A home supply store can place orders for fridges at the start of each month for immediate delivery. A cost of $\$ ...
0
votes
0answers
22 views

Markov probability matrix and steady state solutions

I am currently doing a question and can seem to find where my problem lies with my final solution. The question is based on markov probability and steady state solutions, but the solution I get for ...
0
votes
0answers
24 views

Average service time in a finite state Markov chain system

Consider a system with a large number of individuals that can belong to a finite number of states $s_1,s_2,\dots,s_N$. Transitions between states follow a Markov chain with stationary distribution $\...
2
votes
1answer
35 views

Do there exist diffusions that do not solve any SDE?

Diffusions are continuous time stochastic processess having continuous paths and satisfying the strong Markov property. I know it is possible to characterize some diffusion processes as solutions to ...
1
vote
0answers
16 views

convergent process in finite time

Let $(p_t)_{t\in\mathbb{N}}$ be an stochastic process on a countable (probability measure) space. Supose it has the Markov and the Martingale properties. It converges almost surely to a random ...
0
votes
1answer
18 views

Is there any concrete connection between a regular transition matrix and aperiodicty and irreducibility of a finite-state Markov Process?

The transition matrix T = \begin{bmatrix} 3/4 &1/4 \\ 1 &0 \end{bmatrix} is clearly a regular transition matrix but the chain itself is not aperiodic (although it is irreducible), right? (...
1
vote
3answers
134 views

Markovian decision process and dynamic programming solution

A home supply store can place orders for fridges at the start of each month for immediate delivery. A cost of $\$ 100$ is incurred each time an order is placed. The cost of storage per fridge is $\$ 5....
0
votes
0answers
17 views

Proof of strong Markov continuity of Brownian motion

Let $(B_t)$ a Brownian motion and $\sigma $ a stopping time finite a.s.. I want to prove that $W_t=B_{\sigma +t}-B_\sigma $ is a Brownian motion. The way to prove it is first to prove that for all $0\...
0
votes
1answer
21 views

Definition of Factorization of Conditional Expectation

I believe this is a very silly question or I am overlooking something fairly simple but I cannot make sense of the factorization of the conditional expectation in a very concrete application: I am ...
2
votes
0answers
21 views

Sufficient condition for the Markov property for a process in continuous time with values in a countable space

I would like to verify the folowing: Let $(X_s)_{s \in [0,\infty)}$ be a stochastic process in continuous time with values in a countable space $E$ and let $\mathcal F_s=\sigma(X_r: 0 \le r \le s)$ ...
0
votes
0answers
16 views

Master Equation vs Chapman Kolmogorov

I've been trying to understand the differences between the Chapman Kolmogorov equation and the Master Equation. Wikipedia calls the latter a different form of the latter. Is there any difference ...
3
votes
1answer
24 views

Independent increments and stationary increments, Lévy process

Prop:Let $(X_t)$ be a $\mathbb{R}^d$ valued stochastic process with transition probability $P_t(x,dy)$. We assume there exsist probabilities on $\mathbb{R}^d$ $\{m_t\}_{t\geq 0}$ s.t., $P_t (x,B)=m_t(...
0
votes
1answer
17 views

Discretization of a Continuous Time Markov Chain

Let $(X_t)_t$ be a continuous time markov chain on a finite state space with initial distribution $\alpha$ and transition matrix $A$. Suppose now we only observe $X$ at certain discrete time steps $...
0
votes
0answers
24 views

Markov chain decision problem, understanding solution a)

During any period, a potential customer arrives at a certain facility with probability $1/2$. If there are already two people at the facility (including the one being served), the potential customer ...
0
votes
1answer
30 views

Irreducible Markov chain rotating cyclically

I am going through Knowing the Odds by John B. Walsh, and I am stuck at one of the exercises there (which is important to understand some next theorem). The exercise is as follows: Let $X$ be an ...
0
votes
1answer
19 views

Probability of being at given state in a continuous time Markov chain

Question: Consider a two-state continuous time Markov chain (with states $1$ and $2$) in which the holding rate at state $1$ is $\lambda_1=2$, and the holding rate at state $2$ is $\lambda_2=3$. ...
-1
votes
1answer
20 views

Markov Decision Process - Discounted return

I was reading this article about Discounted return (in the context of MDP): http://deeplizard.com/learn/video/a-SnJtmBtyA I got the section: ...
-1
votes
1answer
20 views

Markov Process question

There are two tennis courts. Pairs of players arrive at a rate of 3 per hour and play for an exponentially distributed amount of time with mean 1 hour. If there are already two pairs of players ...
2
votes
2answers
21 views

How to find the probability that a process never enters a particular state in a Markov Chain?

I am given the following Markov Chain: Assume that the Markov Chain is in state 3 immediately before the first trial. I am asked to find the probability that the process never enters state 1. I ...
0
votes
0answers
17 views

Determine $\lim_{t→∞} \mathbb{P}_i(X_t = 0) $ for $i = 0, 1, 2, 3$.

I have this problem, I figured out the part (a) but I'm having a little trouble with part (b) if anyone can help me with that. Two repairmen serve three machines (that is, at most two machines can ...
0
votes
0answers
8 views

Optimization over transition probabilities

I am interested in the following question: Fix a natural number $n\ge 2$. Let $\Omega=\{\omega_1, \omega_2, \dots, \omega_n\}$. A Markov process with state space $\Omega$ and transition matrix $P$ ...
2
votes
1answer
105 views

I don't understand the proof of Corollary 4.8.7 in the book of Ethier and Kurtz

I'm trying to understand the proof of Corollary 8.7 of Chapter 4 in the book Markov Processes: Characterization and Convergence by Stewart N. Ethier, Thomas G. Kurtz. Here is the theorem and its proof:...