A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
1answer
25 views

What is the (practical) difference between a stationary distribution and an equilibrium distribution of a MC?

I know that, for a Markov Chain, a stationary distribution is the (row) vector $\pi$ such that $\pi \cdot P = \pi$, where $P$ is the one-step transition matrix for the MC. Intuitively, I assume that ...
0
votes
1answer
68 views

prove homogeneous markov chain

$Y_0, Y_1,Y_2,\dots$ are independent and identically distributed random non-negative integer outcomes. Let X_0 = Y_0 $ Let $X_0 = Y_0$ and $X_n = X_{n-1} - Y_n$ if $X_{n-1}>0$, else $X_n = X_{n-1}...
1
vote
1answer
23 views

How to interpret 'arbitrary customer'

My question is about how to interpret 'arbitrary customer' in the following scenario (see question 2. listed below): "At a single server service station two types of arrivals occur. According to a ...
0
votes
0answers
37 views

Defective Markov transition matrix and relation to its limiting distributions

Im trying to come to grips with what the physical interpretation of a non diagonalisable Markov Matrix means in terms of what we can deduce about it having a limiting distribution/ what potential ...
2
votes
1answer
41 views

Theory of Queueing

There's in a banc two identical queues and totally separated : these are two queues of type $M/M/1$. For each of them, the arrivals are separated by exponential times of parameter $\nu$, the time ...
2
votes
2answers
66 views

Renewal process problem, where $X_i$'s are i.i.d. with exponential distribution.

A room is lit by $2$ bulbs. Bulbs are replaced only when both bulbs burn out. Lifetimes of bulb's are i.i.d exponentially distributed with parameter $λ=1$. What fraction of the time is the room only ...
1
vote
2answers
60 views

Prove that $i$ is an accesible state in a markov chain

Let $i$ be a recurrent state of an homogeneous markov chain such that the state $j$ is accesible from $i$ (that is $\exists$ $k\ge 1$ such that $p_{ij}(k)>0$) Prove that $i$ is accesible from $j$ ...
0
votes
0answers
43 views

Computing hitting times from the stationary distribution

My question is whether it is possible to compute hitting times from the previously calculated stationary distribution $\pi$ of a continuous-time Markov process $(X_t)_{t \geq 0}$. I know that, from ...
2
votes
2answers
39 views

Mean number of tosses of a fair dice to get a sum of outcomes being a multiple of $5$

Let $S_n$ denote the sum of the outcomes of the $n$ tosses of a fair dice. Let $T=\inf\{n>0: S_n$ is a multiple of $5\}$. Compute $E(T)$ (by means of markov chains). Attempt. Instead of ...
0
votes
1answer
25 views

Mean number of steps needed to reach a recurrent state in a finite irreducible Markov chain

Let $\mathbb{X}$ be a finite state space of an irreducible markov chain $\{X_n\}$ and let $T_x=\inf\{k\geq 0\mid X_k=x\}$ be the number of steps until $\{X_n\}$ reaches state $x\in \mathbb{X}$. True ...
1
vote
1answer
49 views

Markov chains and queues

I do not understand how may I use the Markov Chain $Y$ and and describe the system $X$ using the states that the exercise suggest. I was searching queue's examples and -i understand this is a M/M/1 ...
1
vote
1answer
40 views

Markov process and conditional probability

Given a Markov Process, that is $$\mathbb{P}(X_{n+1}=x_{n+1}\mid X_{n}=x_{n},X_{n-1}=x_{n-1}, \dots, X_0=x_0)=\mathbb{P}(X_{n+1}=x_{n+1}\mid X_{n}=x_{n})$$I Need to prove that $$\mathbb{P}(X_{n+m}=x_{...
0
votes
0answers
26 views

Binary Hidden Markov Model

Consider a binary HMM with 2 observed variables $O_n \in \{0,1\} \; \forall n \in \mathbb{N}$. Suppose that the hidden Markov process $X_n$ is characterised by a known transition probability matrix ...
0
votes
0answers
21 views

probability a markov chain leaves a transient state in favor of a recurrent state

Let $\{X_n\}$ be a time-homogenuous markov chain, with state space $\mathbb{X}$ and transition matrix $P$. Let $C_1$ be a transient class and $C_2$ be a recurrent class and let also $x\in C_1,~y\...
2
votes
2answers
94 views

Convergence of mean of an irreducible Markov chain / ergodic theorem

Let $\{X_n\}$ be an irreducible Markov chain on a discrete state space $\mathbb{N}$, that has a stationary distribution $\pi$. Prove or disprove : with probability $1$: $$\lim_{n\rightarrow +\...
1
vote
1answer
55 views

Markov Chain Questions

I've been stuck on these problems for a while. I keep banging my head against the wall, but my calculations are incorrect each time. I sum the probabilities together for each possibility (it's a ...
0
votes
1answer
34 views

Number of $1's$ in a string in terms of successive pairs

Problem. Let $X_n=0$ or $1$ and set $Y_n=(X_n,X_{n+1})$. Set also $\displaystyle \sum_{k=1}^{n}\mathbb{I}_{\{X_k=1\}}$ be the number of times $X_k's$ become $1$, from $X_1$ till $X_n$ ($\mathbb{I}_{A}$...
0
votes
0answers
15 views

State space extension

In probability theory, it is very common to assume that the sample space is a Borel subset of complete separable metric space, which is basically a Borel space. In probability applications extending ...
3
votes
1answer
66 views

Stationnary distribution - Time of service [closed]

There's in a banc two identical queues and totally separated : these are two queues of type $M/M/1$. For each of them, the arrivals are separated by exponential times of parameter $\nu$, the time ...
0
votes
1answer
30 views

Understanding birth and death process - Stationnary distribution

Customers come to receive a service at a Poisson process of intensity $\lambda$. They are served one at a time and the service time is exponentially distributed parameter $\mu$. In addition, ...
5
votes
1answer
55 views

Branching Process: generation survival or extinction?

Let $p\in [0,1]$, and consider a branching process where the number of offspring of an individual is zero with probability $p$, and is two with probability $1-p$. Initially there is one ...
0
votes
0answers
92 views

Expected infimum of a 1d random walk

Consider a simple symmetric random walk on $\mathbb{Z}$ starting from $0$, $S_n$. Let $I_n := \inf\{S_0, S_1, S_2, \ldots S_n\}$. Is an explicit formula for $E[I_n]$ known?
1
vote
1answer
30 views

Irreducible and recurrent Markov chain - theorem notation question

In [J. R. Norris] Markov Chains (Cambridge Series in Statistical and Probabilistic Mathematics) (2009), page 35, Theorem 1.7.5 says: In (ii), does it mean $\gamma^k$ is notation for $\gamma^k_i$ ...
1
vote
1answer
148 views

Stochastic Markov Chain Application: Rat in the maze problem, a modification

I am really new to Stochastic processes, and this is one of the supplementary practice questions that I stumbled across whilst studying: Modify the situation as described in http://www.ucl.ac.uk/...
3
votes
1answer
39 views

Markov process - Queuing system - What is the average number of long-term service repairers? [closed]

The operating time of a unit has an exponential law of expectancy $2$ while the repair time follows an exponential law of expectancy $1/2$ . There are $4$ equipments and $2$ repairers who can work ...
0
votes
1answer
28 views

Symmetric Simple Random Walk - Definition Clarification

I'm finding conflicting answers everywhere, including in my own notes. In the phrase "symmetric simple random walk", which part, "symmetric" or "simple" refers to having a probability of $0.5$ to go ...
1
vote
1answer
60 views

Markov DTMC and CTMC: How to build a $Q$ generating matrix using given $2$ state $q$ probabilities? Find Steady State Probabilities

I know that $$q_{(i,j)(i+1,j)}=\lambda_1$$ $$q_{(i,j)(i-1,j+1)}=\lambda_2$$ $$q_{(i,j)(i+1,j-1)}=0.5\lambda_3$$ $$q_{(i,j)(i,j-1)}=0.5\lambda_3$$ where $\lambda_1 , \lambda_2 , \lambda_3 $ are rates ...
0
votes
1answer
49 views

How many stationary distributions does a time homogeneous Markov chain have?

I've been given the following definition: For a THMC with one step transition matrix $\mathbf{P}$, the row vector $\mathbf{\pi}$ with elements $(\pi_{i})_{i \in S}$ (where $S$ is the state space) ...
0
votes
1answer
18 views

Replacing bulb - Poisson process

A bulb at the entrance of a building has a lifetime of exponential expectancy $100$ days. When burned, the concierge immediately replaced with a new one. In addition, an employee who takes care of the ...
1
vote
1answer
22 views

Generator - Markov process - Reorganization of the service

Two employees of a brokerage firm receive calls from customers regarding the purchase or sale of mutual funds. When their telephone lines were independent, each was busy a time of exponential ...
0
votes
1answer
34 views

Create a transition table

I am trying to create a transition table for a markov chain but I have difficulties. Consider a game, where each player (of two, lets call them A and B) has a fixed given probability of scoring 3 ...
0
votes
1answer
56 views

Sampling uniform equilibrium distribution with Markov Chain Monte Carlo

I'm wanting to sample the discrete uniform distribution over $n = 10$ integers using MCMC. My question concerns the transition probability matrix, $P$. As I understand it, any symmetric, irreducible ...
1
vote
1answer
61 views

Please can someone help me to understand stationary distributions of Markov Chains?

I'm currently trying to understand (intuitively) what a stationary distribution of a Markov Chain is? In our lecture notes, we're given the following definition: This was of little benefit to my ...
1
vote
1answer
25 views

What does this notation, used in Markov Chains, mean?

In my module on Markov processes, the following notation is used: $$ p_{ij}^{(m,n)} = P(X_n = j \mid X_m = i) \quad \text{where } \: m<n \\ p_j^{(n)} = P(X_n = j) \\ p_{ij}^{(k)} = \: ??? $$ Does ...
0
votes
1answer
14 views

Generator for the number of shocks to the machine in function - Markov Process

It is assumed that device fails due to a $k$-th shock with probability $\frac{k^2}{9}$ for $k=1, 2, 3$ and, in this case, it is replaced with a new device. If shocks are spaced by independent time ...
0
votes
0answers
20 views

What is a Markovian time evolution model?

Supose I have to constuct a dynamical model for a random variable $X$ . Then I have read that for atmospheric and environmental purposes, a popular and flexible collection of models are (one-step) ...
0
votes
0answers
30 views

An Enquiry Concerning the “reward function” for a Markov Chain

The Statement of the Problem: Consider a Markov chain with state space $$ S = \{1, 2, 3 \} $$ and probability transition matrix $$ P = \left( \begin{matrix} .3 & .7 & 0 \\ ...
7
votes
1answer
103 views

If $A$ is the generator of $(P_t)$, then $A+f$ is the generator of $(P_t^f)$

Let $X=(X_t)_{t\geq0}$ be a Markov process on a state space $\Gamma$ (a Hausdorff topological vector space), let $A$ be the infinitesimal generator of $X$ and let $\mathcal C(\Gamma)$ the space of ...
0
votes
0answers
37 views

How to find the stationary distribution of a discrete time continuous state space Markov chain

How do I find the stationary distribution of a discrete time continuous state space Markov chain? If the state space is $S$, and if the conditional probability density function of state $y\in S$ ...
0
votes
0answers
9 views

Distribution of two correlated multivariate normal distributions

Let $\varepsilon_{s}^{FR}$ follow a Gaussian Markov process so that ${\varepsilon_{s}^{FR} = \rho \varepsilon_{s-1}^{FR} + \xi_{s}^{FR}, \: \xi_{s}^{FR} \sim \mathrm{i.i.d.} \: N(0,\sigma^{2}_{\xi^{...
1
vote
0answers
52 views

Intersection of two simple random walks

Suppose that $X_n$ and $Y_n$ are independent, symmetric, one-dimensional simple random walks, where $X_0 = 0$ and $Y_0 = N$ for some $N \in \mathbb{N}$ where $N$ is even. I would like to show that the ...
0
votes
0answers
34 views

Characterisations of Markov Processes: SDE's , Generators,…

There are different characterisations of a Markov Process: Probability Semigroups, Generators, even in some cases by Jumps Chains and Holding Times... And I know that, in "real life", the only thing ...
2
votes
0answers
56 views

Strong markov property vector valued process from independent strong markov components

I want to prove that if $X$ and $Y$ are (continuous time) independent strong markov $\mathbb{R}$-valued processes w.r.t. their natural filtrations $\mathcal{F}^X_t$ and $\mathcal{F}^Y_t$, that the ...
0
votes
1answer
30 views

Up-to-date or Behind - [Markov Chain]

Alex is taking a bioinformatics class and in each week he can be either up-to-date or he may have fallen behind. If he is up-to-date in a given week, the probability that he will be up-to-date (or ...
0
votes
0answers
48 views

Basic questions on Markov Chain

I'm a beginner of Markov processes and I have some basic questions. Consider two sequences of real-valued random variables $\{X_t\}_t, \{Y_t\}_t$ where $t$ is a discrete time index, $t=0,1,...$, all ...
0
votes
1answer
55 views

Mickey mouse travels in a maze with nine $3 × 3$ cells. Markov Chain involved?

Mickey mouse travels in a maze with nine $3 × 3$ cells. The cells are numbered as $0, 1, ..., 8$ from left to right and top down. Each step Mickey travels from where it is to one of the surrounding ...
0
votes
0answers
32 views

$ (X_n : n = 0, 1, 2, …) $ is a Markov chain with state space $(1,2,…,10)$.

$ (X_n : n = 0, 1, 2, ...) $ is a Markov chain with state space $(1,2,...,10)$. Then which of the following is the correct answer? ($X_n+X_{n-1} : n = 1, 2, ...$) is a Markov chain. ($X_n-X_{n-1} : ...
1
vote
2answers
40 views

Busy times for a M/M/1 queue

I have an M/M/1 queue with people arriving Poisson with parameter $\lambda$ and a service time exponentially distributed rate $\mu$. I have been asked to find the average time between the first ...
3
votes
1answer
69 views

Proof of semigroup property for family of operators

I'm studying a proof of a large deviations principle and I'm having trouble with a part that is concerned with the semigroup property of a family of operators. Assumptions $(X_t)_{t\geq0}$ is a ...
1
vote
0answers
33 views

Irreducible and positive recurrent CTMC — first passage times are finite?

Consider a continuous-time Markov chain (CTMC) $X$ on a countably infinite state space $S$. The CTMC is irreducible and all the states are positive recurrent. Let $T(i,j)$ be the first passage time to ...