A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
0answers
10 views

Are smooth functions with compact support a core for a generator of a Feller process?

I was reading an article, when the following was stated: Assume we have partial generator defined on some open subset $U\subset \mathbb{R}^n$ $$ A=\sum_{i=1}^n ...
0
votes
0answers
19 views

probability a markov chain leaves a transient state in favor of a recurrent state

Let $\{X_n\}$ be a time-homogenuous markov chain, with state space $\mathbb{X}$ and transition matrix $P$. Let $C_1$ be a transient class and $C_2$ be a recurrent class and let also $x\in ...
2
votes
2answers
89 views

Convergence of mean of an irreducible Markov chain / ergodic theorem

Let $\{X_n\}$ be an irreducible Markov chain on a discrete state space $\mathbb{N}$, that has a stationary distribution $\pi$. Prove or disprove : with probability $1$: $$\lim_{n\rightarrow ...
1
vote
1answer
53 views

Markov Chain Questions

I've been stuck on these problems for a while. I keep banging my head against the wall, but my calculations are incorrect each time. I sum the probabilities together for each possibility (it's a ...
0
votes
1answer
33 views

Number of $1's$ in a string in terms of successive pairs

Problem. Let $X_n=0$ or $1$ and set $Y_n=(X_n,X_{n+1})$. Set also $\displaystyle \sum_{k=1}^{n}\mathbb{I}_{\{X_k=1\}}$ be the number of times $X_k's$ become $1$, from $X_1$ till $X_n$ ...
0
votes
0answers
13 views

State space extension

In probability theory, it is very common to assume that the sample space is a Borel subset of complete separable metric space, which is basically a Borel space. In probability applications extending ...
4
votes
1answer
63 views

Stationnary distribution - Time of service [closed]

There's in a banc two identical queues and totally separated : these are two queues of type $M/M/1$. For each of them, the arrivals are separated by exponential times of parameter $\nu$, the ...
0
votes
1answer
22 views

Understanding birth and death process - Stationnary distribution

Customers come to receive a service at a Poisson process of intensity $\lambda$. They are served one at a time and the service time is exponentially distributed parameter $\mu$. In addition, ...
5
votes
1answer
48 views

Branching Process: generation survival or extinction?

Let $p\in [0,1]$, and consider a branching process where the number of offspring of an individual is zero with probability $p$, and is two with probability $1-p$. Initially there is one ...
0
votes
0answers
89 views

Expected infimum of a 1d random walk

Consider a simple symmetric random walk on $\mathbb{Z}$ starting from $0$, $S_n$. Let $I_n := \inf\{S_0, S_1, S_2, \ldots S_n\}$. Is an explicit formula for $E[I_n]$ known?
1
vote
1answer
30 views

Irreducible and recurrent Markov chain - theorem notation question

In [J. R. Norris] Markov Chains (Cambridge Series in Statistical and Probabilistic Mathematics) (2009), page 35, Theorem 1.7.5 says: In (ii), does it mean $\gamma^k$ is notation for $\gamma^k_i$ ...
1
vote
1answer
105 views

Stochastic Markov Chain Application: Rat in the maze problem, a modification

I am really new to Stochastic processes, and this is one of the supplementary practice questions that I stumbled across whilst studying: Modify the situation as described in ...
3
votes
1answer
37 views

Markov process - Queuing system - What is the average number of long-term service repairers? [closed]

The operating time of a unit has an exponential law of expectancy $2$ while the repair time follows an exponential law of expectancy $1/2$ . There are $4$ equipments and $2$ repairers who can ...
1
vote
1answer
25 views

Symmetric Simple Random Walk - Definition Clarification

I'm finding conflicting answers everywhere, including in my own notes. In the phrase "symmetric simple random walk", which part, "symmetric" or "simple" refers to having a probability of $0.5$ to go ...
1
vote
1answer
41 views

Markov DTMC and CTMC: How to build a $Q$ generating matrix using given $2$ state $q$ probabilities? Find Steady State Probabilities

I know that $$q_{(i,j)(i+1,j)}=\lambda_1$$ $$q_{(i,j)(i-1,j+1)}=\lambda_2$$ $$q_{(i,j)(i+1,j-1)}=0.5\lambda_3$$ $$q_{(i,j)(i,j-1)}=0.5\lambda_3$$ where $\lambda_1 , \lambda_2 , \lambda_3 $ are rates ...
0
votes
1answer
35 views

How many stationary distributions does a time homogeneous Markov chain have?

I've been given the following definition: For a THMC with one step transition matrix $\mathbf{P}$, the row vector $\mathbf{\pi}$ with elements $(\pi_{i})_{i \in S}$ (where $S$ is the state space) ...
0
votes
1answer
17 views

Replacing bulb - Poisson process

A bulb at the entrance of a building has a lifetime of exponential expectancy $100$ days. When burned, the concierge immediately replaced with a new one. In addition, an employee who takes care of the ...
1
vote
1answer
21 views

Generator - Markov process - Reorganization of the service

Two employees of a brokerage firm receive calls from customers regarding the purchase or sale of mutual funds. When their telephone lines were independent, each was busy a time of exponential ...
0
votes
1answer
34 views

Create a transition table

I am trying to create a transition table for a markov chain but I have difficulties. Consider a game, where each player (of two, lets call them A and B) has a fixed given probability of scoring 3 ...
0
votes
1answer
33 views

Sampling uniform equilibrium distribution with Markov Chain Monte Carlo

I'm wanting to sample the discrete uniform distribution over $n = 10$ integers using MCMC. My question concerns the transition probability matrix, $P$. As I understand it, any symmetric, irreducible ...
1
vote
1answer
58 views

Please can someone help me to understand stationary distributions of Markov Chains?

I'm currently trying to understand (intuitively) what a stationary distribution of a Markov Chain is? In our lecture notes, we're given the following definition: This was of little benefit to my ...
1
vote
1answer
24 views

What does this notation, used in Markov Chains, mean?

In my module on Markov processes, the following notation is used: $$ p_{ij}^{(m,n)} = P(X_n = j \mid X_m = i) \quad \text{where } \: m<n \\ p_j^{(n)} = P(X_n = j) \\ p_{ij}^{(k)} = \: ??? $$ Does ...
0
votes
1answer
12 views

Generator for the number of shocks to the machine in function - Markov Process

It is assumed that device fails due to a $k$-th shock with probability $\frac{k^2}{9}$ for $k=1, 2, 3$ and, in this case, it is replaced with a new device. If shocks are spaced by independent time ...
0
votes
0answers
17 views

What is a Markovian time evolution model?

Supose I have to constuct a dynamical model for a random variable $X$ . Then I have read that for atmospheric and environmental purposes, a popular and flexible collection of models are (one-step) ...
0
votes
0answers
30 views

An Enquiry Concerning the “reward function” for a Markov Chain

The Statement of the Problem: Consider a Markov chain with state space $$ S = \{1, 2, 3 \} $$ and probability transition matrix $$ P = \left( \begin{matrix} .3 & .7 & 0 \\ ...
7
votes
1answer
102 views

If $A$ is the generator of $(P_t)$, then $A+f$ is the generator of $(P_t^f)$

Let $X=(X_t)_{t\geq0}$ be a Markov process on a state space $\Gamma$ (a Hausdorff topological vector space), let $A$ be the infinitesimal generator of $X$ and let $\mathcal C(\Gamma)$ the space of ...
0
votes
0answers
32 views

How to find the stationary distribution of a discrete time continuous state space Markov chain

How do I find the stationary distribution of a discrete time continuous state space Markov chain? If the state space is $S$, and if the conditional probability density function of state $y\in S$ ...
0
votes
0answers
9 views

Distribution of two correlated multivariate normal distributions

Let $\varepsilon_{s}^{FR}$ follow a Gaussian Markov process so that ${\varepsilon_{s}^{FR} = \rho \varepsilon_{s-1}^{FR} + \xi_{s}^{FR}, \: \xi_{s}^{FR} \sim \mathrm{i.i.d.} \: ...
1
vote
0answers
45 views

Intersection of two simple random walks

Suppose that $X_n$ and $Y_n$ are independent, symmetric, one-dimensional simple random walks, where $X_0 = 0$ and $Y_0 = N$ for some $N \in \mathbb{N}$ where $N$ is even. I would like to show that the ...
0
votes
0answers
31 views

Characterisations of Markov Processes: SDE's , Generators,…

There are different characterisations of a Markov Process: Probability Semigroups, Generators, even in some cases by Jumps Chains and Holding Times... And I know that, in "real life", the only thing ...
2
votes
0answers
52 views

Strong markov property vector valued process from independent strong markov components

I want to prove that if $X$ and $Y$ are (continuous time) independent strong markov $\mathbb{R}$-valued processes w.r.t. their natural filtrations $\mathcal{F}^X_t$ and $\mathcal{F}^Y_t$, that the ...
0
votes
1answer
29 views

Up-to-date or Behind - [Markov Chain]

Alex is taking a bioinformatics class and in each week he can be either up-to-date or he may have fallen behind. If he is up-to-date in a given week, the probability that he will be up-to-date (or ...
0
votes
0answers
48 views

Basic questions on Markov Chain

I'm a beginner of Markov processes and I have some basic questions. Consider two sequences of real-valued random variables $\{X_t\}_t, \{Y_t\}_t$ where $t$ is a discrete time index, $t=0,1,...$, all ...
0
votes
1answer
47 views

Mickey mouse travels in a maze with nine $3 × 3$ cells. Markov Chain involved?

Mickey mouse travels in a maze with nine $3 × 3$ cells. The cells are numbered as $0, 1, ..., 8$ from left to right and top down. Each step Mickey travels from where it is to one of the surrounding ...
0
votes
0answers
32 views

$ (X_n : n = 0, 1, 2, …) $ is a Markov chain with state space $(1,2,…,10)$.

$ (X_n : n = 0, 1, 2, ...) $ is a Markov chain with state space $(1,2,...,10)$. Then which of the following is the correct answer? ($X_n+X_{n-1} : n = 1, 2, ...$) is a Markov chain. ($X_n-X_{n-1} : ...
1
vote
2answers
36 views

Busy times for a M/M/1 queue

I have an M/M/1 queue with people arriving Poisson with parameter $\lambda$ and a service time exponentially distributed rate $\mu$. I have been asked to find the average time between the first ...
3
votes
1answer
63 views

Proof of semigroup property for family of operators

I'm studying a proof of a large deviations principle and I'm having trouble with a part that is concerned with the semigroup property of a family of operators. Assumptions $(X_t)_{t\geq0}$ is a ...
1
vote
0answers
29 views

Irreducible and positive recurrent CTMC — first passage times are finite?

Consider a continuous-time Markov chain (CTMC) $X$ on a countably infinite state space $S$. The CTMC is irreducible and all the states are positive recurrent. Let $T(i,j)$ be the first passage time to ...
0
votes
0answers
21 views

Distribution of right censored observation in an absorbing Markov Chain

Consider a $3$ state Continuous time Process $\{X_t\}_{t \geq0}$ with state space $\mathcal{S} = \{0,1,2\}$ where state $0$ denotes the absorbing state. Let the generator of this process be: $$Q = ...
5
votes
2answers
170 views
0
votes
0answers
9 views

how to resolve the infinite nesting of interactive POMDP

I am reading papers about I-POMDP. I cant understand the finitely nested I-POMDPs given in these papers. The belief update of the algorithm has a problem that agents' belief updates mutually depend ...
0
votes
1answer
57 views

Equality for transient Markov chains

Let $S_n$ be a transient, irreducible random walk starting from $0$. Then I want to prove that $$\sum_{n\geq0}{p_n(0,x)}=P_0[S_n=x\text{ for some }n\geq0]\sum_{n\geq0}{p_n(0,0)}$$ where $p_n$ is the ...
1
vote
0answers
13 views

Rigorous Derivation of Metropolis-Hastings Transition Density

The Metropolis-Hastings MCMC algorithm is as follows. Set $X_0$ to some initial value in the support of the target density $f$ and choose a proposal density $q(y \mid x)$; a density in $y$ for each ...
1
vote
1answer
26 views

Proving linear operators are Markov Generators

I am trying to do the following question from Liggett. Let $A$ be a linear operator defined on the space of continuous functions $C(E)$ for a compact set $E$. Define $A$ by $A=T-I$ where $T$ is a ...
5
votes
0answers
59 views

markov chain: 2 state chain

I have a machine. It has two states, broken or working. If it is working, then it will be broken with probability $q=0.1$. If the machine is working, I will make \$1000 dollar a day. If it is broken, ...
3
votes
1answer
119 views

Simple random walk on $\mathbb Z^d$ and its generator

I'm still trying to figure out definitions and properties of random walks on $\mathbb Z^d$. My goal is to work up to understanding some large deviation principles for the local times of such random ...
0
votes
0answers
10 views

Unconditional COvariance in markov switching model

I'm trying to do a portfolio optimisation within a Markov switching framework for some risky asset returns. My utility function ideally is CRRA (power) utility. However maximising a linear sum of two ...
1
vote
1answer
14 views

Invariant distributions: Applications in the real World

I'm studying about invariant distributions for Markov processes; say in the context of dynamics of Random Neural Networks (biological Networks). I can't fully understand what does an invariant ...
1
vote
0answers
31 views

The “how many pieces do you have buy on average” problem, a markov problem?

I recently discovered a problem similar to this one in a book about Markov chains: Assume you can buy $n-$ different set of cards in a store, but you do not know which one you'll buy: What is the ...
0
votes
0answers
16 views

When does a periodic but positive recurrent markov chain have a limiting distribution

So I know it's a fact that an aperiodic, finite state, irreducible (so positive recurrent) markov chain has a unique stationary distribution which is limiting. However, I am curious if there is a ...