Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
1answer
25 views

Are Markov chains necessarily time-homogeneous?

I've seen a definition of Markov chains as a stochastic process $(X_t)_{t\in I}$ fulfilling the weak Markov property and having index set $I = \mathbb{N}_0$. But the weak Markov property ...
1
vote
2answers
68 views

Difference between conditional expectation and conditional probabilty

These are known definitions: We have a probability space $(\Omega, A, P)$ Conditional probability is defined through $P(A|B) = \frac{P(A \cap B)}{P(B)}, P(B) > 0$. This is a real nunmber. Then ...
0
votes
2answers
33 views

Alternating Markov process

Given the situation: When Bob enters the room and the light is off, he turns it on with $P = 1/2$ when it is on, he does nothing. When Alice enters the room with light on, she turns it off with $P ...
0
votes
1answer
36 views
0
votes
2answers
75 views

what are “filtering” and “smoothing” mentioned in hidden Markov model wikipedia article?

the article mentions "filtering" and "smoothing" tasks, see here http://en.wikipedia.org/wiki/Hidden_Markov_model#Filtering . It gives brief explanation but no motivating examples and no references to ...
1
vote
1answer
36 views

Can You Help Me With This Markov Chain Question?

For a birth and death process with birth rates, $\lambda_i$ and death rates $\mu_i$ $(i=0,1,2...)$ respectively. Show that the transition probabilities, $P_{i,j}(t)$ satisfy the following differential ...
6
votes
1answer
98 views

Confusion in the proof of properties for $\psi$-irreducibility

Let $P$ be a stochastic kernel on a measurable space $(\mathsf X,\mathfrak B(\mathsf X))$. The kernel $P$ is called $\varphi$-irreducible if for a positive measure $\varphi$ and for all measurable ...
1
vote
1answer
14 views

Prove the following r-step transition

Let $X_0, X_1, X_2,...$ be a Markov Chain on state space $S=\{1, 2,..., n\}$ and let $P$ be the Transition Matrix of the above Markov chain Prove that $\Bbb{P}(X_{t+2}=j|X_t=i) = (P^2)_{ij} $ ...
3
votes
2answers
56 views

Markov chains diagram - what are the numbers above arrows?

Most if not all articles describe the numbers above arrows as probabilities of a transition in that direction, such as this one, or this one. But here, for example, something really weird is ...
2
votes
2answers
70 views

Under what conditions does $(I-N)^{-1}$ exist?

Given an nxn matrix N and $I=I_n$, under what conditions does $(I-N)^{-1}$ exist? On one hand $(I-N)(I + N + N^2 + ...) = (I + N + N^2 + ...) - (N + N^2 + ...) = I?$ On the other hand, $(I-N)(I + N ...
0
votes
0answers
24 views

Law of iterated logarithm for Markov Chains

Does anyone know where(or if) I can find a proof of law of iterated logarithm for irreducible and aperiodic Markov chain with finite number of states. All of the proofs I have seen so far are really ...
0
votes
1answer
39 views

Distribution of particles at infinite time

Let any site of $\mathbb{Z}$ host a number of particles $\eta_0(x)$ which is distributed according to some probability distribution independently and identically for any site $x \in \mathbb{Z}$. At ...
0
votes
0answers
31 views

Is this type of Markov chain known?

I am looking at a situation where we have $N$ urns and $K\le N$ balls. Consider some allocation of the balls to the urns. When any urn contains two or more balls, we call it a colliding urn. The ...
0
votes
0answers
6 views

Long time statistics of random functions

I'd like to understand if an average over random functions can be approximated with a Markov process in the long-time limit. Let $$ X_t = \sum_k a_k \cos(\omega_k t + \phi_k) $$ a random function, ...
1
vote
1answer
24 views

A question about the translation property Markov kernel

Given that ${X_n}$ is a Markov chain, and a Markov kernel with translation property $p(y+x,E+x)=p(y,E)$. Question: How to show $Y_n=X_n-X_{n-1}$ are i.i.d? I'm trying to use Markov Property and ...
0
votes
0answers
20 views

Computing Steady State Probability for 3 state markov chain

I have the equation $\frac{d}{dt}\vec{p(t)} = \vec{p(t)}Q$ here Q is a 3x3 transition matrix. $\vec{p} = (p_a,p_b,p_c)$. I have already solved Q where each row sums to 0. I have been trying to find ...
0
votes
1answer
18 views

Formula for average time in Markov chain

I have a model like: A B C A 0.80 0.10 0.10 B 0.20 0.75 0.05 C 0.10 0.10 0.80 How do I get the average time from B to A? I understand that ...
1
vote
0answers
39 views

Understanding the strong Markov property

I have problems to understand the strong Markov property (Klenke, p. 356): Let $I \subset [0,\infty)$ be closed under addition. A Markov process $(X_t)_{t\in I}$ with distributions $(\mathbf{P}_x, ...
0
votes
1answer
27 views

Absorbing Markov chain when less transient states than absorbing states

I have a probability matrix: 1 2 3 1 0.5 0.3 0.2 2 0 1 0 3 0 0 1 I understand that: $$ Q = \left(\begin{array}{c} 0.5 \end{array} ...
0
votes
1answer
21 views

How to calculate steps of a Markov chain with an unknown probability?

I have the matrix: A B C A 0.80 0.10 0.10 B 0.2 0.75 0.05 C 0.10 0.10 0.80 They ask me: if $ A $ is 40% right now, what's the probability of $A$ ...
0
votes
0answers
8 views

Simple Hidden Markov Model with Autoregressive Structure - Estimation?

I observe a two series over time $Y_{1:T}=\{ Y_{1}, \dots, Y_{T}\}$ and $X_{1:T}=\{ X_{1}, \dots, X_{T}\}$ where the $X$ series supposed to be exogenous (I do not define any stochastic proecess for ...
1
vote
1answer
30 views

Bayes: markov chain, serial connection, marginalization

Goal is to check if p(a) is unconditionally independent to p(c) in the markov chain - serial connection. $$ p(a,b,c) = p(a) p(b|a) p(c|b) $$ $$ p(a,c) = \sum_b p(a) p(b|a) p(c|b) = p(a) p(c|a) \neq ...
1
vote
1answer
109 views

Sum of i.i.d. random variables is a markov chain

I think I have some problem understanding markov chains, because we defined them as abstract objects but our professor does proofs with them as if they where just elementary conditional probabilities. ...
2
votes
1answer
19 views

Order $n^{r-1}$ approximation of product given order $(\frac{1}{n^2})$ approximation of terms

I have that $|a_n - (1+\frac{r}n)| \leq \frac c{n^2}$, for $c$ a constant, and am attempting to show that there exist constants $C < \infty$ and $K > 0$ such that the product $b_n = ...
0
votes
1answer
14 views

What does this question about classifying the states of this Markov chain mean?

If $X$ is a discrete Markov chain with state space $S=\{1,2\}$ and transition matrix \begin{equation*} P=\begin{pmatrix} 1-a& a\\ b& 1-b \end{pmatrix}. \end{equation*} I must answer the ...
0
votes
0answers
27 views

Show that in every irreducible and recurrent Markov Chain for all pair of states (i,j) the probability of ever achieving j from i equals 1.

My question is exactly the one in the title. So far I have figured that I can use definition: $$ F_{ij} = P ( \bigcup_{n=1}^{\infty} \{ X_n = j \} | X_0 = i) $$ $$ f_{ij}(n) = P ( X_1 \neq j, \ldots ...
0
votes
0answers
11 views

Example of Semi Markov Process, that isn't a Markov Chain in Continuous Time?

Question says it all I hope. I have an exam in Stochastic Processes tomorrow and one question that may be asked is to give an example of a Semi-Markov Process that isn't a Markov Chain in Continuous ...
4
votes
1answer
72 views

Expectation and limit of a stop-and-go traveler markov chain

The velocity $V(t)$ of a stop and go traveler is a two-state Markov chain whose generator is given by $$ \begin{array}{cc} &\begin{matrix}0&1\end{matrix}\\ \ \begin{matrix}0\\ 1\end{matrix} ...
1
vote
1answer
68 views

Is it worth playing this game of St. Petersburg paradox?

A gambler offers you the following deal. You have to keep tossing a fair coin until you get a heads, at which point you stop and collect your winnings: if it happens after n throws, the gambler will ...
5
votes
1answer
261 views

Does every continuous time minimal Markov chain have the Feller property?

Consider a Q-matrix on a countable state space. (A Q-matrix is a matrix whose rows sum up to $0$, with nonpositive finite diagonal entries and nonnegative off-diagonal entries.) As explained for ...
1
vote
0answers
35 views

Distribution of continuous time markov chain

I'm having trouble understanding the question below. I understand the continuous time markov chain and unique stationary distribution but not sure what it is asking. I have a continuous-time Markov ...
1
vote
1answer
44 views

Existence of steady state distribution for finite state Markov chains

Let's assume a Markov chain has 2 recurrent classes and a transient state from which we can go to either of the recurrent classes. If one of those recurrent classes is periodic, would it effect the ...
0
votes
1answer
19 views

Absorbing states and Irreducible sets

Question on the definition of Markov Chain matrices: Is it possible to have an absorbing state (i.e. a state where the probability of returning to itself is 1) within an irreducible set? I.e., if we ...
0
votes
1answer
17 views

Do the Matrices representing Markov chains need to be square?

I assume so -- I ask in the context of defining an irreducible set. If a set is non-irreducible, you should be able to find a "smaller" Markov chain matrix nested within a larger one. That "smaller" ...
2
votes
1answer
46 views

Stochastic kernel as linear operator

Let $K$ be a stochastic kernel for a set $S$ equipped with a countably generated $\sigma$-Algebra $B(S)$, i.e. $K:S\times B(S)\rightarrow [0,1]$ such that $K(\cdot,A)$ is a measurable function for ...
0
votes
0answers
26 views

Markov Chains and $n$-step transition probabilities

I have learnt that in a Markov Chain, the one step transition probability depends only on the current state, and not any of the previous states, by the definition of a Markov Chain. Now, when we ...
1
vote
1answer
26 views

Birth-death process: What is the distribution of reached states before reaching an absorbing state?

Intro I am working on a birth-death process. For a given choice of parameter ($n=6$, $Wa=1$, $Wb=0.95$, see below), the transition matrix is $$\left( \begin{array}{ccccccc} 1. & 0.144928 & ...
1
vote
0answers
24 views

Merging rates on a CTMC model

first time question here. I'm having a rough time trying to represent the following CTMC. Any help would be gladly appreciated. We consider a server with a infinite buffer connected to a network. ...
1
vote
0answers
33 views

Identifying markov chains

Problem: A regular dice is thrown repeatedly. Which one of the following random variables with values in $\mathbb{N}\cup\{\infty\}\cup\{0\}$ is a Markov chain? For those who are, give the ...
0
votes
0answers
21 views

Recurrent Markov chain: probability of visiting state i precisely k times in N steps

I'm studying this Markov process with transition matrix $P$, given by \begin{equation} P=\left(\begin{array}{cccc} \mu & 1-\mu & 0 & 0\\ 0 & 0 & \mu & 1-\mu\\ \mu & 1-\mu ...
5
votes
3answers
134 views

A recursive formula to approximate $e$. Prove or disprove.

Let the sequence $\{x_n, n=1,2,...\}$ be defined as follows: Let $x_2=x_1=1$ and for $n>2$ let $$x_{n+1}=x_n-\frac{1}{n}x_{n-1}.$$ This sequence, generated by the recursion above, tends to zero ...
1
vote
1answer
34 views

Show a random walk is transient

I was going through some problems related to Markov chains and I got stuck on this bit: We are given a random walk on $Z$, defined by the transition matrix $p_{i,i+1}=p$ and $p_{i,i-1}=1-p$. How to ...
0
votes
1answer
22 views

Limiting distribution of a Markov chain?

I have the problem below. There are n identical machines. They are all operational at time 0. The lifetime of each one is an exponential random variable with rate L. There are r repairmen (1 ≤ r ≤ ...
0
votes
0answers
21 views

Equilibrium Vector For Regular Markov Chain

Let $P$ be a transition matrix for a regular Markov Chain and let $w$ be it's equilibrium vector. Show that $w$ has no zero entries
1
vote
1answer
31 views

Poisson process Probabilities

If I assume that $\{N(t)=: t \ge 0\}$ is a Poisson process with intensity $\lambda$. For $0<s<t$, how would I find the $\Pr\{N(t)>N(s)\}$?
0
votes
0answers
30 views

Transition rates and probabilities of a continuous markov chain

A certain type of component has two states: 0 = OFF and 1 = OPERATING. In state 0 , the process remains there an exponential amount of time with rate $ \alpha$, and then moves to state 1. The ...
1
vote
1answer
15 views

Application $\pi$-$\lambda$ lemma one-sided Markov shift

Let $(S_k^{\mathbb{N}},\Sigma_k^{\mathbb{N}},m,\tau)$ be the probability preserving transformation of the one-sided Markov shift, where $\Sigma_k^{\mathbb{N}}$ is the $\sigma$-algebra generated by the ...
2
votes
1answer
95 views

Is this chain irreducible and/or Aperiodic? What is its equilibrium mass function?

Consider a Markov chain with outcomes $\{0,…,n\}$ and transition probabilities $P_{i,i+1}=p$ $P_{i,i−1}=q$ for $1\le i\le n−1$ and $p+q=1$. Assume also that $P_{0,1} = P_{n,n−1} = 1$. Is this chain ...
0
votes
1answer
78 views

probability, random walk, Markov chain question

Let $P$ be a transition matrix for a regular Markov chain and let $w$ be it’s equilibrium vector. Show that $w$ has no zero entries.
4
votes
1answer
95 views

Find the Stationary Distribution of an infinite state Markov chain

A Markov Chain on states 0,1,..... has transition probabilities $P_{ij}=1/(i+2)$ for j=0,1,....,i,i+1. I'm supposed to find the stationary distribution. So do I take the limit as n goes to ...