Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
19 views

Absorbing states and Irreducible sets

Question on the definition of Markov Chain matrices: Is it possible to have an absorbing state (i.e. a state where the probability of returning to itself is 1) within an irreducible set? I.e., if we ...
0
votes
1answer
19 views

Do the Matrices representing Markov chains need to be square?

I assume so -- I ask in the context of defining an irreducible set. If a set is non-irreducible, you should be able to find a "smaller" Markov chain matrix nested within a larger one. That "smaller" ...
0
votes
0answers
27 views

Markov Chains and $n$-step transition probabilities

I have learnt that in a Markov Chain, the one step transition probability depends only on the current state, and not any of the previous states, by the definition of a Markov Chain. Now, when we ...
1
vote
2answers
36 views

Transition probability matrix

In the article here it had this question. A walker moves on two positions a and b. She begins at a at time 0, and is at a next time as well. Subsequently, if she is at the same position for two ...
1
vote
1answer
75 views

Long term probability in Markov Chains

I was practicing some questions on transition probability matrices and I came up with this question. You have 3 coins: A (Heads probability 0.2),B (Heads probability 0.4), C (Heads probability ...
1
vote
1answer
27 views

Birth-death process: What is the distribution of reached states before reaching an absorbing state?

Intro I am working on a birth-death process. For a given choice of parameter ($n=6$, $Wa=1$, $Wb=0.95$, see below), the transition matrix is $$\left( \begin{array}{ccccccc} 1. & 0.144928 & ...
1
vote
0answers
26 views

Merging rates on a CTMC model

first time question here. I'm having a rough time trying to represent the following CTMC. Any help would be gladly appreciated. We consider a server with a infinite buffer connected to a network. ...
1
vote
0answers
33 views

Identifying markov chains

Problem: A regular dice is thrown repeatedly. Which one of the following random variables with values in $\mathbb{N}\cup\{\infty\}\cup\{0\}$ is a Markov chain? For those who are, give the ...
1
vote
1answer
25 views

A question about the translation property Markov kernel

Given that ${X_n}$ is a Markov chain, and a Markov kernel with translation property $p(y+x,E+x)=p(y,E)$. Question: How to show $Y_n=X_n-X_{n-1}$ are i.i.d? I'm trying to use Markov Property and ...
0
votes
0answers
23 views

Recurrent Markov chain: probability of visiting state i precisely k times in N steps

I'm studying this Markov process with transition matrix $P$, given by \begin{equation} P=\left(\begin{array}{cccc} \mu & 1-\mu & 0 & 0\\ 0 & 0 & \mu & 1-\mu\\ \mu & 1-\mu ...
1
vote
1answer
36 views

Show a random walk is transient

I was going through some problems related to Markov chains and I got stuck on this bit: We are given a random walk on $Z$, defined by the transition matrix $p_{i,i+1}=p$ and $p_{i,i-1}=1-p$. How to ...
0
votes
1answer
22 views

Limiting distribution of a Markov chain?

I have the problem below. There are n identical machines. They are all operational at time 0. The lifetime of each one is an exponential random variable with rate L. There are r repairmen (1 ≤ r ≤ ...
0
votes
0answers
25 views

Equilibrium Vector For Regular Markov Chain

Let $P$ be a transition matrix for a regular Markov Chain and let $w$ be it's equilibrium vector. Show that $w$ has no zero entries
1
vote
1answer
70 views

Is it worth playing this game of St. Petersburg paradox?

A gambler offers you the following deal. You have to keep tossing a fair coin until you get a heads, at which point you stop and collect your winnings: if it happens after n throws, the gambler will ...
0
votes
0answers
41 views

Transition rates and probabilities of a continuous markov chain

A certain type of component has two states: 0 = OFF and 1 = OPERATING. In state 0 , the process remains there an exponential amount of time with rate $ \alpha$, and then moves to state 1. The ...
4
votes
1answer
72 views

Expectation and limit of a stop-and-go traveler markov chain

The velocity $V(t)$ of a stop and go traveler is a two-state Markov chain whose generator is given by $$ \begin{array}{cc} &\begin{matrix}0&1\end{matrix}\\ \ \begin{matrix}0\\ 1\end{matrix} ...
2
votes
1answer
95 views

Is this chain irreducible and/or Aperiodic? What is its equilibrium mass function?

Consider a Markov chain with outcomes $\{0,…,n\}$ and transition probabilities $P_{i,i+1}=p$ $P_{i,i−1}=q$ for $1\le i\le n−1$ and $p+q=1$. Assume also that $P_{0,1} = P_{n,n−1} = 1$. Is this chain ...
0
votes
1answer
79 views

probability, random walk, Markov chain question

Let $P$ be a transition matrix for a regular Markov chain and let $w$ be it’s equilibrium vector. Show that $w$ has no zero entries.
1
vote
1answer
32 views

Poisson process Probabilities

If I assume that $\{N(t)=: t \ge 0\}$ is a Poisson process with intensity $\lambda$. For $0<s<t$, how would I find the $\Pr\{N(t)>N(s)\}$?
1
vote
1answer
16 views

Application $\pi$-$\lambda$ lemma one-sided Markov shift

Let $(S_k^{\mathbb{N}},\Sigma_k^{\mathbb{N}},m,\tau)$ be the probability preserving transformation of the one-sided Markov shift, where $\Sigma_k^{\mathbb{N}}$ is the $\sigma$-algebra generated by the ...
0
votes
1answer
79 views

Mean return time in Markov chain

Given the following Markov chain: $p_{0,1}=1$ (if we are in state 0, we must go to state 1) $p_{i,i+1}=p_{i,i-1}=0.5$ There are infinitely (countably) many states. I assume that $X_0=0$ and define ...
1
vote
2answers
26 views

Prove that markov chain is recurrent

I have the following markov chain : $S=\{0,1,2,3\}$ $p_{i,0} = q$ (if we are in one of the states $0,1,2,3$ we can return to $0$ with probability $q$) $p_{i,i+1} = 1-q , i\in\{0,1,2\}$ (if we are ...
0
votes
0answers
25 views

How to find stationary states of non-homogeneous Markov chain

As the title suggests I am interested in finding the transition matrix after some steps for a non-homogeneous Markov chain. I modeled my problem as stationary first and after some steps this ...
0
votes
0answers
22 views

Using Little's law to solve a Continuous Time Markov Chain problem

The problem listed below utilizes a CTMC in order to find throughput. I don't understand how to use the 'cut method' in order to find the stationary distribution for this problem. In addition to this, ...
0
votes
0answers
9 views

How to make Continuous Tme Markov Chain Transition Diagram?

I am studying for my final and this is one of the problems my professor assigned to us. I do not understand why the diagonal values of the transition matrix are supposed to be negative. I also do not ...
1
vote
0answers
29 views

Markov chain converges to boundary

I am learning martingale and related concepts recently and come across the following problem. Suppose $D$ is a bounded, connected, open subset of $\mathbb{R}^2$ with boundary $\partial D$. ...
4
votes
1answer
34 views

Expected number of turns for SPROUT

As a mathematical father (and with apparently plenty of time on my hands) I long ago computed the expected number of turns for a number of children's games that are effectively Markov maps. (Chutes ...
0
votes
0answers
36 views

If there are two different stationary distributions, then there are infinitely many distributions in reducible markov chain

If there are two stationary distributions μ1 and μ2 there are actually infinitely many stationary distributions: (pμ1 + (1 − p)μ2) is also a stationary distribution for any real number 0 ≤ p ≤ 1. How ...
0
votes
1answer
44 views

Existing of a distribution of three random variables that have conditional mutual information with defined properties.

I have two similar questions: 1)Does exist a distribution of three random variables such that: $I(a:b) = 0$ and $I(a:b|c)>0$ (where $I(a:b)$ is a mutual information and $I(a:b|c)$ is a ...
0
votes
0answers
20 views

How to find the number of transitions, after which the stationary distribution could be found in Markov chain?

Say I have the initial state space vector S = [1 0 0]. and I know both the transition matrix, P and final stationary distribution, S' = [0.3 0.5 0.2]. If I was asked to calculate after how many ...
0
votes
0answers
21 views

“Simple” proof about expected number of visits

Let $X_n$ be a markov chain with state space $\Omega$. Let $G(x,A)$ denote the expected number of visits to $x \in A$ before exiting a subset $A \subset \Omega$. Prove that for all $x,y$ and A, ...
3
votes
1answer
22 views

non-stationary Markov chain n-step

When I search for the long term behaviour of a stationary markov chain I just multiply the transition matrix with itself for the number of steps: P(n) = P(0)^n. But how do you go about doing it ...
0
votes
0answers
17 views

Book recommendation needed: asymptotic behavior of non-stationary Markov chain

Is there any stochastic process textbook which covers some standard results for non-stationary Markov chain? For my purpose, countable state space is enough. Thanks!
0
votes
0answers
6 views

Efficient random sample from Markov chain with known states at two times

Assume a 2-state Markov chain with known transition matrix. Suppose I know, for example, that the chain is in state 1 at time 1, and is also in state 0 at time 10. I want to sample randomly from the ...
0
votes
2answers
155 views

Does there exist a steady state vector of this Markov Matrix?

Does there exist a steady state vector of Markov Matrix $$P=\begin{bmatrix} \frac{1}{2} & \frac{1}{3}\\ \frac{1}{2} & \frac{2}{3} \end{bmatrix}$$ Initially I was not sure whether to answer ...
1
vote
1answer
25 views

Converting second order Markov chain into a first order Markov chain

I'm having some trouble converting a second order Markov chain into a first order Markov chain, namely I want to define some new random variables $Y_i$, that have the property ...
0
votes
1answer
31 views

How to prove that the column sum for a markov matrix is 1?

As is the topic, it is obvious and easy to explain in non-math language but how do I mathematically prove it?
0
votes
0answers
16 views

Finding a One Step Transition Matrix for a Markov Process? (Gambling Application)

I need help finding what a one step transition matrix would look like for the following gambling scenario: Using the bold strategy, say you have a certain amount of money x at any time and you're ...
0
votes
1answer
49 views

Beginner's questions to Hidden Markov Models

I have started reading about Hidden Markov Models, and have some (more or less) minor questions about things I am not sure I understood correctly. I hope asking here is fine: (1) Assumption about the ...
1
vote
0answers
58 views

How to calculate this integral?

I got confused with this Markov Chain problem: suppose the kernel $Q$ is $Q_x=N(cx,1)$, $c$ is a fixed constant with $|c|<1$ and the stationary distribution is $\pi=N(0,\frac{1}{1-c^2})$. I want ...
2
votes
1answer
65 views

Show that a Markov Chain is ergodic

Let $Y_n$ be iid random variables with values 1,2,3..n so that $P[Y_i=j]=p_j>0$, where $i\leq1$ and $1\leq j\leq n$. I think I managed to show that $Y_n$ is a Markov chain using the definition, ...
1
vote
1answer
38 views

Stronger version of Markov Chain

I have just started looking into the concept of Markov chains and I was wondering if anyone could help me with this problem. Let $X_1, X_2, ...$ be a Markov chain with the state space $S$. I need ...
1
vote
0answers
47 views

Expected time of reaching 0 of a simple symmetric random walk

Consider the symmetric, simple random walk on $S = \{0, 1, \ldots , k\}$ for $k \in \mathbb N$. Let $$T = \min \{ n \in \mathbb N_0|X_n = 0\}$$ be the first time where the process reaches $0$ and ...
9
votes
1answer
130 views

Stochastic kernel as linear operator

Let $K$ be a stochastic kernel for a set $S$ equipped with a countably generated $\sigma$-Algebra $B(S)$, i.e. $K:S\times B(S)\rightarrow [0,1]$ such that $K(\cdot,A)$ is a measurable function for ...
0
votes
1answer
92 views

Probability of extinction in branching process

Consider a branching process where the offspring distribution is given by $$P(X = k) = \frac{1}{2^{k+1}}$$ what is the probability that the process becomes extinct at exactly at the nth generation? ...
0
votes
0answers
32 views

Two-state Markov Chains

If I have a two-state Markov chian $V(t)$ with transition probabilities: $P_{00}(t)=(1-\pi) + \pi e^{-\tau t}$ $P_{01}(t)= \pi - \pi e^{-\tau t}$ $P_{10}(t)=(1-\pi) - (1-\pi)e^{-\tau t}$ ...
1
vote
0answers
22 views

Random DFS properties

Have there been any work analyzing some properties of random DFS walks? By that I mean a DFS search, which chooses the next node to visit with uniform probability. i.e, it still refrains from visiting ...
1
vote
2answers
46 views

Aperiodicity of Markov chain

If a markov chain which has many states but only one state has a self-loop edge, then does it mean that the markov chain is aperiodic? Or every state in the markov chain has to have self-loop? For ...
0
votes
1answer
72 views

Can You Help Me With This Continuous Markov Chain Question?

Consider 2 machines, both of which have an exponential lifetime with mean $\frac{1}{\lambda}$. There is a single repairman that can service machines at an exponential rate $\mu$. Set up the ...
0
votes
0answers
64 views

how to determine transient and recurrent state from transition matrix

I wonder how can I determine the transient and recurrent state from transition matrix ? I mean if I have 10 states It would be very hard to draw diagram for them so how to analyse the matrix? For ...