Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
7 views

Show that in every irreducible and recurrent Markov Chain for all pair of states (i,j) the probability of ever achieving j from i equals 1.

My question is exactly the one in the title. So far I have figured that I can use definition: $$ F_{ij} = P ( \bigcup_{n=1}^{\infty} \{ X_n = j \} | X_0 = i) $$ $$ f_{ij}(n) = P ( X_1 \neq j, \ldots ...
0
votes
0answers
18 views

Cartesian product of two markov chains is a markov chain [on hold]

Is a Cartesian product of two Markov chains a Markov chain? And is it true for a countable product as well?
0
votes
1answer
35 views

Sum of i.i.d. random variables is a markov chain

I think I have some problem understanding markov chains, because we defined them as abstract objects but our professor does proofs with them as if they where just elementary conditional probabilities. ...
1
vote
0answers
9 views

implication of positive speed of random walk on a graph

Let $(V,E)$ be a vertex-transitive graph and let one vertex be the origin. Let $d(v,0)$ be the graph distance between $v$ and $0$. Consider $(X_n)$ a simple random walk on the graph. Let $A_n$ be the ...
1
vote
0answers
14 views

Distribution of continuous time markov chain

I'm having trouble understanding the question below. I understand the continuous time markov chain and unique stationary distribution but not sure what it is asking. I have a continuous-time Markov ...
0
votes
1answer
22 views

Existence of steady state distribution for finite state Markov chains

Let's assume a Markov chain has 2 recurrent classes and a transient state from which we can go to either of the recurrent classes. If one of those recurrent classes is periodic, would it effect the ...
0
votes
1answer
13 views

Absorbing states and Irreducible sets

Question on the definition of Markov Chain matrices: Is it possible to have an absorbing state (i.e. a state where the probability of returning to itself is 1) within an irreducible set? I.e., if we ...
0
votes
1answer
13 views

Do the Matrices representing Markov chains need to be square?

I assume so -- I ask in the context of defining an irreducible set. If a set is non-irreducible, you should be able to find a "smaller" Markov chain matrix nested within a larger one. That "smaller" ...
0
votes
0answers
18 views

Markov Chains and $n$-step transition probabilities

I have learnt that in a Markov Chain, the one step transition probability depends only on the current state, and not any of the previous states, by the definition of a Markov Chain. Now, when we ...
0
votes
2answers
19 views

Transition probability matrix

In the article here it had this question. A walker moves on two positions a and b. She begins at a at time 0, and is at a next time as well. Subsequently, if she is at the same position for two ...
1
vote
1answer
44 views

Long term probability in Markov Chains

I was practicing some questions on transition probability matrices and I came up with this question. You have 3 coins: A (Heads probability 0.2),B (Heads probability 0.4), C (Heads probability ...
1
vote
1answer
19 views

Birth-death process: What is the distribution of reached states before reaching an absorbing state?

Intro I am working on a birth-death process. For a given choice of parameter ($n=6$, $Wa=1$, $Wb=0.95$, see below), the transition matrix is $$\left( \begin{array}{ccccccc} 1. & 0.144928 & ...
1
vote
0answers
15 views

Merging rates on a CTMC model

first time question here. I'm having a rough time trying to represent the following CTMC. Any help would be gladly appreciated. We consider a server with a infinite buffer connected to a network. ...
1
vote
0answers
25 views

Identifying markov chains

Problem: A regular dice is thrown repeatedly. Which one of the following random variables with values in $\mathbb{N}\cup\{\infty\}\cup\{0\}$ is a Markov chain? For those who are, give the ...
0
votes
0answers
12 views

A question about the translation property Markov kernel

Given that ${X_n}$ is a Markov chain, and a Markov kernel with translation propert$p(y+x,E+x)=p(y,E)$. Question:How to show $Y_n=X_n-X_{n-1}$ are i.i.d? I'm trying to use Markov Property and ...
0
votes
0answers
12 views

Recurrent Markov chain: probability of visiting state i precisely k times in N steps

I'm studying this Markov process with transition matrix $P$, given by \begin{equation} P=\left(\begin{array}{cccc} \mu & 1-\mu & 0 & 0\\ 0 & 0 & \mu & 1-\mu\\ \mu & 1-\mu ...
1
vote
1answer
26 views

Show a random walk is transient

I was going through some problems related to Markov chains and I got stuck on this bit: We are given a random walk on $Z$, defined by the transition matrix $p_{i,i+1}=p$ and $p_{i,i-1}=1-p$. How to ...
0
votes
1answer
18 views

Limiting distribution of a Markov chain?

I have the problem below. There are n identical machines. They are all operational at time 0. The lifetime of each one is an exponential random variable with rate L. There are r repairmen (1 ≤ r ≤ ...
0
votes
0answers
16 views

Equilibrium Vector For Regular Markov Chain

Let $P$ be a transition matrix for a regular Markov Chain and let $w$ be it's equilibrium vector. Show that $w$ has no zero entries
1
vote
1answer
36 views

Is it worth playing this game of St. Petersburg paradox?

A gambler offers you the following deal. You have to keep tossing a fair coin until you get a heads, at which point you stop and collect your winnings: if it happens after n throws, the gambler will ...
0
votes
0answers
14 views

Transition rates and probabilities of a continuous markov chain

A certain type of component has two states: 0 = OFF and 1 = OPERATING. In state 0 , the process remains there an exponential amount of time with rate $ \alpha$, and then moves to state 1. The ...
0
votes
0answers
38 views

Markov Chain problem application [on hold]

Let $P$ be the transition matrix for a regular Markov chain and $v$ be its equilibruim vector. Show that $v$ has zero entries. How would you prove this? I am struggling in this class. Any help is ...
-4
votes
1answer
48 views

A Markov chain with outcomes {0, … , N} [closed]

A Markov chain with outcomes $\{0, \ldots , N\}$ and transition probabilities: $$p_{i,i+1} = p \\ p_{i,i-1} = q $$ for $1 \leq i \leq N-1$ and $p+q = 1$. Assume $p_{0,1} = p_{N,N-1} = 1$. Is this ...
4
votes
1answer
48 views
+50

Expectation and limit of a stop-and-go traveler markov chain

The velocity $V(t)$ of a stop and go traveler is a two-state Markov chain whose generator is given by $$ \begin{array}{cc} &\begin{matrix}0&1\end{matrix}\\ \ \begin{matrix}0\\ 1\end{matrix} ...
2
votes
1answer
87 views

Is this chain irreducible and/or Aperiodic? What is its equilibrium mass function?

Consider a Markov chain with outcomes $\{0,…,n\}$ and transition probabilities $P_{i,i+1}=p$ $P_{i,i−1}=q$ for $1\le i\le n−1$ and $p+q=1$. Assume also that $P_{0,1} = P_{n,n−1} = 1$. Is this chain ...
0
votes
1answer
65 views

probability, random walk, Markov chain question

Let $P$ be a transition matrix for a regular Markov chain and let $w$ be it’s equilibrium vector. Show that $w$ has no zero entries.
1
vote
1answer
24 views

Poisson process Probabilities

If I assume that $\{N(t)=: t \ge 0\}$ is a Poisson process with intensity $\lambda$. For $0<s<t$, how would I find the $\Pr\{N(t)>N(s)\}$?
1
vote
1answer
15 views

Application $\pi$-$\lambda$ lemma one-sided Markov shift

Let $(S_k^{\mathbb{N}},\Sigma_k^{\mathbb{N}},m,\tau)$ be the probability preserving transformation of the one-sided Markov shift, where $\Sigma_k^{\mathbb{N}}$ is the $\sigma$-algebra generated by the ...
-1
votes
0answers
31 views

Markov Chain States

Given a MC with five states $\{1,2,3,4,5\}$ and transition matrix \begin{bmatrix} 0.5& 0.5 & 0 & 0 & 0 \\ 0.75 & 0.25 & 0 & 0 & 0 \\ 0 & 0.25 & ...
-2
votes
0answers
20 views

Identifying markov chains and the markov property [closed]

Im currently revising for a probability exam and I came across this question: Let $(X_n),n\geq1$ be a sequence of independent identically distributed non- negative random variables taking values in ...
-2
votes
0answers
27 views

Consider 2 Stocks. If Stock 1 sells \$10(0.8) or sells \$20(0.9). If Stock 2 sells \$10(0.9) or \$25(0.8). Which stock sells for higher price? [closed]

Question is Based on Markov Chains. Consider two stocks. Stock 1 always sells for \$10 or \$20. If stock 1 is selling for\$10 today, there is a 0.80 chance that it will sell for \$10 tomorrow. If it ...
0
votes
1answer
50 views

Mean return time in Markov chain

Given the following Markov chain: $p_{0,1}=1$ (if we are in state 0, we must go to state 1) $p_{i,i+1}=p_{i,i-1}=0.5$ There are infinitely (countably) many states. I assume that $X_0=0$ and define ...
1
vote
2answers
21 views

Prove that markov chain is recurrent

I have the following markov chain : $S=\{0,1,2,3\}$ $p_{i,0} = q$ (if we are in one of the states $0,1,2,3$ we can return to $0$ with probability $q$) $p_{i,i+1} = 1-q , i\in\{0,1,2\}$ (if we are ...
0
votes
0answers
17 views

How to find stationary states of non-homogeneous Markov chain

As the title suggests I am interested in finding the transition matrix after some steps for a non-homogeneous Markov chain. I modeled my problem as stationary first and after some steps this ...
0
votes
0answers
11 views

Using Little's law to solve a Continuous Time Markov Chain problem

The problem listed below utilizes a CTMC in order to find throughput. I don't understand how to use the 'cut method' in order to find the stationary distribution for this problem. In addition to this, ...
0
votes
0answers
8 views

How to make Continuous Tme Markov Chain Transition Diagram?

I am studying for my final and this is one of the problems my professor assigned to us. I do not understand why the diagonal values of the transition matrix are supposed to be negative. I also do not ...
1
vote
0answers
13 views

Markov chain converges to boundary

I am learning martingale and related concepts recently and come across the following problem. Suppose $D$ is a bounded, connected, open subset of $\mathbb{R}^2$ with boundary $\partial D$. ...
4
votes
1answer
31 views

Expected number of turns for SPROUT

As a mathematical father (and with apparently plenty of time on my hands) I long ago computed the expected number of turns for a number of children's games that are effectively Markov maps. (Chutes ...
0
votes
0answers
31 views

If there are two different stationary distributions, then there are infinitely many distributions in reducible markov chain

If there are two stationary distributions μ1 and μ2 there are actually infinitely many stationary distributions: (pμ1 + (1 − p)μ2) is also a stationary distribution for any real number 0 ≤ p ≤ 1. How ...
0
votes
1answer
24 views

Existing of a distribution of three random variables that have conditional mutual information with defined properties.

I have two similar questions: 1)Does exist a distribution of three random variables such that: $I(a:b) = 0$ and $I(a:b|c)>0$ (where $I(a:b)$ is a mutual information and $I(a:b|c)$ is a ...
0
votes
0answers
19 views

How to find the number of transitions, after which the stationary distribution could be found in Markov chain?

Say I have the initial state space vector S = [1 0 0]. and I know both the transition matrix, P and final stationary distribution, S' = [0.3 0.5 0.2]. If I was asked to calculate after how many ...
-2
votes
0answers
24 views

Question on Markov Chains [closed]

Let $S=\{1,2,\ldots,d\}$ for some $d\geq 2$. For $i\in(1,d)\cap \mathbb{Z}$ let $p(i,i+1)=p(i,i-1)=\frac{1}{2}$. Let $p(1,1)=p(d,d)=1$. How would I be able to find all invariant probability ...
0
votes
0answers
20 views

“Simple” proof about expected number of visits

Let $X_n$ be a markov chain with state space $\Omega$. Let $G(x,A)$ denote the expected number of visits to $x \in A$ before exiting a subset $A \subset \Omega$. Prove that for all $x,y$ and A, ...
3
votes
1answer
20 views

non-stationary Markov chain n-step

When I search for the long term behaviour of a stationary markov chain I just multiply the transition matrix with itself for the number of steps: P(n) = P(0)^n. But how do you go about doing it ...
0
votes
0answers
15 views

Book recommendation needed: asymptotic behavior of non-stationary Markov chain

Is there any stochastic process textbook which covers some standard results for non-stationary Markov chain? For my purpose, countable state space is enough. Thanks!
0
votes
0answers
6 views

Efficient random sample from Markov chain with known states at two times

Assume a 2-state Markov chain with known transition matrix. Suppose I know, for example, that the chain is in state 1 at time 1, and is also in state 0 at time 10. I want to sample randomly from the ...
0
votes
2answers
77 views

Does there exist a steady state vector of this Markov Matrix?

Does there exist a steady state vector of Markov Matrix $$P=\begin{bmatrix} \frac{1}{2} & \frac{1}{3}\\ \frac{1}{2} & \frac{2}{3} \end{bmatrix}$$ Initially I was not sure whether to answer ...
1
vote
1answer
19 views

Converting second order Markov chain into a first order Markov chain

I'm having some trouble converting a second order Markov chain into a first order Markov chain, namely I want to define some new random variables $Y_i$, that have the property ...
0
votes
1answer
29 views

How to prove that the column sum for a markov matrix is 1?

As is the topic, it is obvious and easy to explain in non-math language but how do I mathematically prove it?
0
votes
0answers
8 views

Finding a One Step Transition Matrix for a Markov Process? (Gambling Application)

I need help finding what a one step transition matrix would look like for the following gambling scenario: Using the bold strategy, say you have a certain amount of money x at any time and you're ...