Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
8 views

Markov Chain First Step Analysis

Given a MC with five states $\{1,2,3,4,5\}$ and transition matrix \begin{bmatrix} 0.5& 0.5 & 0 & 0 & 0 \\ 0.75 & 0.25 & 0 & 0 & 0 \\ 0 & 0.25 & ...
0
votes
0answers
25 views

Markov Chain problem application

Let $P$ be the transition matrix for a regular Markov chain and $v$ be its equilibruim vector. Show that $v$ has zero entries. How would you prove this? I am struggling in this class. Any help is ...
-3
votes
1answer
31 views

A Markov chain with outcomes {0, … , N} [on hold]

A Markov chain with outcomes $\{0, \ldots , N\}$ and transition probabilities: $$p_{i,i+1} = p \\ p_{i,i-1} = q $$ for $1 \leq i \leq N-1$ and $p+q = 1$. Assume $p_{0,1} = p_{N,N-1} = 1$. Is this ...
1
vote
0answers
5 views

Expectation and limit of a stop-and-go traveler markov chain

The velocity V(t) of a stop and go traveler is a two-state Markov chain whose generator is given by $$ \begin{array}{cc} &\begin{matrix}0&1\end{matrix}\\ \ \begin{matrix}0\\ 1\end{matrix} ...
2
votes
1answer
58 views

Is this chain irreducible and/or Aperiodic? What is its equilibrium mass function?

Consider a Markov chain with outcomes $\{0,…,n\}$ and transition probabilities $P_{i,i+1}=p$ $P_{i,i−1}=q$ for $1\le i\le n−1$ and $p+q=1$. Assume also that $P_{0,1} = P_{n,n−1} = 1$. Is this chain ...
0
votes
1answer
47 views

probability, random walk, Markov chain question

Let $P$ be a transition matrix for a regular Markov chain and let $w$ be it’s equilibrium vector. Show that $w$ has no zero entries.
1
vote
1answer
15 views

Poisson process Probabilities

If I assume that {$N(t)=: t >= 0$} is a Poisson process with intensity $\lambda$. For $0<s<t$, how would I find the Pr{$N(t)>N(s)$}?
1
vote
1answer
15 views

Application $\pi$-$\lambda$ lemma one-sided Markov shift

Let $(S_k^{\mathbb{N}},\Sigma_k^{\mathbb{N}},m,\tau)$ be the probability preserving transformation of the one-sided Markov shift, where $\Sigma_k^{\mathbb{N}}$ is the $\sigma$-algebra generated by the ...
0
votes
0answers
21 views

Markov Chain States

Given a MC with five states $\{1,2,3,4,5\}$ and transition matrix \begin{bmatrix} 0.5& 0.5 & 0 & 0 & 0 \\ 0.75 & 0.25 & 0 & 0 & 0 \\ 0 & 0.25 & ...
-1
votes
0answers
51 views

Show the equilibrium vector of a transition matrix for a Markov Chain has no zero entries [duplicate]

Let P be a transition matrix for a regular Markov chain and let w be its equilibrium vector. Show that w has no zero entries. I am thinking using the fundamental limit theorem for regular chains for ...
-2
votes
0answers
15 views

Identifying markov chains and the markov property [on hold]

Im currently revising for a probability exam and I came across this question: Let $(X_n),n\geq1$ be a sequence of independent identically distributed non- negative random variables taking values in ...
-2
votes
0answers
23 views

Consider 2 Stocks. If Stock 1 sells \$10(0.8) or sells \$20(0.9). If Stock 2 sells \$10(0.9) or \$25(0.8). Which stock sells for higher price? [on hold]

Question is Based on Markov Chains. Consider two stocks. Stock 1 always sells for \$10 or \$20. If stock 1 is selling for\$10 today, there is a 0.80 chance that it will sell for \$10 tomorrow. If it ...
0
votes
1answer
39 views

Mean return time in Markov chain

Given the following Markov chain: $p_{0,1}=1$ (if we are in state 0, we must go to state 1) $p_{i,i+1}=p_{i,i-1}=0.5$ There are infinitely (countably) many states. I assume that $X_0=0$ and define ...
1
vote
2answers
20 views

Prove that markov chain is recurrent

I have the following markov chain : $S=\{0,1,2,3\}$ $p_{i,0} = q$ (if we are in one of the states $0,1,2,3$ we can return to $0$ with probability $q$) $p_{i,i+1} = 1-q , i\in\{0,1,2\}$ (if we are ...
0
votes
0answers
15 views

How to find stationary states of non-homogeneous Markov chain

As the title suggests I am interested in finding the transition matrix after some steps for a non-homogeneous Markov chain. I modeled my problem as stationary first and after some steps this ...
0
votes
0answers
9 views

Using Little's law to solve a Continuous Time Markov Chain problem

The problem listed below utilizes a CTMC in order to find throughput. I don't understand how to use the 'cut method' in order to find the stationary distribution for this problem. In addition to this, ...
0
votes
0answers
8 views

How to make Continuous Tme Markov Chain Transition Diagram?

I am studying for my final and this is one of the problems my professor assigned to us. I do not understand why the diagonal values of the transition matrix are supposed to be negative. I also do not ...
1
vote
0answers
12 views

Markov chain converges to boundary

I am learning martingale and related concepts recently and come across the following problem. Suppose $D$ is a bounded, connected, open subset of $\mathbb{R}^2$ with boundary $\partial D$. ...
4
votes
1answer
31 views

Expected number of turns for SPROUT

As a mathematical father (and with apparently plenty of time on my hands) I long ago computed the expected number of turns for a number of children's games that are effectively Markov maps. (Chutes ...
0
votes
0answers
30 views

If there are two different stationary distributions, then there are infinitely many distributions in reducible markov chain

If there are two stationary distributions μ1 and μ2 there are actually infinitely many stationary distributions: (pμ1 + (1 − p)μ2) is also a stationary distribution for any real number 0 ≤ p ≤ 1. How ...
0
votes
1answer
20 views

Existing of a distribution of three random variables that have conditional mutual information with defined properties.

I have two similar questions: 1)Does exist a distribution of three random variables such that: $I(a:b) = 0$ and $I(a:b|c)>0$ (where $I(a:b)$ is a mutual information and $I(a:b|c)$ is a ...
0
votes
0answers
17 views

How to find the number of transitions, after which the stationary distribution could be found in Markov chain?

Say I have the initial state space vector S = [1 0 0]. and I know both the transition matrix, P and final stationary distribution, S' = [0.3 0.5 0.2]. If I was asked to calculate after how many ...
-2
votes
0answers
24 views

Question on Markov Chains [on hold]

Let $S=\{1,2,\ldots,d\}$ for some $d\geq 2$. For $i\in(1,d)\cap \mathbb{Z}$ let $p(i,i+1)=p(i,i-1)=\frac{1}{2}$. Let $p(1,1)=p(d,d)=1$. How would I be able to find all invariant probability ...
0
votes
0answers
20 views

“Simple” proof about expected number of visits

Let $X_n$ be a markov chain with state space $\Omega$. Let $G(x,A)$ denote the expected number of visits to $x \in A$ before exiting a subset $A \subset \Omega$. Prove that for all $x,y$ and A, ...
3
votes
1answer
19 views

non-stationary Markov chain n-step

When I search for the long term behaviour of a stationary markov chain I just multiply the transition matrix with itself for the number of steps: P(n) = P(0)^n. But how do you go about doing it ...
0
votes
0answers
15 views

Book recommendation needed: asymptotic behavior of non-stationary Markov chain

Is there any stochastic process textbook which covers some standard results for non-stationary Markov chain? For my purpose, countable state space is enough. Thanks!
0
votes
0answers
6 views

Efficient random sample from Markov chain with known states at two times

Assume a 2-state Markov chain with known transition matrix. Suppose I know, for example, that the chain is in state 1 at time 1, and is also in state 0 at time 10. I want to sample randomly from the ...
0
votes
2answers
65 views

Does there exist a steady state vector of this Markov Matrix?

Does there exist a steady state vector of Markov Matrix $$P=\begin{bmatrix} \frac{1}{2} & \frac{1}{3}\\ \frac{1}{2} & \frac{2}{3} \end{bmatrix}$$ Initially I was not sure whether to answer ...
1
vote
1answer
18 views

Converting second order Markov chain into a first order Markov chain

I'm having some trouble converting a second order Markov chain into a first order Markov chain, namely I want to define some new random variables $Y_i$, that have the property ...
0
votes
1answer
29 views

How to prove that the column sum for a markov matrix is 1?

As is the topic, it is obvious and easy to explain in non-math language but how do I mathematically prove it?
0
votes
0answers
8 views

Finding a One Step Transition Matrix for a Markov Process? (Gambling Application)

I need help finding what a one step transition matrix would look like for the following gambling scenario: Using the bold strategy, say you have a certain amount of money x at any time and you're ...
0
votes
1answer
36 views

Beginner's questions to Hidden Markov Models

I have started reading about Hidden Markov Models, and have some (more or less) minor questions about things I am not sure I understood correctly. I hope asking here is fine: (1) Assumption about the ...
1
vote
0answers
58 views

How to calculate this integral?

I got confused with this Markov Chain problem: suppose the kernel $Q$ is $Q_x=N(cx,1)$, $c$ is a fixed constant with $|c|<1$ and the stationary distribution is $\pi=N(0,\frac{1}{1-c^2})$. I want ...
2
votes
1answer
52 views

Show that a Markov Chain is ergodic

Let $Y_n$ be iid random variables with values 1,2,3..n so that $P[Y_i=j]=p_j>0$, where $i\leq1$ and $1\leq j\leq n$. I think I managed to show that $Y_n$ is a Markov chain using the definition, ...
1
vote
1answer
35 views

Stronger version of Markov Chain

I have just started looking into the concept of Markov chains and I was wondering if anyone could help me with this problem. Let $X_1, X_2, ...$ be a Markov chain with the state space $S$. I need ...
1
vote
0answers
35 views

Expected time of reaching 0 of a simple symmetric random walk

Consider the symmetric, simple random walk on $S = \{0, 1, \ldots , k\}$ for $k \in \mathbb N$. Let $$T = \min \{ n \in \mathbb N_0|X_n = 0\}$$ be the first time where the process reaches $0$ and ...
2
votes
1answer
28 views

Stochastic kernel as linear operator

Let $K$ be a stochastic kernel for a set $S$ equipped with a countably generated $\sigma$-Algebra $B(S)$, i.e. $K:S\times B(S)\rightarrow [0,1]$ such that $K(\cdot,A)$ is a measurable function for ...
0
votes
1answer
80 views

Probability of extinction in branching process

Consider a branching process where the offspring distribution is given by $$P(X = k) = \frac{1}{2^{k+1}}$$ what is the probability that the process becomes extinct at exactly at the nth generation? ...
0
votes
0answers
26 views

Two-state Markov Chains

If I have a two-state Markov chian $V(t)$ with transition probabilities: $P_{00}(t)=(1-\pi) + \pi e^{-\tau t}$ $P_{01}(t)= \pi - \pi e^{-\tau t}$ $P_{10}(t)=(1-\pi) - (1-\pi)e^{-\tau t}$ ...
1
vote
0answers
12 views

Random DFS properties

Have there been any work analyzing some properties of random DFS walks? By that I mean a DFS search, which chooses the next node to visit with uniform probability. i.e, it still refrains from visiting ...
0
votes
2answers
25 views

Aperiodicity of Markov chain

If a markov chain which has many states but only one state has a self-loop edge, then does it mean that the markov chain is aperiodic? Or every state in the markov chain has to have self-loop? For ...
0
votes
1answer
36 views

Can You Help Me With This Continuous Markov Chain Question?

Consider 2 machines, both of which have an exponential lifetime with mean $\frac{1}{\lambda}$. There is a single repairman that can service machines at an exponential rate $\mu$. Set up the ...
0
votes
0answers
35 views

how to determine transient and recurrent state from transition matrix

I wonder how can I determine the transient and recurrent state from transition matrix ? I mean if I have 10 states It would be very hard to draw diagram for them so how to analyse the matrix? For ...
1
vote
0answers
37 views

Limiting probability of a successful bid

I'm having trouble completing the above question, as my working knowledge of "limiting probabilities" is not very good. For the 1-step transition matrix, I have $$P= \begin{pmatrix} 0.0 & 0.0 ...
0
votes
1answer
53 views

PageRank (power iteration method) convergence rate?

I could not get my head around the idea that the second eigenvalue is the convergence rate. Since the matrix in this application is a Markov matrix (rows/columns sum to one), the largest eigenvalue ...
0
votes
1answer
25 views

Trying to find the markov chain and adjacency matrix of this graph?

This is graph of the problem: Suppose animal x is at node 3 of the graph. It chooses small path labelled s with 2 times probability then long path l. If length is same then probability is same ...
1
vote
1answer
20 views

Period of a Markov Chain: Why is this one aperiodic?

Here is the problem from a stochastic processes book: Consider a Markov Chain on {0,1,2} having transition matrix 0 1 2 0| 0 0 1| 1| 1 0 0| 2|.5 .5 0| ...
0
votes
0answers
11 views

Computing smoothed state distribution in HMM

Suppose we have an HMM with two states: $s_1$ and $s_2$. The transitional model is as follows: $P(s_1|s_1) = 0.5$, and $P(s1|s2) = 0.25$. There are two observations: $P(a|s_1) = 0.25$ and $P(a|s_2) = ...
2
votes
1answer
56 views

Random walks : Hitting and recurrence Times relation

I have trouble understanding that how $$E\left[T_0|X_{0} = 0\right] = 1 + E[H_0|X_0=1] $$ where $T_0 = \inf\{n \geq 1:X_n = 0 \}$ and $H_A =\inf\{ n\geq 0: X_n \in A \}$. In other words $T_0$ is the ...
1
vote
0answers
24 views

Learning about Markov Chains

I am trying to learn about how to use markov chains for complicated probability problems. I have been looking for different materials to learn these but haven't had much luck. Does anyone have any ...