Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
14 views

Exact expression for this Markov chain's stationary distribution

$$M= \begin{pmatrix} 0 & 1 & 0 & 0 & \cdots & 0 \\ a & 0 & 1-a & 0 & \cdots & 0 \\ 0& a & 0 & 1-a & \cdots & 0 \\ \vdots & \vdots & ...
-2
votes
0answers
15 views

Identifying markov chains and the markov property [on hold]

Im currently revising for a probability exam and I came across this question: Let $(X_n),n\geq1$ be a sequence of independent identically distributed non- negative random variables taking values in ...
-2
votes
0answers
21 views

Consider 2 Stocks. If Stock 1 sells \$10(0.8) or sells \$20(0.9). If Stock 2 sells \$10(0.9) or \$25(0.8). Which stock sells for higher price?

Question is Based on Markov Chains. Consider two stocks. Stock 1 always sells for \$10 or \$20. If stock 1 is selling for\$10 today, there is a 0.80 chance that it will sell for \$10 tomorrow. If it ...
0
votes
1answer
36 views

Mean return time in Markov chain

Given the following Markov chain: $p_{0,1}=1$ (if we are in state 0, we must go to state 1) $p_{i,i+1}=p_{i,i-1}=0.5$ There are infinitely (countably) many states. I assume that $X_0=0$ and define ...
1
vote
2answers
20 views

Prove that markov chain is recurrent

I have the following markov chain : $S=\{0,1,2,3\}$ $p_{i,0} = q$ (if we are in one of the states $0,1,2,3$ we can return to $0$ with probability $q$) $p_{i,i+1} = 1-q , i\in\{0,1,2\}$ (if we are ...
0
votes
0answers
15 views

How to find stationary states of non-homogeneous Markov chain

As the title suggests I am interested in finding the transition matrix after some steps for a non-homogeneous Markov chain. I modeled my problem as stationary first and after some steps this ...
0
votes
0answers
6 views

Using Little's law to solve a Continuous Time Markov Chain problem

The problem listed below utilizes a CTMC in order to find throughput. I don't understand how to use the 'cut method' in order to find the stationary distribution for this problem. In addition to this, ...
0
votes
0answers
8 views

How to make Continuous Tme Markov Chain Transition Diagram?

I am studying for my final and this is one of the problems my professor assigned to us. I do not understand why the diagonal values of the transition matrix are supposed to be negative. I also do not ...
1
vote
0answers
11 views

Markov chain converges to boundary

I am learning martingale and related concepts recently and come across the following problem. Suppose $D$ is a bounded, connected, open subset of $\mathbb{R}^2$ with boundary $\partial D$. ...
4
votes
1answer
30 views

Expected number of turns for SPROUT

As a mathematical father (and with apparently plenty of time on my hands) I long ago computed the expected number of turns for a number of children's games that are effectively Markov maps. (Chutes ...
0
votes
0answers
24 views

If there are two different stationary distributions, then there are infinitely many distributions in reducible markov chain

If there are two stationary distributions μ1 and μ2 there are actually infinitely many stationary distributions: (pμ1 + (1 − p)μ2) is also a stationary distribution for any real number 0 ≤ p ≤ 1. How ...
0
votes
1answer
20 views

Existing of a distribution of three random variables that have conditional mutual information with defined properties.

I have two similar questions: 1)Does exist a distribution of three random variables such that: $I(a:b) = 0$ and $I(a:b|c)>0$ (where $I(a:b)$ is a mutual information and $I(a:b|c)$ is a ...
0
votes
0answers
17 views

How to find the number of transitions, after which the stationary distribution could be found in Markov chain?

Say I have the initial state space vector S = [1 0 0]. and I know both the transition matrix, P and final stationary distribution, S' = [0.3 0.5 0.2]. If I was asked to calculate after how many ...
-2
votes
0answers
23 views

Question on Markov Chains

Let $S=\{1,2,\ldots,d\}$ for some $d\geq 2$. For $i\in(1,d)\cap \mathbb{Z}$ let $p(i,i+1)=p(i,i-1)=\frac{1}{2}$. Let $p(1,1)=p(d,d)=1$. How would I be able to find all invariant probability ...
0
votes
0answers
20 views

“Simple” proof about expected number of visits

Let $X_n$ be a markov chain with state space $\Omega$. Let $G(x,A)$ denote the expected number of visits to $x \in A$ before exiting a subset $A \subset \Omega$. Prove that for all $x,y$ and A, ...
3
votes
1answer
19 views

non-stationary Markov chain n-step

When I search for the long term behaviour of a stationary markov chain I just multiply the transition matrix with itself for the number of steps: P(n) = P(0)^n. But how do you go about doing it ...
0
votes
0answers
14 views

Book recommendation needed: asymptotic behavior of non-stationary Markov chain

Is there any stochastic process textbook which covers some standard results for non-stationary Markov chain? For my purpose, countable state space is enough. Thanks!
0
votes
0answers
6 views

Efficient random sample from Markov chain with known states at two times

Assume a 2-state Markov chain with known transition matrix. Suppose I know, for example, that the chain is in state 1 at time 1, and is also in state 0 at time 10. I want to sample randomly from the ...
0
votes
2answers
63 views

Does there exist a steady state vector of this Markov Matrix?

Does there exist a steady state vector of Markov Matrix $$P=\begin{bmatrix} \frac{1}{2} & \frac{1}{3}\\ \frac{1}{2} & \frac{2}{3} \end{bmatrix}$$ Initially I was not sure whether to answer ...
1
vote
1answer
18 views

Converting second order Markov chain into a first order Markov chain

I'm having some trouble converting a second order Markov chain into a first order Markov chain, namely I want to define some new random variables $Y_i$, that have the property ...
0
votes
1answer
29 views

How to prove that the column sum for a markov matrix is 1?

As is the topic, it is obvious and easy to explain in non-math language but how do I mathematically prove it?
0
votes
0answers
8 views

Finding a One Step Transition Matrix for a Markov Process? (Gambling Application)

I need help finding what a one step transition matrix would look like for the following gambling scenario: Using the bold strategy, say you have a certain amount of money x at any time and you're ...
0
votes
1answer
36 views

Beginner's questions to Hidden Markov Models

I have started reading about Hidden Markov Models, and have some (more or less) minor questions about things I am not sure I understood correctly. I hope asking here is fine: (1) Assumption about the ...
1
vote
0answers
58 views

How to calculate this integral?

I got confused with this Markov Chain problem: suppose the kernel $Q$ is $Q_x=N(cx,1)$, $c$ is a fixed constant with $|c|<1$ and the stationary distribution is $\pi=N(0,\frac{1}{1-c^2})$. I want ...
2
votes
1answer
52 views

Show that a Markov Chain is ergodic

Let $Y_n$ be iid random variables with values 1,2,3..n so that $P[Y_i=j]=p_j>0$, where $i\leq1$ and $1\leq j\leq n$. I think I managed to show that $Y_n$ is a Markov chain using the definition, ...
1
vote
1answer
35 views

Stronger version of Markov Chain

I have just started looking into the concept of Markov chains and I was wondering if anyone could help me with this problem. Let $X_1, X_2, ...$ be a Markov chain with the state space $S$. I need ...
1
vote
0answers
31 views

Expected time of reaching 0 of a simple symmetric random walk

Consider the symmetric, simple random walk on $S = \{0, 1, \ldots , k\}$ for $k \in \mathbb N$. Let $$T = \min \{ n \in \mathbb N_0|X_n = 0\}$$ be the first time where the process reaches $0$ and ...
2
votes
1answer
28 views

Stochastic kernel as linear operator

Let $K$ be a stochastic kernel for a set $S$ equipped with a countably generated $\sigma$-Algebra $B(S)$, i.e. $K:S\times B(S)\rightarrow [0,1]$ such that $K(\cdot,A)$ is a measurable function for ...
0
votes
1answer
80 views

Probability of extinction in branching process

Consider a branching process where the offspring distribution is given by $$P(X = k) = \frac{1}{2^{k+1}}$$ what is the probability that the process becomes extinct at exactly at the nth generation? ...
0
votes
0answers
26 views

Two-state Markov Chains

If I have a two-state Markov chian $V(t)$ with transition probabilities: $P_{00}(t)=(1-\pi) + \pi e^{-\tau t}$ $P_{01}(t)= \pi - \pi e^{-\tau t}$ $P_{10}(t)=(1-\pi) - (1-\pi)e^{-\tau t}$ ...
1
vote
0answers
12 views

Random DFS properties

Have there been any work analyzing some properties of random DFS walks? By that I mean a DFS search, which chooses the next node to visit with uniform probability. i.e, it still refrains from visiting ...
0
votes
2answers
25 views

Aperiodicity of Markov chain

If a markov chain which has many states but only one state has a self-loop edge, then does it mean that the markov chain is aperiodic? Or every state in the markov chain has to have self-loop? For ...
0
votes
1answer
35 views

Can You Help Me With This Continuous Markov Chain Question?

Consider 2 machines, both of which have an exponential lifetime with mean $\frac{1}{\lambda}$. There is a single repairman that can service machines at an exponential rate $\mu$. Set up the ...
0
votes
0answers
34 views

how to determine transient and recurrent state from transition matrix

I wonder how can I determine the transient and recurrent state from transition matrix ? I mean if I have 10 states It would be very hard to draw diagram for them so how to analyse the matrix? For ...
1
vote
0answers
37 views

Limiting probability of a successful bid

I'm having trouble completing the above question, as my working knowledge of "limiting probabilities" is not very good. For the 1-step transition matrix, I have $$P= \begin{pmatrix} 0.0 & 0.0 ...
0
votes
1answer
53 views

PageRank (power iteration method) convergence rate?

I could not get my head around the idea that the second eigenvalue is the convergence rate. Since the matrix in this application is a Markov matrix (rows/columns sum to one), the largest eigenvalue ...
0
votes
1answer
25 views

Trying to find the markov chain and adjacency matrix of this graph?

This is graph of the problem: Suppose animal x is at node 3 of the graph. It chooses small path labelled s with 2 times probability then long path l. If length is same then probability is same ...
1
vote
1answer
20 views

Period of a Markov Chain: Why is this one aperiodic?

Here is the problem from a stochastic processes book: Consider a Markov Chain on {0,1,2} having transition matrix 0 1 2 0| 0 0 1| 1| 1 0 0| 2|.5 .5 0| ...
0
votes
0answers
11 views

Computing smoothed state distribution in HMM

Suppose we have an HMM with two states: $s_1$ and $s_2$. The transitional model is as follows: $P(s_1|s_1) = 0.5$, and $P(s1|s2) = 0.25$. There are two observations: $P(a|s_1) = 0.25$ and $P(a|s_2) = ...
2
votes
1answer
56 views

Random walks : Hitting and recurrence Times relation

I have trouble understanding that how $$E\left[T_0|X_{0} = 0\right] = 1 + E[H_0|X_0=1] $$ where $T_0 = \inf\{n \geq 1:X_n = 0 \}$ and $H_A =\inf\{ n\geq 0: X_n \in A \}$. In other words $T_0$ is the ...
1
vote
0answers
24 views

Learning about Markov Chains

I am trying to learn about how to use markov chains for complicated probability problems. I have been looking for different materials to learn these but haven't had much luck. Does anyone have any ...
1
vote
0answers
12 views

Mixing time analysis of time inhomogeneous markov chaons

There are common methods to characterize mixing times of time homogeneous Markov chains through coupling, conductance and strongly stationary times. However, suppose there is a time-inhomogeneous ...
1
vote
1answer
42 views

Question about HMM

I have this HMM model that I need to solve. Unfortunately, my textbook isn't the best and only describes general cases which I have difficulty working with. Consider an HMM with two states: s1 and ...
0
votes
1answer
25 views

Markov Chain--starting states

How do we define the starting states in a Markov Chain. For example if we are asked to calculate the transition matrix for different starting states, what does that mean? I am ultimately asked to ...
-1
votes
0answers
26 views

Conditions for a Markov process to have independent increments [duplicate]

I consistently see "Let $\{X(t)\}$ be a stochastic process with independent increments..." in various texts, though I have yet to find any conditions under which we can guarantee a process to have ...
4
votes
1answer
91 views

$P^n$ transition matrix of a Markov chain

The setup: We have an unlimited supply of balls and $k$ boxes. In every step, we randomly (all of them have the same probability) choose a box and put a ball in it. Let $X_n$ be the number of ...
1
vote
1answer
22 views

Proving irreducibility of Markov chain

I have a Markov chain: state: a permutation of n cards transition: taking the top-most card and randomly choose one of the n possible positions for the card I know it is obviously irreducible ...
0
votes
0answers
20 views

DTMC: repairing the machine

A machine works for $Y_0$ time then fails and takes $X_1$ time to repair. Then again works for $Y_1$ time and then fails and takes $X_2$ time to repair and so on. All the $X_n$'s and $Y_n$'s ...
1
vote
0answers
25 views

Getting stuck in a loop or the probability of hitting all points in a random walk around a circle.

Suppose you are walking around a circular path made up of $n$ tiles. Each tile $i$ is assigned a distinct value $r_i$ by a random variable uniformly distributed on the set of integers $\{1,...,k\}$ ...
2
votes
1answer
56 views

Random walk : probability of reaching value $i$ without passing by negative value $j$

This is just some question that popped out of nowhere while starting studying random walks, and I don't really know how to approach this. Say I have a random walk that starts at zero, and goes up or ...