Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
14 views

Entropy of a Markov chain (right result?)

Consider the Markov chain with state space $E=\left\{0,1,2,3,4,5,6\right\}$ and transition matrix $$ \begin{pmatrix}1/5 & 3/5 & 0 & 0 & 1/5 & 0 & 0\\0 & 0 & ...
1
vote
1answer
33 views

Bound spectral radius of a certain matrix.

Consider a stochastic matrix $P$, i.e. real, non-negative, square, rows sum to one. Consider $\Xi = \text{diag}(\xi)$ to be a diagonal matrix with a principal left eigenvector $\xi$ of $P$ (i.e. ...
0
votes
2answers
13 views

Does the function of a random variable have the same transition matrix as the variable itself?

If I have a variable X, that follows a Markov Chain with a transition density $\rho(X)$ does a function of that variable f(X) have the same density or is there a one to one mapping to the density of ...
3
votes
1answer
29 views

Exercise from Norris' book on Markov chains

Let $(X_n)$ be a Markov chain on $\mathbb{N}$ with transition probabilities satisfying: $$p_{0,1}=1,\quad p_{i,i-1}+p_{i,i+1}=1,\quad p_{i,i+1}=\left(\frac{i+1}{i}\right)^{\alpha}p_{i,i-1}$$ The ...
0
votes
0answers
12 views

Mean time for the renewal process

The system is as below. Energy keeps coming at a node with a constant rate ρ. Node has files of size exponential(λ) to be transmitted (with fixed rate of transmission $r$, time for transmission would ...
1
vote
1answer
25 views

Simple Markov Chain: Random Walk on $\mathbb{Z}$

We are given a random walk on $\mathbb{Z}$, where $p_{i, i+1}= p < \frac{1}{2}$ and $p_{i,i-1}=1-p > \frac{1}{2}$, starting at $0$. Now we have to compute the probability that we eventually ...
1
vote
1answer
34 views

Prove a certain matrix is positive semidefinte.

Consider a stochastic matrix $P$, i.e. real, non-negative, square, rows sum to one. Consider $\Xi$ to be a diagonal matrix with a principal left eigenvector of $P$ on the main diagonal and zeros ...
1
vote
1answer
8 views

Interval of probabilities which satisfy a Markov chain

Given the following markov chain, where T1 is the start state, the labels are shown on the state( 'a' in this case) and p and 1-p are probabilities for that transition happening: Now, for what ...
-1
votes
0answers
52 views

Markov Chain - Steady State behavior problem

I've been asked to solve the following problem. The problem: Let $X_n$ be a Markov chain with states in given space E, given transition matrix $P$ and all states belong to one and only recurrent ...
0
votes
0answers
15 views

What is the limiting distribution of this Markov Chain?

Take a Markov Chain with state space $\left\{ 0, 1, \dots, 20 \right\}$. Then we have the rule that given $X_n$: Compute $Z = X_n + 1$ or $Z = X_n - 1$ with probability $\frac{1}{2}$ each (if the ...
-2
votes
0answers
6 views

Autocorrelation of a Markov Chain?

Is there a general characterization of the autocorrelation metric of a Markov chain? There are some tangential issues as well: do $n$-state transition probabilities obtained through Chapman-Kolmogorov ...
0
votes
1answer
23 views

Markov Chain Expected value

Let $(X_n)_{n\in\mathbb{N}}$ be a Markov chain with State space $E=\{1,2,3\}$ and transission matrix $$P=\begin{bmatrix} 0 & 1/3 & 2/3 \\ 1/4 & 3/4 & 0 \\ 2/5 & 0 & ...
1
vote
1answer
30 views

Does the steady state distribution of a Markov chain change, when minimizing it?

Say I have this Markov chain. Then I perform bisimulation on it, where I find the largest relation between the states of the Markov chain. Finally I can construct a bisimulation quotient/minimization ...
0
votes
1answer
20 views

Markov Chain Expected Value notation.

I have question to answer regarding $X_n$ where $X_n$ is a Markov chain, $n$=$0$,$1$,$2$,... I am loking for What I don't understand is what this $3$ on $X_{n+1}$ is! Any ideas?
1
vote
1answer
73 views

Is this stochastic process a Markov chain?

I have been struggling sometime now with the following question and I feel like I am stacked. Let $X_n : n= 0,1,\ldots$ be a sequence of iid discrete random variables with $$P(X_n=j)=a_j>0 \qquad ...
0
votes
0answers
4 views

Is there a general formula for determining this distribution in a Markov chain?

Let $C$ be an irreducible Markov chain with state set $S$, $\left| S\right| = n$, transition matrix $T$, starting at state $s_0 \in S$, and yielding the states $s_0, s_1, s_2, \ldots$ during a random ...
2
votes
1answer
62 views

A Markov Chain Flea Problem

A Flea moves around the vertices of a triangle in the following manner: Whenever it is at vertex i it moves to its clockwise neighbor vertex with probability $p_i$ and to the counterclockwise neighbor ...
0
votes
0answers
24 views

variance of number of steps in markov chain (rook move to top right)

I encountered this problem while studying Markov chains and I want to calculate the variance of the problem, i.e. variance of number of steps that a random walker rook make to reach from down-left ...
0
votes
1answer
49 views

Gambler's ruin and Markov Chain, coin toss and stakes

I'm considering a classical problem about Markov Chains: A gambler has $£8$ and wishes to get to $£10$. A coin is tossed repeatedly : if it comes down tails, the gambler loses his stake, and if it ...
4
votes
1answer
30 views

A die whose score cannot be as before (Markov chains)

A die is "fixed" so that each time it is rolled the score cannot be the same as the preceding score, all other scores having probability $1/5$. Given that the first score is 6, what is the probability ...
2
votes
2answers
20 views

Markov Chain: Moving on a circle

A particle moves on 12 points situated on a circle. At each step it is equally likely to move one step in the clockwise or in the counterclockwise direction. Find the mean number of steps for ...
0
votes
1answer
8 views

Expected number of visits to a state of a Markov chain up to a certain time

Let $P=\{p_{ij}\}$ be a stochastic matrix (with rows and columns indexed by a countable set) and let $p^{(k)}_{ij}$ be the entries of $P^k$. I'm trying to prove that, if the associated Markov chain is ...
-3
votes
0answers
23 views

show is markov chain [closed]

suppose: that : X=({X}{n}){n\geq 0}: is: M.C(\lambda ,P): y : f:IxI\rightarrow I a function. denote by ${f}^{-1}(j):={i\in I:f(i)=j}\: \: y \:$ suppose for all $i,j \in I$ such that $ f(i)=f(j)$ ...
0
votes
1answer
26 views

Understanding steady state distribution

I need some help verifying that my understanding of steady state distribution is indeed correct. I have a transition diagram (model). With around 100 states and 6 variables. I have used a software ...
0
votes
0answers
24 views

Using long term Markov Chains in Excel to compute time spent in state

I'm trying to calculate the uptime of a computer state in Excel. I'm in way over my head, and it took me over a day to identify my problem as a Markov Chain. The question is: "X pellets per second ...
0
votes
1answer
15 views

Markov Chains (State transitions)

I was wondering which part I am misunderstanding about the individual-by-individual updating scheme from the book of Jackson M. (Social and Economic Networks, 2008) . The full transition matrix in the ...
-1
votes
0answers
13 views

Finding the infinitesimal generator of a M/M/2 queue [closed]

I have a M/M/2 queue with a total population of 5. The arrival times are independent exponential random variables with mean of $\lambda$ and the service times have a mean of $\mu$. The initial number ...
-1
votes
0answers
19 views

Setting up and Solving Kolmogorov Forward Equations

Consider a Markov Chain with $3\times 3$ generator matrix: $$ G = \begin{bmatrix} -1 & 1/2 & 1/2 \\ 1/2 & -1 & 1/2 \\ 1/2 & 1/2 & -1 \end{bmatrix} $$ What are the ...
1
vote
1answer
30 views

Given a Markov chain $X \rightarrow Y \rightarrow Z$, why is $I(X;Y|Z) \leq I(X;Y)$?

A Markov chain $X \rightarrow Y \rightarrow Z$ is given, where $X,Y,Z$ are random variables characterized by the probability distribution $p(x,y,z) = p(x)p(y|x)p(z|y)$. It follows that $I(X;Y) \geq ...
0
votes
1answer
29 views

Markov chain: if $X\rightarrow Y\rightarrow Z$, then why is $Z\rightarrow Y\rightarrow X$ true?

in a Markov chain, given three random variables $X,Y,Z$, we have $X\rightarrow Y\rightarrow Z$, which means $p(x,y,z) = p(x)p(y|x)p(z|y)$. The right arrow symbol $\rightarrow$ is used to denote a ...
2
votes
0answers
9 views

Monte Carlo Markov Chain Simulation Issues

The Markov Chain is uniformly distributed across all $50$x$50$ matrices of entries $0$ and $$1 with no neighboring $1's$. I am supposed to run a MC simulation to check the probability that the ...
0
votes
1answer
22 views

Markov Chain (Learning)

If I have a Matrix like the one below, what is the probability $p_t$ that at a certain time $t$, we are still not able to arrive at state $z$ $$ \begin{array}{c|lcr} \text{States} & x & y ...
0
votes
0answers
26 views

Monte Carlo Markov Chain simulation

I am going to post the python code logic we used however I want someone to look at the number that are printing out. The Markov chain is uniformly distributed across all $50x50$ matrices with entries ...
3
votes
1answer
49 views

Recurrence/Transience of random walk with +2/-1 steps

Consider the Markov chain with state space $S=(0,1,2,...)$ and transition probabilities: $p(x,x+2)=p$ , $p(x,x-1)=1-p$, $\forall$ $x>0$. $p(0,2)=p$ , $p(0,0)=1-p$. For which values of $p$ is this ...
-1
votes
0answers
13 views

How can I calculate value of theta(state-probability distribution) for this markov game with one-sided information? [closed]

It's an infinite horizon markov game, where player 1 observes realization of each state and player 2 doesn't. They both observe each other's actions (typical markov game). I am trying to simulate this ...
0
votes
0answers
31 views

Random Walk, Markov Process

I'm stuck on a homework question and am wondering if anyone can offer some hints. Suppose we have some straight line graph G over the set $ V = \{1, 2, 3, ... , n\} $ of vertices, with an edge between ...
1
vote
1answer
61 views

Finite state Markov chain

Under what conditions a Markov chain can be considered as finite (and not infinite)? Thank you!
0
votes
0answers
12 views

Am I right that $P_0(t(0)=2k)=\frac{1}{k}\binom{2(k-1)}{k-1}\left(\frac{1}{2}\right)^{2k-1}$?

Consider a Markov chain $(X_n)_{n\in\mathbb{N}_0}$ with state space $E$ containing $0$ and $$ p_{00}^{(2n)}=\binom{2n}{n}\left(\frac{1}{2}\right)^{2n-1}~~~\text{ and ...
0
votes
0answers
35 views

Did I show correctly that $0$ is null recurrent or did I produce nonsense?

Suppose that you have a Markov chain with state space $E$ containing $0$. Assume that $$ p_{00}^{(2n)}=\binom{2n}{n}\left(\frac{1}{2}\right)^{2n-1}~~~\text{ and ...
1
vote
1answer
56 views

How to show positive recurrence/ null recurrence?

Suppose that you have a Markov chain with state space $E$ containing $0$. Assume that $$ p_{00}^{(2n)}=\binom{2n}{n}\left(\frac{1}{2}\right)^{2n-1}~~~\text{ and ...
0
votes
1answer
20 views

Recurrence Equation and Markov Chain: How to get the base case

I established the reccurence equation for a Markov Chain but are not able to finde the base cases. We are interested in whether the sum of $t$ throws of a fair die is divisible by $k$ for some $k ...
1
vote
2answers
25 views

Finding steady state probabilities by solving equation system

(I know that there are numerous questions on this, but my problem is in actually solving the equations, which isn't the problem in other questions.) I'm trying to figure out the steady state ...
0
votes
0answers
23 views

Likelihood of a function of different types of random variables

Is there a general way of expressing the likelihood of some known, but non-trivial function of several random varaibles. For example, suppose that we need to calculate the parameters of a process ...
0
votes
0answers
43 views

Is $0$ transient, positive recurrent or null recurrent?

Suppose that you have a Markov chain with state space $E$ containing $0$. Assume that $$ p_{00}^{(2n)}=\binom{2n}{n}\left(\frac{1}{2}\right)^{2n-1}~~~\text{ and ...
0
votes
0answers
35 views

Random walk on positive integers

Consider the random walk on the positive integers with transition probabilities $$ p_{01}=1,~~p_{i,i+1}=a,~~p_{i,i-1}=1-a\text{ for } i\geq 1 $$ for some $a\in [0,1]$. Is it ...
2
votes
0answers
25 views

The independence of random variables

Here is my question: Consider a homogeneous ergodic Markov chain on a finite state space $X=\{1,\ldots,r\} $. Define the random variables $\tau_n \,,n\ge1$ as the consecutive times when the Markov ...
0
votes
0answers
68 views

Random walk on infinite binary tree (recurrence, transience)

Consider a random walk on the infinite binary tree with root $x$ which has the following transition probabilities. $$ p_{x,0}=p_{x,1}=\frac{1}{2},~~~p_{y,y0}=p,~~~p_{y,y1}=q,~~\text{and ...
1
vote
2answers
54 views

On the definition of Markov chains

A Markov chain with discrete time dependence and stationary transition probabilities is defined as follows. Let $S$ be a countable set, $p_{ij}$ be a nonnegative number for each $i,j\in S$ and assume ...
1
vote
1answer
38 views

convergence of nullrecurrent markov chain

Hi guys! At the moment I'm working on this proof. It's in a german book so hopefully you understand everything. I understand everything in the picture without the use of the markov property at ...
1
vote
0answers
26 views

Inequality about the $L_2$ norm of stochastic matrices.

Let $P$ be a $n \times n$ stochastic matrix i.e. square, non-negative, rows sum to one. Let $\Phi$ be any given real matrix of size $n \times k$. We can assume $\Phi$ has independent columns and $k ...