Tagged Questions

Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
13 views

Is this stochastic process a Markov chain?

I have been struggling sometime now with the following question and I feel like I am stacked. Let Xn, n= 0,1,... be a sequence of iid discrete random variables with P(Xn=j)=aj>0, j=0,1,2... Is ...
0
votes
0answers
4 views

Is there a general formula for determining this distribution in a Markov chain?

Let $C$ be an irreducible Markov chain with state set $S$, $\left| S\right| = n$, transition matrix $T$, starting at state $s_0 \in S$, and yielding the states $s_0, s_1, s_2, \ldots$ during a random ...
2
votes
1answer
30 views

A Markov Chain Flea Problem

A Flea moves around the vertices of a triangle in the following manner: Whenever it is at vertex i it moves to its clockwise neighbor vertex with probability $p_i$ and to the counterclockwise neighbor ...
0
votes
0answers
11 views

variance of number of steps in markov chain (rook move to top right)

I encountered this problem while studying Markov chains and I want to calculate the variance of the problem, i.e. variance of number of steps that a random walker rook make to reach from down-left ...
0
votes
1answer
39 views

Gambler's ruin and Markov Chain, coin toss and stakes

I'm considering a classical problem about Markov Chains: A gambler has $£8$ and wishes to get to $£10$. A coin is tossed repeatedly : if it comes down tails, the gambler loses his stake, and if it ...
4
votes
1answer
23 views

A die whose score cannot be as before (Markov chains)

A die is "fixed" so that each time it is rolled the score cannot be the same as the preceding score, all other scores having probability $1/5$. Given that the first score is 6, what is the probability ...
2
votes
2answers
20 views

Markov Chain: Moving on a circle

A particle moves on 12 points situated on a circle. At each step it is equally likely to move one step in the clockwise or in the counterclockwise direction. Find the mean number of steps for ...
0
votes
1answer
8 views

Expected number of visits to a state of a Markov chain up to a certain time

Let $P=\{p_{ij}\}$ be a stochastic matrix (with rows and columns indexed by a countable set) and let $p^{(k)}_{ij}$ be the entries of $P^k$. I'm trying to prove that, if the associated Markov chain is ...
-3
votes
0answers
22 views

show is markov chain [on hold]

suppose: that : X=({X}{n}){n\geq 0}: is: M.C(\lambda ,P): y : f:IxI\rightarrow I a function. denote by ${f}^{-1}(j):={i\in I:f(i)=j}\: \: y \:$ suppose for all $i,j \in I$ such that $ f(i)=f(j)$ ...
0
votes
1answer
23 views

Understanding steady state distribution

I need some help verifying that my understanding of steady state distribution is indeed correct. I have a transition diagram (model). With around 100 states and 6 variables. I have used a software ...
0
votes
0answers
22 views

Using long term Markov Chains in Excel to compute time spent in state

I'm trying to calculate the uptime of a computer state in Excel. I'm in way over my head, and it took me over a day to identify my problem as a Markov Chain. The question is: "X pellets per second ...
0
votes
1answer
14 views

Markov Chains (State transitions)

I was wondering which part I am misunderstanding about the individual-by-individual updating scheme from the book of Jackson M. (Social and Economic Networks, 2008) . The full transition matrix in the ...
-1
votes
0answers
13 views

Finding the infinitesimal generator of a M/M/2 queue [on hold]

I have a M/M/2 queue with a total population of 5. The arrival times are independent exponential random variables with mean of $\lambda$ and the service times have a mean of $\mu$. The initial number ...
-1
votes
0answers
19 views

Setting up and Solving Kolmogorov Forward Equations

Consider a Markov Chain with $3\times 3$ generator matrix: $$ G = \begin{bmatrix} -1 & 1/2 & 1/2 \\ 1/2 & -1 & 1/2 \\ 1/2 & 1/2 & -1 \end{bmatrix} $$ What are the ...
1
vote
1answer
29 views

Given a Markov chain $X \rightarrow Y \rightarrow Z$, why is $I(X;Y|Z) \leq I(X;Y)$?

A Markov chain $X \rightarrow Y \rightarrow Z$ is given, where $X,Y,Z$ are random variables characterized by the probability distribution $p(x,y,z) = p(x)p(y|x)p(z|y)$. It follows that $I(X;Y) \geq ...
0
votes
1answer
28 views

Markov chain: if $X\rightarrow Y\rightarrow Z$, then why is $Z\rightarrow Y\rightarrow X$ true?

in a Markov chain, given three random variables $X,Y,Z$, we have $X\rightarrow Y\rightarrow Z$, which means $p(x,y,z) = p(x)p(y|x)p(z|y)$. The right arrow symbol $\rightarrow$ is used to denote a ...
2
votes
0answers
9 views

Monte Carlo Markov Chain Simulation Issues

The Markov Chain is uniformly distributed across all $50$x$50$ matrices of entries $0$ and $$1 with no neighboring $1's$. I am supposed to run a MC simulation to check the probability that the ...
0
votes
1answer
22 views

Markov Chain (Learning)

If I have a Matrix like the one below, what is the probability $p_t$ that at a certain time $t$, we are still not able to arrive at state $z$ $$ \begin{array}{c|lcr} \text{States} & x & y ...
0
votes
0answers
25 views

Monte Carlo Markov Chain simulation

I am going to post the python code logic we used however I want someone to look at the number that are printing out. The Markov chain is uniformly distributed across all $50x50$ matrices with entries ...
3
votes
1answer
48 views

Recurrence/Transience of random walk with +2/-1 steps

Consider the Markov chain with state space $S=(0,1,2,...)$ and transition probabilities: $p(x,x+2)=p$ , $p(x,x-1)=1-p$, $\forall$ $x>0$. $p(0,2)=p$ , $p(0,0)=1-p$. For which values of $p$ is this ...
-1
votes
0answers
13 views

How can I calculate value of theta(state-probability distribution) for this markov game with one-sided information? [closed]

It's an infinite horizon markov game, where player 1 observes realization of each state and player 2 doesn't. They both observe each other's actions (typical markov game). I am trying to simulate this ...
0
votes
0answers
29 views

Random Walk, Markov Process

I'm stuck on a homework question and am wondering if anyone can offer some hints. Suppose we have some straight line graph G over the set $ V = \{1, 2, 3, ... , n\} $ of vertices, with an edge between ...
1
vote
1answer
55 views

Finite state Markov chain

Under what conditions a Markov chain can be considered as finite (and not infinite)? Thank you!
0
votes
0answers
17 views

How to show that a finite closed communicating class is positive recurrent? [duplicate]

Let $C$ be a finite and closed class. Show that $C$ is positive recurrent. All I know yet is that as a finite and closed class, C is recurrent. That is, every state $i\in C$ is recurrent. ...
0
votes
0answers
12 views

Am I right that $P_0(t(0)=2k)=\frac{1}{k}\binom{2(k-1)}{k-1}\left(\frac{1}{2}\right)^{2k-1}$?

Consider a Markov chain $(X_n)_{n\in\mathbb{N}_0}$ with state space $E$ containing $0$ and $$ p_{00}^{(2n)}=\binom{2n}{n}\left(\frac{1}{2}\right)^{2n-1}~~~\text{ and ...
0
votes
0answers
32 views

Did I show correctly that $0$ is null recurrent or did I produce nonsense?

Suppose that you have a Markov chain with state space $E$ containing $0$. Assume that $$ p_{00}^{(2n)}=\binom{2n}{n}\left(\frac{1}{2}\right)^{2n-1}~~~\text{ and ...
1
vote
1answer
56 views

How to show positive recurrence/ null recurrence?

Suppose that you have a Markov chain with state space $E$ containing $0$. Assume that $$ p_{00}^{(2n)}=\binom{2n}{n}\left(\frac{1}{2}\right)^{2n-1}~~~\text{ and ...
0
votes
1answer
18 views

Recurrence Equation and Markov Chain: How to get the base case

I established the reccurence equation for a Markov Chain but are not able to finde the base cases. We are interested in whether the sum of $t$ throws of a fair die is divisible by $k$ for some $k ...
1
vote
2answers
22 views

Finding steady state probabilities by solving equation system

(I know that there are numerous questions on this, but my problem is in actually solving the equations, which isn't the problem in other questions.) I'm trying to figure out the steady state ...
0
votes
0answers
23 views

Likelihood of a function of different types of random variables

Is there a general way of expressing the likelihood of some known, but non-trivial function of several random varaibles. For example, suppose that we need to calculate the parameters of a process ...
0
votes
0answers
43 views

Is $0$ transient, positive recurrent or null recurrent?

Suppose that you have a Markov chain with state space $E$ containing $0$. Assume that $$ p_{00}^{(2n)}=\binom{2n}{n}\left(\frac{1}{2}\right)^{2n-1}~~~\text{ and ...
0
votes
0answers
33 views

Random walk on positive integers

Consider the random walk on the positive integers with transition probabilities $$ p_{01}=1,~~p_{i,i+1}=a,~~p_{i,i-1}=1-a\text{ for } i\geq 1 $$ for some $a\in [0,1]$. Is it ...
2
votes
0answers
24 views

The independence of random variables

Here is my question: Consider a homogeneous ergodic Markov chain on a finite state space $X=\{1,\ldots,r\} $. Define the random variables $\tau_n \,,n\ge1$ as the consecutive times when the Markov ...
0
votes
0answers
63 views

Random walk on infinite binary tree (recurrence, transience)

Consider a random walk on the infinite binary tree with root $x$ which has the following transition probabilities. $$ p_{x,0}=p_{x,1}=\frac{1}{2},~~~p_{y,y0}=p,~~~p_{y,y1}=q,~~\text{and ...
1
vote
2answers
54 views

On the definition of Markov chains

A Markov chain with discrete time dependence and stationary transition probabilities is defined as follows. Let $S$ be a countable set, $p_{ij}$ be a nonnegative number for each $i,j\in S$ and assume ...
1
vote
1answer
38 views

convergence of nullrecurrent markov chain

Hi guys! At the moment I'm working on this proof. It's in a german book so hopefully you understand everything. I understand everything in the picture without the use of the markov property at ...
1
vote
0answers
24 views

Inequality about the $L_2$ norm of stochastic matrices.

Let $P$ be a $n \times n$ stochastic matrix i.e. square, non-negative, rows sum to one. Let $\Phi$ be any given real matrix of size $n \times k$. We can assume $\Phi$ has independent columns and $k ...
-1
votes
0answers
31 views

Are $T_4$ and $T_5$ stopping times?

Let $(\xi_n)_{n\in\mathbb{N}_0}$ be a sequence of independent identically distributed random variables that take values in $\left\{-1,1\right\}$ with equal probabilities. Define ...
1
vote
2answers
40 views

Is this a stopping time or not?

Let $(\xi_n)_{n\in\mathbb{N}_0}$ be a sequence of independent identically distributed random variables that take values in $\left\{-1,1\right\}$ with equal probabilities. Define ...
-1
votes
0answers
18 views

Given initial state and steady state, how do I find the transition matrix?

The initial vector is $x_0 = \{.5, .5\}$ and the steady state is $x_\infty = \{1/9, 8/9\}$. How do I get the transition matrix, such that $$\lim_{p\to\infty} x_0 A^p = x_\infty $$ where $A$ is a ...
0
votes
0answers
38 views

Show that every finite closed class is positive recurrent

Let $C$ be a finite closed class. Prove or disprove that $C$ is positive recurrent. Note 1: In our lecture we proved that every finite closed class is recurrent. Note 2: (Positive) recurrence is ...
0
votes
0answers
8 views

Recurrence time of persistent state in a markov chain

Let a Markov chain contains a states and let $E_j$ be persistent. There exists a number $q < 1$ such that for $n \ge a$ the probability of the recurrence time of $E_j$ exceeding $n$ is smaller ...
0
votes
1answer
21 views

Is it possible to compute these probabilities concerning a 6-digit password using theory of Markov chains? [duplicate]

Consider the situation of decoding a 6-digit password that consists of the symbols A to Z and 0 to 9, where all possible combinations are tried randomly and uniformly. Consider the ...
1
vote
1answer
19 views

How to transform a process into a Markov Chain?

This problem is in the book Introduction to Probability. The question goes this way. Consider the process {$ X_n, n = 0,1,...$ } with values 0,1 or 2. If P{$X_{n+1} = j | X_n = i, X_{n-1} = ...
0
votes
1answer
24 views

Computer failure with Markov chains and n-step transition matrix

Hi I am struggling with a Markov Chain question: A computer network has two servers, only one of which is in operation at any given time. A server may break down on any given day with probability p. ...
4
votes
0answers
113 views

6-digit password - a special decoding method

Consider the situation of decoding a 6-digit password that consists of the symbols A to Z and 0 to 9, where all possible combinations are tried randomly and uniformly. Consider the ...
1
vote
1answer
73 views

Counterexample for $\| (\Phi^\top \Xi \Phi)^\dagger \Phi^\top \Xi P \Phi \|_2 \leq 1$

All matrices are real. Let $P$ be any stochastic matrix (i.e. square, non-negative, rows summing to one). Let $\Xi$ be a diagonal matrix containing on its diagonal any left eigenvector of $P$ ...
0
votes
0answers
8 views

Decomposition of a communicating class with countably infinite state space

Do you know an easy example of a Markov chain with countably infinite state space $E$, that has a communicating class $C$, that can be decomposed into a disjoint union of sets ...
1
vote
0answers
37 views

Find condition on $X$ so that $P(\exists n\in\mathbb{N}: N_n=0)=1$

Let $X, X_{n,k}$ for $k,n\in\mathbb{N}$ denote independent random variables with values in $\mathbb{N}_0$. $X$ is identically distributed as all $X_{n,k}$. Define $N_0:=1$ and for ...
0
votes
0answers
21 views

Transition probabilities do not sum to $1$

I have a set of weekly probabilities, and in order to convert to monthly probabilities, I have firstly convert the weekly probabilities into rates, $$r = -\frac{\ln (1-p)}{ t} ,\quad t=1\text{ ...