Tagged Questions
0
votes
1answer
45 views
Markov Chain - Snakes and Ladders
A simple game of snakes and ladders is played on a board of nine squares. At each turn a player tosses a fair coin and advances one or two places according to whether the coin lands heads or tails. If ...
1
vote
0answers
22 views
Meaning of $\pi$ in case of irreducible positive recurrent DTMC which is not aperiodic
In case of a irreducible positive recurrent DTMC which is not aperiodic, we know that there exist a positive unique probability mass function $\pi$ satisfying $\pi=\pi p$. The meaning of this can be :
...
0
votes
1answer
21 views
A basic doubt on the sojourn time of a CTMC
By using the memoryless property, I find the CDF of sojourn time of a CTMC as follows :
$$F_X(t) = 1-e^{-F_X'(0)t}$$
I am slightly confused about the intuitive meaning of the term $F_X'(0)$. How ...
-1
votes
0answers
33 views
Why can't finite closed communicating classes be null recurrent?
Why can't finite closed communicating classes be null recurrent ? I want both formal proof and intuitive answer.
I know that if state $j$ is null recurrent then
$$\lim ...
0
votes
1answer
28 views
Markov chain problem in Ross's Introduction to probability models
It is example 4.10 and the problem states that a pensioners receives 2 at the beginning of the each month. The amount of money he needs to spend is independent of the amount he has and is equal to $i$ ...
1
vote
0answers
32 views
Log Moment Generating function of a two-state Markov source
Let's say you have a two-state markovian source whose transition matrix is $P=\begin{pmatrix}1-\sigma & \sigma\\ \tau & 1-\tau\end{pmatrix}$, for the state 0 the data rate is 0 and for the ...
1
vote
1answer
34 views
Hidden Markov Model Coin Toss Problem
Given two coins and transition matrix between them given by:
$\begin{bmatrix}
1-\alpha&\alpha \\
\beta&1-\beta
\end{bmatrix}$
Where the first coin has probability of heads $p$ and tails ...
0
votes
1answer
26 views
Convergence of probabilistic process
Take a random vector $V$ of length $n$ where $V_i \in [m]$. Define a random process as follows. Repeatedly pick two indices $i, j \in [n]$ uniformly at random and let $V_i := V_j$ (that is change the ...
0
votes
1answer
62 views
Random walk with absorbing barriers
Consider a random walk with absorbing barriers at $0$ and $3$. $\mathbb P(S_{n+1}-S_n=1)=0.6$ and $\mathbb P(S_{n+1}-S_n=-1)=0.4$. What is the probability of eventual absorption at $0$, given that the ...
4
votes
2answers
78 views
Prove that a random walk on $\mathbb{Z}_+\cup \{0\}$ is transient
Prove that a random walk on $\mathbb{Z}_+ \cup \{0\}$ is transient with $p_{i,i+1}=\frac{i^2+2i+1}{2i^2+2i+1}$ and $p_{i,i-1}=\frac{i^2}{2i^2+2i+1}$.
So since this Markov chain has only a single ...
0
votes
1answer
22 views
Time Periodic Homogeneous Markov Chain
I want to find a textbook or survey article reference with a treatment of discrete-time, inhomogeneous, yet time periodic, markov chains on finite state spaces.
Elaboration: I have an inhomogeneous ...
3
votes
0answers
53 views
Prove the 2 definitions of the periodicity of Markov Chain are equivalent.
In many textbooks, there are basically 2 ways of defining the periodicity of Markov Chain. One is by partitioning the graph in to subgraph such that transition in one group of state leads to the other ...
0
votes
1answer
56 views
Probability, Markov chain
A teacher leaves out a box of N stickers for children to take home as treats. Children form a queue and look at the box one by one. When a child finds $k \geqslant 1 $ stickers in the box, he or she ...
1
vote
2answers
63 views
Question about Markov chain derived from a Poisson process
Let $(N_t)$ be a Poisson process of rate $\lambda$. Define $$
X_n = N_n − n,\quad\text{for }\; n = 0, 1, 2, \ldots
$$
Explain why $(X_n)$ is a Markov chain and give its transition probabilities.
Using ...
0
votes
1answer
37 views
Calculating probabilities in genetic sequences
I am working with certain recurring sequences in genetics and try to calculate certain probabilities:
Let for instance
$$\langle g_i\rangle :=\{1,1,1,6,1,1,1,6,...,1,1,1,6\}$$ and
$$\langle ...
0
votes
0answers
96 views
Random walk, Cat and mouse
Here is the problem.
In graph G, on different vertices there is cat and mouse. Cat and mouse do independent
random walk, but time is synchronous, in one unit of time both cat and mouse do one step.
...
1
vote
2answers
55 views
How can I calculate the expected number of changes of state of a discrete-time Markov chain?
Assume we have a 2 state Markov chain with the transition matrix:
$$
\left[
\begin{array}
(p & 1-p\\
1-q & q
\end{array}
\right]
$$
and we assume that the first state is the starting state. ...
3
votes
2answers
59 views
Given that $(X_n)_{n\geq 0}$ is a Markov Chain, prove that $(X_{kn})_{n\geq 0}$ is a Markov Chain
Given that $(X_n)_{n\geq 0}$ is a Markov Chain, prove that $(X_{kn})_{n\geq 0}$ is a Markov Chain.
I don't know what this exercise has been so difficult for me, I've been playing around with the ...
1
vote
2answers
101 views
Probability of visiting state $s_1$ of a Markov chain more than $N$ times in $L$ steps.
Assume we have a two-state Markov chain, with $s_1$ and $s_0$ denoting the two states. The initial state of the Markov chain is either $s_1$ or $s_0$ with probability $p_1$ or $p_0$, respectively. The ...
1
vote
1answer
38 views
Hitting times of Markov chain/process have always finite moments?
Consider an irreducible ergodic Markov chain on a finite state space $\Omega$. Then any state is positive recurrent and this should suffice to conclude that the mean hitting time of state $s \in ...
-1
votes
1answer
44 views
Markov chains and conditional probability on subset of state space
Consider the Markov chain $(X_n)$ consisting of the three states $\{1,2,3\}$ and having transition probability matrix
$$\left (\begin{matrix} 1/3 & 2/3 & 0\\ 1/2 & 0 & 1/2 \\0 & 0 ...
1
vote
1answer
32 views
Finding the limiting distribution of a $3 \times 3$ Markov chain
This is a question from a book.
Find $\lim_{n\rightarrow \theta}P^n$ where $$P=\begin{pmatrix}0 & 1 & 0\\
\frac{1}{6} & \frac{1}{2} & \frac{1}{3}\\
0 & \frac{2}{3} & ...
3
votes
1answer
65 views
Probability with Markov chains
I need some hint about Markov chains. So here is my homework.
Let $\{ X_t : t = 0,1, 2, 3, \ldots, n\}? $ be a Markov chain. What is $P(X_0 =i\mid X_n=j)$?
So I need to calculate if it's $j$ ...
0
votes
1answer
73 views
Time to absorption and fraction of time spent in a state in a CTMC
I consider a Markov chain with a single absorbing state with $N$ transient states and I would like to find the expected time to absorption, given an initial state. I write the following equation to ...
3
votes
1answer
40 views
Intuition on Harris recurrence
I am trying to get some intuition on Harris recurrence in Markov chains. Define state space $\mathcal S$ comprising a single communication class, $f_{ii}^{(n)}=P(X_n=i, X_{n-1}\ne i,\ldots X_1\ne ...
3
votes
2answers
60 views
What's the probability that A wins finally
Suppose A has \$2 and B has \$3. They play a game, each game gives the winner \$1 from other. A has a probability $\frac{3}{5}$ to win each game. They play this game until one of them is bankrupt. ...
2
votes
0answers
81 views
Markov chain from Poisson
Let $K_t$ be a Poisson process with rate $1$ and $X_n=K_n-n$ $, \ \ \ n\in \mathbb{N}$
am asked to determine whether it is null or positive recurrent, we already know it is recurrent.
I ...
2
votes
2answers
75 views
$(X_n)$ an irreducible transient Markov chain. Is $f(x) = \mathbb{P}(X_n = x_0 \text{ for some } n > 0 | X_0=x)$ constant?
Let $(X_n)_{n=0}^{\infty}$ be an irreducible transient Markov chain with countably infinite state space $E$. Let $T_x = \inf\{n > 0 : X_n = x\}$. Let $\mathbb{P}_x$ be probability conditioned on ...
2
votes
0answers
120 views
Boundedness of expected reward Markov chain (may be related to discret $M/M/\infty$ queue)
[EDIT]:
I read a bit on $M/M/\infty$ queue and it may not be the right comparison and my notation may be confusing (I'm in discrete time and $\lambda,\mu$ look likes rates when they are probability). ...
0
votes
1answer
39 views
How to show that a stochastic process is Markov
How can I prove that a given stochastic process is a Markov chain.
Assume the following process:
Joey is walking in the woods. at every turn:
if at the previous turn Joey turned left then he will ...
4
votes
1answer
99 views
Chance of being able to quit while ahead in a betting game (Markov chain with gambler's ruin)
Suppose a player starts with $N$ chips, and is playing a game with odds $O$, betting 1 chip in each iteration. When the player reaches 0 chips the betting must end.
What is the probability that at ...
1
vote
2answers
59 views
Simple Symmetric Random Walk : $P_{00}^{2n}=\binom{2n}{n}\left(\dfrac{1}{2}\right)^{2n}$
I was studying Simple Symmetric Random Walks and my notes state (without proof) that $$P_{00}^{2n}=\binom{2n}{n}\left(\dfrac{1}{2}\right)^{2n}$$
That is the probability of going from $0$ to $0$ in ...
6
votes
1answer
99 views
Markov chain stochastic process
Can anyone help me with this question, maybe by giving a hint.
Consider a Markov chain with state space $\{0,1,2....\}$. A sequence of positive numbers $p_1,p_2,...$ is given with $\sum p_i=1$. ...
1
vote
2answers
136 views
How to create a transition matrix that will guarantee an outcome after infinite transitions
Let's assume we have the a transition matrix like:
0 0 0
1 2 0
2 4 0
3 6 0
4 7 2
5 9 3
6 6 6
7 7 7
8 8 8
9 9 9
First ...
1
vote
0answers
49 views
Finding probablity from a markov chain
If I have a markov chain transition matrix for 2 states.
Specifically in my case, it is a transition matrix for a bacterial genome with 4 random variables being A,C,G and T. (The bases)
If I want to ...
0
votes
3answers
230 views
Finding the transition probability matrix, two switches either on or off..
Each of two switches is either on or off during a day. On day n, each switch will independently be on with probability
[1+number of on switches during day n-1]/4
For instance, if both switches are on ...
0
votes
1answer
57 views
Mean Duration of Stochastic/Markov Game
An urn contains five red and three green balls. The balls are chosen at random, one by one, from the urn. If a red ball is chosen, it is removed. Any green ball that is chosen is returned to the urn. ...
0
votes
1answer
100 views
Markov Chains Probability
A Markov chain $X_0$, $X_1$, $X_2$, ... has the transition probability matrix
$$
P = \left[
\matrix
{
0.3&0.2&0.5 \\
0.5&0.1&0.4 \\
0&0&1
}
\right]
$$
and is known to ...
1
vote
1answer
203 views
Irreducible, finite Markov chains are positive recurrent
I am under the impression that an irreducible, finite Markov chain is necessarily positive recurrent. How might I show this?
Regards,
Jon
1
vote
1answer
79 views
Classify the states of a markov chain
a)
P =$\begin{bmatrix}
{1-2p} & 2p & {0} \cr
{p} & {1-2p} & {p} \cr
{0} & 2p & {1-2p} \cr
\end{bmatrix}$
b)
P = $\begin{bmatrix}
0 & p & 0 & 1-p \cr
1-p & 0 ...
1
vote
1answer
113 views
Markov Chain Transition Intensity Conversion
I have a question about converting a 3-state discrete state, continuous-time, markov chain to a 2-state.
My 3-state model has states: Well (state 1), Ill (state 2) and Dead (state 3).
...
2
votes
2answers
50 views
Convergence of an Ergodic process
I'm having trouble working through the math of the problem below. I believe the problem as described is an ergodic process. I've written a simple simulation of the problem, that converges to 66.6...6% ...
0
votes
1answer
47 views
What does $P^3(D,D)$ stand for?
We're on Markov and we're considering the Markov chain on $\{A,B,C,D,E\}$ with a transition matrix $P$.
I am asked to find $P^3(D,D)$ but I am unfamiliar with what the notation stands for.
Thanks
2
votes
0answers
18 views
Ruin time with a maximum purse size
Imagine I have a gambler's ruin scenario where I start with $m$ dollars and I cannot have more than $N$ dollars. For each of however many rounds, I flip a coin, and with probability $p$ I win a ...
1
vote
2answers
87 views
Game with losing and winning a dollar
I found an interesting problem in my book:
There is a game where player starts with $k\$$. In each step he wins or loses $1\$$ (both with probability $p=\frac{1}{2}$). The game ends when player ...
0
votes
0answers
140 views
Hidden Markov Chain- How to use Viterbi/Forward-Backward algorithm to predict?
I am trying to understand the above algorithm in order to implement very basic stock price prediction logic.
I found this example on wikipedia describing the algorithm (I am under the impression the ...
0
votes
0answers
72 views
A conceptual doubt on steady state probability of a DTMC
Consider a DTMC with state space $\{1, 2,\ldots {}{} \}$. Now we want to calculate the probability that state 1 is followed by state 2 in the long run i.e, $P(X_n=1, X_{n+1}=2)$ as $n$ tends to ...
1
vote
2answers
72 views
Using random walks to predict behavior rather than matrix decomposition
I want to create a model that tries to predict a user's behavior based on the random walks of similar users. The problem is similar to Netflix's recommendation challenge. One of the popular solutions ...
1
vote
0answers
94 views
a problem on DTMC
For a Markov chain $\{X_n, n\ge0\}$ with transition probabilities $P_{i,j}$, consider the conditional probability that $X_n = m$ given that the chain started at time $0$ in state $i$ and has not ...
1
vote
2answers
96 views
What's the probability of a gambler losing \$10 in this dice game? What about making \$5? Is there a third possibility?
Can you please help me with this question:
In a gambling game, each turn a player throws 2 fair dice. If the sum of numbers on the dice is 2 or 7, the player wins a dollar. If the sum is 3 or 8, ...
