1
vote
0answers
22 views

Conditional return time of simple random walk

Consider a simple random walk on $\mathbb{Z}$, $(S_t)_{t \geq 0}$, with $S_0 = 0$. The probability to jump to the right neighbour is $p \geq \frac{1}{2}$. Call $\tau_k = \min\{t \in \mathbb{N}\, : \, ...
0
votes
0answers
26 views

Exact probability distribution for hitting time of simple random walk

Consider simple random walk on the line starting from the site $y \in \mathbb{N}$. With probability $p$ the walker moves to the right and with probability $1-p$ to the left. Call $\tau$ the first time ...
0
votes
0answers
21 views

Transition matrix in left-right hidden semi-Markov model

I'm developing a hidden semi-Markov model left-right . In a left-right model a sequence of $M$ states starts in state $1$ and ends in state $M$, with no repetition of states. Since the model is ...
1
vote
0answers
27 views

A Central Limit Theorem to Markov Chains

I am looking for some textbook or paper that treats this question: Let be $X_{1}, X_{2}, \ldots$ the random variables from a Markov Chain (MC). Is there any Central Limit Theorem (CLT) envolving ...
0
votes
0answers
15 views

How to use symmetry of transition rate matrix in a continuous-time Markov chain?

This is part of a bigger question, so I have to change the question a bit to focus on the point. We have a continuous- time Markov chain with the following transition rate matrix: $$Q= \begin{pmatrix} ...
5
votes
3answers
144 views

Expected value of number of draws

We have $5$ number in a bag: $(1,3,5,7,9)$. We draw one from the bag and then put it back. We do this until the sum of the numbers can be divided by $3$. Whats the expected value of the number of ...
0
votes
0answers
15 views

Policy Adjustment in Markov Decision Process

I was using MDP on my work to make optimal decision. I used discrete time, finite state MDP. I assumed that I will have an initial parameters, like the Reward/Cost, state transition probabilities and ...
1
vote
0answers
49 views

How's the damping factor in Google PageRank algorithm calculated

I'm doing some researches about Google's PageRank algorithm for my thesis, I've found that the damping factor x (for example), where x is in : P` = x.P + (1-x)Q where P is the original ...
1
vote
0answers
22 views

How to get from this probability formula to the one I need?

I'm working on a gambler's ruin problem where a player starts out with $i$ money, and 'winning' is when their total money reaches $N$ (ie they will keep playing until they reach N or run out of money, ...
2
votes
2answers
137 views

Professor has 4 umbrellas, Markov chain and Probability

OK this problem is making me tear my hair out. I need someone to walk me through this in baby-steps method like 1 + 1 = 2. I am trying to figure out what I don't understand. I know this is going to be ...
2
votes
1answer
35 views

On track Prerequisite for Statistics and Probability

I do not really have a solid mathematical background because of the range of courses i had back in high school/university that wasn't really scientific oriented. Presently i am doing an MSc in ...
0
votes
0answers
27 views

Convergence of sequence of stationary distributions of Markov chains

I have a sequence of finite, discrete-time ergodic Markov chains indexed by a parameter $N$, and I want to prove that their stationary distributions are converging to a well-defined limit as $N\to ...
0
votes
0answers
64 views

Probability distribution after n-steps with different initiation state in Markov chain

The transition matrix at n-th time step for a discrete time Markov chain with $ S = \{1, 2, 3, 4\} $is given as below: $$ P(n) = \pmatrix{0 & 0.6 & 0.4 & 0 \\ 0.8 & 0 & 0 & ...
3
votes
1answer
68 views

Random walk where increment depend on current position

Consider the following stochastic process, $$b(i+1) = b(i) + \xi_i (b_i),$$ where $\xi_i(b_i) \in \{-1, k \}$ are the independent increments having the following distribution: $$\begin{align} P (\xi ...
4
votes
1answer
55 views

Joint Probability from Marginal Probabilities

$X, Y_1, Y_2$ are random variables with (possibly) different finite alphabets. For given conditional probability mass functions $\mathbb{P}(Y_1|X)$ and $\mathbb{P}(Y_2|X)$, is it always possible to ...
5
votes
1answer
114 views

Log-likelihood function

I'm not sure if this could be asked here, or in math overflow... In the following paper Cho, Jin Seo, and Halbert White. "Testing for regime switching." Econometrica 75.6 (2007): 1671-1720. doi: ...
0
votes
1answer
190 views

Expectation problem in Absorbing Markov Chain(exercise on Grinstead and Snell 11.2 18 )

Hi I encountered this problem. It took me quite long but I could not solve it. The problem is as follows: Assume that a student going to a recently established school in a university has, each year, ...
1
vote
1answer
36 views

Prove that if $a_k=x_{k1}a_{k-1}+x_{k2}a_{k-2}+\cdots+x_{kt}a_{k-t},$ ($x_{ki}$~U(0,b)) then $\dfrac{\log{a_k}}{k}\to^{p} c$

Let $a_1=a_2=\cdots=a_t= 1,a_k=x_{k1}a_{k-1}+x_{k2}a_{k-2}+\cdots+x_{kt}a_{k-t},$ where $x_{ki}$~U(0,b), and $x_{ki}(k>t,i=1,2,\cdots,t)$ are independent each other. Prove that $\exists c\in ...
1
vote
1answer
202 views

Question on M/M/s queue

costumers arrive to a service station according to a poisson prossees and on average 2 during an hour.the service times and independent of the arrivals and internally independent with mean 45 minuts ...
0
votes
1answer
867 views

Markov Chain - Snakes and Ladders

A simple game of snakes and ladders is played on a board of nine squares. At each turn a player tosses a fair coin and advances one or two places according to whether the coin lands heads or tails. If ...
1
vote
0answers
37 views

Estimate on Galton-Watson process distribution

Let $(Zn)_{n\in \mathbb N_0}$ be a Galton-Watson process, i.e. $$ Z_{n+1} = \sum_{k=1}^{Z_n}\xi_{n,k},\qquad (\xi_{n,k})_{n\in \mathbb N_0,k \in \mathbb N} \quad \text{i.i.d } \mathbb N_0 \text{ ...
-1
votes
1answer
144 views

Markov Chain Stationary distribution

I constructed a 4*4 state transition matrix from a discrete-time Markov Chain Model as follows: A=[p0 p0 p0 p0; p*p0+(1-p) p0 p0 p0; p0 p*p0+(1-p) p0 p0; p0 p0 p*p0+(1-p) ...
0
votes
1answer
68 views

Stationary Distribution of T

I'm trying to find the stationary distribution of T, a transition matrix (Markov Chain). After I solve the equations of the matrix, I can't get to their values, does that mean that T doesn't have a ...
0
votes
1answer
112 views

Reversibility of Markov Process and Exponential Distribution of Transition Rates

I am reading the paper Towards Utility-optimal Random Access Without Message Passing by J. Liu, Y. Yi, A. Proutiere, M. Chiang, H. V. Poor. A sentence in Section 3.2 can be paraphrased as follows: ...
1
vote
1answer
103 views

Why does this probability equivalence of events hold?

$P(X_0 = j, X_m \ne j, 1 \le m \le n-1) = P(X_m \ne j, 1 \le m \le n-1) - P(X_m \ne j, 0 \le m \le n-1) $ Where $\{X_n\}$ is an irreducible Markov Chain with a finite state space.
3
votes
1answer
1k views

coin flips and markov chain

Consider the case of an infinite (or finite $n$) string of coin tosses, and let $q$ and $1-q$ be the probabilities that the coin comes up tails and heads, respectively. (For simplicity, we can take ...