Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.
0
votes
0answers
18 views
$\psi$ irreducibility and ergodicity of Markov Processes
How is Markov chain splitting technique useful for inferring ergodicity of a Markov Chain?Assume that I am working with general state space (uncountable say $R^{N}$ but time is discrete. I want to ...
0
votes
1answer
28 views
Relationship between an inhomogeneous Poisson process and Markov chain
What type of Markov process relates to an inhomogeneous Poisson process?
A homogeneous Poisson process-- one where the rate, $\lambda$, is constant-- is a pure birth continuous time Markov chain ...
0
votes
1answer
22 views
Markov jump process
Assume there is a radioactive material which emits particles according to a Poisson process at rate $\lambda$. Each particle stays alive for 1/μ time units (deterministic time).
Define ...
1
vote
0answers
20 views
expected hitting time with two absorbing states
Consider a Markov chain in a finite space and with two absorbing states, each of which is accessible from the other, transient states. Is the expected number of transitions to reach any single ...
4
votes
1answer
48 views
recurrence criterion for random-walk like Markov chain
Suppose we have a random-walk like Markov chain, i.e. state space is the set of all integers from $-\infty$ to $\infty$, the transition probability $P_{ij}$ is nonzero only when $j=i+1$ or $j=i-1$.
...
1
vote
0answers
18 views
literature to learn more on ergodic harris recurrent chains with an atom
I'm trying to learn more on the topic mentioned in the title. Namely I'd like to get more information on the behavior of the boundary terms. ie if I decompose sum of my chain (suppose it's ...
2
votes
2answers
43 views
The second largest eigenvalue for Perron-Frobenius matrix
The Perron-Frobenius theorem is about the largest eigenvalue and eigenvector of a non-negative (irreducible) matrix.
My question: Is there any estimation of the difference between the first and ...
0
votes
0answers
17 views
relation between cosine similarity matrix and markov matrix
Are all cosine similarity matrices Markov matrices?
Is every row in a cosine similarity matrix normalized?
0
votes
1answer
33 views
when would next integer $n+1$ be a prime in a given range of $p_*< p< p_*^2$?
Conjecture
that along the sequence of natural numbers $n\in\Bbb N$, if walking upwards $1,2,3,4,\ldots,n,n+1,\ldots,$ from every integer to the next (starting with $n=1$), the probability $\phi_p$ ...
0
votes
1answer
23 views
How to determine if a process can be modeled by a Markov Chain?
In a version of the popular arcade game “Whack-a-Mole”, the player stands in front
of a board with five holes in it. The (animatronic) mole pops up briefly in one of the
holes each second and the player ...
1
vote
1answer
27 views
Markov Chain Converging in Single Step
I have a markov kernel K. From this I find the invariant probability $\pi$. The question is to design a "dream" matrix K*, that converges in one step. Such that $\lambda_{SLEM}=0$ (second largest ...
-1
votes
0answers
58 views
Does the twin prime constant 0.66 equals the conditional probability upon a Markov Chain versus Euler Product?
Conjecture that
(1) along the sequence of natural numbers $n\in\Bbb N$, if walking upwards $1,2,3,4,\ldots,n,n+1,\ldots,$ from every integer to the next (starting with $n=1$), the probability ...
0
votes
0answers
33 views
A basic question on necessary and sufficient condition for positive recurrence
If state $j$ is recurrent and the following holds can it be called as positive recurrent ?
$$\lim_{n -> \infty}\frac{1}{n}\sum_{k=1}^{n}p_{jj}^{(k)} > 0$$
I know that this a necessary ...
1
vote
2answers
44 views
why is this Markov Chain aperiodic
I have this Matrix:
$$P=\begin{pmatrix} 0 & 1 \\ 0.3 & 0.7 \end{pmatrix}$$
this markov chain is said to be aperiodic, I dont understand how it comes to it. Period $\delta$ is the gcd of ...
-2
votes
1answer
84 views
infinite probability in discrete time
A Markov-chain in discrete time has the state space $I = \{1, 2, 3, 4\}$ and the transition matrix
$$P = \begin{pmatrix}
1 & 0 & 0 & 0 \\
1/4 & 3/8 & 1/8 & 1/4 \\
0 & ...
0
votes
0answers
56 views
Markov chain in discrete time
A Markov-chain in discrete time has the state space I = {1; 2; 3; 4} and the transition matrix $$p = \begin{pmatrix}
1 & 0 & 0 & 0 \\
1/4 & 3/8 & 1/8 & 1/4 \\
0 ...
1
vote
0answers
29 views
A probability problem on periodicity of Markov Chain
Assume for a Markov Chain with period $d$, $\{C_0, C_1, \dots, C_{d−1}\}$ be the equivalence classes induced by $∼$ $(i$~$j$ means all the paths from $i$ to $j$ is of length $0$ mod $d$ )and numbered ...
0
votes
1answer
46 views
Finding the probability from a markov chain with transition matrix
Consider the Markov Chain with state space $S=\{v,w,x,y,z\}$, transition matrix below:
$$\left[\begin{array}{cccccccccc}
0 & 0.4 & 0.6 & 0 & 0\\
0 & 0.5 & 0.5 & 0 & ...
0
votes
1answer
28 views
A markov chain inequality in Billingsley that should be an equation?
In the section on Markov chains in Billingsley's Probability and Measure (3e) we have the following inequality on page 120 in the proof of Theorem 8.3,
$$
\begin{align*}
p_{ji}^{(m)} &=
...
-2
votes
0answers
56 views
Proofs from Markov Chains, JR Norris. [closed]
I need a simple proof for these two theorems (from Markov Chains by JR Norris ) but I couldn't understand it from the book:
$1.$ Every recurrent class is closed
$2.$ Every finite closed class is ...
0
votes
1answer
81 views
Coin toss probability calculation
A gambler bets on coin flips. With each flip, he wins $1$ dollar with probability $p$, and loses $1$ dollar with probability $1-p$. He starts with $2$ dollar and stops when he reaches either $0$ or ...
0
votes
1answer
77 views
Equilibrium distribution of a Markov Chain
Can anyone please tell me or help me with this question shown below?
A drunken chess grandmaster dials a long string of digits on a standard telephone keypad (laid out as shown below). It takes more ...
0
votes
1answer
61 views
m/m/2 question in queueing theory
Customers arrive at a serving-system according to a Poisson process with rate 1. In the system there are two serving stations, A and B, which only take care of one customer at a time. If a customer ...
-1
votes
1answer
83 views
markov chain application
Two workers handle three machines(i.e. we can at most repair two machines at a time). The time until the machine breaks down is exponentially distributed with expectation value 1/2 and independent of ...
1
vote
1answer
84 views
Question on M/M/s queue
costumers arrive to a service station according to a poisson prossees and on average 2 during
an hour.the service times and independent of the arrivals and internally independent with mean 45 minuts ...
0
votes
1answer
23 views
Markov Chain Equilibrium Distribution Question
Suppose I have the following equilibrium probability distribution:
$\vec π = ({2\over5} , {1\over5} , {3\over20},{1\over4})$, corresponding to states 0,1,2,3, respectively.From my possible states of ...
0
votes
2answers
30 views
Markov Chain Transition Probability [duplicate]
When dealing with markov chains, say I am in state 0 on day 1, is the probability that I will be in state 0 on day 4 equal to the probability that I will be in state 0 on all of day 2, day 3 and day ...
0
votes
1answer
21 views
solving an equilibrium equation
I have the following example:
$ pa_1=a_0\\ pa_0+qa_1+pa_2=a_1\\qa_0+qa_2=a_2 $
where p+q=1.
I can see how to get $a_1=(1/p)a_0 $ but from there they say from the third equation they produce $a_2 ...
-1
votes
0answers
39 views
Markov chain and irreducible, aperiodic graph [closed]
In order for markov chain's distribution to converge to a unique stationary distribution $\pi $ no matter what the original distribution $P_0$ is, the Markov Chain graph must be irreducile and ...
1
vote
1answer
58 views
Markov Chain transitional probability query.
Say I have the transitional probability matrix P= $\begin{bmatrix}.8 & .2\\.6 & .4\end{bmatrix}$ And the entry (1,1) denotes the probability that I stay in state 0, (1,2) I move from state 0 ...
1
vote
0answers
34 views
a question about Markov chain and coupling [closed]
Anyone have an idea to solve the following question?
Suppose we have a transition matrix P
Consider a state space S = {0,1} and a Markov chain with transition probabilities P=(Pij) where P00=P01=1/2 ...
1
vote
1answer
27 views
Solving variables for the equilibrium distribution
This question is related to the equilibrium probability distribution of a markov chain. I have: $ \vec {π} * \vec {P} = \vec {π}$
$ [π_0,π_1,π_2,π_3] $ [ $\begin{matrix} 1/3 & 2/3 & 0 & ...
0
votes
0answers
22 views
Discrete Hidden Markov Model Steady State
Excuse me if 'steady state' is the incorrect term - it probably is because Google does not give me a clear answer.
For context - this is an assignment that asks that we write a HMM sampler that ...
1
vote
1answer
62 views
Random walk probability non-symmetric steps
I currently have a probability class tutorial question that I have no idea where to begin. At first instinct, I thought it may have been a CTMC question or branching question, but now I have no idea, ...
1
vote
1answer
61 views
Intuitive argument in case of a problem on Gambler's ruin
We have a gambler who at each step wins and loses $1$ dollar with probability $p$ and $1-p$ respectively. The game ends when he loses everything or wins $m$ dollar. Now starting with $i$ dollar the ...
1
vote
1answer
63 views
Discrete-time Markov chain properties
A Markov chain in discrete time is irreducible, has state space $\{0,1,\dots\}$
and starts at $1$. It is both a branching process and a martingale.
Determine the probability of hitting $0$.
4
votes
0answers
76 views
Conditional probability and integrating out part of a random walk
Suppose that I have a random walk process defined by $\alpha_{t+1}$ ~ N$(\alpha_t, \omega^2)$.
Given $\alpha_t$ and $\alpha_{t+2}$, I understand why the conditional formula for ...
0
votes
1answer
70 views
Markov Chain - Snakes and Ladders
A simple game of snakes and ladders is played on a board of nine squares. At each turn a player tosses a fair coin and advances one or two places according to whether the coin lands heads or tails. If ...
2
votes
0answers
31 views
Intuition behind criterion for an irreducible Markov chain to be transient
I have been looking over my notes for Markov chains, and I have come across the following:
Theorem: An irreducible Markov chain is transient iff for some state $i$ there exists a nonzero vector $y$ ...
0
votes
0answers
26 views
Markov proof that a state is either transitive or ergotic - can it be so simple?
This is the chart associated with a Markov matrix
The equivalence(communication) classes are:
{1,2,3,4} - transitive
{5,6,7} - transitive
{8} - ergotic
My teacher said that "all equivalence ...
1
vote
0answers
28 views
Meaning of $\pi$ in case of irreducible positive recurrent DTMC which is not aperiodic
In case of a irreducible positive recurrent DTMC which is not aperiodic, we know that there exist a positive unique probability mass function $\pi$ satisfying $\pi=\pi p$. The meaning of this can be :
...
0
votes
1answer
22 views
A basic doubt on the sojourn time of a CTMC
By using the memoryless property, I find the CDF of sojourn time of a CTMC as follows :
$$F_X(t) = 1-e^{-F_X'(0)t}$$
I am slightly confused about the intuitive meaning of the term $F_X'(0)$. How ...
0
votes
1answer
20 views
Can an absorbing CTMC be reversible?
Can a CTMC with an absorbing state be reversible? I guess not, as the product of rates through any loop cannot be equal when the loop involves the absorbing state (Kolmogorov criterion). Is my ...
1
vote
1answer
23 views
Mean number of particle present in the system: birth-death process, $E(X_t|X_0=i)$, $b_i=\frac{b}{i+1}$, $d_i=d$
Let $\{X_t\}$ be a birth–and–death process with birth rate
$$
b_i = \frac{b}{i+1},
$$
when $i$ particle are in the system, and a constant death rate
$$
d_i=d.
$$
Find the expected number of particle ...
0
votes
1answer
45 views
three-state Markov chain
a male and a female go to a 2-table restaurant on the same day.
each day the male sits at one or the other of the 2 tables, starting at the table 1, with a Markov chain transition matrix:
...
1
vote
1answer
50 views
Finding Markov chain transition matrix using mathematical induction
Let the transition matrix of a two-state Markov chain be
$$P = \begin{bmatrix}p& 1-p\\
1-p& p\end{bmatrix}$$
Questions:
a. Use mathematical induction to find $P^n$.
b. When n goes to ...
1
vote
1answer
59 views
Limit theorem of Markov chains applied to higher order Markov chains
I have a second order Markov chain with 4 states {A,T,C,G} (the 4 DNA nucleotides).
the transition matrix looks like this:
...
0
votes
1answer
31 views
Markov chain problem in Ross's Introduction to probability models
It is example 4.10 and the problem states that a pensioners receives 2 at the beginning of the each month. The amount of money he needs to spend is independent of the amount he has and is equal to $i$ ...
1
vote
0answers
41 views
Log Moment Generating function of a two-state Markov source
Let's say you have a two-state markovian source whose transition matrix is $P=\begin{pmatrix}1-\sigma & \sigma\\ \tau & 1-\tau\end{pmatrix}$, for the state 0 the data rate is 0 and for the ...
0
votes
1answer
31 views
Question on Markov chains of expected number of states
I am confused with an statement from my probability book that has to do with Markov chains. I hope someone could clarify that, if possible....Consider a Markov chain for which $P_{11}=1$ and ...
