Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.
3
votes
2answers
145 views
Markov chain stationary probability simulation
Having a defined markov chain with a known transition matrix, rather than to calculate the steady state probabilities, I would like to simulate and estimate them.
Firstly, from my understanding there ...
2
votes
2answers
217 views
probability terminology for parameter in a Markov process
Suppose $$P(\text{feature present at time} \ t \ \text{and} \ t+\Delta t) = \beta^{2}+\beta(1-\beta) \exp(\Delta t/\tau)$$
where $\tau = 1/(\pi_{01}+\pi_{10})$. What is $\tau$?
1
vote
2answers
101 views
Probability of visiting state $s_1$ of a Markov chain more than $N$ times in $L$ steps.
Assume we have a two-state Markov chain, with $s_1$ and $s_0$ denoting the two states. The initial state of the Markov chain is either $s_1$ or $s_0$ with probability $p_1$ or $p_0$, respectively. The ...
1
vote
2answers
72 views
Using random walks to predict behavior rather than matrix decomposition
I want to create a model that tries to predict a user's behavior based on the random walks of similar users. The problem is similar to Netflix's recommendation challenge. One of the popular solutions ...
0
votes
2answers
94 views
Markov chain, Q matrix, jump matrix and invariant distribution
For the following Q matrix i want to find the jump matrix and the invariant distribution.
\[
Q= \begin{pmatrix}
-2 &1 &1 &0\\
2 & -4 &1 &1\\
1 &0 &-1 ...
6
votes
1answer
164 views
What happens to a random walk when we increase the probabilities of going right?
Consider a random walk on the integers where the probability of transitioning from $n$ to $n+1$ is $p_n$ (and of course, the probability of transitioning from $n$ to $n-1$ is $1-p_n$); we assume all ...
4
votes
1answer
76 views
Behavior of explosive random process
Inspired somewhat by this problem, I've been investigating the behavior under iteration of the following discrete random process:
Given $n\in\mathbb{N}$, choose an integer from $\{0,1,\ldots,n\}$ ...
3
votes
1answer
41 views
Intuition on Harris recurrence
I am trying to get some intuition on Harris recurrence in Markov chains. Define state space $\mathcal S$ comprising a single communication class, $f_{ii}^{(n)}=P(X_n=i, X_{n-1}\ne i,\ldots X_1\ne ...
2
votes
1answer
50 views
Random Process derived from Markov process
I have a query on a Random process derived from Markov process. I have stuck in this problem for more than 2 weeks.
Let $r(t)$ be a finite-state Markov jump process described by
...
2
votes
1answer
44 views
Functions of a Markov Chains
can does anybody know if the following expectations are available in closed for...
Let $\{ X_t : t = 1, 2, 3 \dots \}$ be a random variable defined on a Markov chain with m -step transition matrix ...
2
votes
1answer
89 views
Solving Discrete Markov Chain with diagonal band matrix.
I am trying to model a certain process as a Discrete Markov Chain.
My system has $N+1$ states: $X=0, \ldots N$, and I can assume that the $(N+1)\times (N+1)$ transition matrix $T$ has the general form
...
2
votes
1answer
166 views
Comparing two Markov chains
I am interested in the question of the positive recurrence of a Markov chain that (in some sense) converges to another Markov chain known to be positive recurrent. The following is a concrete example ...
1
vote
1answer
18 views
Discrete-time Markov chain properties
question :a markov chain in discrite time is irreducible ,has state space i={0,1,...}
and start in 1.it is both a branching process and a martingale .
determine the probability of hitting 0.
1
vote
1answer
37 views
Hidden Markov Model Coin Toss Problem
Given two coins and transition matrix between them given by:
$\begin{bmatrix}
1-\alpha&\alpha \\
\beta&1-\beta
\end{bmatrix}$
Where the first coin has probability of heads $p$ and tails ...
1
vote
1answer
37 views
How can I count clockwise vs counterclockwise cycles in a series of (modular) integers?
I'm working with a Markov process that has a state and transition diagram like in the picture:
The transition probabilities are not listed, but they are always positive, but not necessarily ...
1
vote
1answer
115 views
Markov Chain Transition Intensity Conversion
I have a question about converting a 3-state discrete state, continuous-time, markov chain to a 2-state.
My 3-state model has states: Well (state 1), Ill (state 2) and Dead (state 3).
...
1
vote
1answer
81 views
Markov Chains - How to calculate prob. a state is visited at least N times? what about Expectation?
In Markov chains, if I was given a transition probability matrix with each of the probabilities specified, then how do I determine the following:
1- Probability that state y is visited at least n ...
1
vote
1answer
203 views
Markov Chains as Autoregressive Processes
Is there a simple way to approximate a Markov Chain as an Autoregressive Process, for instance, an AR(1) process?
I am aware that it is easy to approximate an AR(1) process with a Markov Chain, but I ...
0
votes
1answer
50 views
Markov Chain - Snakes and Ladders
A simple game of snakes and ladders is played on a board of nine squares. At each turn a player tosses a fair coin and advances one or two places according to whether the coin lands heads or tails. If ...
0
votes
1answer
20 views
Can an absorbing CTMC be reversible?
Can a CTMC with an absorbing state be reversible? I guess not, as the product of rates through any loop cannot be equal when the loop involves the absorbing state (Kolmogorov criterion). Is my ...
0
votes
1answer
38 views
three-state Markov chain
a male and a female go to a 2-table restaurant on the same day.
each day the male sits at one or the other of the 2 tables, starting at the table 1, with a Markov chain transition matrix:
...
0
votes
1answer
22 views
Time Periodic Homogeneous Markov Chain
I want to find a textbook or survey article reference with a treatment of discrete-time, inhomogeneous, yet time periodic, markov chains on finite state spaces.
Elaboration: I have an inhomogeneous ...
0
votes
1answer
42 views
Identity in Markov Processes
I want to know if my reasoning here is correct, it seems simple enough but I just want clarification (I am considering the proof that if a Markov process satisfies the detailed balance condition, then ...
0
votes
1answer
56 views
Probability, Markov chain
A teacher leaves out a box of N stickers for children to take home as treats. Children form a queue and look at the box one by one. When a child finds $k \geqslant 1 $ stickers in the box, he or she ...
0
votes
1answer
38 views
Calculating probabilities in genetic sequences
I am working with certain recurring sequences in genetics and try to calculate certain probabilities:
Let for instance
$$\langle g_i\rangle :=\{1,1,1,6,1,1,1,6,...,1,1,1,6\}$$ and
$$\langle ...
0
votes
1answer
58 views
Example of continuous transient Markov chain in detailed balance?
I have been thinking of such a chain but I've found none. I thought about random walk on $\mathbb{N}$ with probability p to go to right and $q=1-p$ to go back(i.e. this is the transition probabilities ...
0
votes
1answer
71 views
taxicab in a city with two zones
We recently saw a problem here about a taxicab driver who operates in a city with two zones $A$ and $B$, taking trips that may or may not take him to the other zone. The probabilities are such that if ...
-1
votes
1answer
72 views
Markov Chain Stationary distribution
I constructed a 4*4 state transition matrix from a discrete-time Markov Chain Model as follows:
A=[p0 p0 p0 p0;
p*p0+(1-p) p0 p0 p0;
p0 p*p0+(1-p) p0 p0;
p0 p0 p*p0+(1-p) ...
-1
votes
1answer
143 views
Markov n-step State Transition
I've understood the concept and application of the "recursive n-Step state transition formula" by Chapman/Kolmogorov. For those who don't recall it is this formula:
$$
...
0
votes
0answers
138 views
n-tuple Notation
I am reading a paper (to be able to implement the Baum-Welch algorithm in it) and the following notation is defined:
$$
[ a_k ]_{k=i}^j ≡ (a_i, a_{i+1}, \ldots , a_j)
$$
$$
[a(k)]_{k=i}^j ≡ (a(i), ...
0
votes
0answers
108 views
How to calculate the limit kernel of a non-ergodic Markov Chain?
This question is about finding the limit kernel $P^\infty$ of a non-ergodic Markov Chain.
The problem
We consider a TDMC (Time Discrete Markov Chain) $(X_t)_{t \geq 0}$ with $X \in \mathcal{X}$ ...
0
votes
0answers
495 views
Sum of two geometric brownian motion
Is sum of two geometric brownian motion a markov process or a diffusion process? It is given that the two Weiner processes are independent.
Thanks
-1
votes
0answers
34 views
Why can't finite closed communicating classes be null recurrent?
Why can't finite closed communicating classes be null recurrent ? I want both formal proof and intuitive answer.
I know that if state $j$ is null recurrent then
$$\lim ...
