Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
72 views

Can You Help Me With This Continuous Markov Chain Question?

Consider 2 machines, both of which have an exponential lifetime with mean $\frac{1}{\lambda}$. There is a single repairman that can service machines at an exponential rate $\mu$. Set up the ...
0
votes
0answers
65 views

how to determine transient and recurrent state from transition matrix

I wonder how can I determine the transient and recurrent state from transition matrix ? I mean if I have 10 states It would be very hard to draw diagram for them so how to analyse the matrix? For ...
1
vote
0answers
45 views

Limiting probability of a successful bid

I'm having trouble completing the above question, as my working knowledge of "limiting probabilities" is not very good. For the 1-step transition matrix, I have $$P= \begin{pmatrix} 0.0 & 0.0 ...
0
votes
1answer
172 views

PageRank (power iteration method) convergence rate?

I could not get my head around the idea that the second eigenvalue is the convergence rate. Since the matrix in this application is a Markov matrix (rows/columns sum to one), the largest eigenvalue ...
0
votes
0answers
43 views

Trying to find the markov chain and adjacency matrix of this graph?

This is graph of the problem: Suppose animal x is at node 3 of the graph. It chooses small path labelled s with 2 times probability then long path l. If length is same then probability is same ...
1
vote
1answer
31 views

Period of a Markov Chain: Why is this one aperiodic?

Here is the problem from a stochastic processes book: Consider a Markov Chain on {0,1,2} having transition matrix 0 1 2 0| 0 0 1| 1| 1 0 0| 2|.5 .5 0| ...
0
votes
0answers
15 views

Computing smoothed state distribution in HMM

Suppose we have an HMM with two states: $s_1$ and $s_2$. The transitional model is as follows: $P(s_1|s_1) = 0.5$, and $P(s1|s2) = 0.25$. There are two observations: $P(a|s_1) = 0.25$ and $P(a|s_2) = ...
2
votes
1answer
62 views

Random walks : Hitting and recurrence Times relation

I have trouble understanding that how $$E\left[T_0|X_{0} = 0\right] = 1 + E[H_0|X_0=1] $$ where $T_0 = \inf\{n \geq 1:X_n = 0 \}$ and $H_A =\inf\{ n\geq 0: X_n \in A \}$. In other words $T_0$ is the ...
1
vote
0answers
30 views

Learning about Markov Chains

I am trying to learn about how to use markov chains for complicated probability problems. I have been looking for different materials to learn these but haven't had much luck. Does anyone have any ...
1
vote
0answers
17 views

Mixing time analysis of time inhomogeneous markov chaons

There are common methods to characterize mixing times of time homogeneous Markov chains through coupling, conductance and strongly stationary times. However, suppose there is a time-inhomogeneous ...
1
vote
1answer
49 views

Question about HMM

I have this HMM model that I need to solve. Unfortunately, my textbook isn't the best and only describes general cases which I have difficulty working with. Consider an HMM with two states: s1 and ...
0
votes
1answer
38 views

Markov Chain--starting states

How do we define the starting states in a Markov Chain. For example if we are asked to calculate the transition matrix for different starting states, what does that mean? I am ultimately asked to ...
4
votes
1answer
98 views

$P^n$ transition matrix of a Markov chain

The setup: We have an unlimited supply of balls and $k$ boxes. In every step, we randomly (all of them have the same probability) choose a box and put a ball in it. Let $X_n$ be the number of ...
1
vote
1answer
25 views

Proving irreducibility of Markov chain

I have a Markov chain: state: a permutation of n cards transition: taking the top-most card and randomly choose one of the n possible positions for the card I know it is obviously irreducible ...
0
votes
0answers
24 views

DTMC: repairing the machine

A machine works for $Y_0$ time then fails and takes $X_1$ time to repair. Then again works for $Y_1$ time and then fails and takes $X_2$ time to repair and so on. All the $X_n$'s and $Y_n$'s ...
1
vote
0answers
32 views

Getting stuck in a loop or the probability of hitting all points in a random walk around a circle.

Suppose you are walking around a circular path made up of $n$ tiles. Each tile $i$ is assigned a distinct value $r_i$ by a random variable uniformly distributed on the set of integers $\{1,...,k\}$ ...
2
votes
1answer
62 views

Random walk : probability of reaching value $i$ without passing by negative value $j$

This is just some question that popped out of nowhere while starting studying random walks, and I don't really know how to approach this. Say I have a random walk that starts at zero, and goes up or ...
1
vote
0answers
23 views

Dwell times of an absorbing markov chain conditional on reaching specific absorbing state

The fundamental matrix of a discrete time markov chain with absorbing states dictates the expected amount of time spent in each state $j$, given that you started in state $i$. The equation is $$S = ...
0
votes
1answer
49 views

Left eigenvector of stochastic matrices with eigenvalue 1

I am only talking about matrices for finite number of states. By the existence of unique equilibrium distribution, this surely means there can only be one of such eigenvector (i.e. the eigenvalue 1 ...
0
votes
1answer
49 views

Expected success of trial with conditions

Assume that $n$ people want to achieve a task T. One person can try, and is successful with probability $p$. But when a person try all the other have to do an other trial to have the right to ...
0
votes
0answers
38 views

Markov chains: Condtitional independence implies independence?

In one proof, I encountered the following reasoning: $$P(T_1=n,T_2=m\mid X_0=j)=P(T_1=n\mid X_0=j)P(T_2=m\mid X_0=j)$$ Where $T$s are waiting times between returns to a state, $X_0$ is the state at ...
0
votes
1answer
24 views

Markov-Chain with general state space - recurrent sets

I have an irreducible Markov Chain $(z_n )_{n\in \mathbb N } $ with state space $X$ and with transition-probability-kernel $K$, so $K(x,\cdot)$ is a probability measure (on the $\sigma$-Algebra ...
0
votes
0answers
20 views

Plot probability density and calculat a probability from a markov transition matrix

Let's say we have a vector $v_0 = (-10, -1, 0.2, 0.3, 0.7, 1, 1.5, 2, 3)$ where the elements are possible values of a portfolio at time $0$ (denoed $C_0$), and let's say we have a transition matrix ...
0
votes
0answers
33 views

Canonical Construction of a Markov Chain: Intuition

Let $P=(p_{xy})_{x,y \in E}$ be a transition probability matrix over a discrete state space $E$ and $\mu_0$ any distribution over $E$. We proved in the lecture that there is a unique ...
2
votes
1answer
60 views

Time sampling an ordinary poisson process

My questions will be given at the end, let me just give some definitions first. The counting process $\{ N(t), t \geq 0 \} $ is said to be a non homogenous Poisson process with intensity function ...
0
votes
0answers
31 views

Optimal stopping strategy

I try to solve the following problem : Given a series of random variables : X1,X2,... such that each one can get either -1 or 1 with probability 0.5, give a strategy to maximize the expected value of ...
0
votes
1answer
32 views

conditional probability density function

If the joint probability density function for the waiting times $W_1$ and $W_2$ is given by: $f(w_1,w_2)=\lambda^2$ $exp(-\lambda w_2)$ for $0<w_1<w_2$. How would I determine the conditional ...
3
votes
1answer
66 views

Joint density function Poisson Process

We did an example in class that I'm not sure how we came up with the answer. The problem is: If I let X(t) be a Poisson process of rate $\lambda$. I'm supposed to validate the identity ...
0
votes
2answers
188 views

Markov Chains Question

Markov chains are widely used in modeling several natural and social processes. Consider the following three-state Markov chain modeling the daily weather in Boston. Each day can be sunny, partly ...
3
votes
1answer
47 views

Question about random walk markov chain

For a random walk, let $a$ denote the probability that the markov chain will ever return to state $0$ given that it is currently in state $1$. Because the markov chain will always increase by $1$ with ...
1
vote
1answer
41 views

Independence of random variables derived from a Random walk

Let $w=(w_x)_{x \in \mathbb Z}$ be i.i.d random variables taking values in $(0,1)$. Let $(X_n)_{n \in \mathbb{N}_0} (\mathbb{N} \cup {0})$ be a Markov chain (more specifically a simple random walk ...
1
vote
1answer
30 views

Bayes: markov chain, serial connection, marginalization

Goal is to check if p(a) is unconditionally independent to p(c) in the markov chain - serial connection. $$ p(a,b,c) = p(a) p(b|a) p(c|b) $$ $$ p(a,c) = \sum_b p(a) p(b|a) p(c|b) = p(a) p(c|a) \neq ...
0
votes
1answer
25 views

Probability that Markov chain process has particular state after n steps

If we have a Markov chain X with four discrete states, and we want to find the probability the process is in a certain state (one of the four) n iterations later, would we raise X to the nth power and ...
1
vote
1answer
31 views

Conditional Probabilities Poisson Process

If I let ${X(t); t>=0}$ be a Poisson process having rate parameter $\lambda = 2$. I'm supposed to determine the probability: Pr{${X(1)>=2 | X(1) >=1}$} My solution: I looked at this as ...
1
vote
1answer
25 views

Conditional Distribution Poisson Process

In class, our professor told us to verify this solution on our own time. The problem is: Let $\left\{X(t),t \geq 0 \right\}$ be a Poisson process of rate $\lambda$. For $s,t >0$, determine the ...
1
vote
1answer
39 views

Modelling a continious-time queue which behaves differently when there are more or less people being served.

For my research I am trying to model a continuous-time queue which behaves differently when there are more or less people being served. The arrival rate in the queue is constant, however the departure ...
2
votes
0answers
26 views

$T_n$ stopping time, is $\{X_{T_n}\}$ markov chain

Let $\{X_n\}$ be a Markov Chain with finite state space $S$. Let $T_n$ be the $n$-th hitting time of $A \subset S$ i.e. $n$-th time it hits some state from the set $A$. Is $\{X_{T_n}\}$ a Markov chain ...
4
votes
1answer
39 views

How do stochastic matrices really converge?

We are given the matrix $A=\begin{bmatrix}0.9&0.5\\0.1&0.5\end{bmatrix}$ and any initial vector $X^{(0)}=\begin{bmatrix}a\\b\end{bmatrix}$. The matrix $A$ has the following eigensystem: ...
0
votes
0answers
30 views

mean recurrence time

$E\left[T_j |X_0=i,X_1=k\right]$ \left\ space{\begin{matrix} 1+U_{kj} \space\ k\neq j & \\ 1 \space\ k=j &\end{matrix}\right. Does this mean that the number of steps it takes to get back to ...
1
vote
2answers
26 views

Something about Markov chains

We check $P(X_{n+1}\in B|\mathcal{F}_n)=P(X_{n+1}\in B|X_n)$ when we want to prove $X_n,n=1,2,\dots$ is a Markov chain. Through this equation it seems that $X_n$ is a Markov chain if $X_{n+1}$ is ...
0
votes
1answer
39 views

Proving that the Markov chain is recurrent - Confusion/Help

Giving the following transition matrix [ 0.9 0.1 ] [ 0.8 .2 ] Classify the states From drawing the graph I know that both stats are recurrent. However I'm really failing to prove mathematically ...
1
vote
0answers
15 views

Is the steady state of a uniform markov chain always a vector of proportions?

Given that all edges in a markov chain are bi-directional (though not necessarily equally weighted), and each edge for a given node has equal probability, does the steady state always converge to a ...
1
vote
1answer
24 views

Definition of limiting distribution in a Markov chain — why do we condition on the initial state?

Given a Markov chain $\{X_n \mid n \in \{0, 1, \ldots\}\}$ with states $\{0, \ldots, N\}$, define the limiting distribution as $$ \pi = (\pi_0, \ldots, \pi_N) $$ where $$ \pi_j = \lim_{n \to +\infty} ...
0
votes
1answer
41 views

Expected time to absorption

I have been trying to solve the following problem for quite a while now, but not with much luck. The Question Let $P$ be the TPM(Transition Probability Matrix) of a DTMC with state space ...
0
votes
1answer
39 views

Infinite$-$state absorbing Markov chains

Could someone provide a good reference/book about infinite$-$state absorbing Markov chains? Most of what I've found so far deals only with the finite$-$state case.
0
votes
0answers
45 views

Help with this Markov Chain Proof please

Problem: Consider a finite Markov Chain with N states $(1,2,...,N)$. Let $P(n) = [P_{i,j} (n)]$, be an n-step transition matrix. Suppose that $lim_{n\to\infty} P_{i,j} (n) = \pi_{j} $ for any $1 ...
0
votes
0answers
61 views

When is $A^{+} P^{\top} A$ non-negative?

$P$ is a $n \times n$ stochastic matrix (non-negative, rows sum to one). $A \in \mathbb{R}^{n \times k}$ with $k < n$ has non-negative entries and independent columns. Denote by $A^+ \in ...
0
votes
1answer
30 views

Understanding the proof of stationary distribution of a markov chain

I am reading the proof of existence of stationary distribution in an irreducible markov chain from the book Markov Chains and Mixing Times by P. D. A. Levin, Y. Peres, E. L. Wilmer, and I have the ...
0
votes
0answers
28 views

Repair Chain (Markov Chain Sample Model)

A machine has $3$ critical parts that are subject to failure, but can function as long as two of these parts are working. When two are broken, they are replaced and the machine is back to working ...
0
votes
0answers
18 views

Show if $p^n(i,j)\rightarrow \pi(j)$ as $n\rightarrow\infty$ then $\pi(j)$ is a stationary measure..

Suppose $p(i,j)$ is a transition kernel on $S$ for a countable state markov chain $X_n$ with $$p^n(i,j)\rightarrow \pi(j)$$ as $n\rightarrow\infty$ for all $i,j\in S$. want to verify that $\pi$ is a ...