Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
12 views

Exercise on Markov chains

I'm preparing my Probability exam and I'm having trouble with exercise 2 here. The question is to consider the random walk on $E$ with transition matrix $p$ and find the communication classes (or ...
1
vote
0answers
16 views

Periodicity of Markov chains under cartesian product

Suppose that you have a finite state Markov chain, with $n$ states and characterized by $p_{i,j}$ the probability of reaching state $j$ from state $i$. Consider the new Markov chain with $n^2$ states ...
1
vote
0answers
14 views

Proving criterion for a transient state in Markov Chain

Let $\{X_n\}_n$ be a homogenous Markov chain. Prove that if exist a connected subset of states (means set of states which exist positive probability to move between them), $S$ which is not closed, ...
0
votes
0answers
8 views

continuous markov chain generator

I am trying to learn Markov process with my own. I am a little confused about the generator of markov process. I understand that Markov process consists of embbedded Markov chain matrix and the ...
0
votes
0answers
20 views

Proof of “strong law of large numbers” in Markov Chains

I have been given a theorem stating an analogue of the strong law of large numbers for Markov chains. It states that if $X=(X_n)_{n\in\mathbb{N}}$ is a Markov chain with transition matrix $p$ and ...
0
votes
0answers
18 views

Conditional independence given elementary events implies conditional independence given $\sigma $-algebra

Proposition: Let $X$ be a continuous markov chain with discrete state space $S$. Let $Z$ be the corresponding jump chain and $\left\{ {{W_i},i \in \mathbb{N}} \right\}$ its holding times. Let ...
0
votes
1answer
50 views

Markov chain doesn't sum up to 1

Let $\{X_n\}$ be a Markov chain on $S=\{1,2,3,4,5,6\}$ with the matrix suppose we define a new sequence $\{Y_n\}$ by $$Y_n=\cases{1\quad X_n=1\vee X_n=2\\2\quad X_n=3\vee X_n=4\\3\quad ...
1
vote
0answers
12 views

Marcov Chain confirmation

I am currently having some problems on the following question: Given is the function $f(x)$: $f(x) = 0,1,2$ with probability $\frac{1}{3}$ for each. I have to give the state space, transition ...
0
votes
0answers
14 views

Proof of theorem on markov chains

If $X=(X_n)_{n\in\mathbb N}$ is a Markov chain on a space $E$, it has an initial distribution $(\lambda_i)_{i\in E}$ such that $\sum\lambda_i=1$ and a transition matrix $(p_{ij})_{i,j\in E}$ such that ...
0
votes
0answers
8 views

non-integer powers of transition matrices with complex eigenvalues and resulting negative probabilities

I am currently working on a Markov Chain model for transition probabilities of a certain set of states. I am trying to figure out how to scale my transition matrix to arbitrary time periods by raising ...
0
votes
0answers
12 views

Why does from Perron-Frobenius follow that that constant functions are the only harmonic functions here?

in our reading we had the following example for a Markov chain. State Space $E=\left\{1,2,3,4\right\}$ and Transition Matrix $$ P=\frac{1}{3}\begin{pmatrix}0 & 1 & 1 & 1\\1 ...
-1
votes
0answers
16 views

Calculate all harmonic functions

Let $E$ befinite and suppose that $P$ is irreducible and strictly sub-stochastic. Calculate all harmonic functions. To my understanding, $P$ strictly sub-stochastic means that $\sum_{y\in ...
0
votes
0answers
11 views

Memory less property of a Markov chain- Validation methods

Are there any tests to check the memory less property of a discrete time homogeneous Markov chain? I found a chi squared test to verify the time homogeneity of a Markov chain constructed from a set of ...
1
vote
2answers
241 views

Reversibility of a Markov Chain

Rephrased question: Is it ever possible for a reducible Markov chain to be reversible?
1
vote
1answer
36 views

Calculating the hitting probability using the strong markov property

** This problem is from Markov Chains by Norris, exercise 1.5.4.** A random sequence of non-negative integers $(F)n)_{n\ge0}$ is obtained by setting $F_0=0$ and $F_1=1$ and, once $F_0,\ldots,F_n$ are ...
0
votes
0answers
5 views

Hastings algorithm

Let $Q=\begin{pmatrix} 0 & 1 & 0 & 0 & 0\\0.5 & 0 & 0.5 & 0 & 0\\ 0 & 0.5 & 0 & 0.5 & 0\\ 0 & 0 & 0.5 & 0 & 0.5\\ 0 & 0 & 0 ...
0
votes
1answer
12 views

Transition probability matrix of Markov chain

Given that $g(x)=\begin{cases} 1/3 \quad\text{for } x=0\\ 1/3 \quad \text{for } x=1\\ 1/3 \quad \text{for } x=2\end{cases}$ Explain why independent draws $X_1,X_2,\dots$ from $g(x)$ ...
0
votes
0answers
16 views

question with marknow chain on finding happenings after unspecified amount of time.

I am attaching the questions and what i have done so far. I am having doubt in question from 16. Can anyone please explain me how to solve the rest? question 9 P23 that is equal to 0.1 ...
0
votes
0answers
21 views

Markov chain general help

If I have an absorbing state markov chain (with 2 absorbing states, graduate and dropout), and I know how many people I have in each state (say total for all states is 1000), how would I work out what ...
0
votes
1answer
36 views

Proof that there exists a non-negative eigenvector corresponding to eigenvalue 1 of stochastic matrix

Let $P \in [0,1]^{n \times n}$ be a [irreducible or reducible] stochastic matrix where its rows sum to 1 i.e. $$ \forall i \in \{ 1 , \dots n \} \quad \sum_{j=1}^{n} P_{ij} = 1 $$ It is easy to show ...
3
votes
0answers
19 views

$L^2$ Bounds for Markov Chains.

Consider a non-negative, square stochastic $n \times n$ matrix $P$ (rows sum to one, $P$ is ergodic). We are interested in characterizing the set of $n \times n$ invertible matrices $A$ such that we ...
0
votes
2answers
108 views

Are irreducible, positiv-definite Markov chains aperiodic?

If $M$ is the transition matrix of a discrete Markov chain, and $M$ is both irreducible, symmetric and positiv-definite, is the resulting Markov chain necessarily aperiodic? In my intuition, ...
0
votes
2answers
641 views

Forming the transition matrix for Markov chain, given a word description of transition probabilities

I have just started learning about Markov chain and have a trouble determining appropriate transition matrix: Suppose that whether or not it rains today depends on previous weather conditions ...
0
votes
0answers
36 views

Create a Martingale out of a Markov Chain.

Consider a homogeneous finite state Markov chain $\{X_n\}$ with transition matrix $P$ and state set $S$ consisting of real numbers. How to choose the elements of $S$ so that $\{X_n\}$ be a ...
0
votes
0answers
6 views

Condition under which Markov Chain remains in a compact set a.s.

Let $\{Y_n\}$ be a Marov chain. Is it good question to ask under what conditions this chain will take values from a compact set a.s. ?
0
votes
1answer
18 views

what are “filtering” and “smoothing” mentioned in hidden Markov model wikipedia article?

the article mentions "filtering" and "smoothing" tasks, see here http://en.wikipedia.org/wiki/Hidden_Markov_model#Filtering . It gives brief explanation but no motivating examples and no references to ...
-1
votes
0answers
31 views

Branching Processes

I would like to have some hints about the following problem: Imagine a descendant tree. We assume that each species can give birth to a new specie in a constant rate $\lambda$. (This rate is called ...
0
votes
0answers
11 views

Given stochastic matrices D and K, under what conditions can I find stochastic matrices that satisfy a given equality?

Let $D \in \Re^{n \times m}$ and $K \in \Re^{m \times n}$ be two stochastic matrices, with $n > m$. The problem is to determine under which conditions there exist stochastic matrices $P \in \Re^{m ...
1
vote
0answers
46 views

Couple/Compare two stochastic processes and prove an intuitive proposition

Consider a stochastic process (denoted $X$) $X_0, X_1, X_2, \ldots$ (not necessarily a Markov Chain) over state space $\{0, 1, \cdots, n \}$. The transition probabilities are $$P(X_{i+1} = 1 \mid ...
0
votes
0answers
10 views

Simple Random Walk; Proof hitting theorem; Ballot Theorem

Suppose that $(X_{n}:n\in\mathbb{N})$ is a $\pm1\mbox{-valued sequence.}$ Let $p\in(0,1)$ and $p=\mathbb{P}(X_{i}=1)\mbox{ and}\mathbb{P}(X_{i}=-1)=1-p=q$ . Define the simple random walk ...
2
votes
2answers
279 views

probability terminology for parameter in a Markov process

Suppose $$P(\text{feature present at time} \ t \ \text{and} \ t+\Delta t) = \beta^{2}+\beta(1-\beta) \exp(\Delta t/\tau)$$ where $\tau = 1/(\pi_{01}+\pi_{10})$. What is $\tau$?
0
votes
2answers
31 views

Does from $P^n$ irreducible for all $n\in\mathbb{N}$ follow that $P$ is aperiodic?

let $P$ be a transition matrix of a Markov chain with state space E, that is finite. Does from $P^n$ irreducible for all $n\in\mathbb{N}$ follow that $P$ is irreducible and aperiodic? ...
1
vote
1answer
53 views

What does it mean if $P^n$ is irreducible for every $n\in\mathbb{N}$?

If $P$ is the transition matrix belonging to a markov chain, then what does it mean that $P^n$ is irreducible for every $n\in\mathbb{N}$? For $n=1$ it means that all states communicate with each ...
0
votes
2answers
47 views

Why is P irreducible and aperiodic?

Let $P=(p_{ij})_{i,j\in E}$ denote the transition matrix of a Markov chain with finite state space $E$. Why does the following implication hold: $$ \exists n\in\mathbb{N}: ...
1
vote
1answer
46 views

Probability of a time-dependent set of states in Markov chain

Consider a Markov matrix $P$ defining $m$ states. For each time $n$, define a set of states $S_n$ that contains a predefined subset of the states $\left\{ {1,...,m} \right\}$. For time $n=k$, I would ...
0
votes
0answers
19 views

Stationary solution of a binary Markov chain of order m

Let $X$ be a binary Markov chain of order m. What is the stationary solution of X? In other words, find $\lim_{n\to \infty} P( (X_{n-m+1},X_{n-m},...,X_{n}) =(a_1,a_2,...,a_m))$, for arbitrary values ...
0
votes
0answers
23 views

Steady state of a specific Markov martirx

Let $n = 2^L$ for an arbitrary integer $L>0$ and let $A=(a_{i,j})$ be an $n \times n$ matrix with the following structure: For $1\leq i \leq \frac{n}{2}$, $a_{i,2i-1} = p_i$, $a_{i,2i} = 1- ...
1
vote
1answer
25 views

Combination of Markov chains, existance of limiting distribution

I have two Markov chains described by the stochastic matrices $P_1$ and $P_2$ for which a limiting distribution exists. Now I combine the two stochastic matrices using the cartesian product, this ...
2
votes
1answer
37 views

Eigenvectors of transition matrices in PageRank algorithm

In my probability course, we were discussing applications of Markov Chains to computer science -- in particular, how the PageRank algorithm goes about finding stationary distributions, and thus, ranks ...
3
votes
1answer
148 views

Monotonic convergence of powers of a stochastic matrix

Let $P$ be a stochastic matrix (nonnegative and each row summing to 1). Assuming that $P^n$ converges to $\textbf{1}\pi$ as $n \rightarrow \infty$, where $\pi$ is a row vector (stationary distribution ...
0
votes
0answers
9 views

hidden Markov model with multiple observations

I am wondering if HMM can be used for the case that in a particular state, there are more than one observations. For instance at time t, we can observe the position, velocity and acceleration at the ...
9
votes
1answer
309 views

What happens to a random walk when we increase the probabilities of going right?

Consider a random walk on the integers where the probability of transitioning from $n$ to $n+1$ is $p_n$ (and of course, the probability of transitioning from $n$ to $n-1$ is $1-p_n$); we assume all ...
2
votes
0answers
28 views

Mixing time for metropolis chain on graph coloring

I'm reading the Markov Chains and Mixing Times by David Levin et al.. In section 5.4 page 71 a proof is given for a bound of mixing time for the Metropolis Chain on graph coloring. In the proof, such ...
0
votes
0answers
16 views

Problem related to variance of first passage matrix of a absorbing Markov chain

Consider the below computations taken from Kemeny/Snell Finite Markov Chains. Here $N=(I-Q)^{-1}$ calculated from some absorbing MC. $N_2$ is the variance matrix of $N$ and $N_{sq}$ is taken by ...
0
votes
1answer
62 views

Countable state Markov chain with multiple transitions

I'm searching for hints on how to analyze the following Markov chain. I can solve for the steady state probabilities numerically by using a finite transition matrix. However, I would like to have an ...
0
votes
0answers
21 views

How to prove that this stochastic matrix has a limiting distribution

I have the following stochastic matrix with $p_{ij} > 0$ and $\sum_j p_{ij} = 1$ $$ P = \begin{bmatrix} p_{11} & p_{12} & 0 & 0 & 0 & 0 \\ p_{21} & ...
4
votes
1answer
298 views

Intuition on Harris recurrence

I am trying to get some intuition on Harris recurrence in Markov chains. Define state space $\mathcal S$ comprising a single communication class, $f_{ii}^{(n)}=P(X_n=i, X_{n-1}\ne i,\ldots X_1\ne ...
0
votes
0answers
23 views

A problem on markov chain

I have seen in a book the following : Let $X_n$ be an ergodic markov chain taking values in a complete separable metric space $S$. Now consider the function $\mu(t) = \delta_{Y_n(\omega)}$ when $t ...
1
vote
0answers
29 views

Construction of pure birth process

I am considering a Markov chain $\lbrace X(t) \rbrace_{t≥0}$ in continunous time on the countable state space $S=\lbrace 0 \rbrace\cup \lbrace (i,j) \mid i \in \mathcal{A} , j \in \mathbb{N} \rbrace, ...
1
vote
1answer
317 views

Renewal Processes for Uniform and exponential Distributions

Suppose the lifetime of a component Ti in hours is uniformly distributed on [100, 200]. Components are replaced as soon as one fails and assume that this process has been going on long enough to reach ...