Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
45 views

Markov chain question

Consider an irreducible, recurrent Markov Chain ($X_n$) on a countable state space $S$ with transition probability $p(x,y).$ Pick a sigma-algebra $A \subset S$ and let $T_k=\inf\{n>T_{k-1}:X_n \in ...
1
vote
1answer
80 views

Markov chain problem, Help!

I am stuck on this question for a long time Question: Consider 4 balls, labelled from 1 to 4 and distributed amongst two urns (Urn 1 and Urn 2). At each time $n>1$, a number from 1 to 4 is chosen ...
0
votes
0answers
4 views

How can I calculate distribution of minima of sections of a continuous path (from a stochastic process)?

I have a long slab whose width is defined by a stochastic process, whose complete statistics I am aware of, say. I now cut it into smaller sections of uniform length, and calculate the minimum width ...
0
votes
1answer
26 views

Show $P(S_{2n}=x|S_0=x) \ge \frac{1}{N}$

Let $X_n$ be an aperiodic, discrete-time Markov chain so $S=\{1,...,N\}$ whose transition probability is symmetric. How can I show that for all $x \in S$ and all integers $n$, $P(S_{2n}=x|S_0=x) \ge ...
0
votes
0answers
106 views

understanding submartingale proof with discrete state space

I am reading a text about branching markov chains: My question is about the first half of page 8 where $Q(t)$ is proven to be a submartingale. Briefly the used notation: $t$ is discrete time, $n(t)$ ...
2
votes
0answers
6 views

$\psi$-irreducibility of m-skeletons.

In Proposition 5.4.5 of Meyn and Tweedie's Markov Chains and Stochastic Stability, it is said that if a chain $\Phi$ is $\psi$-irreducible and aperiodic, then every $m$-skeleton of it is also ...
1
vote
2answers
32 views

Looking for good literature on Markov Chains with explicit calculations

I am currently starting my thesis on Markov Chains and am looking for good books and papers that include explicit calculations. I have taken a small course on Markov Chains so the subject is not ...
2
votes
0answers
31 views

markov spectral radius independent of states?

Let $\Pi$ be a stochastic matrix of an irreducible markov chain. We define the spectral radius of $\Pi$ as: $\rho(\Pi) := \limsup_{n \to \infty} \left( \pi^{(n)}_{(a,b)} \right)^{\frac{1}{n}}$ Why ...
0
votes
0answers
9 views

Hidden Markov Model Confidence Interval (preferably in MATLAB)

I'm trying to uncover the transition parameters of data of a hidden Markov Model using MATLAB. Using the built in hmmtrain function, I can estimate the parameters quite well (I already know what they ...
4
votes
0answers
60 views

Spectral gap of mixture of Markov chains

Context Let $P$ be the transition matrix of an irreducible, aperiodic, discrete-time Markov chain. The spectral gap is given by $$\xi = 1 - \lambda_\max$$ where $\lambda_\max = \max\{\lambda_2, ...
2
votes
2answers
40 views

Generate random sample with three-state Markov chain

I have a Markov chain with the transition matrix $$\pmatrix{0 & 0.7 & 0.3 \\ 0.8 & 0 & 0.2 \\ 0.6 & 0.4 & 0}$$ and I would like to generate a random sequence between the three ...
0
votes
0answers
10 views

References for non-homogeneous continuous-time Markov chains

In one applied problem that I'm trying to solve, I want to apply nonhomogeneous continuous-time Markov chains. But cannot find a good reference on these kind of chains. I mean with simple worked-out ...
0
votes
1answer
46 views

Proving a chain is aperiodic, and finding a stationary distribution.

We have an irreducible Markov chain with a not necessarily finite state space. It has a transition matrix $P$ such that $P^2=P$. Prove (1) the chain is aperiodic, and (2) prove $p_{ij}=p_{jj}$ ...
0
votes
1answer
17 views

Question involving an invariant measure on a Markov chain

Suppose $\mu$ is an invariant measure for a Markov chain with state space $S$ with $\mu(i)p_{ij}=\mu(j)p_{ji}$ $\forall i,j \in S$. Describe a Markov chain with this property. Also, show that $\mu$ is ...
1
vote
1answer
44 views

P is transition probability matrix.I is identitiy matrix.A is matrix whose entries are all 1.Then prove I+A-P is invertible

$P$ is the transition probability matrix for a finite irreducible markov chain. $I$ is identitiy matrix. $A$ is the matrix whose entries are all $1$. Prove $I+A-P$ is invertible. I don't have any ...
0
votes
1answer
19 views

Transition rate matrix from transition probability matrix

If I have a $2 \times 2$ continuous time Markov chain transition probability matrix (generated from a financial time series data), is it possible to get the transition rate matrix from this and if ...
5
votes
0answers
73 views

Generating a stochastic matrix with a given second dominant eigenvalue

I need a procedure (iterative or otherwise) that, given a positive integer $N$ and a (possibly complex) number $\lambda$ such that $0 < \vert \lambda \vert < 1$, will be able to generate an $N ...
0
votes
0answers
26 views

Continuous time Markov chains, how would this definition be expanded from time-homogeneous to time-inhomogeneous.

Below I have a picture of how we can view a continuous time Markov chain that is time-homogeneous. Now, I am wondering what happens when we have a inhomogeneous continuous Markov chain. I have ...
0
votes
1answer
31 views

What is the Deterministic Traffic Generation Model?

I am studying Markov chains and queuing theory. I was curious about traffic generation models and actually happened to see the Deterministic Traffic Model, referred to as $D$ in Kendall's notation. ...
1
vote
2answers
43 views

Solving a linear equation to find a stationary matrix

I'm trying to solve the following system of linear equations derived from a transitional matrix for a regular Markov chain. I can't use matrix methods since that would involve finding the inverse of a ...
0
votes
1answer
19 views

Queue system with queue-triggered input process

I have a queue system, a classic system with an input generator, a queue and a servant. The servant is a $M$-servant with a certain serving rate $\mu$. The queue can contain an infinite number of ...
0
votes
1answer
30 views

Markov Chain probabilities

I'm having trouble with this problem from Resnick's Adventures in Stochastic Processes: Consider a Markov Chain on states {0,1,2} with transition matrix $ \left( \begin{array}{ccc} 0.3 & 0.3 ...
0
votes
1answer
19 views

Average time of permanence in a state of a Markov-chain

I know that in a Markov-chain the mean permanence time in a state is a random variable distributed accordingly to: Geometric distribution for Time Discrete Markov Chains Exponential distribution for ...
0
votes
3answers
65 views

Markov Chain: classify states of finite Markov chain

I can easily see the states of this MC, recurrent and transient if I graph them, but how do I prove that a state is recurrent or transient. My book refers to probability to ever returning to state $j$ ...
0
votes
0answers
25 views

Markov renewal process vs Markov Jump process

The venerable wikipedia for "Markov renewal process" says that: "a Markov renewal process is a random process that generalizes the notion of Markov jump processes". So what's the definition of a ...
0
votes
0answers
26 views

Normalizing a co-occurrence matrix with an energy function to create a conditional random field or markov random field.

I currently have a set of factors produced for a co-occurrence matrix : link I want to be able to move towards developing a conditional random field for pixel labelling and multi-class segmentation. ...
3
votes
1answer
55 views

When are stable continuous time Markov chains Feller? Always?

This is a question is similar to this 2 year-old one that never got answered (truthfully it's pretty much the same question except that I'm adding a bit more detail and the assumption that the $Q$ ...
1
vote
0answers
53 views

Markov Chains: Limiting probabilities of positive recurrent states sum to one?

I have a question about Markov chains. I am trying to understand the proof of Proposition 2.6 of http://www.columbia.edu/~ks20/stochastic-I/stochastic-I-MCII.pdf. The setting is: we have a positive ...
3
votes
1answer
57 views

Sum of two Markov processes another Markov process?

Let $dX_{t} = m_1(l_1-X_{t})dt+\sigma_1 dW_{t}$ and $dY_{t} = m_2(l_2-Y_{t})dt+\sigma_2(\rho dW_{t}+\sqrt{1-\rho^2}dW_{t}^{1})$ where the $m_i$'s, $l_i$'s and $\sigma_i$'s are constants, $\rho \in ...
3
votes
1answer
17 views

The expectation of total number of different states in N time points

[Conditions] (1) An object has K possible states. (2) This object can have only one state at a single time point. (3) The probability of each state at any single time point is 1/K, and each time ...
0
votes
0answers
30 views

Continuous time Markov chain

I would like to know if I am on a right track? Continuous time Markov chain on Wikipedia A very new European “Rapid Reaction Force for Fire” has been created today and begins operation between three ...
0
votes
0answers
40 views

Approximating a Markov process by differential equations

I have a system of states, $m_S = 1, 0, -1$. After performing a certain manipulation (it can be assumed to be instantaneous), a transition can happen with probability p. However, not all states can ...
2
votes
1answer
27 views

On track Prerequisite for Statistics and Probability

I do not really have a solid mathematical background because of the range of courses i had back in high school/university that wasn't really scientific oriented. Presently i am doing an MSc in ...
1
vote
1answer
84 views

Elementary proof of geometric / negative binomial distribution in birth-death processes

The birth-death process concerns a population of $n_0$ individuals, each of which reproduce and die at a constant rate as time $t$ increases from $t=0$. Each individual splits into two individuals ...
0
votes
1answer
44 views

Markov chains example

Your exam could be marked with a range of possible grades, simplified as on the following state diagram: To begin with the chances are that you will pass with a standard result. Each 45 minutes ...
0
votes
2answers
57 views

expected value of this markov chain

Question: A bag contains 3 white chips and 3 red chips. You repeatedly draw a chip at random from the bag. If it's white, you set it aside; if it's red, you put it back in the bag. After removing all ...
0
votes
0answers
19 views

Can an irreducible countable state Markov chain have transient states?

I've been studying Markov chains, and I came up with a question: Can an irreducible countable state Markov chain have all transient states? I know the fact that for finite-state Markov chains, there ...
1
vote
0answers
18 views

Finite state Markov chains and expectations

Let $X_t$ be a finite state Markov chain with generator matrix $Q$. For a give function $f(x)>0$ define: $$ u(t, i) = \mathbb{E}\left[\int_t^T f(X_s) \mathrm{d} s| X_t = i\right] $$ Are there any ...
1
vote
2answers
64 views

Expected number of steps between states in a Markov Chain

Suppose I am given a state space $S=\{0,1,2,3\}$ with transition probability matrix $\mathbf{P}= \begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 \\[0.3em] \frac{2}{3} ...
0
votes
0answers
18 views

Poisson processes are the only renewal processes which are Markov Chains.

How would one prove the Proposition: "Poisson processes are the only renewal processes which are Markov Chains." A renewal process $N=(N(t))$ is a process for which $$N(t)=\max\{n : T_n \leq t\}$$ ...
2
votes
1answer
49 views

Coarser cyclic decomposition of Markov chain

For a irreducible Markov chain with period $d$ there is a standard construction which shows that the state space can be partitioned into $d$ sets $C_1, \ldots, C_d$ such that $P(x,y)>0$ only if $x ...
1
vote
2answers
38 views

Counterintuitive Markov chain problem

In class, my professor said that given a Markov chain $\{X_k\}$ it intuitively should be true that $P(X_{k+1} = x_{k+1} \, \mid \, X_0 = a_0, \dots, X_{k-1}= a_{k-1}) = P(X_{k+1} = x_{k+1}\, \mid ...
0
votes
1answer
62 views

Exercise on Markov chain

Prove, or give an explicit counterexample to refute, the following assertion: if $\{X_n\}$ is a Markov chain, then $\{X_n^2\}$ is also a Markov chain. It's easy to show that ...
3
votes
1answer
50 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is ...
0
votes
0answers
18 views

Computing standard errors using EM algorithm

I'm applying the EM algorithm to a hidden markov chain (the $\mathbf{Z}=\{Z_1,...,Z_n\}$ variable), with observations(the $\mathbf{Y}=\{Y_0,...,Y_n\}$ variable) dependent not only on the hidden markov ...
1
vote
1answer
16 views

Question regarding Markov chains

I have this problem: Random variables $U_1,U_2,...$ are i.i.d with the distribution, $P(0)=0.1, P(1)=0.3,P(2)=0.2,P(3)=0.4.$ Consider a new sequence $X=(X_n=X(n))$ defined as $X(0)=0, ...
2
votes
1answer
92 views

Markov chain problem with finite states

Consider any Markov chain on a state space with exactly $r$ states. We want to find the largest $N>0$ such that there exists states $i,j$ for every $n < N$ we have $p_{ij}^{(N)} > 0$ and ...
1
vote
1answer
45 views

The “maximum” of a simple random walk

Suppose $S_n$ is a simple random walk started from $S_0=0$. Denote $M_n$ to be the maximum of the walk in the first $n$ steps, i.e. $M_n=\max_{k\leq n}S_k$. Show that $M_n$ is not a Markov chain, but ...
1
vote
0answers
32 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
0
votes
2answers
33 views

Markov chain, successive running average

Here's my question: Take three numbers $x_1$, $x_2$, and $x_3$, and form the successive running averages $$x_n = \frac{x_{n-3} + x_{n-2} + x_{n-1}}{3}$$ starting with $x_4$. Prove that $$\lim_{n \to ...