Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
26 views

Symmetric Simple Random Walk - Definition Clarification

I'm finding conflicting answers everywhere, including in my own notes. In the phrase "symmetric simple random walk", which part, "symmetric" or "simple" refers to having a probability of $0.5$ to go ...
0
votes
0answers
15 views

Markovian Model: scheduling jobs to servers

I have the following problem. I tried to look at queuing theory, but it probably fits better as a scheduling problem. I have a set of $C$ servers: each one can perform 1 job. Processes arrive ...
0
votes
1answer
14 views

Does an embedded discrete-time Markov chain preserve its properties in continuous time?

Given a discrete-time Markov chain without independent increments, is the embedding of it into a continuous time Markov chain (i.e. via the use of exponential waiting times) an example of a continuous ...
1
vote
0answers
43 views

For a finite state irreducible aperiodic MC, show that $P^{d^2}$ has all coordinates positive.

Suppose $X_n$ is an irreducible aperiodic finite state MC, with $P$ being the transition matrix. Then we know that $P^n$ has all positive entries for some $n\in\mathbb N$. If the state space $S$ of ...
0
votes
0answers
34 views

Question regarding regular stochastic matrix

We say that a stochastic matrix is regular iff $\exists n\in \mathbb N$ such that $p_{ij}(n)>0$ for all states $i,j$ How many powers of a matrix do we need to compute at most in order to verify ...
1
vote
1answer
40 views

Show that a function of a markov chain is not a markov chain [closed]

How do you show that a function, $Y(n)=g(X(n))$ if some Markov chain $X(n)$ cannot be a Markov chain?
-1
votes
0answers
34 views

A few questions related to a Markov chain.

In the picture I have a Markov chain, call it $X(n)$. I have a few questions about this Markov chain. First, is it aperiodic? Second, what is the value of $P[X(1)=1,X(2)=0,X(3)=0,X(4)=1\mid ...
0
votes
0answers
8 views

How to find transition matrix in cascaded FSMs?

I am considering a 4-state system Equivalently, I can use a cascade approach to represent the same system as In cascade approach, If I consider independent condition on input on both ...
1
vote
1answer
82 views

Markov Chains where Time Spent in State Matters

I have done a good bit of research on the subject, and cannot seem to find many materials. I was just wondering if you all knew of a good resource regarding chains which are Markov excepting the fact ...
0
votes
0answers
6 views

How to make Markov Chain model from sequence of data in MATLAB?

Markov Chain model considers only 1-step transition probabilities i.e. probability distribution of next state depends only on current state and not on previous state. I have a sequence and from that I ...
0
votes
1answer
43 views

Markov Chain that isn't Irreducible

What is an example of a Markov chain that isn't irreducible but has a unique distribution, such that its distribution converges to that unique invariant distribution for any initial distribution.
1
vote
0answers
25 views

How to test quality of probability estimates?

I have a Markov chain model which produces a probability distribution for absorption in 4 possible absorbing states. I.e. the model estimates the probability distribution for a discrete random ...
3
votes
1answer
55 views

Example of a Continuous-Time Markov Process which does NOT have Independent Increments

1. Given a discrete-time Markov chain without independent increments, is the embedding of it into a continuous time Markov chain (i.e. via the use of exponential waiting times) an example of a ...
1
vote
1answer
37 views

Markov dynamic programming recursion

I'm learning Markov dynamic programming problem and it is said that we must use backward recursion to solve MDP problems. My thought is that since in a Markov process, the only existing dependence is ...
1
vote
0answers
57 views

Prediction Interval from Markov Chains

Thank you for taking the time to look at my question. Short, less involved question: How do you find Prediction Intervals with non-Gaussian residuals? Here is the situation: I have made a model that ...
0
votes
1answer
26 views

What does it mean for an object to not be following a definition based on some implication

I want to get a deeper understanding of what being an object that doesn't follow a definition means in terms of predicates and logical operators. Suppose the following definition of the closedness ...
1
vote
1answer
30 views

probability of which alarm clock goes off first

I'm learning about continuous time markov chains, in the text I am reading, they are setting up the discussion by talking about a series of alarm clocks that are set: Suppose $T_1,...,T_n$ are ...
1
vote
1answer
29 views

distance from the origin in a simple random walk on $\mathbb Z^2$

let $S_{n}= \sum_{i=1}^{n}X_i$ be a simple random walk on $\mathbb{Z}$, with $S_0 = 0$. $X_i = 1$ with probability $p$ and $X_i = -1$ with probability $1-p$. It can be shown that ...
3
votes
1answer
26 views

Show that a matrix satisfying certain conditions is non-singular

I have a square matrix $A$ satisfying the following conditions: The elements on the diagonal are negative; All other elements are non-negative; All row sums are less than or equal to $0$; There is ...
-1
votes
1answer
17 views

Initial point and initial distribution of the Markov chains

I am reading about Markov chains on a general state space and the ergodicity theory. Some of the ergodic theorems are presented when we consider n-step transition probability conditional on initial ...
0
votes
1answer
21 views

Period of an irreducible Markov Chain is given by the number of eigenvalues with unit modulus

Suppose $\{X_n\}$ is an irreducible Markov Chain on finite state space $S$. Then, the number of eigenvalues of the transition matrix with unit modulus is precisely equal to the period of the chain. ...
1
vote
1answer
21 views

Markov chain - Expectation $+1$?

Let the transition matrix of a markov chain with states $\{0,1,2\}$ : \begin{equation} A=\begin{bmatrix} \frac{1}{3} & \frac{1}{3} & \frac{1}{3} \\ 0 & \frac{1}{2} & ...
0
votes
1answer
27 views

Stationary distribution vs invariant distribution of a Markov chain

Lets $p$ be a distribution on a finite sample space with $n$ points. I wish to find a transition matrix that is invariant with respect to $p$, that is $$p^T T = p^T$$. The problem is clearly ...
0
votes
0answers
34 views

Mixing time for lazy random walk on hypercube.

I am studying for a probability exam and am having trouble with the following exercise: Let $X_n$ be a lazy random walk on $\{0,1\}^d$ starting at $(0,\ldots,0)$ (stays put with probability ...
2
votes
1answer
38 views

Stability of a dimer on a square grid after $n$ random steps

On a white square grid there are two black cells. Each step consists of each of the cells 'moving' in one of the four directions with equal probability $p_0=1/4$ (a cell can't stay in the same place). ...
1
vote
0answers
24 views

When can an embedded Markov chain X for a Markov process Y be reducible?

It's pretty widely documented that a Markov process Y is reducible/irreducible if and only if the embedded Markov chain X is reducible/irreducible. However I'm not sure this works in reverse. I'm ...
0
votes
0answers
30 views

Modeling the joint distribution of stream statistics

I have a question regarding computing the joint discrete probability distribution of statistics in a number stream. I tried searching all previous posts to look at different forms of this problem ...
1
vote
1answer
473 views

Transition rate matrix from transition probability matrix

If I have a $2 \times 2$ continuous time Markov chain transition probability matrix (generated from a financial time series data), is it possible to get the transition rate matrix from this and if ...
1
vote
1answer
41 views

Expected steps in a Markov chain

I have two jars; initially one is empty while the other holds $n$ red and $n$ blue marbles. Every minute, I do the following procedure: take any pair of marbles of different colors but from the same ...
0
votes
0answers
43 views

Random walk with reflection and skips in linear system

Let's take a case of simple and linear Random walk (0, 1...n) with only one absorbing state n and reflecting state -1, which we can define as: P (move right at state i) = 1/2 and P (moving left at ...
3
votes
0answers
48 views

This is a Markov Chain?

Consider two irreducible ergodic Markov chains with the same state space $\{0, 1, . . . , N\}$, with transition matrices $P$ and $Q$ and respective stationary distributions $\pi$ and $\rho$. We ...
5
votes
0answers
44 views

Poisson: Conditional Probability on Pizza order

I am not sure about my answer. In particular, part b of the following question. Pizza orders arrive according to a Poisson process of rate 20 per hour. Orders are independently for a vegetarian ...
1
vote
1answer
25 views

M/M/1 queue derivation: how to “recursively solve in dependence on $p_0$”

I want to sketch out the derivation of the equations for an M/M/1 queue for a presentation I'm giving. I can understand most of the derivation from Willig but I don't understand this section from p10 ...
0
votes
0answers
28 views

Getting different results with two methods for a Markov Chain

Given the below Markov transition matrix, calculate $p_{0,1}^9$ \begin{matrix} 0.5 & 0.5 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \\ \end{matrix} Method ...
3
votes
1answer
34 views

Poisson: $P(N(4)-N(2)=5|N(4)=8)$

Let {N(t), t ≥ 0} be a Poisson process of rate 2. Determine: a) $P(N(4)-N(2)=5|N(4)=8)$ Attempt: The conditional poisson distribution is uniformly distributed between the interval [0,4]. Therefore, ...
1
vote
0answers
19 views

Harris Chain and Stationary Distribution

Consider a Harris chain given by $\{X_n\}$ with the following transition function, $X_{n+1}=\max \{0,X_n-b\} $ with probability $p$ and $X_{n+1}=\max \{0,a-\tau\} $ with probability $1-p$, where $\tau ...
2
votes
0answers
40 views

Quasistationary distribution for the Moran model

The Moran model is a model for genetic drift. Basically, it is a finite Markov chain (more precisely, a birth-death chain) with state space $S:=\{0,...,N\}$ and the following transition probabilites: ...
0
votes
0answers
52 views

What is the difference between “state-transition-matrix” and a transition matrix?

What's the difference between a state-transition-matrix and a transition matrix (say, for an ergodic Markov Chain) that is typically taught in a basic probability theory course? This is the first ...
2
votes
1answer
410 views

three-state Markov chain

a male and a female go to a $2$-table restaurant on the same day. each day the male sits at one or the other of the $2$ tables, starting at the table $1$, with a Markov chain transition matrix: ...
3
votes
1answer
36 views

Random walk on the positive integers with reflecting boundary

Consider a Markov chain $X$ on the positive integers where for each $n$: $$n\longrightarrow 1,\;2,\;3\;\dots \;n,\;n+1$$ with equal probability, and $n\longrightarrow m$ with zero probability if ...
1
vote
1answer
158 views

Probability of extinction in branching process

Consider a branching process where the offspring distribution is given by $$P(X = k) = \frac{1}{2^{k+1}}$$ what is the probability that the process becomes extinct at exactly at the $n$th generation? ...
0
votes
2answers
950 views

Branching Process Extinction Probability

I'm doing a branching process problem and am not sure I did it correctly. Suppose $X_0 = 1$ and $p_0 = .5, p_1 = .1,$ and $p_3 = .4$ represent the probabilities that zero, one, and three individuals ...
0
votes
1answer
17 views

Random Walk Markov Chain Long run distribution

In the question above do I have to calculate the stationary distribution? I've been learning about the ergodic theorem but I'm not sure if it's applicable here. I know that the probability that Xn ...
1
vote
1answer
29 views

Transition matrix and communicating classes

Firstly I wanted to check if they have a mistake in the solution. So for the transition matrix, the element p11 should be 0 and p12 should be 1, but they have it the other way round so I just ...
1
vote
1answer
29 views

Constructing transition graph from transition matrix

Ok so for this question I'm having trouble understanding how the transition graph has been drawn from the given transition matrix. This is what I understand and hopefully someone can correct the ...
0
votes
1answer
30 views

Computation of n-step transition matrix : method of matching coefficients

For the third step I don't understand how they have worked out $C_1^1=0$? where did they get the value of 0 from? p11(n)=(c01+c11n)lamda 1^n. Using p11(1)=1-a and c01=1, n=1,lamda 1=1 and lamba ...
0
votes
2answers
33 views

Soft Question - book recommendation - Stochastic Processes

My mother language book on stochastic processes is pretty much complete(~500 pages) but would like another one in English, to have in my library. I'm looking for a similar book containing the ...
1
vote
0answers
13 views

How to determine the transition probability in Sequential Importance Sampling (SIS) for Particle Filter

Given a state-space model \begin{align} x_k &= f_k(x_{k-1}, v_{k-1}),\\ z_k &= h_k(x_k, w_k), \end{align} where $x_k \in {\mathbb R}^{n}$ and $y_k \in {\mathbb R}^{m}$ are the system state ...
-2
votes
1answer
24 views

Markov's matrix into stationary Distribution [closed]

How do I know if this Markov's Transition Matrix converges into a stationary distribution? $$P= \begin{bmatrix} .8 & .2 & 0 \\ .3 & .4 & .3 \\ .2 & .1 & .7 \\ ...
0
votes
1answer
66 views

prove homogeneous markov chain

$Y_0, Y_1,Y_2,\dots$ are independent and identically distributed random non-negative integer outcomes. Let X_0 = Y_0 $ Let $X_0 = Y_0$ and $X_n = X_{n-1} - Y_n$ if $X_{n-1}>0$, else $X_n = ...