Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
10 views

Markov Model of Exponential State Transitions

I am trying to derive the stationary distribution of a system with 2 states, in which transitions between the two states occur as Poisson Processes. That is, if the only two edges are 0 -> 1 with ...
3
votes
1answer
50 views

A linear growth model with immigration

Ill give some background first before asking questions.(the text below is straight out of the book) Each individual in the population is assumed give birth at an exponential rate of $\lambda$ in ...
1
vote
0answers
46 views

Continuous transition kernel for Markov Chains

I am trying to show that the stationary distribution for a Markov Chain on a continuous state space can be obtained by building a transition density kernel, which obeys the detailed balance rule where ...
0
votes
1answer
21 views

“Number of passages from the state $i$”: a strange equality.

Consider a homogenenous Markov chain $\{X_n\,:\, n\in \mathbb N\}$ ($0\in\mathbb N$). The state space is $S$ with $|S|\le |\mathbb N|$ and $i\in S$. Consider moreover the function ...
0
votes
0answers
14 views

Mean time spent in transient states (markov)?

Book https://www.dropbox.com/s/qtcef6g03fuj3bh/Screenshot%202014-04-26%2019.57.42.png Where he writes "condition..." I'd like to have some more calculations; I do not understand that step.
0
votes
1answer
57 views

M/M/1 Queuing Theory Question

Lets say I have packets arrive to a terminal at Poisson rate $\lambda$ per hour and my terminal has an exponential service rate $\mu$ per hour (so the mean service time is $\frac{1}{\mu}$). So this is ...
2
votes
0answers
46 views

Markov Chain with Normal Transition Matrix

Consider a (sub)-stochastic matrix $P$, and the associated Markov chain $X$ with \begin{align*} \mathbf P [X_n =y | X_0 = x] = P_{xy}^n. \end{align*} Suppose we have the condition $P^T P = P P^T$, ...
0
votes
1answer
33 views

Why does markov chains power method converge at the rate of |λ_2/λ_1 |

I'm doing some researches on Markov Chains, and every time I meet this statement, that The rate of convergence of the power method is given by |λ_2/λ_1 |^k→0, when k→inf. And where λ_1 and λ_2 are ...
1
vote
1answer
34 views

Probability error

I perform $N$ independent trials with $M$ successes. The probability of success is therefore $P=M/N$. Can I assign a sample-size-dependent error to the probability based only on this information? i.e. ...
0
votes
0answers
23 views

Simple random walk problem, time homogeneity

Similar to a question I've asked before, consider a simple symmetric random walk, $\left\{X_n\right\}$, where the following hold; $$X_n = X_{n-1} + Y_n$$ for $n = 1, 2, 3,\ldots$ $$Y_1, Y_2, Y_3, ...
1
vote
0answers
24 views

Analysis of Steady State Probability for Markov Process

I have a balance equation, representing a Markov Chain, which yields $$ (K - z) \pi(Z_c = z) = (\lambda_c + (z+1)x) \pi(Z_c = z+1) $$ where K is the maximum state of the server. The term $\lambda_c$ ...
2
votes
1answer
27 views

How to get transition matrix of markov process?

I am monitoring a Markov process with ~21 states. I know all the states, initial state and what states transitions can/cannot be, so that zero elements of transition matrix are known. I know the ...
0
votes
0answers
24 views

Restarting a Markov chain

I'm reading an article and having difficulty understanding some basic stochastic processes (I'm new to the subject so please pardon my wording of the question). Let $S$ be a set of states and $Q$ be a ...
0
votes
2answers
49 views

Weather Transition Matrix

Based on observation, I've gathered some data: Day 1 2 3 4 5 6 7 8 9 10 S R S F S R F F R S (S) = Sunny (R) = Rainy (F) = Foggy How do I construct this into a transition matrix, for markov ...
0
votes
0answers
11 views

Hidden Markov Model to Markov Model

Can every HMM be converted into some $nth$ order Markov model? My thoughts are that you could just multiply the emission probabilities in each state by the probability of state transitions. For ...
0
votes
0answers
42 views

Markov Transition Matrix

I have some data, shown below. How do I construct a transition matrix, for Markov Chain ? I need the formula to calculate observation data into transition matrix. Thanks! Accumulative ...
1
vote
0answers
43 views

How to solve this simple 2D Markov chain

I have 2D Markov chain like below : I have already solved for $p_{b,0}$ and $p_{0,d}$ like this : $$p_{b,0} = \rho^bp_{0,0}$$ $$p_{0,d} = \rho^dp_{0,0}$$ But I don't know how to get $p_{b,d}$. So ...
0
votes
1answer
23 views

Computing the PMF for N(t) in a renewal process

I'm in a stochastic processes course, and we just started on renewal theory. Unfortunately, we skipped the section on queuing theory, and nearly example in my textbook for renewal processes uses ...
1
vote
1answer
32 views

Strict proof of markovity of queing system of type $M/M/n/\infty$

I have a queing system of type $M/M/n/\infty$. The service time is exponential, and the arrival process is poisson. I do understand that because of these two facts the future of the system in ...
1
vote
1answer
46 views

Markov Chain Ergodic Theorem (Proof references)

Where can I find a proof of the erogidc theorem for Markov chains that doesn't use Birkhoff? The theorem states the following : Let $(X_n)_{n\in \mathbb{N}}$ be an irreducible and positively ...
0
votes
0answers
22 views

Discrete Time Markov Chain Proof Question

My instructor stated this result without proof, and I don't know enough about Markov Chains to 'Google' the name of theorem, but if anyone has a reference or a method of proof I'd appreciate it. The ...
1
vote
1answer
30 views

$X_n, n> 0$ is a Markov Chain, how to interpret $Z_n = (X_n,X_{n+1}), n > 0$?

Am a newbie to Markov Chain. So, this might be incredibly naive/stupid question. If $X_n, \, n > 0$ is MC, am having difficulty imagining/interpreting process $Z_n = (X_n,X_{n+1}), n > 0$. I ...
0
votes
1answer
22 views

Recurrence of infinite markov chain

I have a Markov chain with state space $S=\{0,1,2...\}$ and a sequence of positive numbers $p_1,p_2,...$ where $\sum p_i=1$. The transition probabilities are based on these where $p(x,x-1)=1, ...
0
votes
1answer
9 views

First property of discrete time homogenous markov chain

I'm trying to understand the properties of a DTHMC. I am having trouble understanding with the first one. My textbook says - "$X_t$ takes values in $X$ for all $t$ (i.e. $X_t$ is a random variable ...
1
vote
0answers
33 views

markov chain computation

I consider a 2 state Markov chain: $X = \{1,2\}$, transitions are $M(i,j)$ and the matrix has a unique stationnary distribution $\pi$: $$ \pi(1) = \frac{M(2,1)}{2-M(1,1) - M(2,2)} \\ ...
0
votes
0answers
37 views

Markov Chain Geometric Convergence

Perhaps due to my non-mathematician background (lack of Measure Theory knowledge), I have some difficulties about the Markov Chain Theory. Given an ergodic (irreducible and aperiodic) Markov Chain ...
0
votes
1answer
26 views

Markov chain probability that a state changes

For the Markov chain given below what is the best way to find the probability P{$x_n = x_{n-1}$} and P{$x_n \neq x_{n-1}$} The transition matrix of the chain is \begin{array}{} 1/3 & 2/3 & ...
0
votes
1answer
54 views

M/M/1 queue with probability of new client leaving

I'm looking at a M/M/1 queue system and trying to show that $\{M_t\}_{t\geq}0$, the number of clients in the system, is a birth-death process. In the simplest of cases this is true if $\lambda_i = ...
-1
votes
1answer
18 views

radius of symmetric random walk on $\mathbb{Z}$

How to calculate the radius of the symmetric random walk on $\mathbb{Z}$, i.e. $\limsup_k (p^{(k)}(0,0))^\frac{1}{k}$? ($p^{(k)}(0,0)$ denotes the probability to get from $0$ to $0$ in $k$ steps and ...
1
vote
1answer
70 views

Proving that Markov Chain Monte Carlo converges

I am trying to understand how the very basic Markov Chain Monte Carlo approach works: We try to approximately calculate the expected value $E_{\pi(x)}[X]$ by drawing sequential samples from a Markov ...
1
vote
0answers
14 views

Are there any models that have mean $\sqrt{t\log(t)}$?

R. Arratia (The Motion of a Tagged Particle in the Simple Symmetric Exclusion System on Z) shows in theorem 2 that for step initial condition in the SSEP, the position of the lead particle, $x_1(t)$ ...
3
votes
0answers
42 views

Invariant measure of Euler-Maruyama Discretisation of an Ito diffusion

Let $(X_t)_{t \geq 0}$ be a diffusion process with dynamics governed by the stochastic differential equation \begin{equation} dX_t = b(X_t)dt + \sigma(X_t)dW_t, ~~ X_0 = x_0, \end{equation} where ...
0
votes
1answer
45 views

Conditional mutual information and Markov chain.

If we have the Markov chain $X \to Y \to Z$, or equivalently $$I(X;Z| Y)=0, \tag{1}$$ where $I(\cdot)$ denotes the mutual information. Does the Markov chain $X \to (Y,W) \to Z$ also hold? Or ...
1
vote
1answer
46 views

Stickers in a box Markov chain problem

I'm revising for exams in June and my university, very irritatingly, doesn't provide mark schemes for past questions. I'm stuck a few parts into a question and am not totally confident of my preceding ...
0
votes
0answers
27 views

Markov chain definition: should the conditional probability also hold if $P(X_{n} = i_{n}, \ldots, X_0 = i_0) = 0$? Is $S$ a set of real numbers?

Definition of Markov Chain (as it is stated in my textbook): Let $S$ be a set of states and $\mathbb P = \{p_{i,j}\}$, $i,j \in S$ a transistion matrix . Then the sequence of RV's $(X_{n})_{n \ge 0}$ ...
1
vote
1answer
62 views

rate of convergence of absorbing markov chain

Let $G$ be a biconnected and non-bipartite graph. I can simulate a random walk on this graph with a markov chain. The stochastic matrix is $M = AD^{-1}$, where $A$ is the adjacency matrix of $G$ and ...
0
votes
1answer
27 views

Steady state of a $4 \times 4$ transition matrix

Normally I just take $q(M_{m\times n} - I_{m\times n})$ to workout the steady state, but here I have: $$\left(\begin{array}{rrrr} 0 & 0 & .8 & .2 \\ .4 & .6 & 0 & 0 \\ .2 ...
0
votes
1answer
23 views

Limiting probability of Markov chain(Terminology)

If I am asked to find the limiting probability of a Markov chain, what does this pertain to? $\lim \limits_{n \to \infty} P^n$? Where $P$ is the stepping matrix and $n$ is the number of steps. "What ...
1
vote
0answers
49 views

How's the damping factor in Google PageRank algorithm calculated

I'm doing some researches about Google's PageRank algorithm for my thesis, I've found that the damping factor x (for example), where x is in : P` = x.P + (1-x)Q where P is the original ...
1
vote
0answers
25 views

showing a condition implies convergence to invariant distribution

For an arbitrary Markov chain, I'm trying to show that $\lVert{ P_{\mu}\circ \theta_n^{-1} - \nu \rVert}\to 0$ iff $\sum_j \lvert{ P_{ij}^n - \nu(j)\rvert}\to 0$, where $\theta_n^{-1}$ is a shift ...
0
votes
1answer
21 views

limiting probability - what matrix to write

To find the limiting probability you solve the systems of equations: $\vec{\pi}=P\vec{\pi}$ $\Sigma \pi_j = 1$ and my teacher told us "you could rewrite this as matrices". Having just completed a ...
1
vote
0answers
68 views

Markov Process Feller Construction

I have this assignment question and I am stumped on how to complete a Feller construction: A system consisting of two components is subject to a series of shocks . The time be- tween consecutive ...
2
votes
2answers
60 views

Transition Matrix eigenvalues constraints

I have a Transition Matrix, i.e. a matrix whose items are bounded between 0 and 1 and either rows or columns sum to one. I would like to know if it is possible that in any such matrices the ...
1
vote
1answer
79 views

Mean time spent in transient states/Markov chain

I dont get this in my book: For transient states $i$ and $j$ , let $s_{ij}$ denote the expected number of time periods that the markov chain is in state $j$ , given that it starts in state $i$. Let ...
4
votes
0answers
46 views

Markov chains and natural filtration

I have the following problem Consider a homogeneous Markov chain $(X_n)$ with countable state set $E$. Suppose that $A$ is a proper subset of $E$ and consider the stopping times $\tau^0=0 $ and ...
0
votes
1answer
24 views

How many iterations are generally required when using the power iteration method?

Suppose I have an n x n matrix and I want to find the dominant eigenvalue and its associated eigenvector. Given these dimensions, what is the minimum number of iterations of the power iteration ...
1
vote
0answers
22 views

How to get from this probability formula to the one I need?

I'm working on a gambler's ruin problem where a player starts out with $i$ money, and 'winning' is when their total money reaches $N$ (ie they will keep playing until they reach N or run out of money, ...
0
votes
0answers
16 views

Markov chains - Proof of how to check recurrent states

Question 1 I read a proof of how to check recurrent states. There is one = sign that I do not understand, see the image. ...
3
votes
0answers
43 views

Standard deviation of a quantum walk?

The standard deviation of a classical random walk with $n$ steps is $\sqrt n$ - Standard deviation of a random walk. I have read in many places that the standard deviation of a quantum walk $n$ with a ...
0
votes
1answer
36 views

Markov chains for group decision making

I am new to Markov chains since I am doing my own studying on it recently. I was doing some questions and came across this one that got me stuck. Suppose there are four employees and they need to ...