Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
4 views

Why does a Markov chain with one irreductible class has a lower triangular transition matrix?

Given a Markov chain on an infinite and countable set of states, with one irreductible class that has a finite number of states, why can its transition matrix be put in a lower triangular form ? ...
1
vote
0answers
76 views

Exercise on Markov chains

I'm preparing my Probability exam and I'm having trouble with exercise 2 here. The question is to consider the random walk on $E$ with transition matrix $p$ and find the communication classes (or ...
1
vote
0answers
14 views

Marcov Chain confirmation

I am currently having some problems on the following question: Given is the function $f(x)$: $f(x) = 0,1,2$ with probability $\frac{1}{3}$ for each. I have to give the state space, transition ...
1
vote
0answers
34 views

Construction of pure birth process

I am considering a Markov chain $\lbrace X(t) \rbrace_{t≥0}$ in continunous time on the countable state space $S=\lbrace 0 \rbrace\cup \lbrace (i,j) \mid i \in \mathcal{A} , j \in \mathbb{N} \rbrace, ...
1
vote
0answers
64 views

Explosion of a Markov chain

I am considering a Markov chain i continunous time on the countable state space $S=\lbrace 0 \rbrace\cup \lbrace (i,j) \mid i \in \lbrace A,B,C,D,E,F \rbrace , j \in \mathbb{N} \rbrace$. The ...
1
vote
0answers
56 views

How would I find this constant?

I have this equation, and I'm not sure how to solve for the constant $\nu$, since everything else is known: $$\begin{equation} a + \sqrt{a_i + 4 b_i \nu} + \sum^N_{j=1} (\sqrt{a_j + 4 b_j \nu}) ...
1
vote
0answers
65 views

Compute transition probability in n step in infinte markov chain

I want to calculate the probability of transition in n step from state 0 to state 0 ($p_{00}^{(n)}$) in below Markov-Chain : if self loop in state 0 doesn't exist, probability computed with Catalan ...
1
vote
0answers
19 views

Checking the closeness of probability distributions

Suppose I have a Markov chain that satisfies all the conditions of ergodicity and has a stationary distribution pi. I want to find the time when the probability distribution of the markov chain is ...
1
vote
0answers
39 views

Find condition on $X$ so that $P(\exists n\in\mathbb{N}: N_n=0)=1$

Let $X, X_{n,k}$ for $k,n\in\mathbb{N}$ denote independent random variables with values in $\mathbb{N}_0$. $X$ is identically distributed as all $X_{n,k}$. Define $N_0:=1$ and for ...
1
vote
0answers
52 views

Check if $(N_n)$ is a Markov chain

Let $X, X_{n,k}$ for $k,n\in\mathbb{N}$ denote independent random variables with values in $\mathbb{N}_0$. Define $N_0:=1$ and for $n\in\mathbb{N}$ set $$ N_n:=\begin{cases}0, & \text{ ...
1
vote
0answers
76 views

Kolmogorov backward equations for Birth-Death

I'm trying to solve the Kolmogorov backward equations for a Birth-Death Markov chain with three states. I have 2 equations: $$P_{00}'(t) = \lambda_0 (P_{10}(t)-P_{00}(t))$$ $$P_{10}'(t) = ...
1
vote
0answers
82 views

Chapman-Kolmogorov equations of time inhomogenous Markov chains

Let us assume that we are given a time inhomogenous Markov chain in continuous time (ICTMC) $(X(t))_{t \geq0}$ with a finite state space $\{1,\ldots,n\}$. Set $P(t)_{i,j} := \mathbb{P}(X(t) = j \mid ...
1
vote
0answers
26 views

Closed communicating class and stochastic matrix

Let $(X_n)_{n\in\mathbb{N}_0}$ be a Markov chain with state space $E$ and transition matrix $(p_{ij})_{i,j\in E}$. Let $C\subseteq E$ be a closed communicating class. Show that $$ ...
1
vote
0answers
35 views

Comparing frequencies in stationary distribution

Do there exist theorems for comparing frequencies in the stationary distribution of a (say) aperiodic, positive recurrent Markov chain? i.e. given the transition probability matrix $\mathbf{P}$ with ...
1
vote
0answers
27 views

Is there a single matrix norm such that for all stochastic $P$, $\| P \| = 1$?

By a stochastic matrix I mean a square real non-negative matrix with rows summing to one. Denote the set of all such matrices $\mathcal{S}$. By matrix norm I mean a norm in the vector space of ...
1
vote
0answers
32 views

Markov chains (which book can be recommended?)

This semester I am learning about Markov chains, mainly including basic definitions & properties Recurrence & Transience Perron-Frobenius Theory equilibrium states convergence to ...
1
vote
0answers
31 views

Random Walk and strong law

I want to prove that a Random Walk in 1 dimension is transient when $p\neq\frac{1}{2}$ but i want to prove it by the strong law of large numbers, so i have this: Define a random variable $$X_i = ...
1
vote
0answers
143 views

Can we find a correlation between states of a Markov chain?

I have a fair bit of knowledge on Markov chains but I recently wondered if there is a way to find out a correlation between the states of a finite Markov chain. I could not find any material on this. ...
1
vote
0answers
50 views

Linear Filtering Problem (Keynman Fac/Particle Model)

$lienar Filtering Problem $$X_n^1 = X_{n-1}^1 + \epsilon_n *W_n $$ $$X_n^2 = (1-\alpha* \delta) X_{n-1}^2 + \beta*\delta X_n^1 $$ $$X_n^3 = X_{n-1}^3 + \delta*X_n^2$$ above is $$\approx$$ $$dX_n^1 ...
1
vote
0answers
51 views

Stochastic process using Markov chain (thief on the run!!)

I'm given an exercise where we are to simulate a thief escaping from an officer. The thief (let's call him/her T for simplicity) and an officer (O) have four cities to be in. Let's call the cities A, ...
1
vote
0answers
102 views

Decomposing a stochastic matrix into a product of stochastic matrices.

It is well-known that any square real matrix of small rank $k$ can be decomposed into a product of a skinny matrix with $k$ columns and a fat matrix with $k$ rows by means of an SVD. This question is ...
1
vote
0answers
19 views

Calculate ultimate survival when more than 1 survival curve is needed to determine outcome

I would like to know if it is possible to combine multiple survival curves via an equation (e.g., via matrix multiplication or whatever) rather than stepping through multiple equations. E.g., assume ...
1
vote
0answers
50 views

Cereal boxes - Mean time spent in transient states

Problem: A cereal company gives 2 images in each cereal box it has. There are a total of 5 images. Once a buyer have 5 images she wins a prize. No box contains 2 images that are the same. What is the ...
1
vote
0answers
62 views

Follow-up on solution to Markov process equation

I asked a question here about solving a system related to an absorbing Markov chain. I now have a variation where there are $m$ types (of student, job seeker, etc) each of which applies to ...
1
vote
0answers
30 views

A question about Markov

There is a continuous-time markov chain,and we know the probability transition matrix P.The time between 2 states can be formulated as a exponential distribution whose u is related to the 2 states.Now ...
1
vote
0answers
59 views

Use Hasting-Metropolis to generate a random element from a large complicated combinatorial set L

Let $P$ the set of all permutations $(x_1, \ldots, x_n)$ of numbers $(1, \ldots,10)$ and $L$ the subset of $P$ where $\sum_{j=1}^{n}jx_j> a$. To use Hasting-Metropolis algorithm I followed the ...
1
vote
0answers
34 views

Finding a probability measure

Could someone helpme with this problem? First, consider the transition kernel in $\mathbb{R^2}\times B(\mathbb{R})$ given by $K(x,A)=U_{S^1}(A-x)$. We can than define an other kernel in ...
1
vote
0answers
29 views

Markov chains mixing time

Informally, the mixing time of a Markov chain is the time it takes to reach “nearly uniform” distribution from any arbitrary starting distribution. What does it mean by nearly uniform? I hope some one ...
1
vote
0answers
56 views

A Central Limit Theorem to Markov Chains

I am looking for some textbook or paper that treats this question: Let be $X_{1}, X_{2}, \ldots$ the random variables from a Markov Chain (MC). Is there any Central Limit Theorem (CLT) envolving ...
1
vote
0answers
16 views

Birkhoff-Neumann like result for stochastic matrices?

during my research I came along a nice lemma which looks like a Birkhoff-Neumann-theorem result, but in a version for stochastic matrices. Namely, I have: Lemma. Let $M$ be a stochastic matrix, then ...
1
vote
0answers
33 views

Changes in the transition matrix of a Markov chain

In most or all Markov chain theories that I know of assumes that the transition matrix does not change over time. But what if certain changes are expected to occur at certain times in the transition ...
1
vote
0answers
83 views

Problem with stochastic processes book - should I switch.

I've been reading "Essentials of Stochastic Processes" (second edition) by "Richard Durrett" and I quite liked it, it's a nice size book and it's very easy to read. However, and this is quite a big ...
1
vote
0answers
51 views

iterates of generalized matrix system

This may be somewhat of an underspecified question but I'll nonetheless give it a try. In the context of applied work, I've recently come across systems of the form $$ ...
1
vote
0answers
80 views

References for time-inhomogeneous Markov jump processes?

In some central models in life insurance mathematics, the state of the insured is modeled using a continuous-time time-inhomogeneous Markov process with finitely many states. While many results for ...
1
vote
0answers
84 views

Continuous transition kernel for Markov Chains

I am trying to show that the stationary distribution for a Markov Chain on a continuous state space can be obtained by building a transition density kernel, which obeys the detailed balance rule where ...
1
vote
0answers
31 views

Analysis of Steady State Probability for Markov Process

I have a balance equation, representing a Markov Chain, which yields $$ (K - z) \pi(Z_c = z) = (\lambda_c + (z+1)x) \pi(Z_c = z+1) $$ where K is the maximum state of the server. The term $\lambda_c$ ...
1
vote
0answers
212 views

How to solve this simple 2D Markov chain

I have 2D Markov chain like below : I have already solved for $p_{b,0}$ and $p_{0,d}$ like this : $$p_{b,0} = \rho^bp_{0,0}$$ $$p_{0,d} = \rho^dp_{0,0}$$ But I don't know how to get $p_{b,d}$. So ...
1
vote
0answers
48 views

markov chain computation

I consider a 2 state Markov chain: $X = \{1,2\}$, transitions are $M(i,j)$ and the matrix has a unique stationnary distribution $\pi$: $$ \pi(1) = \frac{M(2,1)}{2-M(1,1) - M(2,2)} \\ ...
1
vote
0answers
15 views

Are there any models that have mean $\sqrt{t\log(t)}$?

R. Arratia (The Motion of a Tagged Particle in the Simple Symmetric Exclusion System on Z) shows in theorem 2 that for step initial condition in the SSEP, the position of the lead particle, $x_1(t)$ ...
1
vote
0answers
121 views

How's the damping factor in Google PageRank algorithm calculated

I'm doing some researches about Google's PageRank algorithm for my thesis, I've found that the damping factor x (for example), where x is in : P` = x.P + (1-x)Q where P is the original ...
1
vote
0answers
26 views

showing a condition implies convergence to invariant distribution

For an arbitrary Markov chain, I'm trying to show that $\lVert{ P_{\mu}\circ \theta_n^{-1} - \nu \rVert}\to 0$ iff $\sum_j \lvert{ P_{ij}^n - \nu(j)\rvert}\to 0$, where $\theta_n^{-1}$ is a shift ...
1
vote
0answers
117 views

Markov Process Feller Construction

I have this assignment question and I am stumped on how to complete a Feller construction: A system consisting of two components is subject to a series of shocks . The time be- tween consecutive ...
1
vote
0answers
24 views

How to get from this probability formula to the one I need?

I'm working on a gambler's ruin problem where a player starts out with $i$ money, and 'winning' is when their total money reaches $N$ (ie they will keep playing until they reach N or run out of money, ...
1
vote
0answers
15 views

condition for recurrence of the chain

Let $\{X_n\}$ be an irreducible Markov chain with transition probability $P=(p_{ij})$ on a countable state space $S=\{0,1,2,\dots\}$.Suppose $s\in S$.Show that $s$ is a recurrent state if the there is ...
1
vote
0answers
59 views

Parametric transition matrix in Markov Chains

I am trying to model a discrete-time MC with transition probabilities that depend on some function of parameters i.e $p_{ij} = f(X_0,X_1)$. Suppose we take a log-linear model where $p_{ij} = ...
1
vote
0answers
70 views

Markov chain question

Consider an irreducible, recurrent Markov Chain ($X_n$) on a countable state space $S$ with transition probability $p(x,y).$ Pick a sigma-algebra $A \subset S$ and let $T_k=\inf\{n>T_{k-1}:X_n \in ...
1
vote
0answers
135 views

Markov Chains: Limiting probabilities of positive recurrent states sum to one?

I have a question about Markov chains. I am trying to understand the proof of Proposition 2.6 of http://www.columbia.edu/~ks20/stochastic-I/stochastic-I-MCII.pdf. The setting is: we have a positive ...
1
vote
0answers
61 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
1
vote
0answers
40 views

period of product markov chain

Consider $Z_n := (X_n,Y_n)$ where $(X_n)_{n\in \mathbb{N}}$ and $(Y_n)_{n\in \mathbb{N}}$ are irreducible markov chains with periods $\lambda$ and $\mu$. We know that $(Z_n)_{n\in \mathbb{N}}$ is a ...
1
vote
0answers
36 views

Prove equilibrium theorem without irreducibility and aperiodicity

I have to solve the following question: Consider a random walk Markov chain on $S = \{1, 2, \ldots, 100\}$. If the chain is between 2 and 99, it selects one of the adjacent states with equal ...