Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
47 views

Linear Filtering Problem (Keynman Fac/Particle Model)

$lienar Filtering Problem $$X_n^1 = X_{n-1}^1 + \epsilon_n *W_n $$ $$X_n^2 = (1-\alpha* \delta) X_{n-1}^2 + \beta*\delta X_n^1 $$ $$X_n^3 = X_{n-1}^3 + \delta*X_n^2$$ above is $$\approx$$ $$dX_n^1 ...
1
vote
0answers
46 views

Stochastic process using Markov chain (thief on the run!!)

I'm given an exercise where we are to simulate a thief escaping from an officer. The thief (let's call him/her T for simplicity) and an officer (O) have four cities to be in. Let's call the cities A, ...
1
vote
0answers
88 views

Decomposing a stochastic matrix into a product of stochastic matrices.

It is well-known that any square real matrix of small rank $k$ can be decomposed into a product of a skinny matrix with $k$ columns and a fat matrix with $k$ rows by means of an SVD. This question is ...
1
vote
0answers
18 views

Calculate ultimate survival when more than 1 survival curve is needed to determine outcome

I would like to know if it is possible to combine multiple survival curves via an equation (e.g., via matrix multiplication or whatever) rather than stepping through multiple equations. E.g., assume ...
1
vote
0answers
47 views

Cereal boxes - Mean time spent in transient states

Problem: A cereal company gives 2 images in each cereal box it has. There are a total of 5 images. Once a buyer have 5 images she wins a prize. No box contains 2 images that are the same. What is the ...
1
vote
0answers
61 views

Follow-up on solution to Markov process equation

I asked a question here about solving a system related to an absorbing Markov chain. I now have a variation where there are $m$ types (of student, job seeker, etc) each of which applies to ...
1
vote
0answers
29 views

A question about Markov

There is a continuous-time markov chain,and we know the probability transition matrix P.The time between 2 states can be formulated as a exponential distribution whose u is related to the 2 states.Now ...
1
vote
0answers
59 views

Use Hasting-Metropolis to generate a random element from a large complicated combinatorial set L

Let $P$ the set of all permutations $(x_1, \ldots, x_n)$ of numbers $(1, \ldots,10)$ and $L$ the subset of $P$ where $\sum_{j=1}^{n}jx_j> a$. To use Hasting-Metropolis algorithm I followed the ...
1
vote
0answers
34 views

Finding a probability measure

Could someone helpme with this problem? First, consider the transition kernel in $\mathbb{R^2}\times B(\mathbb{R})$ given by $K(x,A)=U_{S^1}(A-x)$. We can than define an other kernel in ...
1
vote
0answers
26 views

Markov chains mixing time

Informally, the mixing time of a Markov chain is the time it takes to reach “nearly uniform” distribution from any arbitrary starting distribution. What does it mean by nearly uniform? I hope some one ...
1
vote
0answers
47 views

A Central Limit Theorem to Markov Chains

I am looking for some textbook or paper that treats this question: Let be $X_{1}, X_{2}, \ldots$ the random variables from a Markov Chain (MC). Is there any Central Limit Theorem (CLT) envolving ...
1
vote
0answers
15 views

Birkhoff-Neumann like result for stochastic matrices?

during my research I came along a nice lemma which looks like a Birkhoff-Neumann-theorem result, but in a version for stochastic matrices. Namely, I have: Lemma. Let $M$ be a stochastic matrix, then ...
1
vote
0answers
31 views

Changes in the transition matrix of a Markov chain

In most or all Markov chain theories that I know of assumes that the transition matrix does not change over time. But what if certain changes are expected to occur at certain times in the transition ...
1
vote
0answers
71 views

Problem with stochastic processes book - should I switch.

I've been reading "Essentials of Stochastic Processes" (second edition) by "Richard Durrett" and I quite liked it, it's a nice size book and it's very easy to read. However, and this is quite a big ...
1
vote
0answers
51 views

iterates of generalized matrix system

This may be somewhat of an underspecified question but I'll nonetheless give it a try. In the context of applied work, I've recently come across systems of the form $$ ...
1
vote
0answers
70 views

References for time-inhomogeneous Markov jump processes?

In some central models in life insurance mathematics, the state of the insured is modeled using a continuous-time time-inhomogeneous Markov process with finitely many states. While many results for ...
1
vote
0answers
73 views

Continuous transition kernel for Markov Chains

I am trying to show that the stationary distribution for a Markov Chain on a continuous state space can be obtained by building a transition density kernel, which obeys the detailed balance rule where ...
1
vote
0answers
30 views

Analysis of Steady State Probability for Markov Process

I have a balance equation, representing a Markov Chain, which yields $$ (K - z) \pi(Z_c = z) = (\lambda_c + (z+1)x) \pi(Z_c = z+1) $$ where K is the maximum state of the server. The term $\lambda_c$ ...
1
vote
0answers
163 views

How to solve this simple 2D Markov chain

I have 2D Markov chain like below : I have already solved for $p_{b,0}$ and $p_{0,d}$ like this : $$p_{b,0} = \rho^bp_{0,0}$$ $$p_{0,d} = \rho^dp_{0,0}$$ But I don't know how to get $p_{b,d}$. So ...
1
vote
0answers
47 views

markov chain computation

I consider a 2 state Markov chain: $X = \{1,2\}$, transitions are $M(i,j)$ and the matrix has a unique stationnary distribution $\pi$: $$ \pi(1) = \frac{M(2,1)}{2-M(1,1) - M(2,2)} \\ ...
1
vote
0answers
15 views

Are there any models that have mean $\sqrt{t\log(t)}$?

R. Arratia (The Motion of a Tagged Particle in the Simple Symmetric Exclusion System on Z) shows in theorem 2 that for step initial condition in the SSEP, the position of the lead particle, $x_1(t)$ ...
1
vote
0answers
108 views

How's the damping factor in Google PageRank algorithm calculated

I'm doing some researches about Google's PageRank algorithm for my thesis, I've found that the damping factor x (for example), where x is in : P` = x.P + (1-x)Q where P is the original ...
1
vote
0answers
26 views

showing a condition implies convergence to invariant distribution

For an arbitrary Markov chain, I'm trying to show that $\lVert{ P_{\mu}\circ \theta_n^{-1} - \nu \rVert}\to 0$ iff $\sum_j \lvert{ P_{ij}^n - \nu(j)\rvert}\to 0$, where $\theta_n^{-1}$ is a shift ...
1
vote
0answers
86 views

Markov Process Feller Construction

I have this assignment question and I am stumped on how to complete a Feller construction: A system consisting of two components is subject to a series of shocks . The time be- tween consecutive ...
1
vote
0answers
24 views

How to get from this probability formula to the one I need?

I'm working on a gambler's ruin problem where a player starts out with $i$ money, and 'winning' is when their total money reaches $N$ (ie they will keep playing until they reach N or run out of money, ...
1
vote
0answers
15 views

condition for recurrence of the chain

Let $\{X_n\}$ be an irreducible Markov chain with transition probability $P=(p_{ij})$ on a countable state space $S=\{0,1,2,\dots\}$.Suppose $s\in S$.Show that $s$ is a recurrent state if the there is ...
1
vote
0answers
58 views

Parametric transition matrix in Markov Chains

I am trying to model a discrete-time MC with transition probabilities that depend on some function of parameters i.e $p_{ij} = f(X_0,X_1)$. Suppose we take a log-linear model where $p_{ij} = ...
1
vote
0answers
65 views

Markov chain question

Consider an irreducible, recurrent Markov Chain ($X_n$) on a countable state space $S$ with transition probability $p(x,y).$ Pick a sigma-algebra $A \subset S$ and let $T_k=\inf\{n>T_{k-1}:X_n \in ...
1
vote
0answers
127 views

Markov Chains: Limiting probabilities of positive recurrent states sum to one?

I have a question about Markov chains. I am trying to understand the proof of Proposition 2.6 of http://www.columbia.edu/~ks20/stochastic-I/stochastic-I-MCII.pdf. The setting is: we have a positive ...
1
vote
0answers
58 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
1
vote
0answers
284 views

Gradient-descent and Hidden Markov Models

I would like to use gradient-descent to fit the parameters of a simple 2-state HMM. This paper Levinson, S. E., Rabiner, L. R. and Sondhi, M. M. (1983), An Introduction to the Application of the ...
1
vote
0answers
39 views

period of product markov chain

Consider $Z_n := (X_n,Y_n)$ where $(X_n)_{n\in \mathbb{N}}$ and $(Y_n)_{n\in \mathbb{N}}$ are irreducible markov chains with periods $\lambda$ and $\mu$. We know that $(Z_n)_{n\in \mathbb{N}}$ is a ...
1
vote
0answers
33 views

Prove equilibrium theorem without irreducibility and aperiodicity

I have to solve the following question: Consider a random walk Markov chain on $S = \{1, 2, \ldots, 100\}$. If the chain is between 2 and 99, it selects one of the adjacent states with equal ...
1
vote
0answers
63 views

Two independent renewal processes

We have two urns (blue and red) that are connected, and two particles, $p_1$ and $p_2$, are traveling between these urns independently. The amount of time $Z_1$ that $p_1$ spends in blue urn is iid ...
1
vote
0answers
50 views

probability with exponential distribution

there is a barber shop with two waiting chair and one barber.suggest that,when a customer comes in,and two waiting chair are full,he goes to another barber shop. The time between entering of two ...
1
vote
0answers
62 views

A probability problem on Markov Chain

I am copying one example problem from Sheldon Ross which I am not understanding : Suppose we are given a set of $n$ elements, numbered $1$ through $n$, which are to be arranged in some ordered list. ...
1
vote
0answers
108 views

An urn probability problem

Suppose that $M$ balls are distributed among two urns and at each time point one of the balls is chosen at random, removed from its urn and placed in the other one. Now, in the long run the ...
1
vote
0answers
52 views

Is a deterministic path a Markov process

I have a question about what classifies as a Markov chain and what does not. Consider a system with state space $\left\{ 1,\ldots,n \right\}$ and a trajectory for the system defined by the following ...
1
vote
0answers
84 views

Rate of convergence of $n$-fold convolution over $\mathbb{Z}_p$.

Suppose that $\mu$ is a probabilist measure on $\mathbb{Z}_p$ such that $d_{TV}(\mu,\mu_U)\leqslant \delta < 1$. What are the best upper bound on $$ d_{TV}(\underbrace{\mu*\mu*\ldots*\mu}_{n ...
1
vote
0answers
108 views

First and second moments of recurrence time in a finite two-dimensional Markov chain

I have a two dimensional finite Markov chain with $(m+1)^2$ states, and with transition rates: $q_x((x,y)\to (x+1,y))=(m-x)\lambda,\quad 0\leq x< m, 0\leq y \leq m$, $q_x((x,y)\to ...
1
vote
0answers
47 views

Finding Steady state using markov chains. Am I right?

Suppose that there are two doctors in a country town, Dr Black and Dr White. Each year, 13% of patients move from Dr Black to Dr White, while 19% of patients move from Dr White to Dr Black. Suppose ...
1
vote
0answers
98 views

A matrix-multiplication random walk

Let $x \in \mathbb{R}^n$. Consider an $n\times n$ matrix $A$. Suppose we're interested in how $||A^nx||$ grows with $n$, the answer (excluding pathological cases) is that it scales exponentially with ...
1
vote
0answers
89 views

Stationary distribution behavior - Markov chain

I have modeled a process with a Markov chain with K+1 states which is irreducible and apperiodic. The transition matrix is a centrosymmetric matrix where all it's entries has a positive probability. ...
1
vote
0answers
82 views

Expected distance traversed between 2 vertices on probabilistic graph

Let $V = \{1,2,3,...,N\}$ be the vertex set of a graph. Let $d(i,j)>=0$ represent the $(i,j)$ vertex distance between vertices $i$ and $j$, $i \in V, j \in V$. Now, define a non-negative number ...
1
vote
0answers
90 views

Intuitive meaning of spectral radius of a Markov chain transition matrix?

What is the intuitive meaning of the eigenvalues and in particular of the spectral radius of the transition matrix corresponding to a Markov chain?
1
vote
0answers
58 views

Conditions for Markov chain

Let $\{X_n\}$ be a Markov chain with transition matrix $P$, and $Y_n := X_{m-n}$, $m\ge n$. Under what conditions is $\{Y_n\}_{n\ge 0}$ Markov chain? I stared by proving that conditional probability ...
1
vote
0answers
81 views

Markov kernels and update functions

I would like to prove, that for any Markov kernel $K$ on a Polish space $(F,\mathcal{F})$ (with a $\sigma$-field) you can find a measurable space $(S,\mathcal{S})$, a random element $Z$ on $S$ and an ...
1
vote
0answers
59 views

Gibbs / MCMC sampling for sum of parameters - how to improve slow mixing?

Suppose I have a hierarchical Bayesian model, where my observational prediction, $y'$, is calculated as the sum of other parameters, ${\alpha_i}$. My observation equation (the likelihood) is: $P(y | ...
1
vote
0answers
94 views

Markov chain weak convergence

consider a sequence of Markov chains $\Phi^{(n)}$whose transition kernel $P^{(n)}$ converges to $P$. Now let $\Phi$ be the Markov chain with the limiting kernel $P$. How do I show that $\Phi^{(n)}\to ...
1
vote
0answers
41 views

how to describe this case with markov-chain

I want to describe this case in markov chain: The case: Mr. Meier reads NYTimes everyday and puts the newspaper on news rack. His wife sometimes cleans the house(with prob $1/3$ each day) and throws ...