Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
0answers
44 views

How to sample the walk which visits each vertex of a graph specific number of times?

Is there any MCMC mathod that allow me to uniformly sample from all feasible walks where the following restrictions apply: ...
1
vote
0answers
64 views

Positive recurrence of a continuous-time jump process, from its jump chain

I am looking at an irreducible, continuous-time jump process $(X_t)_{t\geq0}$ with the following jump times. Let a Poisson process $(T_i)_{i=1}^\infty$ determine the event times. With probability ...
1
vote
0answers
147 views

Continuous-time Markov process - need help with flux and discrete-time equivalent

I have a continuous-time markov process and I need to calculate the following transition frequencies matrix (aka intensity matrix) transition probabilities all parameters which define permanence ...
1
vote
0answers
79 views

Quasi-stationary distribution of a state in a birth-and-death MC

I need to find an expression for the first state in an MC with transition matrix $P$ with tridiagonal entries. The state space is $U={1,2,..n}$ with the last state being absorbing. Expressions for ...
1
vote
0answers
50 views

Green kernel of Hunt processes

Let $X$ be a regular Hunt process on $R^+$ with starting at $x$ and $T_y :=\inf\{t>0: X_t=y\}, y\in R^+$. $G_q(\cdot,\cdot), q >0$ denotes Green kernel of the process $X$. We have the following ...
1
vote
0answers
159 views

Markov Chains - Using Gibbs & Metropolis algorithm.

Suppose $f_x,_y$ is bivariate normal distribution. I was given the parameters $(μ_1, μ_2, σ_1^2, σ_2^2)$ and $ρ=0.95$ the correlation coefficient. I want to generate $(x_1,y_1), ...
1
vote
0answers
21 views

How many observations is the minimum?

I want to estimate model transition matrix for a process (Markov chain). How much observiations of state do I need? I would prefer this as a function dependant of n, where n is number of possible ...
1
vote
0answers
36 views

Integrating with respect to an MCMC Kernel

I'm currently trying to evaluate an integral with respect to an MCMC kernel, and I wanted someone else to read what I `proved' to see if it's really correct. It just doesn't feel right. Suppose I ...
1
vote
0answers
126 views

Cesaro mixing time: Show $t_m(2^{-k}) \le k t_m(1/4), k \ge 1$

Let $(X_t)_{t \ge 0}$ be a finite Markov chain with state space $\Omega$, transition matrix $P$ and stationary distribution $\pi$. Let $\| \cdot \|$ denote the total variation distance and define  ...
1
vote
0answers
105 views

Irreducible Markov: harmonic function based on stationary distribution

Let $P$ be the transition matrix of an irreducible Markov chain on a finite state space $\Omega$. Let $\pi_1$ and $\pi_2$ be two stationary distributions for $P$. Is the function $$h(x)={\pi_1(x) ...
1
vote
0answers
119 views

Expectation of an event

Let $S[4]$ be a binary array with elements of $S$ are taken uniformly and independently from $\{0,1\}$. Also take $k$ uniformly from $\{0,1\}$. Take $i=1$. Now run the following process: Take $a,b$ ...
0
votes
0answers
5 views

Computing the PMF for N(t) in a renewal process

I'm in a stochastic processes course, and we just started on renewal theory. Unfortunately, we skipped the section on queuing theory, and nearly example in my textbook for renewal processes uses ...
0
votes
0answers
10 views

Discrete Time Markov Chain Proof Question

My instructor stated this result without proof, and I don't know enough about Markov Chains to 'Google' the name of theorem, but if anyone has a reference or a method of proof I'd appreciate it. The ...
0
votes
0answers
22 views

Markov Chain Geometric Convergence

Perhaps due to my non-mathematician background (lack of Measure Theory knowledge), I have some difficulties about the Markov Chain Theory. Given an ergodic (irreducible and aperiodic) Markov Chain ...
0
votes
0answers
12 views

radius of symmetric random walk on $\mathbb{Z}$

How to calculate the radius of the symmetric random walk on $\mathbb{Z}$, i.e. $\limsup_k (p^{(k)}(0,0))^\frac{1}{k}$? ($p^{(k)}(0,0)$ denotes the probability to get from $0$ to $0$ in $k$ steps and ...
0
votes
0answers
11 views

Gibbs sampler for MCMC is equilibrium distribution

Consider a Gibbs sampler, as in https://en.wikipedia.org/wiki/Gibbs_sampling, which we use to generate a Markov chain $x^0, x^1, x^2,...$ which samples from a distribution $\pi$. If we call ...
0
votes
0answers
25 views

Markov chain definition: should the conditional probability also hold if $P(X_{n} = i_{n}, \ldots, X_0 = i_0) = 0$? Is $S$ a set of real numbers?

Definition of Markov Chain (as it is stated in my textbook): Let $S$ be a set of states and $\mathbb P = \{p_{i,j}\}$, $i,j \in S$ a transistion matrix . Then the sequence of RV's $(X_{n})_{n \ge 0}$ ...
0
votes
0answers
19 views

Discrete Markov Chain: probabilities

I'm confused about these: steady-state transition probabilities limiting probabilities stationary probabilities how are they different? I know the question is pretty vague, but I feel like I'm ...
0
votes
0answers
20 views

How to get from this probability formula to the one I need?

I'm working on a gambler's ruin problem where a player starts out with $i$ money, and 'winning' is when their total money reaches $N$ (ie they will keep playing until they reach N or run out of money, ...
0
votes
0answers
11 views

Markov chains - Proof of how to check recurrent states

Question 1 I read a proof of how to check recurrent states. There is one = sign that I do not understand, see the image. ...
0
votes
0answers
11 views

radius of markov chain

For an irreducible markov chain one can show, that $\limsup \sqrt[n]{p_{ij}^{(n)}}$ is independent of the choice of the states $i$ and $j$ where $p_{ij}^{(n)}$ is the probability to get from $x$ to ...
0
votes
0answers
14 views

Sapling/ counting order ideals

Is there anything known about sampling or counting ordered ideals in Posets of special cases?
0
votes
0answers
16 views

Random Walk Prediction on a grid using markov chain

We have a m*n Grid and n no of robots in that grid which would perform a random walk simultaneously.Each robot can move in 4 direction specifically Up, Left, Down, Right. After x steps the random ...
0
votes
0answers
22 views

Transition matrix, stationary distribution and expected number

A company wants to operate s identical machines, but they are subject to failure according to a given probability law. To replace them, the company orders new machines at the beginning of each week to ...
0
votes
0answers
41 views

Expected time to reach 6th return time of Markov Chain

I'm having a hard time figuring out this problem from Resnick's Adventures in Stochastic Processes: Harry is negotiating a new tv show and the negotiations follow a discretely indexed Markov chain. ...
0
votes
0answers
8 views

How are Markov chains used in simulated annealing?

How are Markov chains used in simulated annealing? Is it only that the cooling scheme can provide ergodicity?
0
votes
0answers
61 views

proving null recurrence of random walk (Markov chain)

How would I prove that the zero state of a random walk with a positive probability of staying in the same state is null recurrent. (sorry if this isn't a random walk and just a Markov chain.) eg. ...
0
votes
0answers
4 views

How can I calculate distribution of minima of sections of a continuous path (from a stochastic process)?

I have a long slab whose width is defined by a stochastic process, whose complete statistics I am aware of, say. I now cut it into smaller sections of uniform length, and calculate the minimum width ...
0
votes
0answers
107 views

understanding submartingale proof with discrete state space

I am reading a text about branching markov chains: My question is about the first half of page 8 where $Q(t)$ is proven to be a submartingale. Briefly the used notation: $t$ is discrete time, $n(t)$ ...
0
votes
0answers
9 views

Hidden Markov Model Confidence Interval (preferably in MATLAB)

I'm trying to uncover the transition parameters of data of a hidden Markov Model using MATLAB. Using the built in hmmtrain function, I can estimate the parameters quite well (I already know what they ...
0
votes
0answers
10 views

References for non-homogeneous continuous-time Markov chains

In one applied problem that I'm trying to solve, I want to apply nonhomogeneous continuous-time Markov chains. But cannot find a good reference on these kind of chains. I mean with simple worked-out ...
0
votes
0answers
28 views

Continuous time Markov chains, how would this definition be expanded from time-homogeneous to time-inhomogeneous.

Below I have a picture of how we can view a continuous time Markov chain that is time-homogeneous. Now, I am wondering what happens when we have a inhomogeneous continuous Markov chain. I have ...
0
votes
0answers
26 views

Markov renewal process vs Markov Jump process

The venerable wikipedia for "Markov renewal process" says that: "a Markov renewal process is a random process that generalizes the notion of Markov jump processes". So what's the definition of a ...
0
votes
0answers
27 views

Normalizing a co-occurrence matrix with an energy function to create a conditional random field or markov random field.

I currently have a set of factors produced for a co-occurrence matrix : link I want to be able to move towards developing a conditional random field for pixel labelling and multi-class segmentation. ...
0
votes
0answers
30 views

Continuous time Markov chain

I would like to know if I am on a right track? Continuous time Markov chain on Wikipedia A very new European “Rapid Reaction Force for Fire” has been created today and begins operation between three ...
0
votes
0answers
41 views

Approximating a Markov process by differential equations

I have a system of states, $m_S = 1, 0, -1$. After performing a certain manipulation (it can be assumed to be instantaneous), a transition can happen with probability p. However, not all states can ...
0
votes
0answers
19 views

Can an irreducible countable state Markov chain have transient states?

I've been studying Markov chains, and I came up with a question: Can an irreducible countable state Markov chain have all transient states? I know the fact that for finite-state Markov chains, there ...
0
votes
0answers
19 views

Poisson processes are the only renewal processes which are Markov Chains.

How would one prove the Proposition: "Poisson processes are the only renewal processes which are Markov Chains." A renewal process $N=(N(t))$ is a process for which $$N(t)=\max\{n : T_n \leq t\}$$ ...
0
votes
0answers
18 views

Computing standard errors using EM algorithm

I'm applying the EM algorithm to a hidden markov chain (the $\mathbf{Z}=\{Z_1,...,Z_n\}$ variable), with observations(the $\mathbf{Y}=\{Y_0,...,Y_n\}$ variable) dependent not only on the hidden markov ...
0
votes
0answers
24 views

Convergence of sequence of stationary distributions of Markov chains

I have a sequence of finite, discrete-time ergodic Markov chains indexed by a parameter $N$, and I want to prove that their stationary distributions are converging to a well-defined limit as $N\to ...
0
votes
0answers
34 views

Probability distribution after n-steps with different initiation state in Markov chain

The transition matrix at n-th time step for a discrete time Markov chain with $ S = \{1, 2, 3, 4\} $is given as below: $$ P(n) = \pmatrix{0 & 0.6 & 0.4 & 0 \\ 0.8 & 0 & 0 & ...
0
votes
0answers
22 views

Calculation of the Gallavotti-Cohen fluctuation theorem made by Lebowitz

I have a problem understanding a calculation in this paper (another form of the theorem an be found here at equation 11). For those who want to read the paper, I have difficulties with formula 2.14 in ...
0
votes
0answers
23 views

Is a CTMC recurrent or positive recurrent?

Let $(X_t)_{t\geq 0}$ be the continuous-time Markov chain on $\mathbb{Z}$ with transition rates: $$\begin{align}q_{i,i-1} &= i^2 + 1\\ q_{i,i} &= -2(i^2 + 1)\\ q_{i,i+1} &= i^2 + ...
0
votes
0answers
209 views

Gradient-descent and Hidden Markov Models

I would like to use gradient-descent to fit the parameters of a simple 2-state HMM. This paper Levinson, S. E., Rabiner, L. R. and Sondhi, M. M. (1983), An Introduction to the Application of the ...
0
votes
0answers
27 views

References for Hidden Markov Chains

I'm looking for some nice introductions to Hidden Markov Chains. Preferably some that begin from the basic definitions. I would like some of these references to be papers published in journals. Any ...
0
votes
0answers
37 views

Is this two dimensional Markov chain correct for this queueing system?

The problem that I have two single server station with no queuing space a customer goes to station 1 if it is available else it goes to station 2 if it is available or it will be lost output from ...
0
votes
0answers
19 views

propability of largest samples of repeated random divisions

I have a bunch of numbers $A_1=\{a_{1,1},\dots,a_{n,1}\in\mathbb R^+\}=\{1,\dots,1\}$, that get multiplied by independent uniformly $[0,1]$ distributed samples, e.g. $a_{i,2}=X_ia_{i,2}$. This process ...
0
votes
0answers
26 views

Combine transition and fertility matrices for youngest stage groups

I would like to ask how to combine the following information into the projection matrix. I do have data for transition (T) and fertility (F) stage matrices, so the projection matrix (A) is equal to ...
0
votes
0answers
58 views

transition matrix for Markov chain

Can any one help me to solve this home work please? The city of Sacramento recently completed a new light rail system to bring commuters and shoppers into the downtown area and relieve freeway ...
0
votes
0answers
31 views

Does time homogeneity imply strong Markov property in a Markovian process

Does a time homogeneous Markovian process necessarily have strong Markovian property? Does continuity in state space, time, or path make a difference? What are the examples if it does not?