Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
17 views

Is there a way to “normalise” probability values of Markov chain transitions for comparison?

Suppose I have a series of states and I've a database of frequencies (probabilities) of the states transiting from one to another. I've a set of states ${A...G}$. Let's say I've a state transition (...
0
votes
0answers
18 views

Stationary distribution of finite-state Markov chain in terms of determinants/products of eigenvalues

I have an $M$-state continuous-time Markov chain with transition-rate matrix $K$ (the column sums are zero), which has $M$ distinct eigenvalues $\lambda_i$, $i=1,\dots,M$. $\lambda_M=0$, so $K$ has ...
0
votes
1answer
32 views

Equilibrium distribution exponentially fast

I need to prove that for an aperiodic, irreducible Markov Chain $X_n$ with stationary distribution $\pi$ holds that $P_x[X_n=j]\to\pi(j)$ exponentially fast. I found some proof of that statement but ...
1
vote
0answers
93 views

Probability of losing lotteries needed

A person earns $x_i$ amount of money every month where $x_i$ is an exponential random variable with parameter $\lambda_1$. The amount $x_i(1-p)y$, here $0 \leq p\leq 1$ and $y$ is exponentially ...
2
votes
1answer
40 views

Bayesian Estimator and Markov Chains

This is Exercise 6.1.14 from Dembo's notes found here. At this point, we are just beginning a discussion of Markov chains. I have no prior experience with estimators and so I am a bit lost with this ...
0
votes
1answer
28 views

Check that stopping time is a.s. finite

I have the following situation. Let $(X_i)_{i\geq1}$ be a sequence of iid random variables in $\mathbb{Z}$ and consider the random walk $S_n=\sum_{i=1}^n{X_i}$, $S_0=x$. Let $y>x$ and consider ...
2
votes
1answer
52 views

Find the mean number of steps in a Markov chain

Let $S = \{1,2,3,4\}$ be a state space like this $$\begin{array}\\ 1 & - & 2 \\ | & & |\\ 3 & - & 4 \end{array} $$ and let $P$ be the transition matrix given by $$P = \begin{...
0
votes
0answers
33 views

Error term in the definition of the transition rates of a continuous time Markov chain

I'm studying G.F.Lawler's stochastic process book. There he defines the transition rates $\alpha(x,y)$ from the state $x$ to state $y$ (the state space is countable) of a continuous time Markov chain $...
0
votes
1answer
38 views

Stopping times of random walk with time dependent absorbing barriers

I have a Bern$(p)$ random walk ($Y_i = 1$ with probability $p$ and Y_i = 0 with $1-p$) with two absorbing boundaries, $A: Y^i \leq t_i$ and $B:Y^i \geq d_i-t_i$. Now, both $d_i$ and $t_i$ are evolving ...
0
votes
2answers
46 views

Recurrence Relation with two parameters and Summation

This is a recurrence relation with two parameters which came up in a problem I was trying to solve. Given $$\begin{align}&A_n=pB_{n-1};\qquad &&B_n=q(A_{n-1}+B_{n-1})\\ &A_4=p; \...
0
votes
0answers
26 views

HMM limiting distribution

Consider a hidden markov model (HMM) with two hidden states $A$ and $B$ and emission support $1$ and $2$ fitted with initial state distribution $$\lambda = [\begin{array}{cc} .7&.3\end{array}]$$ ...
0
votes
1answer
28 views

Long-run fraction Markov Chains

A machine has three critical parts (1,2,3) but can function as long as two of these parts are functional. When two are broken, they are replaced and the machine is functional the next day. The state ...
1
vote
2answers
38 views

General Two-State Markov Chain: $P(X_{n}=1)=\frac{b}{a+b}+(1-a-b)^n \big(P(X_0=1)-\frac{b}{a+b}\big)$

Consider a general chain with the state space $S=\{1,2\}$ and write the transition probability as $$\begin{pmatrix} 1-a&a\\ b&1-b\end{pmatrix}$$ Use the Markov property to show that $$P(X_{n}=...
0
votes
0answers
38 views

Lebesgue Integral vs. Lebesgue Stieltjes Integral

Forgive me if this has been addressed in a question already on here (and for my lack of comfort with measure theory), but is there any difference between the Lebesgue integral and the Lebesgue-...
0
votes
0answers
23 views

implementing particle filters without a priory distribution

i am implrmrnting the particle filter, and i have some problem understanding the algorithm. given the state equations: $$ x_k = f(x_{k-1},v_k) $$ $$ z_k=h(x_k,u_k) $$ where $v_k, u_k$ are process ...
1
vote
1answer
27 views

Let $\mathbf{X}$ be a Markov chain on a square find $p_{1,1}(n)$

Consider a square like this $$\begin{array}\\ 1 & - & 2\\ | & & |\\ 3 & - & 4 \end{array} $$ such that you can go from each state with chance $\tfrac{1}{2}$ to the ...
0
votes
1answer
33 views

Why is this Markov chain's stationary distribution not (1/2, 0, 0, 1/2)?

I have the Markov chain 1, 0, 0, 0 1/2, 0, 1/2, 0 0, 1/2, 0, 1/2 0, 0, 0, 1 I understand how to build the system with which I am supposed to find that ...
1
vote
2answers
34 views

Help on some results with Markov chains

I currently have a markov chain represented by the following matrix: $$\left[\begin{array}{ccc} 0&1&0 \\ .99&0&.01 \\ 0&0&1 \end{array}\right],$$ i.e. a row-stochastic matrix, ...
2
votes
1answer
37 views

How does a Markov process inherit its homogeneity to the embedded Markov chain?

A homogenous Markov process $\lbrace X(t),t\geq 0\rbrace $ is given and the embedded Markov chain $Y_0,Y_1,\ldots$ is defined as $Y_n:=X(T_n)$, where the $0=T_0<T_1<\ldots$ are the moments where ...
0
votes
0answers
17 views

Started Counts method for estimating transition probabilities of a discrete time markov chain

I would be very pleased if you could help me with a problem I'm having for my Bachelor's thesis. I'm working on some inventory forecasting methods and one of the method's I'd like to apply is a method ...
3
votes
0answers
32 views

Distribution on number of revisits in past $k$ steps of Markov chain

Consider a finite-state Markov chain with transition matrix $P$. The chain starts in a state chosen uniformly over all the states and runs indefinitely from there. We're going to examine only the $k ...
1
vote
1answer
19 views

Conditioning on invariant sigma algebra with respect to ergodic measure

So this question arose to me while applying the Ergodic theorem. If $X$ is a finite state (in $ \{1,\dots,d\}$) continuous-time Markov chain, which is ergodic, then $X$ has a unique invariant ...
2
votes
0answers
58 views

Is this transformation of a Markov process again Markovian?

Let $(X_t)_{t\in\mathbb{N}_0}$ be a stationary Markov process valued in $\mathbb{R}$ and $c\in\mathbb{R}$. Is the process $(Y_t)_{t\in\mathbb{N}_0}$ defined by $$ Y_t={\bf 1}{(X_t<c)} $$ again a ...
1
vote
1answer
48 views

Doubly stochastic matrix positive recurrent?

Let $\{X_n, n \ge 0\}$ be a discrete-time markov chain with a doubly stochastic transition matrix $P$ and a finite state space $S$. Prove that all states in $S$ are positive recurrent. My work: It is ...
0
votes
1answer
36 views

Proving specific formula for stationary markov process [closed]

In my probability class, right now we are dealing with Markov chains and I was stumbled by parts of this problem: Given a $ \{ X_n \}_{n=0}^{\infty} $ be a homogeneous Markov chain (the transition ...
1
vote
0answers
32 views

Statement of the strong Markov property in Norris' book

In J.R.Norris' Markov chains book, the strong Markov property for discrete-time, Markov chains is stated and proved as follows: Let $(X_n)_{n \geqslant 0}$ be a Markov chain with transition ...
1
vote
1answer
58 views

On the Markov chain defined by $X_n=U_nU_{n+1}$, where $(U_n)$ is i.i.d. symmetric Bernoulli

I came across this problem in homework: $U_n$ are i.i.d random variables with $P[Un=1]=P[Un=−1]=0.5$. a) Show that $X_n=U_nU_{n+1}$ is a Markov Chain. b) Show that $X_n=(U_n+U_{n+1})/2$ is not a ...
0
votes
1answer
33 views

About the expected transitions in Markov Chain

The problem is here: The given answer is here: K = $2+ X_1 + X_2$, where $X_1$ and $X_2$ are independent exponential random variables with parameters $2/3$ and $3/5$. $$ E[K] = 2=2+1/p_1 +1/p_2 = ...
0
votes
1answer
17 views

Finding Transition Probabilities using Metropolis Hastings

I want to find the $4$x$4$ Probability Transition Matrix under the temperature parameter T=2 of Metropolis Hastings. I know that, if x and y are neighbors, $p(x,y) =$ $$ f(x) = \left\{ \...
1
vote
2answers
60 views

Given irreducible transition matrix $P$, why does the matrix $(P−I|\mathbb{1})(P−I|\mathbb{1})^T$ have full rank?

Given irreducible transition matrix $P$, why does the matrix $(P−I|\mathbb{1})(P−I|\mathbb{1})^T$ have full rank? I know this is partially due to the fact that since $P$ is irreducible, there exists ...
3
votes
0answers
31 views

How to handle Finite-state-machine with correlated inputs?

My system can be represented by the following state-diagram. where each arch represents Input/Output when a transition is made from one state to the other. The inputs to this FSM are correlated. ...
0
votes
1answer
28 views

Show that if a Markov chain is irreducible and has a state $s_i$ such that $P_{ii}>0$, then it is also aperiodic.

Show that if a Markov chain is irreducible and has a state $s_i$ such that $P_{ii}>0$, then it is also aperiodic. Proof: Let $X=(X_0, X_1, X_2, \dots)$ be an irreducible Markov chain with a state ...
0
votes
0answers
8 views

Ergodicity coefficient of block matrix

I have a stochastic matrix of the following form \begin{equation} X=\begin{bmatrix}A/3&B/3&C/3\\I_n&&\\&I_n&&\\\end{bmatrix}, \end{equation} where $A,B,C$ are all $n$ by $n$...
1
vote
1answer
29 views

Finding the mean given the probability

I'm doing some work on branching processes and would like to know where the process becomes extinct. If $X$ is the number of offspring of an individual, then the process goes extinct when $\mathbb{E}[...
3
votes
0answers
48 views

Hitting probabilities in a random walk on a graph

Consider a random walk $(X_n)$ on the graph below, where we jump from a given vertex to one of its adjacent vertices with equal probability. I want to find: the probability that we hit $A$ before ...
0
votes
2answers
30 views

Show that this MC is ergodic?

Suppose I have a Markov Chain with States, $S = {1,2,3,4}$ and a PTM given by $P =$ $\begin{pmatrix} .25 & .25 & .25 & .25 \\ 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0\\ 0 &...
0
votes
1answer
66 views

Markov Chain Transition Matrix Question

Ok, so my question is pretty simple, the question states: A spider web is only big enough to hold 2 flies at a time. Assuming that the flies fly into the web independently: -The probability that no ...
1
vote
1answer
22 views

Strong Markov Property for Markov Chains - Statement Verification

I suspect that my handwritten lecture notes for the Strong Markov Property are wrong. I'd appreciate corrections to them. We first define the following: A random variable $\tau$ is called a ...
0
votes
0answers
9 views

Markov Chain: Aperiodicity => Primitivity

Hellooo, I would like to know how I can show that the transition Matrix $P$ of an aperiodic Markov chain is primitive. Any suggestions?
0
votes
0answers
15 views

Markov chain nulls

hope the question is ok for this forum. I am a developer and not a mathematician but realise your group is likely to know the answer for these questions. The background is that I am writing a program ...
1
vote
1answer
30 views

Limiting Distribution of a Gibbs Distribution

I know that the Gibbs distribution at a particular state, x, is given by $\frac{e^{-\beta E_i}}{\sum_j e^{-\beta E_j}}$ with $\beta = \frac{1}{T}$, but I do not understand what a limiting distribution ...
0
votes
1answer
19 views

Induction proof for a stochastic process.

Let $(X_n)_{n\in\mathbb{N}}$ be a Markovchain. How can I then show following equation for all $ n \in \mathbb{N}$, $ \displaystyle\bigcup_{k=1}^n \lbrace X_k = j \rbrace = \biguplus_{k=1}^n \lbrace ...
0
votes
1answer
32 views

Finding Initial state vector with given values

I am not sure how to use a given values to form a initial state vector. There are 30% of customers from Company A switch to company B every month, and 35% customers from Company B switch to company A ...
0
votes
0answers
24 views

Correlation Matrix Question

Why is this not a possible correlation matrix for any three random variables X, Y, and Z? $\begin{pmatrix} 1 & -1 & -1 \\ -1 & 1 & -1\\ -1 & -1 & 1\end{pmatrix}$
0
votes
0answers
30 views

Partition Theorem and Markov Chains

Suppose a Markov chain has $s$ states, $S = {1, 2, . . . , s}$, with PTM $P =$ ($p_{ij}$). That is, $p_{ij} = P[X_{n+1} = j | X_n = i]$. Use the Partition Theorem to verify that if $X_n ∼ ν$, then $X_{...
2
votes
1answer
19 views

Simple Bayes? Probability of a state at time t in hidden markov model

Suppose we have a HMM with $2$ states -- $A$ and $B$, with $P(A) = 0.4$ and $P(B) = 0.6$. $A$ has a probability of $0.9$ of outputting "hot," and $B$ has a probability of $0.1$ of outputting "hot." ...
0
votes
0answers
14 views

Analysis of incomplete system of diferential equations

I need to find information about the kinetics of a reaction. I tried to solve this problem first generalizing the equations for the different kind of reactions yielding an equation like: $$ \dot{x} =...
3
votes
1answer
40 views

Probability that two random walks on $\mathbb{Z}^2$ meet at the origin

Suppose $X,Y$ are symmetric, independent random walks on the lattice $\mathbb{Z}^2$. I am trying to find the probability: $$\mathbb{P}\big(X_n=Y_n=(0,0)\;\text{for some}\;n\,\big|\,X_0=Y_0=(0,N)\big)$$...
0
votes
0answers
23 views

MCMC and Metropolis-Hastings problem(s)

What does it mean for a particular state to be a "ground" state or a "stable" state? I should make clear that this is final exam review material and not homework. Also, how does one compute a Gibbs ...
0
votes
1answer
66 views

A Question Regarding Markov Chains and Ergodicity

Suppose the Markov chain with Probability Transition Matrix, $P$ = ($p{_x}{_y}$) is ergodic and $p{_m}(x, y) > 0$ for all states $x$ and $y$. If $n ≥ m$, show that $p_n(x, y) > 0$ for all ...