Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Classification of states and stationary distribution in a continuous Markov chain

My question is about a continuous-time Markov chain $ X_{t} $ with state space $E=\{0,1,2\}$ and infinitesimal matrix: $$Q=\begin{pmatrix} -6 & 6 & 0\\ 1 & -7 & 6\\ 0 & 2 & -2 ...
0
votes
0answers
24 views

How to calculate the following probability with two Markov chains?

I have two independent Markov chains at discrete time $X_{n}$ y $Y_{n}$, and I know the transition probabilities of both. I would like to know an easy way to calculate probabilities of the type: $$P(...
0
votes
2answers
22 views

Invariant measures that aren't reversible (or vice versa)?

It seems like every discussion of reversible Markov chains assumes that the measure is invariant. A reversed chain has the transition probabilities $p'$ satisfying $$ p'_{xy} = \frac{\pi(y)}{\pi(x)} ...
1
vote
1answer
24 views

no stationary distribution with none-vanishing limit of transition probability

We know that irreducible Markov chains can be separated into the two cases: (1) All limits of transition probabilities vanish: $lim_{n\rightarrow\infty} p_{ij}^{(n)} = 0$ for all i,j in state space S. ...
1
vote
2answers
36 views

Generating function, stopping time and Markov chain

Suppose we have a simple random walk $ X_{n} $ and let $ f $ be the generating function of $ T_{0} = \min \{n \geqslant 0: X_{n} = 0 \} $ starting at $1$, that is $f(x)=\mathbb{E}_{1}(x^{T_{0}}):=\...
1
vote
0answers
18 views

Computation of domain of infinitesimal generator of Brownian motion

Let $L:C_{0}(\mathbb{R}) \supseteq D(L) \rightarrow C_{0}(\mathbb{R})$ be the generator of the standard one dimensional Brownian motion. I calculated that $\{f \in C_{0}(\mathbb{R}):f'' \in C_{0}(\...
1
vote
2answers
55 views

Calculating probability for exponentials

Let $T_1, T_2$ be exponentials with rate $\lambda_1, lambda_2$. We want to find $P\left(T_{1}<T_{2}+T\right)$. I did: $P\left(T_{1}<T_{2}+T\right) = \int_0^\infty P(T_1 < t + T) f_{T_2}(t) dt ...
2
votes
1answer
39 views

First time two independent Markov chains reach same state

Let $\{X_n\}$ and $\{Y_n\}$ two independent Markov chains with the same state space $\{0,1,2\}$ and same transition probability matrix: $$ P=\begin{pmatrix} 0.5&0.25&0.25\\ 0.25&0.25&0....
0
votes
1answer
37 views

About random walks, stopping times and Markov chains

Suppose we have a simple random walk $ X_{n} $ and let $ f $ be the generating function of $ T_{0} = \min \{n \geqslant 0: X_{n} = 0 \} $ starting at $1$, that is $f(x)=\mathbb{E}_{1}(x^{T_{0}}):=\...
0
votes
0answers
22 views

Birth and death chains and Markov property

My problem is about a birth and death string $ X_{n} $ with $ P_{00} = 1 $, $ P_{i, i + 1} = p_{i} $ and $ P_{i, i- 1} = 1-p_i $. They ask me to find $ h_{i} = P(T_{0} < \infty | X_{0} = i) $, ...
0
votes
1answer
53 views

Simple Random Walk, Generating Function and Markov Property

My problem is about a simple random walk $ X_{n} $ with transition probabilities $ P_{i,i + 1} = p $ and $ P_{i, i-1} = 1-p $. I must prove that $ [\mathbb{E}_{1}(s^{T})]^2 = \mathbb{E}_{2}(s^{T}) $, ...
1
vote
0answers
21 views

Right continuity of the potential

Let $X=(X_t)_{t \geq 0}$ be a Hunt process with transition semigroup $(P_t)_{t \geq 0}$. We define for $\alpha >0$ and a bounded Borel-function $f$ the potential $U^{\alpha}f(x)=\int^{\infty}_0 e^{-...
0
votes
0answers
8 views

stationary vector for an unbreakable markov chain with period 3

I need to find an unbreakable markov chain with period 3 on all the natural numbers such that it's stationary vector $\Pi =(\pi_0,\pi_1,\ldots)$ follows: $\pi_1 = \pi_2 = 1/3$ my attempt was that $0$ ...
4
votes
2answers
131 views

Random Walk Markov Process — Probability of Return to the Origin

I'm struggling with a problem I was asked by a friend a few days ago. It goes as follows: You start at the origin. On the first iteration you walk right with probability $p$ and left with probability ...
2
votes
0answers
17 views

PGF and the Kolmogorov backward equation

$(N_t,t\ge 0)$ is a poisson process with rate $\lambda$ Using the Kolmogorov backward equation, find the PGF: $$G_i(z,t)=\Bbb E[z^{N_t}|N_o=i]$$ The answer is $G_i(z,t)=e^{\lambda t(z-1)}$ but I am ...
0
votes
0answers
16 views

Initial Probability Distribution of a Markov Chain

c. If the initial probability distribution is $Pr [𝑋_0 = 𝑖] = 1/ 3; i= 1,2,3.$ Find the probability distribution of $𝑋_1.$ d. Suppose the process begins at in state $𝑋_0 = 1.$ Find the probability ...
0
votes
0answers
26 views

Harmonic function, Markov chain and martingales

I must prove that if $ X_{n} $ is a Markov chain and $ S $ is countable (at most) then $ h(X_{n}) $ is martingale, where $ h $ is harmonic, that is $ h(x) = \sum_{y \in S} P_{xy} h(y) $ (my chain is ...
1
vote
0answers
38 views

Conditional expectation of a function of an Itô-process.

Assume that $S_t$ is an Itô-process with dynamics $dS_t=\mu(S_t,t)dt+\sigma(S_t,t)dB_t,$ where $\mu,\sigma$ are jointly Borel measurable and are such that $|\mu(x,t)|+|\sigma(x,t)|\le K(1+|x|)$, $|\mu(...
-1
votes
0answers
12 views

Interpretation of the probability kernel in the jump process

The infinitesimal generator of a pure jump process with kernel J(x, y) is given by $$Lu(x)=\int J(x, y)(u(y)-u(x))dy$$. Which is the interpretation of the function $J(x, y)$? In which sense it is the ...
1
vote
1answer
43 views

Continuous Time Markov Chain to find probabilities

I got part a, since the rows must sum to 1 $a = 2$ and $b = 6$. I am confused on b,c, and d. I would love some hints on how to approach these parts of the problem. For b, I was thinking that $P(X_{4.7}...
0
votes
0answers
13 views

Best way to build Markov model with multivariate data

Most of Markov model examples has only on variate variable for on point of time. Famous weather example for each data point has information about one state [sunny, rainy, cloudy]. From this We should ...
2
votes
1answer
34 views

Modified M/M/1 Queue

I have queueing system in which the arrival rate is Poisson with rate $\lambda$ and the service times are exponentially distributed with rate $\mu > \lambda$. The twist in this problem that a user ...
0
votes
1answer
24 views

one-point-compactification of a Polish space is a Polish space?

Is it true that the one-point-compactification of a Polish space is again a Polish space? I am currently learning something about Feller Processes and I think at some point this is needed.
0
votes
0answers
15 views

Markov property implies that expression ???

My professor has told us that Markov property means that: Let $\{X_n\}_n$ be a markovian sequence in a countable stte space $X$ then $\{Y_n\} = X_{n+k}$ is also a markovian sequence with the same ...
1
vote
0answers
26 views

Mean time to go from state i to state j in following discrete-time Markov chain (DTMC)

Let the following discrete-time Markov chain (DTMC) be defined with its transition probabilities as follows: $$ \begin{aligned} &p_{0,0}=\frac{b_{0}}{b_{0}+b_{1}}, \quad p_{i, i+1}=\frac{b_{i+1}}{...
0
votes
0answers
5 views

Discounted occupancy of Markov Chain states

Assuming a Markov Process with states $S$, transition probabilities in the form $p(s_{t+1} | s_t)$ and discount factor $\gamma$, i.e. at any step there is a probability $p(\emptyset | s_t) = 1-\gamma$ ...
1
vote
0answers
20 views

Cover time in Markov Chain from transition matrix

Given a process on a graph $X_{n} = \{x_{1}, ..., x_{n}\}$, is there a way to obtain the cover time, starting at any state $x_{i}$, from the transition matrix $\mathbf{P}$? I've obtained the expected ...
1
vote
2answers
27 views

Markov Chain Derivation question

This page defines the Markov Property as the following. Does anyone how form (b) becomes (c)? Is it due to $(X_{n-1} = i_{n-1}) \subset (X_{n-1} = i_{n-1}) \cap ... \cap (X_0 = i_0)$? Is it due to ...
0
votes
0answers
14 views

Continuous-time Markov Chain Long term probability

A Markov chain (Xt)t≥0 on {1, 2, 3, 4} has generator matrix Q: $$ \begin{matrix} -2 & 1 & 1 & 0 \\ 1 & -3 & 1 & 1 \\ 2 & 2 & -4 & 0 \\ 1 & 2 ...
0
votes
1answer
110 views

Stationary distribution defined using Riemann-Stieltjes integrals

Apologies for a long post ahead. I encountered a theorem from a 1975 paper (Theorem 1) on the existence of a unique stationary limiting distribution, defined using a sequence of monotone non-...
0
votes
0answers
73 views
+50

Brownian Motion has generator $\frac{1}{2} f''$

For $t \geq 0$ let $P_{t}(x,\cdot)=N(x,t)$ be the transition kernels of a one dimensional Brownian Motion. Then $(\bar{P}_{t})_{t \geq 0}$ with $ \bar{P}_{t}:C_{0}(\mathbb{R},\mathbb{R}) \rightarrow ...
1
vote
1answer
20 views

Relation between the strong Markov property of a process and the strong Markov property of the associated canonical process on the path space

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space; $(E,\mathcal E)$ be a measurable space; $\pi_I$ denote the projection from $E^{[0,\:\infty)}$ onto $I\subseteq[0,\infty)$ and $\pi_t:=...
0
votes
1answer
26 views

Stationary Measures For Markov Chains

I would like to know why in theory of Markov Chains we are always interested in to know (if it exits) the stationary measure (s) of the Markov Chains ? Could you list some results that clarify the ...
1
vote
0answers
20 views

Master equation of a continuous-time continuous-space linear random walk

A particle continuously moves in $\mathbb{R}^+ = \{x \in \mathbb{R}, x\ge 0\}$. It starts from $0$ at time $0$, and it has a restart probability defined by a pdf on $\mathbb{R}^+$. If it is at $0$ at ...
1
vote
1answer
63 views

Exponential decay of tail probability for Brownian stopping time

A previous question, Exit Time of an Interval Brownian Motion - Distribution, asked about the tail probabilities for exit times from a region $(-a,b)$. In particular, the question was about ...
0
votes
1answer
33 views

Markov Chain - Absorption

I am interested in learning about Markov chains, for that I am doing the following exercise and I am generating the following questions: I have the following matrix of one-step transition ...
0
votes
0answers
33 views

State space for Markov chain.

I'm trying to do this exercise from Probability by John Walsh: Let ${X_{n}, n = 0,1, 2, ... }$ be a Markov chain whose transition probabilities MAY NOT be stationary. Define $X'_{n}$ to be the n-tuple ...
1
vote
1answer
30 views

Solution of general Dirichlet problem

Let $(E,\mathcal E)$ be a measurable space, $\kappa$ be a Markov kernel on $(E,\mathcal E)$ and $X_I$ denote the projection from $E^{\mathbb N_0}$ onto $E^I$ for $I\subseteq\mathbb N_0$. We know that ...
1
vote
0answers
32 views

Derivation formula for piecewise-deterministic Markov process.

I was reading a formulation of piecewise-deterministic Markov process $\Pi_t$, $ t \in \mathbb{R}_+$. In particular $\Pi_t$ is defined as $\Pi_t = P (X_t | \mathcal{F}_{\lfloor t/ \Delta \rfloor}) $, ...
2
votes
1answer
64 views

Markov Chains and Stochastic Stability Tweedie Notation

I am trying to understand the notation used in the MC and Stochastic Stability book. Specifically, on page 63. If I have a markov chain $\Phi$ defined on $(\Omega,\mathcal{F},P_{\mu})$ where $\Omega$=$...
2
votes
0answers
19 views

Why is $E[T_A] = \sum_{y \in X} P[X_1=y|Xo=x]E[T_A|X_0 =x, X_1 =y]$

I am studying a proof of a theorem in stohastic processes and there is this part thas confuses me : $E[T_A] = \sum_{y \in X} P[X_1=y|Xo=x]E[T_A|X_0 =x, X_1 =y]$ where $T_A = inf\{k \geq 0: X_k \in A\}$...
1
vote
1answer
60 views

Martingale Problem - Markov process

Let $X$ be a right continuous Markov process with left limits and generator $L$. Why is $f(X_t)-f(X_0) - \int Lf(X_s) ds$ a martingale for every $f \in D(L)$? Let s<t. $E^x[M_t^f|F_s]= E^x[f(X_t)-f(...
0
votes
0answers
19 views

Markov change average number of transitions

I am self studying Markov chain to prepare for an exam, and I would like to verify my answer to a question. Given a Markov chain with 3 states, whose transition matrix is given by $P=\begin{pmatrix} ...
1
vote
0answers
21 views

on Markov chains and limits of probability distributions

I was faced by this problem a while ago and I can't think how to do it: Let ${e_{1},\dots,e_{n}}$ be a finite set of states and $P = p_{ij}$ a stochastic matrix. Suppose that there is a constant $\...
2
votes
2answers
103 views

Is the period of an irreducible Markov Chain the same for all states?

I'm a bit confused about this theorem about irreducible Markov Chains: If a Markov chain is irreducible, then all its states have the same period $d(i) := g.c.d.\{n > 0|P^n(i, i) > 0\}$.[source]...
0
votes
0answers
21 views

Finite irreducible Markov chain and it elements of stationary distribution [duplicate]

Finite irreducible Markov chain with stacionary distribution $\pi=\left(\pi_i\right)_{i=1}^n$ has all $\pi_i\neq0$ for $i=1,2,\dots,n$. It is true? If yes how to prove it. I know that there is only ...
2
votes
0answers
21 views

Interpretation of the Sinkhorn-Knopp algorithm applied to a (singly stochastic) transition matrix of a Markov process?

Say I have a discrete-time Markov process (and let's say discrete states too, for simplicity). If $\mathbf p_t$ is a vector of probabilities over states at time $t$, then the probability distribution ...
1
vote
0answers
24 views

Understanding Yosida approximation of operators

Definition 129 in these notes states: The Yosida approximation to a semigroup $K_t$ with generator $G$ is given by $$ K_t^\lambda := e^{tG^\lambda}$$ $$ G^\lambda := \lambda GR_\lambda = \lambda (\...
1
vote
1answer
49 views

Probability a given markov chain is in two different states at two different times

Let us say that our state space $S = \{1, 2, 3, 4\}$ Now let us say our transition matrix $P$ is given by: \begin{bmatrix} 1/2 & 1/2 & 0 & 0 \\ 1/3 & 0 & 1/3 & 1/3 \\ ...
1
vote
1answer
53 views

Bounds for finite Markov chains

Problem Let $G$ be a strongly connected directed graph with $n$ vertices, and let $d(v)$ denote the out-degree of vertex $v$. Let $M_G$ be a finite discrete-time homogeneous Markov chain defined over $...

1
2 3 4 5
42