# Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

2,080 questions
Filter by
Sorted by
Tagged with
10 views

22 views

37 views

8 views

### stationary vector for an unbreakable markov chain with period 3

I need to find an unbreakable markov chain with period 3 on all the natural numbers such that it's stationary vector $\Pi =(\pi_0,\pi_1,\ldots)$ follows: $\pi_1 = \pi_2 = 1/3$ my attempt was that $0$ ...
131 views

### Random Walk Markov Process — Probability of Return to the Origin

I'm struggling with a problem I was asked by a friend a few days ago. It goes as follows: You start at the origin. On the first iteration you walk right with probability $p$ and left with probability ...
17 views

### PGF and the Kolmogorov backward equation

$(N_t,t\ge 0)$ is a poisson process with rate $\lambda$ Using the Kolmogorov backward equation, find the PGF: $$G_i(z,t)=\Bbb E[z^{N_t}|N_o=i]$$ The answer is $G_i(z,t)=e^{\lambda t(z-1)}$ but I am ...
16 views

### Initial Probability Distribution of a Markov Chain

c. If the initial probability distribution is $Pr [𝑋_0 = 𝑖] = 1/ 3; i= 1,2,3.$ Find the probability distribution of $𝑋_1.$ d. Suppose the process begins at in state $𝑋_0 = 1.$ Find the probability ...
26 views

### Harmonic function, Markov chain and martingales

I must prove that if $X_{n}$ is a Markov chain and $S$ is countable (at most) then $h(X_{n})$ is martingale, where $h$ is harmonic, that is $h(x) = \sum_{y \in S} P_{xy} h(y)$ (my chain is ...
38 views

13 views

### Best way to build Markov model with multivariate data

Most of Markov model examples has only on variate variable for on point of time. Famous weather example for each data point has information about one state [sunny, rainy, cloudy]. From this We should ...
34 views

### Modified M/M/1 Queue

I have queueing system in which the arrival rate is Poisson with rate $\lambda$ and the service times are exponentially distributed with rate $\mu > \lambda$. The twist in this problem that a user ...
24 views

### one-point-compactification of a Polish space is a Polish space?

Is it true that the one-point-compactification of a Polish space is again a Polish space? I am currently learning something about Feller Processes and I think at some point this is needed.
15 views

### Markov property implies that expression ???

My professor has told us that Markov property means that: Let $\{X_n\}_n$ be a markovian sequence in a countable stte space $X$ then $\{Y_n\} = X_{n+k}$ is also a markovian sequence with the same ...
26 views

110 views

### Stationary distribution defined using Riemann-Stieltjes integrals

Apologies for a long post ahead. I encountered a theorem from a 1975 paper (Theorem 1) on the existence of a unique stationary limiting distribution, defined using a sequence of monotone non-...
73 views
+50

26 views

### Stationary Measures For Markov Chains

I would like to know why in theory of Markov Chains we are always interested in to know (if it exits) the stationary measure (s) of the Markov Chains ? Could you list some results that clarify the ...
20 views

### Master equation of a continuous-time continuous-space linear random walk

A particle continuously moves in $\mathbb{R}^+ = \{x \in \mathbb{R}, x\ge 0\}$. It starts from $0$ at time $0$, and it has a restart probability defined by a pdf on $\mathbb{R}^+$. If it is at $0$ at ...
63 views

### Exponential decay of tail probability for Brownian stopping time

A previous question, Exit Time of an Interval Brownian Motion - Distribution, asked about the tail probabilities for exit times from a region $(-a,b)$. In particular, the question was about ...
33 views

### Markov Chain - Absorption

I am interested in learning about Markov chains, for that I am doing the following exercise and I am generating the following questions: I have the following matrix of one-step transition ...
33 views

### State space for Markov chain.

I'm trying to do this exercise from Probability by John Walsh: Let ${X_{n}, n = 0,1, 2, ... }$ be a Markov chain whose transition probabilities MAY NOT be stationary. Define $X'_{n}$ to be the n-tuple ...
30 views

### Solution of general Dirichlet problem

Let $(E,\mathcal E)$ be a measurable space, $\kappa$ be a Markov kernel on $(E,\mathcal E)$ and $X_I$ denote the projection from $E^{\mathbb N_0}$ onto $E^I$ for $I\subseteq\mathbb N_0$. We know that ...
32 views

### Derivation formula for piecewise-deterministic Markov process.

I was reading a formulation of piecewise-deterministic Markov process $\Pi_t$, $t \in \mathbb{R}_+$. In particular $\Pi_t$ is defined as $\Pi_t = P (X_t | \mathcal{F}_{\lfloor t/ \Delta \rfloor})$, ...
64 views

19 views

103 views

### Is the period of an irreducible Markov Chain the same for all states?

I'm a bit confused about this theorem about irreducible Markov Chains: If a Markov chain is irreducible, then all its states have the same period $d(i) := g.c.d.\{n > 0|P^n(i, i) > 0\}$.[source]...
21 views

### Finite irreducible Markov chain and it elements of stationary distribution [duplicate]

Finite irreducible Markov chain with stacionary distribution $\pi=\left(\pi_i\right)_{i=1}^n$ has all $\pi_i\neq0$ for $i=1,2,\dots,n$. It is true? If yes how to prove it. I know that there is only ...
21 views

### Interpretation of the Sinkhorn-Knopp algorithm applied to a (singly stochastic) transition matrix of a Markov process?

Say I have a discrete-time Markov process (and let's say discrete states too, for simplicity). If $\mathbf p_t$ is a vector of probabilities over states at time $t$, then the probability distribution ...
24 views

### Understanding Yosida approximation of operators

Definition 129 in these notes states: The Yosida approximation to a semigroup $K_t$ with generator $G$ is given by $$K_t^\lambda := e^{tG^\lambda}$$  G^\lambda := \lambda GR_\lambda = \lambda (\...
Let us say that our state space $S = \{1, 2, 3, 4\}$ Now let us say our transition matrix $P$ is given by: \begin{bmatrix} 1/2 & 1/2 & 0 & 0 \\ 1/3 & 0 & 1/3 & 1/3 \\ ...
Problem Let $G$ be a strongly connected directed graph with $n$ vertices, and let $d(v)$ denote the out-degree of vertex $v$. Let $M_G$ be a finite discrete-time homogeneous Markov chain defined over \$...