# Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

2,256 questions
Filter by
Sorted by
Tagged with
12 views

### Weak convergence against upper invariant measure

Setting I am studying invariant measures and their weak limits. In a book about probability on graphs the following setting is presented in chapter 6.3 (this is a short form of the actual presentation)...
18 views

### Proof of Bellman error

How can I prove the Bellman error using fix point and norm properties? If $$|| V_{k+1}-V_k || < \epsilon (1-\gamma)/\gamma$$ then $$|| V_{k+1}-V^* || < \epsilon.$$
13 views

### Gibbs sampling from Multinomial distribution

I need to use Gibbs sampling to sample from $(X_1, X_2, \ldots, X_n)$ that is distributed according to Mult $\left( N, 1/n, \ldots, 1/n\right)$. For this I need to compute the conditional pmf ...
1 vote
16 views

### Is this Markov process Gaussian ? $Y_t= X_t - (1-t) X_0 - tX_1, \, t \in [0,1]$

Let $(X_t)$ be a real sample continuous stochastic process with density function $f_t$. Let $$Y_t= X_t - (1-t) X_0 - tX_1, \, t \in [0,1].$$ Suppose that $(Y_t)$ is Markov with regard to its own ...
1 vote
27 views

### Finding stationary distribution of random process

Suppose we are given $x_t, \bar{x_t}, t\in \mathbb{Z_+}$ independent 2-states $\{0, 1\}$ Markov chains with positive transition probabilities. Initial states are $x_0 = 0; \bar{x}_0 = 1$. For which ...
14 views

### Deriving Transition Probabilities of a Markov Process

I was reading this article about Markov Processes (https://www.jstatsoft.org/article/view/v066i06) and saw the following equation (4): I am trying to understand why equation (4) is true. I can ...
1 vote
7 views

### Can we construct an equivalent markov process with newer states, some of which are combination of previous states?

Let there be a continuous time Markov chain with three possible states $C_1, C_2, C_3$, and the rate of going from configuration $C_i$ to $C_j$ be $r_{ij}$. A very simple markov chain could be such ...
19 views

38 views

### Markov processes with positive jump probability at fixed time

I would expect this question to be answered somewhere but I can't find it. I have the following conjecture. Let $X=(X_t)_{t\geq0}$ be a time homogeneous (strong?) Markov process with cadlag paths on ...
18 views

### Probabilities involving the superposition of poisson processes

Say customers arrive at an ice cream shop at an average rate of 100 customers per day. Independently, the ice cream shop receives a restock of their ice cream at an average rate of one shipment every ...
15 views

### Difference between semi-Markov and Markov process

I have seen the definition of semi-Markov process as a rcll process such that \begin{align*} &P[X_{T_n+1}=k',T_{n+1}-T_n \leq y|(X_{T_0},T_0),(X_{T_1},T_1),\ldots,(X_{T_n},T_n)]\\ &...
100 views

### Drunken king on chessboard: Why is the probability that the king is on each square proportional to the number of adjacent squares?

On a chessboard there is a (drunken) king. The king moves at the beginning of each minute, in a random direction: up, down, left, right, or the four diagonal directions (unless the king is on an edge ...
1 vote
38 views

59 views

### Dual stochastic process

I'm trying to understand a proof that involves the dual of stochastic processes. The definition: Let $X^x_t$ and $Y^y_t$ be two stochastic processes starting from $x$ and $y$ respectively. They are ...
372 views

### Possibility that all lights $\mathbf{X}=(X_1,X_2,\cdots)$ turn off again with every time turn a light with its number $n\sim\text{geom}(\frac{1}{2})$.

Problem: Let $\mathbf{X} = (\mathbb{Z}_2)^\mathbb N$, i.e., $\mathbf{X} = (X_1,X_2,\cdots,X_N,\cdots)$, $X_i\in \{0,1\}$. It can be considered as countable lightbulbs. $0$ means off, $1$ means on. We ...
1 vote
52 views

### Applying theorem to gambler coin toss problem (Norris , Markov Chain exercise 1.3.2)

This's the problem I'm facing (from Norris , Markov Chain p.18) : " A gambler has £2 and needs to increase it to £10 in a hurry. He can play a game with the following rules: a fair coin is tossed;...
91 views

### How do we show that the concatenation of Markov processes is Markov?

Let $(\Omega^n,{\mathcal A}^n,{\operatorname P}^n)$ be a probability space and \begin{align}\Omega&:=\Omega^1\times\Omega^2;\\\mathcal A&:=\mathcal A^1\otimes\mathcal A^2\\\operatorname P&...
1 vote
35 views

### What is the probability that a Markov chain transitions between states if it passes through a specified intermediate transition?

Consider a discrete-time finite Markov chain with transition probability matrix $P$. One of the foundational results of Markov theory is, of course, that the probability that the chain transitions ...
8 views

### Does the two-stage method for describing a semi-Markov chain give enough information to reconstruct the kernel?

The chapter at https://link.springer.com/chapter/10.1007/978-1-4615-2367-3_8 describes two different ways of specifying a semi-Markov chain: The "kernel method" specifies a time-dependent ...
20 views

1 vote
57 views

### Expected hitting time of a random walk on a complete graph

I have a random walk defined on a complete graph with n vertices(there is an edge between any pair of nodes). I need to compute the expected hitting time $E(\tau)$ of the set $A=\{1,2,3\}$ given some ...
48 views

### If $Z$ is i.i.d. and independent of $X_0$, then $X_n:=f(X_{n-1},Z_n)$ is Markov

Let $(E_i,\mathcal E_i)$ be a measurable space; $f:E_1\times E_2\to E_1\to E_1$ be $(\mathcal E_1\otimes\mathcal E_2,\mathcal E_1)$-measurable; $(\Omega,\mathcal A,\operatorname P)$ be a probability ...
28 views

### The variation of transition probability was determined by the states

I'm working on a random process problem. At first, I have 2 variables, $W_{a}$ and $W_{b}$ (0 < $W_{a}$ < 1, 0 < $W_{b}$ < 1); and 4 states, $S_{1}$, $S_{2}$, $S_{3}$, $S_{4}$. The ...
I have trouble to integrate the function $F(x)=P' [\sum_{t=0}^{\infty} (A+Bx)^t]x D= P'(I-A-Bx)^{-1}x D$, where P and D are $n \times 1$ vector, A and B are $n \times n$ matrix. I is identical matrix....