Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Weak convergence against upper invariant measure

Setting I am studying invariant measures and their weak limits. In a book about probability on graphs the following setting is presented in chapter 6.3 (this is a short form of the actual presentation)...
user avatar
  • 21
-1 votes
0 answers
18 views

Proof of Bellman error

How can I prove the Bellman error using fix point and norm properties? If $$|| V_{k+1}-V_k || < \epsilon (1-\gamma)/\gamma $$ then $$|| V_{k+1}-V^* || < \epsilon.$$
user avatar
  • 47
0 votes
0 answers
13 views

Gibbs sampling from Multinomial distribution

I need to use Gibbs sampling to sample from $(X_1, X_2, \ldots, X_n)$ that is distributed according to Mult $\left( N, 1/n, \ldots, 1/n\right)$. For this I need to compute the conditional pmf ...
user avatar
  • 319
1 vote
0 answers
16 views

Is this Markov process Gaussian ? $Y_t= X_t - (1-t) X_0 - tX_1, \, t \in [0,1]$

Let $(X_t)$ be a real sample continuous stochastic process with density function $f_t$. Let $$Y_t= X_t - (1-t) X_0 - tX_1, \, t \in [0,1].$$ Suppose that $(Y_t)$ is Markov with regard to its own ...
user avatar
  • 2,050
1 vote
0 answers
27 views

Finding stationary distribution of random process

Suppose we are given $x_t, \bar{x_t}, t\in \mathbb{Z_+}$ independent 2-states $\{0, 1\}$ Markov chains with positive transition probabilities. Initial states are $x_0 = 0; \bar{x}_0 = 1$. For which ...
user avatar
0 votes
0 answers
14 views

Deriving Transition Probabilities of a Markov Process

I was reading this article about Markov Processes (https://www.jstatsoft.org/article/view/v066i06) and saw the following equation (4): I am trying to understand why equation (4) is true. I can ...
user avatar
1 vote
0 answers
7 views

Can we construct an equivalent markov process with newer states, some of which are combination of previous states?

Let there be a continuous time Markov chain with three possible states $C_1, C_2, C_3$, and the rate of going from configuration $C_i$ to $C_j$ be $r_{ij}$. A very simple markov chain could be such ...
user avatar
-1 votes
0 answers
19 views

A proof question ralated to Markov chain and transition probability [closed]

Suppose ${X_n,n\ge0}$ is a Homogeneous Markov Chains taking the value of $S=\{0,1,2,...\}$,which has the transition probability of $p_{0,1} = 1, p_{i,i+1}+p_{i,i-1}=1,p_{i,i+1}=(\frac{i+1}{i})^2p_{i,i-...
user avatar
0 votes
0 answers
10 views

Why does converting the Chordal Markov network to Bayesian network need the concept of clique tree?

In the book Probabilistic Graphical Models Principles and Techniques, Page141 mentioned Theorem 4.13 as: Theorem 4.13: H is a chordal Markov network, then there is a Bayesian network G such that I(G)=...
user avatar
2 votes
1 answer
46 views

A statement about finite Markov chains

The following quantity $\tilde\pi$ is defined in the textbook Markov chains and mixing times by David A. Levin. Here $\tau_z^+ = \min \left\{t\geq 1| X_t = z\right\}$. Let $z \in \mathcal{X}$ be an ...
user avatar
4 votes
0 answers
57 views

Using PDE to represent a markov process

Consider a population of constant size $N + 1$ that is suffering from an infectious disease. We can model that spread of the disease as Markov process. Let $X(t)$ be the number of healthy individuals ...
user avatar
0 votes
0 answers
8 views

Are branching processes infinite state space markov processes?

Is it correct to view a branching process as an infinite state space markov process, with the states being the total population which can be 0,1,2,...?
user avatar
1 vote
1 answer
28 views

$M/M/2/3$ Queuing Theory Word-Problem

A service center consists of two servers, each working at an exponential rate of two services per hour. If customers arrive at a Poisson rate of three per hour, then, assuming a system capacity of at ...
user avatar
  • 11
0 votes
0 answers
32 views

Is it a Markov Chain? If yes, find its transition matrix

I'm doing some exercises on Markov Chains. Let $S_n$ be the simple walk of $\mathbb{Z}$, then are the following processes Markov chains? and if yes find its transition Matrix: $(S_n + n)_{n \geq 1}$ ...
user avatar
  • 337
0 votes
0 answers
63 views

How to determine the number of distinct deterministic policies in a MDP?

I'm trying to tackle this question to understand MDP. Can someone explain how can you determine or calculate the number of distinct deterministic policies in the below MDP? Or resources where I can ...
user avatar
  • 1
1 vote
1 answer
20 views

How to reorder matrix to its canonical form (Markov chain)?

Is it possible to rearrange this matrix in its canonical form (link below)? I have searched numerous websites and videos and have only found answers for small matrices where you automatically get the ...
user avatar
2 votes
1 answer
23 views

Confusion about definition of ergodicity in Markov Chains

On Wikipedia I found the following section on ergodicity Ergodicity A state i is said to be ergodic if it is aperiodic and positive recurrent (*). In other words, a state i is ergodic if it is ...
user avatar
0 votes
0 answers
22 views

Concatenation of Markov proccesses is Markov

Let $(E_n,\mathcal E_n)$ be a measurable space with $E^1\cap E^2=\emptyset$ and \begin{align}E&:=E^1\cup E_2;\\\mathcal E&:=\sigma(\mathcal E_1\cup\mathcal E_2);\end{align} $(\Omega^n,{\...
user avatar
  • 12.8k
2 votes
0 answers
38 views

Markov processes with positive jump probability at fixed time

I would expect this question to be answered somewhere but I can't find it. I have the following conjecture. Let $X=(X_t)_{t\geq0}$ be a time homogeneous (strong?) Markov process with cadlag paths on ...
user avatar
0 votes
1 answer
18 views

Probabilities involving the superposition of poisson processes

Say customers arrive at an ice cream shop at an average rate of 100 customers per day. Independently, the ice cream shop receives a restock of their ice cream at an average rate of one shipment every ...
user avatar
-1 votes
0 answers
15 views

Difference between semi-Markov and Markov process

I have seen the definition of semi-Markov process as a rcll process such that \begin{align*} &P[X_{T_n+1}=k',T_{n+1}-T_n \leq y|(X_{T_0},T_0),(X_{T_1},T_1),\ldots,(X_{T_n},T_n)]\\ &...
user avatar
  • 319
2 votes
1 answer
100 views

Drunken king on chessboard: Why is the probability that the king is on each square proportional to the number of adjacent squares?

On a chessboard there is a (drunken) king. The king moves at the beginning of each minute, in a random direction: up, down, left, right, or the four diagonal directions (unless the king is on an edge ...
user avatar
  • 1,192
1 vote
1 answer
38 views

If $\mathbb E[f\circ X_t\circ\theta_s\mid F_s]= \mathbb E_{X_s}[f\circ X_t]$, can we show $\mathbb E[g\circ\theta_s\mid F_s]=\mathbb E_{X_s}[g]$?

Let $(\Omega,\mathcal A)$ be a measurable space; $(E,\mathcal E)$ be a measurable space; $\pi$ be a Markov kernel with source $(E,\mathcal E)$ and target $(\Omega,\mathcal A)$; $\operatorname P_\mu:=\...
user avatar
  • 12.8k
0 votes
0 answers
13 views

infinitesimal generator of a Markov process

Problem: There are $n$ identical components in a system that operate independently. When a component fails, it undergoes repair, and after repair is placed back into the system. Assume that for a ...
user avatar
0 votes
1 answer
63 views

Poyla URN - computing the probability to pick a black ball at the nth turn

Consider the Polya urn model with $a$ black balls and $b$ red balls initially in an urn where $a$ and $b$ are arbitrary positive integers. At each turn you pick a ball at random and return it to the ...
user avatar
  • 319
2 votes
0 answers
11 views

Composite Stochastic Process: Evolution equation of the probability density

Consider a Markov process $x(t)$ that takes the discrete values $\{x_i\}_{i=1,\cdots,n}$ and whose probability density evolves according to a master equation: $$ \partial_t p_i(t) = \sum_{j=1}^n[w(i|j)...
user avatar
0 votes
0 answers
53 views

How can we construct a canonical Markov process with càdlàg paths?

Let $(E,\mathcal E)$ be a measurable space; $\pi_I$ denote the projection from $E^{[0,\:\infty)}$ onto $E^I$ for $I\subseteq[0,\infty)$ for nonempty $I\subseteq[0,\infty)$; $\pi_t:=\pi_{\{t\}}$ for $...
user avatar
  • 12.8k
-2 votes
1 answer
27 views

Distribution of $X_n$ Stochastic process [closed]

I am thinking of finding $P(X_3=r|X_0=s)$ i.e $P_{s,r}^3$. But with the given data i am unable to go ahead. Am I thinking correctly? How to solve this problem?
user avatar
1 vote
0 answers
32 views

$M/M/c/c$ queue is PASTA

Consider a $M/M/c/c$ queue, e.g. arrivals to a shop form a Poisson process rate $\lambda$, each customer spends an iid exponential time mean $1/\mu$ in the shop. The shop has maximum capacity of $c$, ...
user avatar
  • 183
1 vote
3 answers
94 views

Expected number of flips vs probability in a Markov Chain

The problem is the following: (a) We keep flipping coins until we see the sequence HTHH. Find the expected number of flips. (b) Alice and Bob play the following game. They keep flipping coins until ...
user avatar
0 votes
0 answers
56 views

Find transition matrix for Markov Chain

The problem is the following: An individual has three umbrellas, some at her office, and some at home. If she is leaving home in the morning (or leaving work at night) and it is raining, she will ...
user avatar
0 votes
0 answers
6 views

MDPs: Does the value function of each state $V(s)$ depend on the initial distribution over states?

Given an infinite-horizon discounted MDP $(\mathcal{S}, \mathcal{A}, \gamma, r , \rho)$, where $\rho:\mathcal{S}\rightarrow [0, 1]$ and $\sum_s \rho(s) = 1$ is the initial distribution over states; ...
user avatar
  • 115
0 votes
0 answers
38 views

How do we prove Dynkin's formula for the resolvent of a strong Markov process?

In eq. (1.1) of this paper it is claimed that if $X$ is strongly Markov at a stopping time $\tau$, then it would follow from Dynkin's formula that $$(U_\alpha f)(x)=\operatorname E_x\left[\int_0^\tau ...
user avatar
  • 12.8k
0 votes
0 answers
8 views

Exsitence of stationary distribution for M/M/1 with non-homogeneous poisson arrival rate

Consider an M/M/1 queue where arrivals occur at rate $\lambda(t)$ according to a Poisson process at time $t$ and move the process from state $i$ to $i+1$, and service times have an exponential ...
user avatar
  • 462
1 vote
1 answer
52 views

If $\text E\left[1_F\int_a^bf(X_{s+t})\:{\rm d}t\right]=\text E\left[1_F\int_a^b(κ_tf)(X_s)\right]$, then $X$ is Markov with transition semigroup $κ$

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space; $(\mathcal F_t)_{t\ge0}$ be a filtration on $(\Omega,\mathcal A)$; $(E,\mathcal E)$ be a measurable space; $(X_t)_{t\ge0}$ be an $(E,\...
user avatar
  • 12.8k
0 votes
0 answers
59 views

Dual stochastic process

I'm trying to understand a proof that involves the dual of stochastic processes. The definition: Let $X^x_t$ and $Y^y_t$ be two stochastic processes starting from $x$ and $y$ respectively. They are ...
user avatar
  • 1,404
14 votes
3 answers
372 views

Possibility that all lights $\mathbf{X}=(X_1,X_2,\cdots)$ turn off again with every time turn a light with its number $n\sim\text{geom}(\frac{1}{2})$.

Problem: Let $\mathbf{X} = (\mathbb{Z}_2)^\mathbb N$, i.e., $\mathbf{X} = (X_1,X_2,\cdots,X_N,\cdots)$, $X_i\in \{0,1\} $. It can be considered as countable lightbulbs. $0$ means off, $1$ means on. We ...
user avatar
  • 804
1 vote
1 answer
52 views

Applying theorem to gambler coin toss problem (Norris , Markov Chain exercise 1.3.2)

This's the problem I'm facing (from Norris , Markov Chain p.18) : " A gambler has £2 and needs to increase it to £10 in a hurry. He can play a game with the following rules: a fair coin is tossed;...
user avatar
  • 335
0 votes
0 answers
91 views

How do we show that the concatenation of Markov processes is Markov?

Let $(\Omega^n,{\mathcal A}^n,{\operatorname P}^n)$ be a probability space and \begin{align}\Omega&:=\Omega^1\times\Omega^2;\\\mathcal A&:=\mathcal A^1\otimes\mathcal A^2\\\operatorname P&...
user avatar
  • 12.8k
1 vote
1 answer
35 views

What is the probability that a Markov chain transitions between states if it passes through a specified intermediate transition?

Consider a discrete-time finite Markov chain with transition probability matrix $P$. One of the foundational results of Markov theory is, of course, that the probability that the chain transitions ...
user avatar
  • 5,401
0 votes
0 answers
8 views

Does the two-stage method for describing a semi-Markov chain give enough information to reconstruct the kernel?

The chapter at https://link.springer.com/chapter/10.1007/978-1-4615-2367-3_8 describes two different ways of specifying a semi-Markov chain: The "kernel method" specifies a time-dependent ...
user avatar
  • 5,401
0 votes
1 answer
20 views

How to make a Poisson process time-homogeneous?

A continuous-time Markov Chain $(X_t)_{t\ge 0}$ is called time-homogeneous if $$ \qquad (*) \qquad P(X_{s+t}=j|X_s=i)=P(X_t=j|X_0=i), \forall s\ge 0 $$ Strictly speaking, a Poisson process $(N_t)_{t\...
user avatar
  • 1,936
0 votes
0 answers
12 views

If a sequence of random vectors $X_k\in R^d$ is Markovian, is $a^TX_k$ Markovian for any deterministic $a\in R^d$

I have a question when I study the definition of the Markov Process. If a sequence of random vectors $X_k\in R^d$ is Markovian, is it the case that $a^TX_k$ is Markovian for any deterministic $a\in R^...
user avatar
1 vote
1 answer
57 views

Expected hitting time of a random walk on a complete graph

I have a random walk defined on a complete graph with n vertices(there is an edge between any pair of nodes). I need to compute the expected hitting time $E(\tau)$ of the set $A=\{1,2,3\}$ given some ...
user avatar
  • 319
0 votes
1 answer
48 views

If $Z$ is i.i.d. and independent of $X_0$, then $X_n:=f(X_{n-1},Z_n)$ is Markov

Let $(E_i,\mathcal E_i)$ be a measurable space; $f:E_1\times E_2\to E_1\to E_1$ be $(\mathcal E_1\otimes\mathcal E_2,\mathcal E_1)$-measurable; $(\Omega,\mathcal A,\operatorname P)$ be a probability ...
user avatar
  • 12.8k
0 votes
0 answers
28 views

The variation of transition probability was determined by the states

I'm working on a random process problem. At first, I have 2 variables, $W_{a}$ and $W_{b}$ (0 < $W_{a}$ < 1, 0 < $W_{b}$ < 1); and 4 states, $S_{1}$, $S_{2}$, $S_{3}$, $S_{4}$. The ...
user avatar
0 votes
0 answers
25 views

The integration of a function written in matrix form

I have trouble to integrate the function $F(x)=P' [\sum_{t=0}^{\infty} (A+Bx)^t]x D= P'(I-A-Bx)^{-1}x D$, where P and D are $n \times 1$ vector, A and B are $n \times n$ matrix. I is identical matrix....
user avatar
1 vote
0 answers
18 views

Markov property of $X_t + \sigma Y_t$ when $\sigma \rightarrow 0$

Let $(X_t)$ and $(Y_t)$ be sample continuous stochastic processes on $[0,1]$ such that $Z_t (\sigma):= X_t + \sigma Y_t$, where $\sigma >0$, is Markov with regard to the filtration generated by $...
user avatar
  • 2,050
1 vote
0 answers
17 views

Is the jump measure of a space homogeneous Markov process translation invariant?

Let $E$ be a normed $\mathbb R$-vector space and $(X_t)_{t\ge0}$ be an $E$-valued Lévy process. $X$ is a space- and time-homogeneous Markov process with transition semigroup $$\kappa_t(x,B):=\...
user avatar
  • 12.8k
2 votes
2 answers
114 views

Loss networks and Poisson processes

Consider a shop with capacity C, where customers spend an i.i.d exponential(1) amount of time before leaving through the back door. Customers arrive through the front door as a Poisson($\lambda$) ...
user avatar

1
2 3 4 5
46