Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

0
votes
0answers
20 views

Markov Memoryless Property confusion with counterexample

Consider the following transitions: $$P(X_2=d|X_1=b)=P(X_1=d|X_0=b)=3/4$$ By Markov property, all what happened before $X_1=b$ doesn't matter: So when I see $P(X_2=d|X_1=b)$ I know that I can ...
0
votes
0answers
13 views

Convergence of Feller semigroups

Let $E$ be a locally compact separable metric space $(T_t)_{t\ge0},\left(T^{(n)}_t\right)_{t\ge0}$ be Feller semigroups$^1$ on $C_0(E)$ for $n\in\mathbb N$ Assume $$\left\|T^{(n)}_tf-T_tf\right\|_{...
1
vote
1answer
26 views

Difference between non-homogeneous Markov and Semi-Markov?

I have been going around with Markov family lately and now bit confused. As per my understanding, to satisfy Markov property, state holding time distribution needs to be exponential otherwise the ...
0
votes
0answers
28 views

First-Visit vs Every-Visit Monte Carlo

I have recently been looking into reinforcement learning. For this, I have been reading the famous book by Sutton, but there is something I do not fully understand yet. For Monte-Carlo learning, we ...
1
vote
1answer
21 views

Notation for conditional expectation using integral measure

Hi I am struggling to understand this notation for conditional expectation: (Say $X_{t}$ is a process that takes values in $\mathbb{R}$) then $$E[f(X_{t})|X_{0}=x]=\int_{\mathbb{R}}f(y)p_{t}(x,dy)$$ ...
0
votes
1answer
28 views

Relation between waiting time distribution and probability that an event occurs within time $dt$

Waiting time distribution is defined as the distribution of the time interval between two successive events. I'm looking at stochastic processes in discrete space and continuous time with non-...
1
vote
1answer
26 views

Finite-dimensional conditional distributions of a Markov process

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $I\subseteq\mathbb R$ $(\mathcal F_t)_{t\in I}$ be a filtration on $(\Omega,\mathcal A)$ $(E,\mathcal E)$ be a measurable space $X$ be ...
1
vote
0answers
24 views

Why are diffusion equations typically formulated using $-\partial_{x}(D\partial_{x}u)$ when Fokker--Planck has $-\partial_{xx}^{2}(Du)$

Is there any physical process that naturally gives rise to equations of the form $$ \partial_{t}u -\partial_{x}(D\partial_{x}u) $$ or is this form really used in all my textbooks (e.g. Evans) and ...
0
votes
0answers
21 views

Does the theory of Gittins Indices solve the Multi-armed Bandit problem?

For example, both Wikipedia and Reinforcement Learning: An Introduction (page 33) seem to claim as much, which would suggest that the problem has been solved for over 40 years. However, doing as ...
1
vote
1answer
31 views

Show some property of a Markov process

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $I\subseteq\mathbb R$ $(\mathcal F_t)_{t\in I}$ be a filtration on $(\Omega,\mathcal A)$ $(E,\mathcal E)$ be a measurable space $X$ be ...
0
votes
0answers
14 views

Probability of reaching a trajectory in a Markov chain before another?

Suppose I have a markov chain given by the transistion probability matrix $Q$. Suppose I also have $2$ trajectories, $t_1, t_2$. I want to calculate the probability that I would get trajectory $t_1$ ...
0
votes
0answers
14 views

Prove the sufficient and necessary condition for Markov chains to have a unique stationary distribution?

I learned that the sufficient and necessary condition for a finite state Markov chain to have a unique stationary distribution is there's only one closed communication class. For example from this ...
0
votes
0answers
51 views
+100

Steady states to this generalized TASEP?

The standard setup of a Totally Asymmetric Simple Exclusion Process is pictured below: We have a one-dimensional lattice of length $n$ populated with particles($p_1,p_2,p_3$ in this case) that hop to ...
0
votes
0answers
13 views

The u* rate for M/M/2 queue to have same “average waiting time in queue of a customer” with M/M/1 system with rate u

Ok here is the question, in a supermarket I have one cashier with u and a single M/M/1 queue, Suppose I want to get 2 cashier with same rate u*.What would should be the u* value for me to have same" ...
0
votes
0answers
12 views

Probability measure on $A^\mathbb{N}$ for Markov chains

Let $A$ be a finite set, $\Omega = A^N$, with $N \in \mathbb{N}$ or $N = \infty$ and let $\mathcal{B}(\Omega)$ be a $\sigma$-algebra of $\Omega$. Consider $\omega \in \Omega$ such that $\omega = (x_1, ...
0
votes
0answers
26 views

Distributions over probability simplex

I was wondering if there any probability distributions over the probability simplex ($p \in \mathbb{R^n_+}:\sum_i^n p_i=1,p_i\geq 0$). In particular, what are the distributions which can be used to ...
0
votes
0answers
12 views

Relation between v(s) and q(s,a) in a Markov Decision Process?

I was solving questions related to backup diagrams from Reinforcement Learning: An Introduction by Barto and Sutton. Are these 4 equations mathematically correct ? Are there any shortcomings in terms ...
1
vote
1answer
32 views

Applications of the Markov Property

The Markov property for a discrete state time stochastic process is defined as: $$\mathbb{P}(X_n=x_n\mid X_{n-1}=x_{n-1}, \dots, X_0=x_0)=\mathbb{P}(X_n=x \mid X_{n-1}=x_{n-1})$$ A corollary is $$\...
0
votes
0answers
13 views

Multidimensional squared Ornstein-Uhlenbeck process

We know that for any $b\in\mathbb{R}$, the generator of squared Ornstein-Uhlenbeck process with parameter $\delta>0$ is given by: $$Af(x)=\frac{1}{2}xf''(x)+(\delta-bx)f'(x).$$ Now, let $B$ be a $d\...
2
votes
1answer
38 views

Showing continuity of integrals of Feller process

Let $\{P^x\}$ be the probability distribution for a Feller process $\{X\}_t$. Then, how does one show that $$F(x) = \int_Df(X_t)g(X_s)P^x(d\omega)\quad (\dagger)$$ is a continuous function of $x$ ...
0
votes
0answers
19 views

Rank of a stochastic matrix

Are there any findings about the rank of a square or fat ($m\times n$-dimensional, $m<n$) stochastic matrix? I would appreciate any pointers.
1
vote
0answers
16 views

Existence of solution to system of linear equations with stochastic coefficients matrix

Given an underdetermined or determined system of linear equations, can anything be said about existence of a solution if the coefficients matrix is a stochastic matrix, i.e., its rows sum up to 1? I ...
0
votes
1answer
13 views

Why $P^x \left[X_{t+s} \in \Gamma \mid \mathcal{F}_s\right]=P^x \left[X_{t+s} \in \Gamma \mid X_s\right]$?

If we can show that $P^x \left[X_{t+s} \in \Gamma \mid \mathcal{F}_s\right]$ has a $\sigma(X_s)$-msb version then why is it true that $P^x \left[X_{t+s} \in \Gamma \mid \mathcal{F}_s\right]=P^x \left[...
-1
votes
0answers
19 views

Characteristics of a Markov Proces

Description of the process A two-states Markov parocess, where $X$ = {$State$ 1, $State$ 2}, can be represented graphically as where $[P]_{i,j} = Pr (X_{n+1} = State$ j$|X_{n} = State$ i$)$. The ...
2
votes
1answer
38 views

Random walk Catalan sum

Starting at $0$ on the number line, you go right $1$ unit with probability $p$ and left $1$ unit with probability $1-p$. What's the probability of ever getting to $n>0$, and how many steps are ...
1
vote
1answer
41 views

A die is rolled repreatedly and … is it a Markov Chain?

A die is rolled repeatedly. Determine whether the sequence: "at time n the time Xn of the most recent appearance of a six" is a Markov Chain. /// My attempt: in class it was explained that a ...
0
votes
0answers
31 views

Arrival and departure probabilities in an Erlang Loss system

Consider an M/G/c/c system; that is, an Erlang loss system with a general service time. If the system is in the steady state situation, what is the probability that a randomly observed event is an ...
2
votes
2answers
54 views

Example of a stochastic non Markov process?

A Markov process is defined as: $$P(X_t| X_{1:t-1}) = P(X_t|X_{t-1})$$ Is there a non Markov process that can be generated by a computer and cannot be converted to a Markov process by changing the ...
-1
votes
1answer
38 views

Gambler´s ruin problem

A gambler starts with an initial fortune of 9. He wins 1 with p=1/3 and losses 1 with q=2/3. The game ends when the gambler looses all of his money or when we reaches 15. What is the probability ...
0
votes
1answer
26 views

Stationary distribution of DTMC with infinite state space

I am solving the stationary distirbution of a Discrete time Markov Chain with infinite state space. The state space is $\{\pi^H_0,\pi^H_1,...,\pi^L_0,\pi^L_1,...\}$. The transition matrix has ...
0
votes
1answer
21 views

Markov chain with three states and equal fractions

This is a homework problem and I have been stuck at it for over an hour. Any hint will be appreciated. The question states that a town is running a bike sharing program. A bike could be grabbed at a ...
0
votes
1answer
22 views

Transition matrix of “Three machines, one repairman”

This is an example given in Durrett's textbook. I can only understand half of it. Suppose that an office has three machines that each break with probability $.1$ each day, but when there is at ...
3
votes
0answers
39 views

Markov Property fos Ising type Models

We are interested in proving the Markov property for the long range Ising type model in $\mathbb{Z}^d$. Setting: Define $\Omega = \{-1, +1\}^{\mathbb{Z}^d}$ the space of all possible configurations ...
-1
votes
1answer
22 views

Show that $\mathbb{E}[T_A\mid Z_0=k] = \sum_{\ell\in S} \mathbb{E}[T_A \mathbb{1}_{\{Z_1 = \ell\}}\mid Z_0=k].$

I am reading Notes on Markov Chain. Consider a Markov chain $(Z_n)_{n=1}^\infty$ with state space $S$ and let $A\subseteq S.$ Define $$T_A = \inf\{n\geq 0: Z_n\in A\}.$$ Mean hitting time is defined ...
0
votes
0answers
13 views

Reversible distribution of a CTMC on a graph, where the edges are cut according to another process.

Hi : I am trying to formulate a theorem i know to be true, but i cant find the proof anywhere, could anyone finish my proof or point me in the direction of the theorem stated properly. THEOREM : ...
0
votes
1answer
16 views

How to prove that $P(Z_1 = j| Z_0 = i) = P(Z_{n+1} = j| Z_n = i)?$

Currently I am reading this Markov Chains' notes. At page $112,$ section $4.2$ Transition matrix, I fail to understand how author obtains the following. As seen above, the random evolution of a ...
0
votes
1answer
30 views

Clarification regarding stopping time

Let $(X_n)_{n\geq 1}$ be a Markov process on a finite state space $S$, and let $N$ be a stopping time with respect to the natural filtration $(\mathcal{F}_n)_{n\geq 1}$ defined by \begin{equation} \...
1
vote
0answers
58 views

Transition Probabilities in Discrete Time Markov Chain

If someone would be able to point me in a starting direction, I would be greatly appreciative. Question I am attempting to derive a closed form for the transition probabilities of the discrete time ...
1
vote
1answer
30 views

Simple example of numerical integration using the Metropolis-Hastings algorithm

Let $(E,\mathcal E)$ be a measurable space $\kappa$ be a Markov kernel with source and target $(E,\mathcal E)$ $\mu$ be a probability measure on $(E,\mathcal E)$ I know how the Metropolis-Hastings ...
1
vote
0answers
43 views

Long-term ball distribution in urns

Suppose, we have $k$ urns each starting with $n$ balls (all balls are of the same color). At each step, we draw balls from each urn, where each ball could be drawn independently at random with ...
3
votes
2answers
77 views

Existence of finitely presented group with exactly $k$ subgroups?

Given any natural number $k$, does there necessarily exist a finitely presented group $G$ with $k$ subgroups? I'm trying to show that having $k$ subgroups is a markov property, but google doesn't seem ...
0
votes
0answers
79 views

Harmonic functions and Markov processes

I was reading some lecture notes, in which it is stated that, given a discrete time Markov process $X_t$, for its Markov semigroup $P_t$ it holds that: $$ P_tf(x) = E_x[f(X_t)]=E[f(X_t)|X_0=x] $$ For ...
0
votes
0answers
24 views

the stationary distribution maximal entropy random walk on the irregular lattice

I was reading the paper 0810.4113v2, burda, which analyzed the stationary distribution maximal entropy random walk on the irregular lattice. I am confused on some of the steps. Description: The ...
0
votes
0answers
15 views

Finding new expectation using Chebyshev.

I've tried to use similar questions for this, but I ended up with just a horrible answer. I have a question about a woman serving soup, she originally had 30 litres for 80 pupils, with expectation 0....
0
votes
0answers
17 views

Function of Markov process

I have the following transition matrix: $$P = \left[ \begin{array}{ccc} \frac{1}{2} & \frac{1}{4} & \frac{1}{4} \\\frac{1}{4} & \frac{1}{2} & \frac{1}{4} \\ \frac{1}{4} & \frac{1}{...
1
vote
2answers
40 views

Markov Property and FDDs

Let $X,Y$ be two discrete time $\mathbb{R}^n$-valued stochastic processes with the same finite dimensional distributions. It may be that $X,Y$ are defined on two different probability spaces. Now, if $...
-2
votes
3answers
79 views

Notation in expectation

I know this question has been asked several times before, anyway I don't find the answer to my problem - on some lecture notes I read the following statement: Let$ P_t $ be a Markov semigroup acting ...
3
votes
0answers
91 views

Will simple random walk on $n$-cycle converges to Brownian motion on $S^1$?

I know that, by Donsker's theorem, simple random walk on $\mathbb{Z}$ will converge to Brownian motion on $\mathbb{R}$. Here, simple random walk means that the Markov chain with probability from $n$ ...
1
vote
0answers
44 views

A question about Markov Chain.

Suppose $\{X_n\}$ has Markov property. Show that for any $n,r \in \Bbb N,i \in S,A \subset S^n, B \subset S^r$ $P[(X_{n+1}, \cdots , X_{n+r}) \in B\ \mid X_n=i,(X_0,\cdots,X_{n-1}) \in A]=P[(...
1
vote
0answers
19 views

How to find the intensity matrix $Q$ in a reliability process (Markovs chains)?

I'm trying to understand how the intensity matrix can be set up in this problem: Question A system has two components such that if one of them breaks, it will stop working. Each of the components' ...