A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
1answer
18 views

Distribution Stopping time under Brownian motions

Considering $W$ the canonical process on $C([0,1],\mathbb{R})$ and the row filtration generated by the coordinate process of $W$, I want to prove that ...
-1
votes
2answers
52 views

Markov process which is not martingale

I have seen the examples of a discrete time martingale that is not a Markov Process. Can you construct me an example of discrete time Markov Process that is not a martingale?
1
vote
0answers
28 views

Semigroup associated to a Markov process

I'm studying the transition semigroup associated to a Markov Process, in particular the Hille-Yosida theorem and the Martingale Problem. In my notes I found : "If $\{T_t\}_t$ is a strongly continuous ...
1
vote
0answers
45 views

Autocorrelated, discrete, bounded and symmetric random walk with no edge attraction

I need to move over a discrete set of linearly organized.. let's say "Japan steps" $S=\{0,\dots,c\}, c \in \mathbb{N}^*$. My current position is given by $d \in S$. On each time step, I need to draw ...
1
vote
1answer
34 views

Properties of a transient state in a Markov Chain

I have been trying to solve this problem for a while now Prove that if $j$ is transient state, then $\displaystyle\sum_{n=1}^\infty p_{ij}^{(n)}<\infty \ \forall i \in S$, with $S$ the state ...
1
vote
1answer
26 views

Proving technique used to show an equivalence to the definition of a Markov process

Let $X=(X_t)_{t\in I}$ be Markov process with values in a Polish space $E$. I want to show, that there exists a stochastic kernel $\kappa:E\times\mathcal{B}(E)^{\otimes I}\to [0,1]$ such that ...
5
votes
1answer
3k views

Transition Kernel of a Markov Chain

Supposing $X_t$ is a Markov Process, can the transition kernel be defined by $$K_t(x,A):= P(X_{t+1} \in A | X_t = x)?$$ Assume that $X_t : \Omega \to \mathbb{R}^n$. The issue is that under the normal ...
0
votes
1answer
42 views

Assume a die is rolled repeatedly. Find the markov matrix $P$ for the random variable of the time until the next $6$.

Assume a die is rolled repeatedly. Find the markov/transition matrix $P$ for the random variable $X_r$ = the time until the next six at time $r$. My solution was: For $i,j \geq 0$, $P$ is given ...
2
votes
1answer
35 views

Markov processes and $C_0$-semigroups

A Markov process $(X_t)_{t \geq 0}$ in continuous time on $\mathbb{R}^d$ can be described by a semigroup of Markov kernels $(p_t(x,A))_{t \geq 0}$ with $p_0(x,A) = 1_{A}(x)$ and which fulfill the ...
3
votes
2answers
56 views

Markov Chains - Strong Markov Property

I'm struck with an exercise. I tried, but the results don't seem to fit to those proposed. Exercise: Two players play the following game. The one who begins draws two cards from a deck of 40 cards ...
0
votes
1answer
24 views

How to understand this kind of Markov chain?

There are two unidirectionally coupled processes $X_t$ and $Y_t$. The coupling is $Y_t=g(X_{t-\delta},\dots)$. The Bayesian network of the two process is described in the following figure: Now this ...
1
vote
0answers
12 views

Factorization of the Fokker-Planck semigroup

"In the classical theory of Markov processes, the Fokker-Planck semigroup $\{T_t:t\ge 0\}$ can be factored as $T_t=N\circ U_t \circ j$, $t\ge 0$, where $j$ is an embedding, $U_t$ is a group of ...
-1
votes
0answers
37 views

$\sup B_t$ has the same distribution as $\sup C_t$ for two brownian motions $B_t, C_t$

Let $(B_t)_{t \ge 0}$ and $(C_t)_{t \ge 0}$ be two standardized brownian motions. Now why is $\sup_{t \ge 0} B_t$ distributed same as $\sup_{t \ge 0} C_t$? This is a result we assumed as trivial ...
4
votes
0answers
39 views

Martingale converges to the boundary

I asked an almost same question before and it is solved by considering adjacent $Z_n$ can not be far away and obtain a contradiction. However, if the setting is altered a bit, I wonder whether it is ...
1
vote
0answers
25 views

Simple Markov property

I want to prove the simple Markov property but I come to a point where I do not see how to conclude. I want to prove $\mathbb{E}_\nu[Z\circ\Theta_t\mid ...
2
votes
1answer
71 views

Expression for the quotient between two stationary states in a Markov process

I've been thinking about this problem and I would appreciate some help. Consider a finite number of states ($n$) Markov process with transition matrix $Q_{n\times n}$ with the usual properties and ...
0
votes
0answers
14 views

model a system with finite users as a Markov Chain

I have to model a system M/M/2 with finite users (4 users) as a Markov Chain (and then find the probality an incoming users would enter the queue being the servers busy but that's not the problem). I ...
0
votes
0answers
5 views

Looking for resources on Harris recurrence

I'm working on a problem (in a not countable space) and it seems that I could get much further with it if I can prove that a certain Markov chain is Harris recurrent (I strongly suspect that it is). I ...
2
votes
0answers
43 views

Electrostatic capacity of two spheres with changing radii

Although I have read a lot of questions and answers here, this is my first time actually posting. Feel free to suggest needed edits. My question is the following (in a simplified setting). All this ...
0
votes
0answers
8 views

rigorous definition of adjoint infinitesimal generator for Markov process?

A Markov process on a $\sigma$-addive measurable space $(E,\Sigma,\mu)$ can be described by a family of operators $(P_t)_{t \geq 0} ,\, P_t: L^\infty(\mu) \to L^\infty(\mu)$. These operators define a ...
0
votes
0answers
32 views

Markov Chains and transition semigoups

I'm trying to figure out what the following statement refers to. A process $X$ is markov with transitions semigroup $(K_t)_{t\geq0}$ and initial distribution $\mu$ if and only if for all ...
0
votes
1answer
580 views

Expected state of a Markov chain

Let's start with a slightly trivial Markov chain defined as follows: the beginning state is called $1$ and the set of states is $\mathbb{N}$. At each step, when the current state is $n$, the ...
0
votes
0answers
13 views

How does one estimate the order of a Markov chain empirically (given the data)?

I have a string of symbols $x_1, x_2, ...., x_n$, ($n$ very large), belonging to a finite alphabet. I know that they are a result of a Markov process, but I want to find out the order of the process. ...
1
vote
0answers
66 views

Doubt Concerning Markov Property

Given a Markovian process $(X_t )_{t\geq 0 }$, is the following property accurate? $$\mathbb E \left[ f(X_{t_1}, X_{t_2},X_{t_3}) \mid \mathcal F ^X_{t_2}\right] = \mathbb E \left[ f(X_{t_1}, ...
3
votes
0answers
39 views

Correct steps in rewriting expectation to a probability

My knowledge of measure theory and probability spaces is limited, so please keep it relatively simple. Let $\{X(t), ~ t \ge 0\}$ be a Markov process on the countable state space $\mathbb{N}_0$ with ...
1
vote
1answer
21 views

Condition for the existence of a Markov process from the properties of a semigroup $\{T_t\}$

In the article Diffusion processes with continuous coefficients I (1969, Stroock and Varadhan) one finds the following arguments in pages 26-27 "$(\cdots)$ for any $ \epsilon >0, \sup_{x \in ...
0
votes
0answers
41 views

Calculating the generator of a weighted transition function

Let $(P_t)_t$ be the transition function of a Feller-Dynkin process $X$. The usual Banach space of functions that the semigroup $(P_t)_t$ is working on is $C_0(E)$, i.e. continuous functions that ...
0
votes
0answers
16 views

Ergodicity property for continuous-time Harris positive Markov process

The following theorem is Theorem 13.3.3 of Meyn and Tweedie's Markov Chains and Stochastic Stability on page 328 Theorem 13.3.3. If $\Phi$ is positive Harris and aperiodic, then for every initial ...
0
votes
0answers
29 views

Can solution to Rubik's cube be seen from the point of view of Markov Decision Process?

Solving Rubik's cube can be thought of as a Planning problem which has : a state space $S$ a set $G \subseteq S$ of goal states (in this case singleton) actions $A(s) \subseteq A$ applicable in ...
0
votes
0answers
25 views

Probability of hitting zero

Suppose time is discrete. $X_{t+1} = X_t + x_t$. $x_t$ is of continuous value, iid with mean zero and finite variance. Let initial condition $X_0>0$, how can I prove that the probability of $X_t$ ...
1
vote
1answer
55 views

Is this PMF or PDF?

I am reading a technical report on expectation-maximization (EM) algorithm (http://melodi.ee.washington.edu/people/bilmes/mypapers/em.pdf) and I am confused about something. For HMMs, it defines ...
2
votes
0answers
17 views

Discrete-Time Stochastic Calculus and Stopping Times: Resources

In my measure-theoretic probability course we covered what the professor called "discrete-time stochastic calculus". Essentially, it was a three part method for computing certain quantities such as ...
0
votes
1answer
55 views

The probability of going bankrupt

The question looks like a gambling problem. But I am not sure whether it is the same or similar to gambler's ruin problem. Assume I have wealth $W_t$. At each step $t$, I encounter a random shock ...
0
votes
0answers
94 views

Server farm model and non-explosiveness

Consider the following server farm model. Customers arrive at a server farm according to a Poisson process at rate $\lambda$, each requesting a machine for an amount of time that has an exponential ...
1
vote
1answer
17 views

Ornstein-Uhlenbeck-Model with irregular time intervals

I have data of the form: Time t Price x(t) 0 80 21 82 24 82.3 32 81.5 ... ... The point is, that the time intervals are highly ...
0
votes
0answers
20 views

Partial differential equation involving a random process (literature advice)

In articles like this one (end of page one and page two), physicists often tend to treat a random process with discrete time and countable space set as a differentiable function (whose domains are ...
4
votes
1answer
50 views

Simple random walk: What is the probability that the hitting time is exactly 2n?

I refer to the random walk $(S_n)_{n \geq 1}$ where $S_n = X_1 + \cdots + X_n$ and $X_i$ are i.i.d random variables taking values $\pm 1$ with equal probability. I want to know how to show that ...
2
votes
0answers
24 views

Limit of decreasing sequences of markov time (stopping time) is markov time?

Let $(\Omega, \mathcal{F}, \{\mathcal{F}_t\}_{t \geqslant 0}, \mathbb{P})$ be a filtered probability space and let $\tau_n \geqslant \tau_{n+1}$ be a markov time (stopping time) with respect to ...
1
vote
0answers
22 views

Discrete time Pure-Birth process with population fraction

I would like to solve difference equations for a pure-birth process, where the rate of adding new nodes depends on the fraction of population. $$x_i(t+1)-x_i(t)=\alpha ...
0
votes
0answers
18 views

What is the axiomatic definition of potential kernel?

In the enter link description here, a kernel V is a Markov potential kernel if the operator I+V is invertible and its inverse is of the form I􀀀-N with N a sub-Markov kernel.The definition of Markov ...
0
votes
1answer
42 views

Escape probabilities in a random walk.

So, a lot of theory in symmetric random walks seems to concentrate on 'hitting/stopping times' and things like that. So I started wondering... How would I go about calculating the probability of ...
1
vote
0answers
40 views

Hidden Markov Model Transition Probability

I am doing my assignment and I am asked to derive transition probability of a HMM. There are Three states. H, E and T. They initially gave me the information as follow. E is followed by an H 40% ...
2
votes
1answer
70 views

Understanding a proof of the strong Markov property of Lévy processes

I don't understand the the last sentence of a proof of the Markov property for Lévy processes given in Jochen Wengenroth's textbook "Wahrscheinlichkeitstheorie" (de Gruyter, 2008). I will appreciate ...
1
vote
1answer
109 views

Stationary Markov process properties

Let $X$ be a right-continuous process with values in $(E,\mathcal{E})$, defined on $(\Omega, \mathcal{F}_t,P)$. Suppose that $X$ has stationary, independent increments. I now want to show the ...
0
votes
0answers
46 views

markov process and markov chains

I have learned that Markov processes are stochastic processes possessing certain mathematical properties (memoryless, etc). My question is, if you say that a process is Markov, is it automatic (as a ...
0
votes
1answer
46 views

Shannon's definition of ergodicity

In A Mathematical Theory of Communication (1948) Shannon gives a definition of ergodicity for a Markov process. In order to be ergodic the directed graph of the process must have the following ...
1
vote
1answer
64 views

Escape time for a not absorbing state

Let $X$ be a right-continuous Feller Dynkin process. For $r>0$ we define the $\{\mathcal{F}_t\}_t$ stopping time (which is called escape time) $$\eta_r=\inf\{t\geq 0: \|X_t -X_0\|\geq r\}$$ We have ...
1
vote
0answers
28 views

Finding the generating function of $H_{0}$ probability of hitting 0 in Markov Chain

Let $Y1 , Y2,...$ be independent identically distributed random variables with $\mathbb{P}(Y1 =1)=\mathbb{P}(Y1 =-1)=1/2$ and set $Xo=1,Xn =Xo+Y1+...+Yn$ for $n\geq1$. Define; $$H_o= inf\{n\geq0:Xn = ...
1
vote
1answer
39 views

What is the probability of arrive either A or B at starting point K?

There are two points which are $A$ and $B$. The distance between $A$ and $B$ is $50$ meter. One person goes to $A$ with probability $\frac{1}{6}$, he goes to $B$ with probability $\frac{3}{6}$. And he ...
5
votes
4answers
5k views

Why Markov matrices always have 1 as an eigenvalue

Also called stochastic matrix. Let $A=[a_{ij}]$ - matrix over $\mathbb{R}$ $0\le a_{ij} \le 1 \forall i,j$ $\sum_{j}a_{ij}=1 \forall i$ i.e the sum along each column of $A$ is 1. I ...