A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
1answer
32 views

Continuous-time Markov Question

I have a question about a continuous-time Markov process on the discrete space. I am given the generator and asked for find the expected time the Markov process needs to get back to state 3, given ...
0
votes
1answer
66 views

Showing the square of a Markov process is or isn't Markov

Hi I am trying to show that if $X_n$ is a markov process, whether or not $X_n^2$ is a markov process. $X_n$ is a markov process if $P\{X_k = a_k|X_{k-1} = a_{k-1}, X_{k-2} = a_{k-2}, ..., X_k = a_1 ...
0
votes
2answers
66 views

Find the generator of Markov Process

Homework question: Consider the Markov process $X_t=B_t-t^2+t$ where $B_t$ is the Brownian motion. Find the generator $Q$ of this process. I am completely confused how to find the generator for ...
0
votes
1answer
37 views

Steady state of a $4 \times 4$ transition matrix

Normally I just take $q(M_{m\times n} - I_{m\times n})$ to workout the steady state, but here I have: $$\left(\begin{array}{rrrr} 0 & 0 & .8 & .2 \\ .4 & .6 & 0 & 0 \\ .2 ...
0
votes
1answer
27 views

Limiting probability of Markov chain(Terminology)

If I am asked to find the limiting probability of a Markov chain, what does this pertain to? $\lim \limits_{n \to \infty} P^n$? Where $P$ is the stepping matrix and $n$ is the number of steps. "What ...
0
votes
0answers
32 views

Error in thinking: Poisson Process is a Markov Process

I am a bit confused on proving the Markov property for Poisson processes. That is, we want to prove, if $X = (X_t: t \in \mathbb{R})$ is a Poisson process with rate $\lambda$: $P(X_{t_n} = a_n | ...
1
vote
0answers
77 views

How's the damping factor in Google PageRank algorithm calculated

I'm doing some researches about Google's PageRank algorithm for my thesis, I've found that the damping factor x (for example), where x is in : P` = x.P + (1-x)Q where P is the original ...
0
votes
0answers
71 views

Stationary distribution of a “birth-death model” that does not have Markov property

A typical birth-death process is defined such as the probability of going from any state $j$ to any state $i$ is given by: $$ p_{ij}= \begin{cases} b_i & \text {if $j = i+1$} \\ ...
1
vote
0answers
25 views

showing a condition implies convergence to invariant distribution

For an arbitrary Markov chain, I'm trying to show that $\lVert{ P_{\mu}\circ \theta_n^{-1} - \nu \rVert}\to 0$ iff $\sum_j \lvert{ P_{ij}^n - \nu(j)\rvert}\to 0$, where $\theta_n^{-1}$ is a shift ...
0
votes
1answer
22 views

inequality for finite state markov chains

Let $X$ be a discrete-time Markov process in $S$ with invariant distribution $\nu$. Show that for any measurable set $B\subset S$ such that $$P_{\nu}\{X_n \in B\, \textrm{i.o.} \}\geq \nu B.$$ I'm ...
2
votes
1answer
67 views

continuous time Markov chain, something the book does not explain

I have a problem with something in my book, under the chapter of continuous time Markov chains. I will post a link to what the book does. They do something which they seem to take for granted, but I ...
0
votes
0answers
17 views

radius of markov chain

For an irreducible markov chain one can show, that $\limsup \sqrt[n]{p_{ij}^{(n)}}$ is independent of the choice of the states $i$ and $j$ where $p_{ij}^{(n)}$ is the probability to get from $x$ to ...
0
votes
1answer
55 views

Representation of Markov process adapted to given Filtration

Let $X$ be continuous Markov process adapted to a filtration generated by Brownian motion $B$. Does there exist a function $f$ such that $X_t = f(t,B_t)$? My guess is that it should have such ...
0
votes
1answer
92 views

Ornstein-Uhlenbeck processs: Markov, but not martingale?

I'm puzzled about properties of the Ornstein-Uhlenbeck process, given by the Itō integral $$ X_t = x e^{-\lambda t} + \sigma \int_0^t e^{-\lambda(t-s)} d W_s \,. $$ I compute that $\{X_t\}$ is not ...
0
votes
1answer
55 views

Markov chain with infinite number of transient and positive recurrent states?

Is it possible to have a markov chain with an infinite number of transient states, and an infinite number of positive recurrent states? Thank you!
1
vote
2answers
247 views

Prove markov chain is null recurrent

Two fair coins are tossed repeatedly. Let Xn denote (Total Number of Heads from Coin 1)-(Total Number of Heads from Coin 2) after n tosses. Thus the state space is {0, ±1, ±2, .... }. Show that the ...
1
vote
0answers
20 views

Three-State Markov Process with Differential Equations

This question is from a take home quiz and I could really use the help. Thanks in advance
2
votes
2answers
383 views

Mean recurrence time and stationary distribution of a Markov chain?

In a Markov chain is there a theorem relating the existence of the stationary distribution and the mean recurrence time? E.g. impossible for stationary distribution to exist therefore mean recurrence ...
0
votes
0answers
77 views

proving null recurrence of random walk (Markov chain)

How would I prove that the zero state of a random walk with a positive probability of staying in the same state is null recurrent. (sorry if this isn't a random walk and just a Markov chain.) eg. ...
3
votes
1answer
139 views

What is the value of this game?

We have 3 black and 2 red balls in an urn. If we pick a black ball, we lose 1 USD. If we pick a red ball we win 1 USD. We can chose to start the game or not. If we start the game we can stop after ...
0
votes
1answer
28 views

Show $P(S_{2n}=x|S_0=x) \ge \frac{1}{N}$

Let $X_n$ be an aperiodic, discrete-time Markov chain so $S=\{1,...,N\}$ whose transition probability is symmetric. How can I show that for all $x \in S$ and all integers $n$, $P(S_{2n}=x|S_0=x) \ge ...
0
votes
0answers
27 views

Maximal principle for elliptic or linear integro-differential operator

Consider $L$ the operator forming as $$ Lg= -g^{'}(x)+(g(x+1)-g(x)) $$. $h$ on $[0,\infty)$ satisfies the following integro-differential equation $$ Lh \geq 0 $$ with boundary condition: $$ ...
0
votes
1answer
90 views

Transition rate matrix from transition probability matrix

If I have a $2 \times 2$ continuous time Markov chain transition probability matrix (generated from a financial time series data), is it possible to get the transition rate matrix from this and if ...
0
votes
0answers
78 views

Continuous time Markov chains, how would this definition be expanded from time-homogeneous to time-inhomogeneous.

Below I have a picture of how we can view a continuous time Markov chain that is time-homogeneous. Now, I am wondering what happens when we have a inhomogeneous continuous Markov chain. I have ...
0
votes
1answer
63 views

Markov processes and semimartingales

Semimartingales and Markov processes are two fundamental families in probability theory. There are many specific processes that belongs to the intersection of those two families, e.g. Levy processes. ...
1
vote
1answer
65 views

What is the Deterministic Traffic Generation Model?

I am studying Markov chains and queuing theory. I was curious about traffic generation models and actually happened to see the Deterministic Traffic Model, referred to as $D$ in Kendall's notation. ...
0
votes
1answer
37 views

Queue system with queue-triggered input process

I have a queue system, a classic system with an input generator, a queue and a servant. The servant is a $M$-servant with a certain serving rate $\mu$. The queue can contain an infinite number of ...
0
votes
1answer
35 views

Average time of permanence in a state of a Markov-chain

I know that in a Markov-chain the mean permanence time in a state is a random variable distributed accordingly to: Geometric distribution for Time Discrete Markov Chains Exponential distribution for ...
0
votes
0answers
25 views

Is the exit time independent of the state jumped to in a Markov chain?

Let $X$ be a continuous time Markov chain on a countable state space $S$, and let $\tau_n$ be the $n^{th}$ time at which the chain jumps out of a set $D$ (i.e. times $t$ at which, for some $\epsilon ...
0
votes
0answers
76 views

Markov renewal process vs Markov Jump process

The venerable wikipedia for "Markov renewal process" says that: "a Markov renewal process is a random process that generalizes the notion of Markov jump processes". So what's the definition of a ...
0
votes
0answers
35 views

How to compute the probability of a node given its Markov Blanket?

I'm trying to find P(a|b,e,~j,~m) in the image below. I think it's .3774, but when I try doing it by using the markov blanket formula (also in the picture) I get: P(a | b,e)*P(~j | a) * P(~m | a) = ...
0
votes
0answers
158 views

The difference between Dynamic Optimization, Stochastic Programming, Optimal control and Markov Decision Processes

I've seen the following terms thrown around somewhat interchangeably, and I'm confused. What are the distinctions between them, and what are some representative problems that each deals with? ...
3
votes
1answer
275 views

Elementary proof of geometric / negative binomial distribution in birth-death processes

The birth-death process concerns a population of $n_0$ individuals, each of which reproduce and die at a constant rate as time $t$ increases from $t=0$. Each individual splits into two individuals ...
10
votes
2answers
581 views

What is the importance of the infinitesimal generator of Brownian motion?

I have read that the infinitesimal generator of Brownian motion is $\frac{1}{2}\small\triangle$. Unfortunately, I have no background in semigroup theory, and the expositions of semigroup theory I have ...
1
vote
2answers
133 views

Expected number of steps between states in a Markov Chain

Suppose I am given a state space $S=\{0,1,2,3\}$ with transition probability matrix $\mathbf{P}= \begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 \\[0.3em] \frac{2}{3} ...
0
votes
0answers
29 views

Poisson processes are the only renewal processes which are Markov Chains.

How would one prove the Proposition: "Poisson processes are the only renewal processes which are Markov Chains." A renewal process $N=(N(t))$ is a process for which $$N(t)=\max\{n : T_n \leq t\}$$ ...
0
votes
0answers
27 views

Invariant measure for gradient flows

I found here that a stochastic process from the SDE $dX_t=-\nabla V(X_t)dt + \sqrt{2 \beta^{-1}} dB_t $ has a unique invariant measure (if $V$ is suitable smooth), also called Gibbs distribution ...
0
votes
1answer
79 views

Exercise on Markov chain

Prove, or give an explicit counterexample to refute, the following assertion: if $\{X_n\}$ is a Markov chain, then $\{X_n^2\}$ is also a Markov chain. It's easy to show that ...
3
votes
1answer
89 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is ...
3
votes
1answer
114 views

Markov chain problem with finite states

Consider any Markov chain on a state space with exactly $r$ states. We want to find the largest $N>0$ such that there exists states $i,j$ for every $n < N$ we have $p_{ij}^{(N)} > 0$ and ...
1
vote
1answer
72 views

Stochastic processes with independent increments

If $\{X_{t}:t\geq 0\}$ is a real-valued stochastic process with independent increments then $\{X_{t}:t\geq0\}$ is a Markov process? Let $\{ \mathcal{F}_{t} \}_{t\geq0} $ be a natural filtration of ...
1
vote
0answers
49 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
3
votes
1answer
97 views

Strong Markov property of Brownian motion

I was able to understand Brownian Motion $\{B(t):t\geq0\}$ has Strong Markov Property i.e. For any stopping time $\tau$, $P(B(t+\tau)\leq y | \mathcal{F}_{\tau})=P(B(t+\tau)\leq y|B(\tau))$ a.s. , $y ...
1
vote
2answers
39 views

Calculation of the Gallavotti-Cohen fluctuation theorem made by Lebowitz

I have a problem understanding a calculation in this paper (another form of the theorem an be found here at equation 11). For those who want to read the paper, I have difficulties with formula 2.14 in ...
1
vote
1answer
32 views

Consider a Stochastic process X which moves between the corners of a regular tetrahedron. Find P(Xn = 1)

Consider a Stochastic process $X$ which moves between the corners of a regular tetrahedron. The process starts at time 0 in node 1. At each time step, the process chooses an one of the connected edges ...
3
votes
1answer
76 views

Random walk where increment depend on current position

Consider the following stochastic process, $$b(i+1) = b(i) + \xi_i (b_i),$$ where $\xi_i(b_i) \in \{-1, k \}$ are the independent increments having the following distribution: $$\begin{align} P (\xi ...
1
vote
1answer
71 views

Library chain stationary distribution

This is an exercise 1.47 from Richard Durrett's Essentials of Stochastic Processes p.85 (doi: 10.1007/978-1-4614-3615-7_1 or Google Books). On each request the ith of the $n$ possible books is the ...
0
votes
1answer
56 views

Poisson Process (Easy)

I'm stuck at the following question: Customers with items to repair arrive at a repair facility according to a Poisson process with rate λ. The repair time of an item has a uniform distribution on ...
1
vote
2answers
65 views

Markov chain simulation

I'm wonder if there is an algorithm to simulate a discrete Markov chains with a specific number of occurence of state knowing the transition matrixway. For example, how to simualte in $\mathbb R$ a ...
1
vote
1answer
169 views

Equivalence Classes of a Markov Chain with Transition Matrix

I have the following transition probability matrix for a markov chain with state space S={0,1,2,3,4,5,6}: $\begin{bmatrix} \frac13 & \frac13 &0 & 0 & \frac16 & 0 & \frac16\\ ...