1
vote
1answer
79 views

A Markov Chain Problem.(Change the color of ball)

There are $n$ different color balls in a box. Take two balls in turns, and change color of the second ball to the first. (This is one operation). Let $k$ be the (random) number of operations needed to ...
0
votes
1answer
18 views

Markov property for a stochastic process with discrete state space.

Consider a stochastic process $\{X_s\}_{s\in\mathcal S\subseteq\mathbb R}$ with value in $(\mathbb R,\mathcal B(\mathbb R))$ adapted to a filtration $\{\mathcal F_s\}$ (we can suppose that ...
0
votes
0answers
15 views

Probabilities in a Markov Model

I am reading a paper on Markov Models and I am trying to figure out how to compute the probabilities for the $\alpha$-pass. I am given an $N\times N$ matrix $A$, that has the probabilities of ...
0
votes
1answer
30 views

Proving that a process has the Markov property

Let $X_t=xe^{ct+aB_t}$ where $B_t$ is one dimensional Brownian motion. How would I prove this is a Markov process using the expectation definition of a Markov process, i.e., ...
0
votes
1answer
38 views

Can the transition probabilities of an inhomogeneous Markov chain be written as an exponential?

If $Z_t$ is a homogeneous continuous-time Markov chain with finite state space $E=\{1,\ldots,p\}$, transition matrices $(P(t))$ and intensity matrix $Q$, it holds that $$ P(t) = \exp(tQ), $$ see for ...
0
votes
1answer
28 views

Q-matrix vs. P-matrix description of a Markov chain

Consider a continuous time Markov chain $(X_t)_{t \geq 0}$ on some state space $S$ with transition matrix (P-matrix) $p_t(x,y)$, the probability density of jumping from $x$ to $y$ in time $t$. The ...
2
votes
2answers
23 views

Probability of returning to a given state after n transitions-Markov chains

Let us denote $f_j^{(n)}$ denote the probability of the first return to state $j $after n transitions. Let $p_{jj}^{(n)}$ be the probability of returning to the state $j$ after $n$ transitions when ...
1
vote
1answer
43 views

Property of Wiener process sample path

What is a mean of time, when the trajectory of wiener process $W_t$ is over the line $y=t$? We need to find $\mathbb{E}\tau$, where $\tau=\sum\limits_{a,b:\forall t\in(a,b) ; ...
1
vote
0answers
13 views

Branching Brownian Motion and KPP equation

I have troubles understanding the proof of the connection between BBM and KPP equation. I mean the proof of the next lemma from the lecture notes of Anton Bovier about BBM, link. This is almost whole ...
0
votes
0answers
40 views

Can ergodic Markov chains be periodic?

I found a statement in one of my notes which said If a state is persistent, aperiodic and not null the it is said to be ergodic Is it necessary that it should be aperiodic? This statement ...
1
vote
1answer
25 views

Markov chains: An issue in classification of states

I recently came across a lemma which goes as follows. Suppose a Markov chain has N states. Let i and j be pair of states. Then j can be reached from i iff there is an integer $ 0 ≤n< N$ such ...
0
votes
2answers
53 views

Expected number of steps to absorbtion - Markov chain

I want to calculate the expected number of steps (transitions) needed for absorbtion. So the transition probability matrix $P$ has exactly one (lets say it is the first one) column with a $1$ and the ...
0
votes
0answers
16 views

A question in a textbook about Blumenthal 0-1 law for a general Markov process

This question came up as a result of reading this question . Here is the Blumenthal 0-1 law in the book Stochastic Processes by Richard F. Bass. Proposition 20.8 Let $(X_t , \Bbb{P}^x)$ be a ...
0
votes
1answer
25 views

Writing down the transition matrix of a discrete Markov chain

Please consider the following scenario: One person is walking along a discrete circle induced by $\mathbb{Z}/n\mathbb{Z}$ In each round we roll a dice with $w\in\left\{2,\ldots n\right\}$ sides If ...
0
votes
1answer
30 views

Inferring transition rates from continuous markov chain question

A house has 2 rooms of similar sizes with identical air conditioners equipped with thermostats which turn on and off as needed to maintain the temperature in each room to a desired level of 22 ...
1
vote
1answer
59 views

random walk with sticky barriers

Consider a random walk on the line 1,...,d. You start at point 1. At each step you flip a coin: heads means go left, tails means go right. If you're at 1 and get a heads, just stay where you are ...
0
votes
0answers
22 views

Transition matrix in left-right hidden semi-Markov model

I'm developing a hidden semi-Markov model left-right . In a left-right model a sequence of $M$ states starts in state $1$ and ends in state $M$, with no repetition of states. Since the model is ...
1
vote
2answers
23 views

Can the ergodic theorem for Markov chains be proved with linear algebra?

This theorem is in my book, let me just say that it is for discrete-time Markov chains, that are time-homogeneous. Ergodic is defined in the book as being positive recurrent and aperiodic. The ...
0
votes
0answers
30 views

Probability of not reaching completion in Markov process

This question is supposed to be easy but is very hard for me. The Norwegian Skating Association has mass produced certain "collectors' cards" with all $N$ speedskaters (Norwegian as well as ...
0
votes
1answer
36 views

Markov chain exercise

Hello i have this Markov chain exercise: Basically we can always move up 1 step, but there is always a possibility that we will go down to the first state 0, the Markov chain consists of N states. ...
2
votes
1answer
39 views

Question regarding Notes on Strong Markov Property

I wrote the following notes from a lecture a couple of weeks ago and I don't understand a particular line. Suppose $B_t$ is a Brownian Motion. Now look at $B^x_t = x + B_t$ which is a BM starting ...
2
votes
1answer
63 views

Monotonicity and Convexity of Stochastic Matrices

The definition of stochastic monotonicity and convexity is given by "Stochastic Orders and Their Applications" by Moshe Shaked and George Shanthikumar (1994) as: Let $P = \{p_{i,j} \}$ be a ...
0
votes
3answers
89 views

How fast does this Markov chain converge?

Observe the above a Markov chain and limiting matrix of it. Finding the limiting matrix if it exists is easy but I am curious as to how fast this given matrix converges to its limiting matrix. Is ...
0
votes
0answers
52 views

Strong Markov property for Poisson point process

The question is thoroughly contained in the title. I just say that I would only like to find a reference for this question. I have searched in some books, to no avail. Just to avoid misunderstanding, ...
1
vote
1answer
128 views

Probability of a trajectory in Markov processes

I need help with a simple formula! (My question is taken from here, pag 26 eq 1.112. ) Consider a Markov Process with associated Master Equation: \begin{equation*} ...
0
votes
0answers
28 views

Poisson process different type of events

Suppose that it arrives people to a store according to a poisson process with rate $\lambda = 6$/hour , females arrive with probability $0.6$ and male with $0.4$. What is the probability that there ...
2
votes
1answer
62 views

Inverse of a regular stochastic matrix

Is it true that the inverse of a regular stochastic matrix is also regular? Are there any other interesting features that the inverse may have of a regular stochastic matrix? Hope someone could answer ...
1
vote
0answers
33 views

References for time-inhomogeneous Markov jump processes?

In some central models in life insurance mathematics, the state of the insured is modeled using a continuous-time time-inhomogeneous Markov process with finitely many states. While many results for ...
0
votes
0answers
36 views

relations between properties of stochastic processess

If we have an integer valued stochastic process, are these implications correct? independent increments $\rightarrow$ Markov property Markov property $\nrightarrow$ independent increments stationary ...
2
votes
1answer
57 views

time-homogeneous continuous time Markov chain

I have a question about the continuous time Markov chain. In the Poisson process we have independent and stationary increments. Do we have this in a continuous time Markov chain that is ...
0
votes
1answer
23 views

Computing the PMF for N(t) in a renewal process

I'm in a stochastic processes course, and we just started on renewal theory. Unfortunately, we skipped the section on queuing theory, and nearly example in my textbook for renewal processes uses ...
0
votes
1answer
35 views

Showing the square of a Markov process is or isn't Markov

Hi I am trying to show that if $X_n$ is a markov process, whether or not $X_n^2$ is a markov process. $X_n$ is a markov process if $P\{X_k = a_k|X_{k-1} = a_{k-1}, X_{k-2} = a_{k-2}, ..., X_k = a_1 ...
0
votes
0answers
28 views

Error in thinking: Poisson Process is a Markov Process

I am a bit confused on proving the Markov property for Poisson processes. That is, we want to prove, if $X = (X_t: t \in \mathbb{R})$ is a Poisson process with rate $\lambda$: $P(X_{t_n} = a_n | ...
2
votes
1answer
63 views

continuous time Markov chain, something the book does not explain

I have a problem with something in my book, under the chapter of continuous time Markov chains. I will post a link to what the book does. They do something which they seem to take for granted, but I ...
0
votes
1answer
45 views

Representation of Markov process adapted to given Filtration

Let $X$ be continuous Markov process adapted to a filtration generated by Brownian motion $B$. Does there exist a function $f$ such that $X_t = f(t,B_t)$? My guess is that it should have such ...
0
votes
1answer
75 views

Ornstein-Uhlenbeck processs: Markov, but not martingale?

I'm puzzled about properties of the Ornstein-Uhlenbeck process, given by the Itō integral $$ X_t = x e^{-\lambda t} + \sigma \int_0^t e^{-\lambda(t-s)} d W_s \,. $$ I compute that $\{X_t\}$ is not ...
0
votes
1answer
28 views

Show $P(S_{2n}=x|S_0=x) \ge \frac{1}{N}$

Let $X_n$ be an aperiodic, discrete-time Markov chain so $S=\{1,...,N\}$ whose transition probability is symmetric. How can I show that for all $x \in S$ and all integers $n$, $P(S_{2n}=x|S_0=x) \ge ...
0
votes
1answer
37 views

Transition rate matrix from transition probability matrix

If I have a $2 \times 2$ continuous time Markov chain transition probability matrix (generated from a financial time series data), is it possible to get the transition rate matrix from this and if ...
0
votes
0answers
59 views

Continuous time Markov chains, how would this definition be expanded from time-homogeneous to time-inhomogeneous.

Below I have a picture of how we can view a continuous time Markov chain that is time-homogeneous. Now, I am wondering what happens when we have a inhomogeneous continuous Markov chain. I have ...
0
votes
1answer
51 views

Markov processes and semimartingales

Semimartingales and Markov processes are two fundamental families in probability theory. There are many specific processes that belongs to the intersection of those two families, e.g. Levy processes. ...
0
votes
0answers
51 views

Markov renewal process vs Markov Jump process

The venerable wikipedia for "Markov renewal process" says that: "a Markov renewal process is a random process that generalizes the notion of Markov jump processes". So what's the definition of a ...
0
votes
0answers
115 views

The difference between Dynamic Optimization, Stochastic Programming, Optimal control and Markov Decision Processes

I've seen the following terms thrown around somewhat interchangeably, and I'm confused. What are the distinctions between them, and what are some representative problems that each deals with? ...
3
votes
1answer
184 views

Elementary proof of geometric / negative binomial distribution in birth-death processes

The birth-death process concerns a population of $n_0$ individuals, each of which reproduce and die at a constant rate as time $t$ increases from $t=0$. Each individual splits into two individuals ...
9
votes
2answers
309 views

What is the importance of the infinitesimal generator of Brownian motion?

I have read that the infinitesimal generator of Brownian motion is $\frac{1}{2}\small\triangle$. Unfortunately, I have no background in semigroup theory, and the expositions of semigroup theory I have ...
3
votes
1answer
69 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is ...
1
vote
1answer
61 views

Stochastic processes with independent increments

If $\{X_{t}:t\geq 0\}$ is a real-valued stochastic process with independent increments then $\{X_{t}:t\geq0\}$ is a Markov process? Let $\{ \mathcal{F}_{t} \}_{t\geq0} $ be a natural filtration of ...
1
vote
0answers
37 views

Markov Chain depicting unruly customer behavior

A store has 2 bins of balls. 1 bin is red, and contains 3 red balls. The other bin is gray and contains 2 gray balls. Every minute, on the minute, exactly one customer comes by the bins, picks up ...
3
votes
1answer
81 views

Strong Markov property of Brownian motion

I was able to understand Brownian Motion $\{B(t):t\geq0\}$ has Strong Markov Property i.e. For any stopping time $\tau$, $P(B(t+\tau)\leq y | \mathcal{F}_{\tau})=P(B(t+\tau)\leq y|B(\tau))$ a.s. , $y ...
1
vote
2answers
36 views

Calculation of the Gallavotti-Cohen fluctuation theorem made by Lebowitz

I have a problem understanding a calculation in this paper (another form of the theorem an be found here at equation 11). For those who want to read the paper, I have difficulties with formula 2.14 in ...
1
vote
1answer
65 views

Library chain stationary distribution

This is an exercise 1.47 from Richard Durrett's Essentials of Stochastic Processes p.85 (doi: 10.1007/978-1-4614-3615-7_1 or Google Books). On each request the ith of the $n$ possible books is the ...