A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
1answer
99 views

Understanding detailed balance equations

I'm trying to understand how the equilibrium distribution satisfy the detailed balance equation. To my understanding, I only understand that a detailed balance equation would only be satisfied if ...
2
votes
1answer
85 views

Monotonicity and Convexity of Stochastic Matrices

The definition of stochastic monotonicity and convexity is given by "Stochastic Orders and Their Applications" by Moshe Shaked and George Shanthikumar (1994) as: Let $P = \{p_{i,j} \}$ be a ...
0
votes
3answers
119 views

How fast does this Markov chain converge?

Observe the above a Markov chain and limiting matrix of it. Finding the limiting matrix if it exists is easy but I am curious as to how fast this given matrix converges to its limiting matrix. Is ...
0
votes
0answers
171 views

Application of Markov Chain to Game of Life Board Game

I need to calculate the expected outcomes for the Game of Life. I believe that if I multiply the probability of landing on a particular square with the payoff of said square and add up all these ...
0
votes
1answer
22 views

Markov Processes: How to show $\int P(X_t\in B\mid X_0)dPX_0^{-1}=P(X_t\in B)$?

Let $\left\{X_t \right\}_{t\in T}$ be a time homogeneous Markov process with state space $S$. How do I formally demonstrate$$P(X_t\in B)=\int_S P(X_t\in B\mid X_0)dPX_0^{-1}$$(here $PX_0^{-1}$ is the ...
1
vote
1answer
63 views

Markov: Expected time of first visit to a state starting from that state.

Question: Calculate the expected time of first visit to state 2 given we start in state 2. Is the answer to this the mean recurrence time of 2 or simply zero? I at first thought that the answer ...
0
votes
0answers
26 views

The second eigenvalue of a reducible stochastic matrix

The magnitude of the second dominant eigenvalue of a reducible matrix, as I know, is supposed to be 1, why it's not the case for this matrix : $$ \begin{matrix} 0 & 1 & 0 ...
1
vote
1answer
36 views

Markov processes limiting probability questions

I am going over previous mock exams in preparation for an upcoming exam and am having problems with parts (ii) and (iv) and was looking for some guidance. For part (ii), my thinking was that the ...
0
votes
1answer
59 views

How to use symmetry of transition rate matrix in a continuous-time Markov chain?

This is part of a bigger question, so I have to change the question a bit to focus on the point. We have a continuous- time Markov chain with the following transition rate matrix: $$Q= \begin{pmatrix} ...
0
votes
0answers
44 views

Positive probability distribution for Markov Random Fields

While stating results for Markov Random Fields, like the Hammersley-Clifford theorem, why there is emphasis on the requirement that the joint probability distribution should be positive?
0
votes
0answers
64 views

Strong Markov property for Poisson point process

The question is thoroughly contained in the title. I just say that I would only like to find a reference for this question. I have searched in some books, to no avail. Just to avoid misunderstanding, ...
1
vote
1answer
155 views

Probability of a trajectory in Markov processes

I need help with a simple formula! (My question is taken from here, pag 26 eq 1.112. ) Consider a Markov Process with associated Master Equation: \begin{equation*} ...
0
votes
0answers
34 views

Poisson process different type of events

Suppose that it arrives people to a store according to a poisson process with rate $\lambda = 6$/hour , females arrive with probability $0.6$ and male with $0.4$. What is the probability that there ...
0
votes
1answer
43 views

Continuous time Markov chain. proportion of time spent in state i

If a question asks for the proportion of time spent in a specific state is this the same as the stationary distribution or something else? For continuous time Markov chain with finite state space.
0
votes
2answers
106 views

Continuous Markov chains, arriving pairs

I have been trying to sort out this exercise but really stuck on this. Preparing myself for exams and found many exercise on continuous Markov chains but I am always stuck when it comes to transition ...
0
votes
1answer
21 views

A problem on Markov process

Suppose, $\Pi_{\theta}$ be the transition probability function of a Markov chain. For any function $f$ define $$\Pi_{\theta}f_{\theta}(x) = \int f(y,\theta)\Pi_{\theta}(x,dy).$$ Is there any ...
1
vote
1answer
49 views

Markov chain property

I would like to make clarification and show my curiosity about markov process. I will show some part of definition related to markov process from here. The Markov property is the dependence ...
1
vote
1answer
39 views

A doubt on markov decision process

Given that a policy is a function from a state action pair to probabilities, the set of policies for a MDP forms a POSET (the partial order is due to value function for a policy). Why there should be ...
1
vote
1answer
77 views

Reflection Principle interpretation

Given a standard Brownian motion $(\Omega,\mathcal{F},(\mathcal{F}_t)_t,\mathbb{P},(B_t)_t)$ (the standard filtration $(\mathcal{F}_t)_t$), we define $$\forall t\ge 0: M_t:=\max_{0\le s\le t} B_s$$ ...
5
votes
1answer
169 views

How is the Chapman-Kolmogorov Equation not a fancy name for matrix multiplication?

The Chapman-Kolmogorov Equation: $$p^{m+n}(i,j)=\sum_kp^m(i,k)p^n(k,j)$$ Matrix Multiplication (with $[A]_{i,j}=a_{i,j}$ where $A$ is a linear map "" for B) $$[AB]_{i,j}=\sum_ka_{i,k}b_{k,j}$$ In ...
0
votes
2answers
107 views

Prove that something is a Markov chain

Let $\xi_0, \xi_1, \xi_2, ...$be independent, identically distributed, integer valued random variables. Define $Y_n$ = max{$\xi_i: 0 \leq i \leq n$}. Show that $(Y_{n)n\geq0}$ is a Markov chain and ...
2
votes
1answer
251 views

Inverse of a regular stochastic matrix

Is it true that the inverse of a regular stochastic matrix is also regular? Are there any other interesting features that the inverse may have of a regular stochastic matrix? Hope someone could answer ...
1
vote
1answer
99 views

General birth and death process

hi i need some help to understand the following (from the general birth and death process).I'll give some context first , then i ask questions. Consider general birth and death process with birth ...
1
vote
0answers
76 views

References for time-inhomogeneous Markov jump processes?

In some central models in life insurance mathematics, the state of the insured is modeled using a continuous-time time-inhomogeneous Markov process with finitely many states. While many results for ...
0
votes
0answers
42 views

relations between properties of stochastic processess

If we have an integer valued stochastic process, are these implications correct? independent increments $\rightarrow$ Markov property Markov property $\nrightarrow$ independent increments stationary ...
2
votes
1answer
118 views

time-homogeneous continuous time Markov chain

I have a question about the continuous time Markov chain. In the Poisson process we have independent and stationary increments. Do we have this in a continuous time Markov chain that is ...
2
votes
0answers
98 views

Markov Chain with Normal Transition Matrix

Consider a (sub)-stochastic matrix $P$, and the associated Markov chain $X$ with \begin{align*} \mathbf P [X_n =y | X_0 = x] = P_{xy}^n. \end{align*} Suppose we have the condition $P^T P = P P^T$, ...
0
votes
1answer
44 views

Why does markov chains power method converge at the rate of |λ_2/λ_1 |

I'm doing some researches on Markov Chains, and every time I meet this statement, that The rate of convergence of the power method is given by |λ_2/λ_1 |^k→0, when k→inf. And where λ_1 and λ_2 are ...
0
votes
1answer
22 views

Markovian Coefficients Unclear Definitiondtdt

I have come across the following unclear definition: Consider $dS(t) = S(t)[\mu(t)dt + \sigma(t) dW(t)]$ "Assume that the coefficient $\sigma$ is Markovian. That is, (with abuse of notation) ...
1
vote
0answers
31 views

Analysis of Steady State Probability for Markov Process

I have a balance equation, representing a Markov Chain, which yields $$ (K - z) \pi(Z_c = z) = (\lambda_c + (z+1)x) \pi(Z_c = z+1) $$ where K is the maximum state of the server. The term $\lambda_c$ ...
0
votes
1answer
47 views

Markov process which is not martingale

I have seen the examples of a discrete time martingale that is not a Markov Process. Can you construct me an example of discrete time Markov Process that is not a martingale?
1
vote
0answers
25 views

When are the marginals of an extremal invariant measure also extremal invariant?

Let's suppose that $X$ is a compact metric space, and thus as is $X \times X$. If given a Markov process on $X \times X$ with marginals that are Markov processes on $X$, then we know that the ...
0
votes
1answer
32 views

Computing the PMF for N(t) in a renewal process

I'm in a stochastic processes course, and we just started on renewal theory. Unfortunately, we skipped the section on queuing theory, and nearly example in my textbook for renewal processes uses ...
1
vote
1answer
56 views

$X_n, n> 0$ is a Markov Chain, how to interpret $Z_n = (X_n,X_{n+1}), n > 0$?

Am a newbie to Markov Chain. So, this might be incredibly naive/stupid question. If $X_n, \, n > 0$ is MC, am having difficulty imagining/interpreting process $Z_n = (X_n,X_{n+1}), n > 0$. I ...
1
vote
1answer
35 views

Continuous-time Markov Question

I have a question about a continuous-time Markov process on the discrete space. I am given the generator and asked for find the expected time the Markov process needs to get back to state 3, given ...
0
votes
1answer
83 views

Showing the square of a Markov process is or isn't Markov

Hi I am trying to show that if $X_n$ is a markov process, whether or not $X_n^2$ is a markov process. $X_n$ is a markov process if $P\{X_k = a_k|X_{k-1} = a_{k-1}, X_{k-2} = a_{k-2}, ..., X_k = a_1 ...
0
votes
2answers
84 views

Find the generator of Markov Process

Homework question: Consider the Markov process $X_t=B_t-t^2+t$ where $B_t$ is the Brownian motion. Find the generator $Q$ of this process. I am completely confused how to find the generator for ...
0
votes
1answer
69 views

Steady state of a $4 \times 4$ transition matrix

Normally I just take $q(M_{m\times n} - I_{m\times n})$ to workout the steady state, but here I have: $$\left(\begin{array}{rrrr} 0 & 0 & .8 & .2 \\ .4 & .6 & 0 & 0 \\ .2 ...
0
votes
1answer
33 views

Limiting probability of Markov chain(Terminology)

If I am asked to find the limiting probability of a Markov chain, what does this pertain to? $\lim \limits_{n \to \infty} P^n$? Where $P$ is the stepping matrix and $n$ is the number of steps. "What ...
0
votes
0answers
36 views

Error in thinking: Poisson Process is a Markov Process

I am a bit confused on proving the Markov property for Poisson processes. That is, we want to prove, if $X = (X_t: t \in \mathbb{R})$ is a Poisson process with rate $\lambda$: $P(X_{t_n} = a_n | ...
1
vote
0answers
117 views

How's the damping factor in Google PageRank algorithm calculated

I'm doing some researches about Google's PageRank algorithm for my thesis, I've found that the damping factor x (for example), where x is in : P` = x.P + (1-x)Q where P is the original ...
0
votes
0answers
136 views

Stationary distribution of a “birth-death model” that does not have Markov property

A typical birth-death process is defined such as the probability of going from any state $j$ to any state $i$ is given by: $$ p_{ij}= \begin{cases} b_i & \text {if $j = i+1$} \\ ...
1
vote
0answers
26 views

showing a condition implies convergence to invariant distribution

For an arbitrary Markov chain, I'm trying to show that $\lVert{ P_{\mu}\circ \theta_n^{-1} - \nu \rVert}\to 0$ iff $\sum_j \lvert{ P_{ij}^n - \nu(j)\rvert}\to 0$, where $\theta_n^{-1}$ is a shift ...
0
votes
1answer
24 views

inequality for finite state markov chains

Let $X$ be a discrete-time Markov process in $S$ with invariant distribution $\nu$. Show that for any measurable set $B\subset S$ such that $$P_{\nu}\{X_n \in B\, \textrm{i.o.} \}\geq \nu B.$$ I'm ...
2
votes
1answer
110 views

continuous time Markov chain, something the book does not explain

I have a problem with something in my book, under the chapter of continuous time Markov chains. I will post a link to what the book does. They do something which they seem to take for granted, but I ...
0
votes
1answer
65 views

Representation of Markov process adapted to given Filtration

Let $X$ be continuous Markov process adapted to a filtration generated by Brownian motion $B$. Does there exist a function $f$ such that $X_t = f(t,B_t)$? My guess is that it should have such ...
0
votes
1answer
135 views

Ornstein-Uhlenbeck processs: Markov, but not martingale?

I'm puzzled about properties of the Ornstein-Uhlenbeck process, given by the Itō integral $$ X_t = x e^{-\lambda t} + \sigma \int_0^t e^{-\lambda(t-s)} d W_s \,. $$ I compute that $\{X_t\}$ is not ...
0
votes
1answer
64 views

Markov chain with infinite number of transient and positive recurrent states?

Is it possible to have a markov chain with an infinite number of transient states, and an infinite number of positive recurrent states? Thank you!
1
vote
2answers
267 views

Prove markov chain is null recurrent

Two fair coins are tossed repeatedly. Let Xn denote (Total Number of Heads from Coin 1)-(Total Number of Heads from Coin 2) after n tosses. Thus the state space is {0, ±1, ±2, .... }. Show that the ...
1
vote
0answers
21 views

Three-State Markov Process with Differential Equations

This question is from a take home quiz and I could really use the help. Thanks in advance