A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
0answers
11 views

Is the value of the discrete Green's function on a box independent of the position of the points within the box?

I currently contemplate over the discrete Green's function on a box and am trying to gain an intuition for its behaviour. Consider the box $B := \{-N,\cdots,-1,0,1,\cdots,N\}^2$ and let $(X_i)_{i \in ...
0
votes
1answer
52 views

If $N_t$ is a Poisson process and $Y\in\{-1,1\}$, then $X_t = Y(-1)^{N_t}$ is a Markov process

Let $Y$ be a random variable with values in $\{-1,1\}$ independent of a Poisson process $\{N_t\}$ with intensity $\lambda>0$. Set the process $X = \{X_t\}$ by $X_t=Y(-1)^{N_t}$. Show that $X$ is a ...
0
votes
3answers
69 views

markov chains and coin flips

A coin that comes up heads with probability p is continually flipped until the pattern T T T H appears. Let X denote the number of flips, find EX. If I use Markov chains is there a simpler way to ...
2
votes
0answers
21 views

Why do we look only for the first and second derivative when dealing with diffusions?

I would like to understand, from an analytical point of view, why is it that we only take the first and second derivatives into account when computing the generator of a diffusion. This question is ...
0
votes
0answers
36 views

Importance of uniform stationary distribution

When I study Markov chain (or sampling) related papers, most of them emphasize "uniform stationary distribution". But, I can't sure why it is important for Markov chain problems or randomized ...
1
vote
1answer
28 views

Transition Probability of M/M/1 Queue given any constant observing period T

I am trying to find the transition probability for $M$/$M$/$1$ queue given any constant observing period $T$ (if $T$ go to infinity the transition matrix will degenerate to a matrix with identical ...
1
vote
0answers
35 views

The differences between the return times to a recurrent state of a discrete Markov chain are independent and identically distributed

Let $(\Omega,\mathcal A)$ be a measurable space and $\mathbb F=(\mathcal F_n)_{n\in\mathbb N_0}$ be a filtration on $(\Omega,\mathcal A)$ $E$ be an at most countable set equipped with the discrete ...
1
vote
1answer
39 views

Markov process, compute how much time spent in each state in average before absorption.

This problem has three states for a person, which are either employed, unemployed or early retirement. The probability that a working person goes unemployed is 0.2 (ie with intensity 0.2 per year). ...
1
vote
0answers
23 views

Fluctuation of a martingale conditioned to return

Consider a martingale $M_t$ on $\mathbb{Z}$ starting from $M_0=0$ and such that $Var[M_t] \leq C \, t$, where $C>0$ is some constant. For a given $n \in \mathbb{N}$ and $t \leq n$, define a process ...
0
votes
0answers
19 views

Property of an irreducible Markov Chain

How can we prove that if a Markov Chain is irreducible (does not contain any closed set), then every state can be reached from every other state in the chain ?
0
votes
1answer
28 views

model with markov chain

Suppose to have the following situation: At a bar at each time unit arrives a certain number of customer with probabilities $p_1,p_2,...,p_n$. In the bar there are 3 bartenders so 3 customer can be ...
0
votes
1answer
59 views

I think I've found an invariant distribution for a transient discrete Markov chain - Where is my mistake?

Let $E$ be an at most countable set equipped with the discrete topology $\mathcal E$ $X=(X_n)_{n\in\mathbb N_0}$ be a discrete Markov chain with values in $(E,\mathcal E)$, distributions ...
1
vote
0answers
38 views

An irreducible Markov chain is positive recurrent if and only if there is an invariant distribution

Let $E$ be an at most countable set equipped with the discrete topology $\mathcal E$ $X=(X_n)_{n\in\mathbb N_0}$ be a discrete Markov chain with values in $(E,\mathcal E)$, distributions ...
-1
votes
1answer
20 views

If the probability measure $\mu$ is a left-eigenvector to the eigenvalue $1$ of a stochastic matrix $p$, then $\mu p^n=\mu$

Let $E$ be an at most countable set and $\mathcal E$ be the discrete topology on $E$ $p=\left(p(x,y)\right)_{x,y\in E}$ be a stochastic matrix $\mu$ be a probability measure on $(E,\mathcal E)$ ...
0
votes
0answers
30 views

Find the backward equation for the random walk on the integers in the form $f_n(x) = \alpha_n + (x - \beta_n)^2$

Consider the homogeneous Markov process given by the random walk on the integers with probabilities $a$, $b$, and $c$ of moving one step backward, staying in the same place, and one step forward. ...
0
votes
1answer
20 views

Many Markov chains with one graph

I am studying Markov chain as a beginner. When I read some documents, I often find a sentence as follows. "For an undirected graph, many finite irreducible Markov chains can be generated." But, it ...
2
votes
0answers
50 views

Excursion of random walk conditioning on return

Consider a simple random walk in one dimension starting from the origin. Let $\epsilon>0$. How to prove that, conditioning on the event that the random walk is at the origin at time $n$, the ...
0
votes
0answers
57 views

how to show if {$Z_n$} is a Markov chain given {$X_n$}

I'm currently working on an practice question from my notes. But I'm not quite understanding the idea of how to prove that something is a Markov chain. Let {$X_i$}, $i = 1,2,...$, be a Markov chain ...
1
vote
0answers
45 views

Blackwell's example in Markov process theory and Kolmogorov's extension theorem

I'm reading Continuous Time Markov Processes: An Introduction by Thomas M. Liggett. Chapter 2.4 is devoted to Blackwell's example. Let $E=\left\{0,1\right\}$, $\mathcal E:=2^E$ and $X$ be the ...
1
vote
0answers
16 views

Distribution of $(X(t_1),X(t_2))$ of a diffusion process $X(t)$

I am very new to SDE's and diffusion processes, I came across this diffusion process given by $dX(t)=[\alpha-(\alpha + \beta)X(t)]dt + \sqrt{2X(t)(1-X(t))}dB(t)$ where $B(t)$ is a continuous Brownian ...
0
votes
2answers
53 views

Convergence of Markov Chain

Could you give me an intuition for the statement: "The Markov chain converges to its stationary distribution"? I know the math behind it. I'm asking for an intuition without using mathematical ...
0
votes
1answer
37 views

For a discrete Markov process $X$, the probability that $X$ started in $x$ returns to $x$ is always positive. So, there are no absorbing states?!

Let $E$ be an at most countable set equipped with the discrete topology and $\mathcal E=2^E$ $X=(X_t)_{t\ge 0}$ be a discrete Markov process with values in $(E,\mathcal E)$ and distributions ...
0
votes
1answer
29 views

How can I find out the missing Markov transition probabilities given an incomplete transition graph?

I'm given a transition graph as shown below. I need to fill in the two missing probabilities. Is there a general method for doing this?
3
votes
2answers
100 views

If $X$ is a right-continuous, discrete Markov process, then $\displaystyle\lim_{t\downarrow 0}\operatorname P_x\left[X_t=x\right]=1$

Let $E$ be an at most countable Polish space and $\mathcal E$ be the discrete topology on $E$ $X=(X_t)_{t\ge 0}$ be a discrete Markov process with values $(E,\mathcal E)$ and distributions ...
0
votes
0answers
17 views

Deriving/Fitting Origin-Destination Matrix of Directed Graph Flow

Let me preface this by saying that my this area of Mathematics is not my specialty (so pardon me if this is an easy question that I just cannot articulate correctly). I am trying to find a way to ...
5
votes
1answer
106 views

Is a Markov process uniquely determined?

Let $E$ be a Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $I\subseteq[0,\infty)$ be closed under addition and $0\in I$ Please consider the following result: Let ...
0
votes
0answers
41 views

Under which conditions on a Markov process $X$ does $\frac 1t\lim_{t\downarrow 0}\operatorname P_x\left[X_t\in B\right]$ exist?

Let $I\subseteq[0,\infty)$ be closed under addition and $0\in I$ $(\Omega,\mathcal A)$ be a measurable space $E$ be a Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ ...
2
votes
2answers
43 views

If $x$ is a recurrent state of a discrete Markov chain and the probability to go from $x$ to $y$ is positive, then $y$ is recurrent

Let $E$ be an at most countable Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_n)_{n\in\mathbb N_0}$ be a Markov chain with values $(E,\mathcal E)$, distributions ...
1
vote
0answers
29 views

A non-absorbing state of a discrete Markov chain is recurrent if and only if the Green's function explodes at this state

Let $E$ be an at most countable Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_n)_{n\in\mathbb N_0}$ be a Markov chain with values $(E,\mathcal E)$, distributions ...
0
votes
1answer
44 views

Expected value of visits in a state of a discrete Markov chain [duplicate]

Let $X=(X_n)_{n\in\mathbb N_0}$ be a Markov chain with values in a at most countable Polish space $E$ and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $(\operatorname P_x)_{x\in E}$ be the ...
0
votes
1answer
67 views

Periodicity of a Markov Chain.

A class property of Markov Chain is periodicity. But I do not understand how is to calculate the period of a state from a transition probability matrix. I am following the book "An Introduction to ...
4
votes
1answer
70 views

If $τ_x^k$ is the time of the $k$-th entrance of a Markov chain into $x$, then $\text P_x[τ_y^k<∞]=\text P_x[τ_y^1<∞](\text P_y[τ_y^1<∞])^{k-1}$

Let $E$ be at most countable and equipped with the discrete topology and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_n)_{n\in\mathbb N_0}$ be a discrete Markov chain with values in ...
3
votes
1answer
41 views

Does the measurability of $x\mapsto\operatorname P_x[A]$ imply the measurability of $x\mapsto\operatorname E_x[X]$?

Let $(\Omega,\mathcal A)$ and $(E,\mathcal E)$ be measurable spaces $(\operatorname P_x)_{x\in E}$ be a family of probability measures on $(\Omega,\mathcal A)$ such that $$E\ni x\mapsto\operatorname ...
0
votes
0answers
81 views

Waiting times independence and distribution

I am struggling with that: We have irreducible and aperodic Markov Chain on finite state space. There is a state $\alpha$ which is recurrent. We define $\tau_n = \min (m >{\tau_{n-1} : X_m = ...
1
vote
2answers
40 views

Possibly broken definition of the strong Markov property

Let $I\subseteq [0,\infty)$ be closed under addition and $0\in E$ $E$ be a Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $X=(X_t)_{t\in I}$ be a Markov process with values in ...
0
votes
1answer
96 views

Flea on a triangle

"A flea hops randomly on the vertices of a triangle with vertices labeled 1,2 and 3, hopping to each of the other vertices with equal probability. If the flea starts at vertex 1, find the probability ...
0
votes
0answers
20 views

Steady-state sensitivity analysis based on semi-Markov process with absorbing state

I'm looking for some references in which steady-state sensitivity analysis through perturbation and based on semi-Markov process with absorbing state, have been studied. Is there any references about ...
1
vote
1answer
38 views

Markov processes in paper “Recent Contributions to The Mathematical Theory of Communication”

I was reading the well-known paper by Warren Weaver, "Recent Contributions to The Mathematical Theory of Communication", I stumpled upon the following sentence(p. 5)" A system which produces a ...
0
votes
1answer
37 views

Transition intensities of Markov process

In Markov process, transition intensities from state i to j are defined as derivatives of transition probabilities at zero: $$q_{ij}=p_{ij}'(0)$$ However I can't somehow catch the interpretation of ...
0
votes
2answers
67 views

How do I construct transition matrix for the following?

A shopkeeper runs his shop in an area that typically gets heavy rains. He has three umbrellas. Every day, he goes to his shop in the morning and comes back home in the evening. If it is raining in ...
0
votes
0answers
19 views

A question involving kernels on measurable spaces

Let $(E, \mathcal{B}(E)), (F, \mathcal{B}(F))$ measurable spaces. A $\it{kernel}$ from $(E, \mathcal{B}(E))$ to $(F, \mathcal{B}(F))$ is a map $N : p\mathcal{B}(E) \to p\mathcal{B}(F)$ such that: $$N ...
1
vote
0answers
89 views

Determining if a stochastic process is a markov chain

Let $\{X_t\}_{t \geq 0}$ be two-state Markov Chain with state space $S=\{0,1\}$, transition matrix $$ P= \begin{bmatrix} 1-p & p \\ q & 1-q \end{bmatrix} $$ and initial ...
0
votes
1answer
24 views

Recurrence of $\pi$ irreducible chains with invariant distribution

In Tierney's paper (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.110.5995&rep=rep1&type=pdf) on page 1712 in the first paragraph there's this proposition that if a chain $P$ is ...
4
votes
2answers
99 views

What is the probability this Markov chain does not reach state $r$?

Consider a random walk on the non-negative integers. You start at $0$, and in each step you either move $1$ higher, or $2$ lower (but can't go below $0$). The direction is picked w.p. $1/2$ ...
0
votes
0answers
27 views

Continuous Markov Decision Processes: approximate value iteration vs least squares fitting of sampled value function

This is my first question here and I just started getting interested in MDPs, so forgive me for both inaccuracies or unclear questions. Let's talk about continuous MDPs. Sampled based approaches ...
2
votes
0answers
41 views

Compute the value of $\mathbb{P}_{0}\left[\tau\geq n\right]$ and $\mathbb{P}_{0}\left[\tau=+\infty\right]$

We consider the Markov chain $\left(X_{n}\right)_{n\geq0}$ on $\mathbb{N}$ whose transition matrix $P=\left(p_{k,l}\right)_{k,l\geq0}$ given by $$\forall k\in\mathbb{N} ...
0
votes
0answers
37 views

Definition of Past in Markov Property

Usually in textbooks, the definition of the Markov Property reads as: \begin{equation} P(X_n\leqslant x_n|X_{n-1}\leqslant x_{n-1},...,X_{0}\leqslant x_{0})=P(X_n\leqslant x_n|X_{n-1}\leqslant ...
1
vote
1answer
60 views

Prove that this Markov chain is irreducible if and only if there exist infinitely many $k\geq0$ such that $q_{k}>0$

We consider the Markov chain $\left(X_{n}\right)_{n\geq0}$ on $\mathbb{N}$ whose transition matrix $P=\left(p_{k,l}\right)_{k,l\geq0}$ given by $$\forall k\in\mathbb{N} ...
0
votes
1answer
69 views

Gambler's Ruin with changing probabilities

I have the following Markov Chain and am trying to evaluate the probability that the Chain reaches state 4 before it returns to state 1, given it starts in state 1. I've seen many typical problems ...
2
votes
1answer
103 views

Markov chain with infinitely many states

I am stumped on the following infinite Markov Chain. Given the this transition matrix for a Markov chain, how do I determine what values of $x$ the chain is positive recurrent/null ...