A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
0answers
27 views

Stochastic scheduling to maximize the expected number of customers arrived at the root of a Jackson tree

In a Jackson network, organized as a tree rooted at queue r, several customers are queued at time 0 and there is no new customer arrival. The service time of each customer in queue i is geometrically ...
1
vote
1answer
45 views

convergence of nullrecurrent markov chain

Hi guys! At the moment I'm working on this proof. It's in a german book so hopefully you understand everything. I understand everything in the picture without the use of the markov property at ...
1
vote
1answer
70 views

How to connect the deterministic and probabilistic descriptions of the SIR model

I am a 17 year old student and I was reading up on epidemic modelling for a math project, specifically the SIR model and I came across this: "This" refers to the assumptions to which the Markov Chain ...
0
votes
1answer
56 views

Computer failure with Markov chains and n-step transition matrix

Hi I am struggling with a Markov Chain question: A computer network has two servers, only one of which is in operation at any given time. A server may break down on any given day with probability p. ...
0
votes
0answers
41 views

A Counter Example of Doeblin Condition

The question is to prove that the following Markov process doesn't satisfy the Doeblin Condtion. Let $X=\{\ldots,-n,\ldots,-1,0,1,\ldots,n,\ldots \} $, The Markov Transition Matrix $P$ is defined as ...
1
vote
0answers
77 views

Kolmogorov backward equations for Birth-Death

I'm trying to solve the Kolmogorov backward equations for a Birth-Death Markov chain with three states. I have 2 equations: $$P_{00}'(t) = \lambda_0 (P_{10}(t)-P_{00}(t))$$ $$P_{10}'(t) = ...
2
votes
1answer
82 views

Is $(B_t^2)$ Markov where $(B_t)$ is Brownian motion?

I am pretty sure $(B_{t}^{2})$ not Markov because the squared random walk is not. Showing the square of a Markov process is or isn't Markov I guess I can repeat the method since to be Markov it ...
1
vote
0answers
83 views

Chapman-Kolmogorov equations of time inhomogenous Markov chains

Let us assume that we are given a time inhomogenous Markov chain in continuous time (ICTMC) $(X(t))_{t \geq0}$ with a finite state space $\{1,\ldots,n\}$. Set $P(t)_{i,j} := \mathbb{P}(X(t) = j \mid ...
0
votes
1answer
47 views

Bayesian Network vs Markov Decision Process

I am wondering if somebody can tell me anything about the practical differences between using Markov Decision Processes and and Bayesian Networks in reasoning about probabilistic processes?
2
votes
2answers
56 views

Verifying the Markov property

We throw a dice infinitely often. Define $U_n$ to be the maximal number shown up to time $n$. How can I verify that $$ ...
1
vote
1answer
100 views

Why is the Stochastic Process in the HJM model non-Markovian?

I want to understand exactly what my title asks "Why is the Stochastic Process for the short rate in the HJM model of interest rates non-Markovian?" That process is the following: ...
2
votes
2answers
151 views

Connections between SDE and PDE

I have encountered a number of situations where the solution of a PDE and a certain expectation associated to a Markov process are equal. Two examples include: The heat equation $u_t = \frac{1}{2} ...
1
vote
1answer
118 views

Brownian Motion inequality (related to Dvoretzky-Erdoes test)

i have the following question: Let $B(t)$ be a d-dimeansional Brownian motion $d\ge 3$, and $f$ be a monoton increasing function from the positive reals to the positive reals. Let $A_n=(\exists t\in ...
0
votes
1answer
25 views

Brownian Motion and Progressive Process

Let $B_t$ be a Brownian motion. Define sign function as follows. $sign(0) = 0$ and $sign(x) = \frac{x}{|x|}, \forall x \neq 0$. I do not know how to show the following two questions, especially on the ...
1
vote
1answer
86 views

Application of Lévy–Khinchine formula

How can we express the characteristic functions of Wiener and Poisson processes by using the Lévy–Khinchine formula? I don't know how to find the characteristic functions of particular Levy ...
6
votes
4answers
281 views

Discover where Bob is sleeping using hidden Markov chains

Bob lives in four different houses $A, B, C$ and $D$ that are connected like the following graph shows: Bob likes to sleep in any of his houses, but they are far apart so he only sleeps in a house ...
1
vote
0answers
144 views

Can we find a correlation between states of a Markov chain?

I have a fair bit of knowledge on Markov chains but I recently wondered if there is a way to find out a correlation between the states of a finite Markov chain. I could not find any material on this. ...
1
vote
1answer
34 views

Application of Strong Markov Property

Theorem SMP (Strong Markov Property) Let $X$ be a time homogenous Markov process with $T=\mathbb R_+$ or $\mathbb Z_+$ and let $\tau$ be a stopping time taking countably many values. Then ...
1
vote
0answers
50 views

Linear Filtering Problem (Keynman Fac/Particle Model)

$lienar Filtering Problem $$X_n^1 = X_{n-1}^1 + \epsilon_n *W_n $$ $$X_n^2 = (1-\alpha* \delta) X_{n-1}^2 + \beta*\delta X_n^1 $$ $$X_n^3 = X_{n-1}^3 + \delta*X_n^2$$ above is $$\approx$$ $$dX_n^1 ...
0
votes
1answer
28 views

Ergodic behaviour for bounded random dynamical system

Considering an iterated system described by $$ X_n =\gamma_nX_{n-1} , $$ where $\gamma_i$ are non-negative i.i.d. variables. It is easy to show that the expectation will grow unbounded for $\mathbb ...
0
votes
0answers
46 views

Stationary VS. limiting probability

I'm just wondering what the difference between stationary probability and limiting probability is. And, if any of you know: What does it mean that some elements exist and some elements doesn't, when ...
1
vote
0answers
52 views

Stochastic process using Markov chain (thief on the run!!)

I'm given an exercise where we are to simulate a thief escaping from an officer. The thief (let's call him/her T for simplicity) and an officer (O) have four cities to be in. Let's call the cities A, ...
0
votes
1answer
156 views

Covariance of states of a finite Markov chain

I know it is possible to construct a covariance matrix for states of a Markov chain but I cannot seem to find a proper way to compute it. I will attach some theories I found from Kemeny and Snell's ...
1
vote
0answers
253 views

Hidden Markov Model, transition probabilities which are modeled with an exponential distribution

I'm looking at implementing an algorithm described in a paper, but I'm having trouble understanding how the transition probabilities for a Hidden Markov Model are defined. In the first sections, I ...
0
votes
0answers
54 views

Strong Markov property of continous time Markov process

In the book "Applied probability and queues" which is available here http://books.google.de/books?id=BeYaTxesKy0C&pg=PA32&hl=de&source=gbs_toc_r&cad=3#v=onepage&q&f=false , ...
0
votes
1answer
78 views

Including non-markovian processes in a birth-death process

Current model I want to model a stochastic system with a birth-death (Markovian) model. I therefore have this kind of $n$ times $n$ (where $n$ is the number of possible states) transition matrix: ...
1
vote
0answers
15 views

A simple question on the inverse z-transformation of $\frac{z}{1-z}\mathscr{T}(z)\mathbf{q}$

I'm wondering if anyone who is familiar with the book Dynamic Programming and Markov Processes by Ronald Howard or simply z-transform can help me figure out an inverse z-transformation on page 23 of ...
0
votes
1answer
26 views

Recurrence of states in a function of a Markov chain

Suppose $X$ is a Markov chain (or process, for that matter) and suppose further $f(X)$ is also a Markov chain. Let $s$ be a recurrent state in $X$. Is there a general way to determine the recurrence ...
0
votes
1answer
110 views

Probability transition matrix for maximum of iid random variables

I have a homework problem that goes as follows: Let $\xi_i, \ i=0,1,2,\ldots$ be i.i.d. random variables of discrete type. The distribution of $\xi_0$ is given by: $$\mathbb{P}\{\xi_0=i\} = a_i, \ ...
1
vote
1answer
34 views

A certain formulation of the Chapman-Kolmogorov equation.

I am reading a book by Taira called Semigroups, Boundary Value Problems and Markov Processes. It is a nice read, but there is one thing I don't understand regarding the Chapman-Kolmogorov equation. A ...
0
votes
1answer
56 views

Show that this Markov chain is recurrent

So I have a Markov chain on the nonnegative integers such that, starting from $x$, the chain goes to $x+1$ with probability $p$, $0<p<1$, and goes to state $0$ with probability $1-p$. I'm ...
0
votes
1answer
14 views

Markov Processes: $P_x$ and $E_x$

In the study of Markov processes, one usually introduces the measures $P_{\pi}$ on the path space of the process where $\pi$ is an initial distribution of the process $X$ i.e $\pi=\mathcal L(X_0)$. ...
0
votes
0answers
24 views

Equivalent Formulation of Markov Property for Homogeneous Chains

In Shiryaev's Probability (just above the strong Markov property, p.568), the author says that an equivalent formulation of the usual Markov property for homogeneous chains is $$P[\theta_nX\in B\mid ...
0
votes
0answers
37 views

Strong Markov Property for Discrete Stopping Times

I'm having a hard time deciphering a particular proof of the following strong Markov property. Theorem (Strong Markov Property) Let $X$ be a time homogenous Markov process with $T=\mathbb ...
0
votes
1answer
26 views

The “on $\left\{ \tau <\infty \right\}$” in the Strong Markov Property

The strong Markov property is often formulated as $$P[\theta _{\tau}X\in A\mid \mathscr F_{\tau}]\overset{\text {a.s on }\left\{ \tau <\infty \right\} }{=}P_{X_\tau}(X\in A)$$ What exactly does ...
4
votes
2answers
245 views

Interpretation for the determinant of a stochastic matrix?

Is there a probabilistic interpretation for the determinant of a stochastic matrix (i.e. an $n \times n$ matrix whose columns sum to unity)?
2
votes
1answer
86 views

Is this a Markov chain? [duplicate]

Let $\{\xi_n \}_{n \geq 1}$ be i.i.d random variables taking values on $\mathbb{Z}$. Let $\xi_0 = 0$. $S_n = \sum\limits_{i=1}^{n} \xi_i,$ where $S_0=0$ $Y_n = \sum\limits_{i=0}^{n} S_i$. My ...
0
votes
0answers
18 views

markov process with extra boundary

In a markov process a random walker has to reach N (absorbing boundary) from $x_o$ on a $[0,N]$ lattice, where $0$ is the reflecting boundary. To find the first exit time of the random walker via N, i ...
3
votes
0answers
58 views

Simple random walk conditioning on non-return

Consider a simple symmetric random walk on $\mathbb{Z}$, $(S_t)_{t \geq 0}$, with $S_0=0$. Let $k$ and $j$ be two positive integers. Let $P_{k,j}$ be the probability that the walker hits the vertex ...
2
votes
1answer
56 views

Markov processes on function spaces

Is there any reference on Continuous time Markov process whose state space is infinite dimensional function spaces, such as the space of continuous functions $C(R^d)$? It seems Dirichlet Form is a ...
1
vote
1answer
68 views

Markov Chains : Can anything be said about what happens in between two transition?

In time homogeneous discrete Markov chains we take a set period for a single transition. In examples we see sometimes depending on the examples the transition period being a a month a week etc. I'm ...
1
vote
1answer
134 views

A Markov Chain Problem.(Change the color of ball)

There are $n$ different color balls in a box. Take two balls in turns, and change color of the second ball to the first. (This is one operation). Let $k$ be the (random) number of operations needed to ...
1
vote
1answer
85 views

Parental Markov Condition Example

I'm currently reading a text on Bayesian networks and the text is giving some very crude interpretations of what appear to be some of the most important foundations of the subject. It states the ...
0
votes
3answers
382 views

Transition Matrix of M/M/1 Queue

We know that for an M/M/1 queue the state space is $S=\{0,1,2,... \}$. Further the probability to go from state $i$ to $i+1$ is $\lambda$ for all $i$ in $S$. Moreover, to go from $i$ to $i-1$ is the ...
1
vote
0answers
62 views

Follow-up on solution to Markov process equation

I asked a question here about solving a system related to an absorbing Markov chain. I now have a variation where there are $m$ types (of student, job seeker, etc) each of which applies to ...
0
votes
0answers
22 views

Markov Models and Applications

I am looking for resources in Markov models and its applications. I'm looking for tutorials, videos, books etc which provide the following Explain Markov chains in layperson terms and provide ...
0
votes
1answer
60 views

Is there a solution to this system for the diagonal matrix?

I'm trying to find a solution to a system of equations, but its quite different from anything I've come across before. I believe there is a solution, but I could be wrong. $\mathbf{A} = ...
0
votes
1answer
45 views

Markov property for a stochastic process with discrete state space.

Consider a stochastic process $\{X_s\}_{s\in\mathcal S\subseteq\mathbb R}$ with value in $(\mathbb R,\mathcal B(\mathbb R))$ adapted to a filtration $\{\mathcal F_s\}$ (we can suppose that ...
0
votes
0answers
35 views

How to solve “So Who's Counting” problem using Markov Decision Process?

In Martin Puterman's book Markov Decision Processes, one of the problems he gives is "So Who's Counting". In that problem, 5 random digits are generated. After each digit is generated, it is placed in ...
0
votes
0answers
28 views

Probabilities in a Markov Model

I am reading a paper on Markov Models and I am trying to figure out how to compute the probabilities for the $\alpha$-pass. I am given an $N\times N$ matrix $A$, that has the probabilities of ...