Tagged Questions

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
1answer
7 views

markov property in Brownian motion

there are two statements about markov property: $B_t $ is Brownian motion and $\mathcal{F}$ is generated by $B$ $s>0,Y$ is bounded and measuable.then ...
0
votes
1answer
29 views

Gambler's ruin and Markov Chain, coin toss and stakes

I'm considering a classical problem about Markov Chains: A gambler has $£8$ and wishes to get to $£10$. A coin is tossed repeatedly : if it comes down tails, the gambler loses his stake, and if it ...
4
votes
1answer
19 views

A die whose score cannot be as before (Markov chains)

A die is "fixed" so that each time it is rolled the score cannot be the same as the preceding score, all other scores having probability $1/5$. Given that the first score is 6, what is the probability ...
-1
votes
0answers
19 views

Setting up and Solving Kolmogorov Forward Equations

Consider a Markov Chain with $3\times 3$ generator matrix: $$ G = \begin{bmatrix} -1 & 1/2 & 1/2 \\ 1/2 & -1 & 1/2 \\ 1/2 & 1/2 & -1 \end{bmatrix} $$ What are the ...
0
votes
0answers
12 views

HMM - forward algorithm (Part-of-Speech Tagging)

In order to understand the Forward algorithm for Hidden Markov Models, I created a Little example of Part-of-Speech Tagging. Consider the Hidden Markov Model with states $N$ (Noun), $V$ ...
1
vote
0answers
8 views

Proportion of arrivals taking a particular path in a Routing Matrix

I have a routing matrix with Node-0 being the source/sink (outside world) and there are service Nodes 1,2..k in the system. The matrix has entries R_ij = Probability of an arrival at Node-i moving to ...
0
votes
3answers
37 views

an exercise about mean and probability

Let $(\Omega, \mathcal{F}, P)$ be a probability space, $X : \Omega \rightarrow \mathbb{R}$ be a discrete random variable and $$\phi : [0, \infty) \rightarrow (0, \infty)$$ an increasing function (so ...
-1
votes
0answers
11 views

How can I calculate value of theta(state-probability distribution) for this markov game with one-sided information? [on hold]

It's an infinite horizon markov game, where player 1 observes realization of each state and player 2 doesn't. They both observe each other's actions (typical markov game). I am trying to simulate this ...
1
vote
1answer
54 views

Finite state Markov chain

Under what conditions a Markov chain can be considered as finite (and not infinite)? Thank you!
0
votes
0answers
22 views

proof of Bernstein inequality using Markov

I have to solve the following problem: Use the Markov's inequality to prove the Bernstein's inequality: $$P(X\geq x) \leq e^{-tx}E[e^tX],$$ for any t>0. Suppose $X_{1}$,$X_{2}$,.... are independent ...
1
vote
0answers
18 views
+50

Stochastic scheduling to maximize the expected number of customers arrived at the root of a Jackson tree

In a Jackson network, organized as a tree rooted at queue r, several customers are queued at time 0 and there is no new customer arrival. The service time of each customer in queue i is geometrically ...
-1
votes
1answer
59 views

How to fix the embedded discrete time Markov chain of a continuous time Markov chain …

Suppose $X$ is a continuous time Markov chain with three states $\{0,1,2\}$ and suppose that $\lambda_0<\lambda_1<\lambda_2$ be all positive such that $\lambda_i$ is the rate of transitions out ...
1
vote
1answer
38 views

convergence of nullrecurrent markov chain

Hi guys! At the moment I'm working on this proof. It's in a german book so hopefully you understand everything. I understand everything in the picture without the use of the markov property at ...
1
vote
1answer
26 views

How to connect the deterministic and probabilistic descriptions of the SIR model

I am a 17 year old student and I was reading up on epidemic modelling for a math project, specifically the SIR model and I came across this: "This" refers to the assumptions to which the Markov Chain ...
-1
votes
0answers
18 views

Given initial state and steady state, how do I find the transition matrix?

The initial vector is $x_0 = \{.5, .5\}$ and the steady state is $x_\infty = \{1/9, 8/9\}$. How do I get the transition matrix, such that $$\lim_{p\to\infty} x_0 A^p = x_\infty $$ where $A$ is a ...
0
votes
1answer
23 views

Computer failure with Markov chains and n-step transition matrix

Hi I am struggling with a Markov Chain question: A computer network has two servers, only one of which is in operation at any given time. A server may break down on any given day with probability p. ...
0
votes
0answers
29 views

A Counter Example of Doeblin Condition

The question is to prove that the following Markov process doesn't satisfy the Doeblin Condtion. Let $X=\{\ldots,-n,\ldots,-1,0,1,\ldots,n,\ldots \} $, The Markov Transition Matrix $P$ is defined as ...
1
vote
0answers
26 views

Kolmogorov backward equations for Birth-Death

I'm trying to solve the Kolmogorov backward equations for a Birth-Death Markov chain with three states. I have 2 equations: $$P_{00}'(t) = \lambda_0 (P_{10}(t)-P_{00}(t))$$ $$P_{10}'(t) = ...
2
votes
1answer
42 views

Is $(B_t^2)$ Markov where $(B_t)$ is Brownian motion?

I am pretty sure $(B_{t}^{2})$ not Markov because the squared random walk is not. Showing the square of a Markov process is or isn't Markov I guess I can repeat the method since to be Markov it ...
1
vote
0answers
30 views

Chapman-Kolmogorov equations of time inhomogenous Markov chains

Let us assume that we are given a time inhomogenous Markov chain in continuous time (ICTMC) $(X(t))_{t \geq0}$ with a finite state space $\{1,\ldots,n\}$. Set $P(t)_{i,j} := \mathbb{P}(X(t) = j \mid ...
0
votes
1answer
10 views

Bayesian Network vs Markov Decision Process

I am wondering if somebody can tell me anything about the practical differences between using Markov Decision Processes and and Bayesian Networks in reasoning about probabilistic processes?
2
votes
2answers
46 views

Verifying the Markov property

We throw a dice infinitely often. Define $U_n$ to be the maximal number shown up to time $n$. How can I verify that $$ ...
1
vote
1answer
47 views

Why is the Stochastic Process in the HJM model non-Markovian?

I want to understand exactly what my title asks "Why is the Stochastic Process for the short rate in the HJM model of interest rates non-Markovian?" That process is the following: ...
2
votes
1answer
44 views

Connections between SDE and PDE

I have encountered a number of situations where the solution of a PDE and a certain expectation associated to a Markov process are equal. Two examples include: The heat equation $u_t = \frac{1}{2} ...
1
vote
1answer
102 views

Brownian Motion inequality (related to Dvoretzky-Erdoes test)

i have the following question: Let $B(t)$ be a d-dimeansional Brownian motion $d\ge 3$, and $f$ be a monoton increasing function from the positive reals to the positive reals. Let $A_n=(\exists t\in ...
0
votes
1answer
18 views

Brownian Motion and Progressive Process

Let $B_t$ be a Brownian motion. Define sign function as follows. $sign(0) = 0$ and $sign(x) = \frac{x}{|x|}, \forall x \neq 0$. I do not know how to show the following two questions, especially on the ...
1
vote
1answer
50 views

Application of Lévy–Khinchine formula

How can we express the characteristic functions of Wiener and Poisson processes by using the Lévy–Khinchine formula? I don't know how to find the characteristic functions of particular Levy ...
6
votes
4answers
261 views

Discover where Bob is sleeping using hidden Markov chains

Bob lives in four different houses $A, B, C$ and $D$ that are connected like the following graph shows: Bob likes to sleep in any of his houses, but they are far apart so he only sleeps in a house ...
1
vote
0answers
53 views

Can we find a correlation between states of a Markov chain?

I have a fair bit of knowledge on Markov chains but I recently wondered if there is a way to find out a correlation between the states of a finite Markov chain. I could not find any material on this. ...
1
vote
0answers
13 views

Application of Strong Markov Property

Theorem SMP (Strong Markov Property) Let $X$ be a time homogenous Markov process with $T=\mathbb R_+$ or $\mathbb Z_+$ and let $\tau$ be a stopping time taking countably many values. Then ...
1
vote
0answers
44 views

Linear Filtering Problem (Keynman Fac/Particle Model)

$lienar Filtering Problem $$X_n^1 = X_{n-1}^1 + \epsilon_n *W_n $$ $$X_n^2 = (1-\alpha* \delta) X_{n-1}^2 + \beta*\delta X_n^1 $$ $$X_n^3 = X_{n-1}^3 + \delta*X_n^2$$ above is $$\approx$$ $$dX_n^1 ...
0
votes
1answer
22 views

Ergodic behaviour for bounded random dynamical system

Considering an iterated system described by $$ X_n =\gamma_nX_{n-1} , $$ where $\gamma_i$ are non-negative i.i.d. variables. It is easy to show that the expectation will grow unbounded for $\mathbb ...
0
votes
0answers
22 views

Stationary VS. limiting probability

I'm just wondering what the difference between stationary probability and limiting probability is. And, if any of you know: What does it mean that some elements exist and some elements doesn't, when ...
1
vote
0answers
34 views

Stochastic process using Markov chain (thief on the run!!)

I'm given an exercise where we are to simulate a thief escaping from an officer. The thief (let's call him/her T for simplicity) and an officer (O) have four cities to be in. Let's call the cities A, ...
0
votes
0answers
37 views

Covariance of states of a finite Markov chain

I know it is possible to construct a covariance matrix for states of a Markov chain but I cannot seem to find a proper way to compute it. I will attach some theories I found from Kemeny and Snell's ...
1
vote
0answers
208 views

Hidden Markov Model, transition probabilities which are modeled with an exponential distribution

I'm looking at implementing an algorithm described in a paper, but I'm having trouble understanding how the transition probabilities for a Hidden Markov Model are defined. In the first sections, I ...
0
votes
0answers
27 views

Strong Markov property of continous time Markov process

In the book "Applied probability and queues" which is available here http://books.google.de/books?id=BeYaTxesKy0C&pg=PA32&hl=de&source=gbs_toc_r&cad=3#v=onepage&q&f=false , ...
0
votes
1answer
63 views

Including non-markovian processes in a birth-death process

Current model I want to model a stochastic system with a birth-death (Markovian) model. I therefore have this kind of $n$ times $n$ (where $n$ is the number of possible states) transition matrix: ...
1
vote
0answers
9 views

A simple question on the inverse z-transformation of $\frac{z}{1-z}\mathscr{T}(z)\mathbf{q}$

I'm wondering if anyone who is familiar with the book Dynamic Programming and Markov Processes by Ronald Howard or simply z-transform can help me figure out an inverse z-transformation on page 23 of ...
0
votes
1answer
22 views

Recurrence of states in a function of a Markov chain

Suppose $X$ is a Markov chain (or process, for that matter) and suppose further $f(X)$ is also a Markov chain. Let $s$ be a recurrent state in $X$. Is there a general way to determine the recurrence ...
0
votes
1answer
40 views

Probability transition matrix for maximum of iid random variables

I have a homework problem that goes as follows: Let $\xi_i, \ i=0,1,2,\ldots$ be i.i.d. random variables of discrete type. The distribution of $\xi_0$ is given by: $$\mathbb{P}\{\xi_0=i\} = a_i, \ ...
1
vote
1answer
24 views

A certain formulation of the Chapman-Kolmogorov equation.

I am reading a book by Taira called Semigroups, Boundary Value Problems and Markov Processes. It is a nice read, but there is one thing I don't understand regarding the Chapman-Kolmogorov equation. A ...
0
votes
0answers
8 views

Finding a discrete Kalman-type process that produces a given Frequency spectrum

Given a power spectral density from f = -1/2 .. 1/2, is it possible to find a 1st order process that produces this series? In other words, x_i+1 = G x_i + W r_i ...
0
votes
1answer
28 views

Show that this Markov chain is recurrent

So I have a Markov chain on the nonnegative integers such that, starting from $x$, the chain goes to $x+1$ with probability $p$, $0<p<1$, and goes to state $0$ with probability $1-p$. I'm ...
0
votes
1answer
11 views

Markov Processes: $P_x$ and $E_x$

In the study of Markov processes, one usually introduces the measures $P_{\pi}$ on the path space of the process where $\pi$ is an initial distribution of the process $X$ i.e $\pi=\mathcal L(X_0)$. ...
0
votes
0answers
12 views

Equivalent Formulation of Markov Property for Homogeneous Chains

In Shiryaev's Probability (just above the strong Markov property, p.568), the author says that an equivalent formulation of the usual Markov property for homogeneous chains is $$P[\theta_nX\in B\mid ...
0
votes
0answers
24 views

Strong Markov Property for Discrete Stopping Times

I'm having a hard time deciphering a particular proof of the following strong Markov property. Theorem (Strong Markov Property) Let $X$ be a time homogenous Markov process with $T=\mathbb ...
0
votes
1answer
20 views

The “on $\left\{ \tau <\infty \right\}$” in the Strong Markov Property

The strong Markov property is often formulated as $$P[\theta _{\tau}X\in A\mid \mathscr F_{\tau}]\overset{\text {a.s on }\left\{ \tau <\infty \right\} }{=}P_{X_\tau}(X\in A)$$ What exactly does ...
3
votes
2answers
48 views

Interpretation for the determinant of a stochastic matrix?

Is there a probabilistic interpretation for the determinant of a stochastic matrix (i.e. an $n \times n$ matrix whose columns sum to unity)?
2
votes
1answer
47 views

Is this a Markov chain? [duplicate]

Let $\{\xi_n \}_{n \geq 1}$ be i.i.d random variables taking values on $\mathbb{Z}$. Let $\xi_0 = 0$. $S_n = \sum\limits_{i=1}^{n} \xi_i,$ where $S_0=0$ $Y_n = \sum\limits_{i=0}^{n} S_i$. My ...