A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
0answers
32 views

Strong Markov property of continous time Markov process

In the book "Applied probability and queues" which is available here http://books.google.de/books?id=BeYaTxesKy0C&pg=PA32&hl=de&source=gbs_toc_r&cad=3#v=onepage&q&f=false , ...
0
votes
1answer
65 views

Including non-markovian processes in a birth-death process

Current model I want to model a stochastic system with a birth-death (Markovian) model. I therefore have this kind of $n$ times $n$ (where $n$ is the number of possible states) transition matrix: ...
1
vote
0answers
10 views

A simple question on the inverse z-transformation of $\frac{z}{1-z}\mathscr{T}(z)\mathbf{q}$

I'm wondering if anyone who is familiar with the book Dynamic Programming and Markov Processes by Ronald Howard or simply z-transform can help me figure out an inverse z-transformation on page 23 of ...
0
votes
1answer
24 views

Recurrence of states in a function of a Markov chain

Suppose $X$ is a Markov chain (or process, for that matter) and suppose further $f(X)$ is also a Markov chain. Let $s$ be a recurrent state in $X$. Is there a general way to determine the recurrence ...
0
votes
1answer
47 views

Probability transition matrix for maximum of iid random variables

I have a homework problem that goes as follows: Let $\xi_i, \ i=0,1,2,\ldots$ be i.i.d. random variables of discrete type. The distribution of $\xi_0$ is given by: $$\mathbb{P}\{\xi_0=i\} = a_i, \ ...
1
vote
1answer
26 views

A certain formulation of the Chapman-Kolmogorov equation.

I am reading a book by Taira called Semigroups, Boundary Value Problems and Markov Processes. It is a nice read, but there is one thing I don't understand regarding the Chapman-Kolmogorov equation. A ...
0
votes
0answers
8 views

Finding a discrete Kalman-type process that produces a given Frequency spectrum

Given a power spectral density from f = -1/2 .. 1/2, is it possible to find a 1st order process that produces this series? In other words, x_i+1 = G x_i + W r_i ...
0
votes
1answer
32 views

Show that this Markov chain is recurrent

So I have a Markov chain on the nonnegative integers such that, starting from $x$, the chain goes to $x+1$ with probability $p$, $0<p<1$, and goes to state $0$ with probability $1-p$. I'm ...
0
votes
1answer
13 views

Markov Processes: $P_x$ and $E_x$

In the study of Markov processes, one usually introduces the measures $P_{\pi}$ on the path space of the process where $\pi$ is an initial distribution of the process $X$ i.e $\pi=\mathcal L(X_0)$. ...
0
votes
0answers
15 views

Equivalent Formulation of Markov Property for Homogeneous Chains

In Shiryaev's Probability (just above the strong Markov property, p.568), the author says that an equivalent formulation of the usual Markov property for homogeneous chains is $$P[\theta_nX\in B\mid ...
0
votes
0answers
28 views

Strong Markov Property for Discrete Stopping Times

I'm having a hard time deciphering a particular proof of the following strong Markov property. Theorem (Strong Markov Property) Let $X$ be a time homogenous Markov process with $T=\mathbb ...
0
votes
1answer
20 views

The “on $\left\{ \tau <\infty \right\}$” in the Strong Markov Property

The strong Markov property is often formulated as $$P[\theta _{\tau}X\in A\mid \mathscr F_{\tau}]\overset{\text {a.s on }\left\{ \tau <\infty \right\} }{=}P_{X_\tau}(X\in A)$$ What exactly does ...
3
votes
2answers
66 views

Interpretation for the determinant of a stochastic matrix?

Is there a probabilistic interpretation for the determinant of a stochastic matrix (i.e. an $n \times n$ matrix whose columns sum to unity)?
2
votes
1answer
64 views

Is this a Markov chain? [duplicate]

Let $\{\xi_n \}_{n \geq 1}$ be i.i.d random variables taking values on $\mathbb{Z}$. Let $\xi_0 = 0$. $S_n = \sum\limits_{i=1}^{n} \xi_i,$ where $S_0=0$ $Y_n = \sum\limits_{i=0}^{n} S_i$. My ...
0
votes
0answers
17 views

Markov vs reinforcement learning

What's the different between markov chain ,markov decision process and reinforcement learning? when we can apply these theories?
0
votes
0answers
16 views

markov process with extra boundary

In a markov process a random walker has to reach N (absorbing boundary) from $x_o$ on a $[0,N]$ lattice, where $0$ is the reflecting boundary. To find the first exit time of the random walker via N, i ...
1
vote
0answers
18 views

Simple random walk conditioning on non-return

Consider a simple symmetric random walk on $\mathbb{Z}$, $(S_t)_{t \geq 0}$, with $S_0=0$. Let $P_{k,j}$ be the probability that the walker hits the point $k$ without returning to the origin in ...
2
votes
1answer
39 views

Markov processes on function spaces

Is there any reference on Continuous time Markov process whose state space is infinite dimensional function spaces, such as the space of continuous functions $C(R^d)$? It seems Dirichlet Form is a ...
1
vote
1answer
60 views

Markov Chains : Can anything be said about what happens in between two transition?

In time homogeneous discrete Markov chains we take a set period for a single transition. In examples we see sometimes depending on the examples the transition period being a a month a week etc. I'm ...
1
vote
1answer
104 views

A Markov Chain Problem.(Change the color of ball)

There are $n$ different color balls in a box. Take two balls in turns, and change color of the second ball to the first. (This is one operation). Let $k$ be the (random) number of operations needed to ...
0
votes
0answers
37 views

Parental Markov Condition Example

I'm currently reading a text on Bayesian networks and the text is giving some very crude interpretations of what appear to be some of the most important foundations of the subject. It states the ...
0
votes
3answers
172 views

Transition Matrix of M/M/1 Queue

We know that for an M/M/1 queue the state space is $S=\{0,1,2,... \}$. Further the probability to go from state $i$ to $i+1$ is $\lambda$ for all $i$ in $S$. Moreover, to go from $i$ to $i-1$ is the ...
1
vote
0answers
59 views

Follow-up on solution to Markov process equation

I asked a question here about solving a system related to an absorbing Markov chain. I now have a variation where there are $m$ types (of student, job seeker, etc) each of which applies to ...
0
votes
0answers
16 views

Markov Models and Applications

I am looking for resources in Markov models and its applications. I'm looking for tutorials, videos, books etc which provide the following Explain Markov chains in layperson terms and provide ...
0
votes
1answer
49 views

Is there a solution to this system for the diagonal matrix?

I'm trying to find a solution to a system of equations, but its quite different from anything I've come across before. I believe there is a solution, but I could be wrong. $\mathbf{A} = ...
0
votes
1answer
35 views

Markov property for a stochastic process with discrete state space.

Consider a stochastic process $\{X_s\}_{s\in\mathcal S\subseteq\mathbb R}$ with value in $(\mathbb R,\mathcal B(\mathbb R))$ adapted to a filtration $\{\mathcal F_s\}$ (we can suppose that ...
0
votes
0answers
20 views

What are the techniques to solve finite-horizon Markov decision process with large yet finite state and action space?

I formulate a problem into a Markov decision process with finite horizon and plan to solve it with the Backward Induction algorithm. However, both the state and action space are large (on the order of ...
0
votes
0answers
26 views

How to solve “So Who's Counting” problem using Markov Decision Process?

In Martin Puterman's book Markov Decision Processes, one of the problems he gives is "So Who's Counting". In that problem, 5 random digits are generated. After each digit is generated, it is placed in ...
0
votes
0answers
19 views

Probabilities in a Markov Model

I am reading a paper on Markov Models and I am trying to figure out how to compute the probabilities for the $\alpha$-pass. I am given an $N\times N$ matrix $A$, that has the probabilities of ...
0
votes
0answers
33 views

Treatment of Markov process with absolute states

In the standard treatment of a markov process, the state vector is a probability vector, whose elements can be between zero and one. But I have a need to constrain the state vector to zeros or ones. ...
0
votes
0answers
51 views

Exact probability distribution for hitting time of simple random walk

Consider simple random walk on the line starting from the site $y \in \mathbb{N}$. With probability $p$ the walker moves to the right and with probability $1-p$ to the left. Call $\tau$ the first time ...
0
votes
1answer
50 views

Mean absorption time for pure birth process

Let $\xi_t$, $t\geq0$, be a pure birth process, with $P\{\xi_{t+h} = i +1 | \xi_t = i\} = \lambda^ih + o(\lambda)$, as $h \downarrow 0$. At $t=0$, $\xi_0 =1$. Let $\tau = \min\{t ~|~ \xi_t = N\}$. ...
0
votes
1answer
38 views

Proving that a process has the Markov property

Let $X_t=xe^{ct+aB_t}$ where $B_t$ is one dimensional Brownian motion. How would I prove this is a Markov process using the expectation definition of a Markov process, i.e., ...
1
vote
1answer
30 views

A question about Markov process

Let a stochastic process $(x(t),\theta(t))$ be given by $$ \dot{x}(t)=f(x(t),\theta(t)) $$ for a well defined continuous function $f(\cdot,\cdot)$. Let $\mathcal{F}_t$ denote the natural filtration ...
0
votes
1answer
101 views

Continuous time markov chains, is this step by step example correct

I have some questions regarding CTMC... and most importantly whether the step-by-step example I provide below is correct. My main sources about CTMC are: ([1], and [2]). Let's assume 3 possible ...
0
votes
1answer
69 views

Can the transition probabilities of an inhomogeneous Markov chain be written as an exponential?

If $Z_t$ is a homogeneous continuous-time Markov chain with finite state space $E=\{1,\ldots,p\}$, transition matrices $(P(t))$ and intensity matrix $Q$, it holds that $$ P(t) = \exp(tQ), $$ see for ...
0
votes
0answers
28 views

Stopped strong Markov process again strong Markov?

Following setting: I have a right-continuous strong Markov process X in a right-continuous filtration >$\mathbb{F}=(F_t)$ and a P-a.s. finite stopping time $\tau$. My question is: Is the ...
0
votes
1answer
25 views

Markov process states adding.

I got a question from my exam paper. In its third (c) question the transition lime and orange is combined to a single drink called Li-Ora. How can I add the transition probabilities in this case? Will ...
1
vote
2answers
73 views

Resolvent operators and inverses proof

I am trying to prove for myself that $A(R_{\alpha}g)=\alpha R_{\alpha}g-g$ which is proving problematic. The definition of $A$, the generator, is $\displaystyle Af(x)= \lim_{t \rightarrow 0} ...
3
votes
2answers
38 views

Always null recurrence at the boundary between positive recurrence and transience?

I have the following theorem: Let $\rho$ be the traffic intensity. a) If $\rho<1$, then $X$ is positive recurrent. b) If $\rho>1$, then $X$ is transient. c) If $\rho=1$, ...
0
votes
1answer
53 views

Board Game Markov Process - Transient Probabilities

I need to write an essay on the Game of Life board game, and so I studied up on Markov Chains to help me calculate the probabilities and average payoffs for the spaces; however I'm not sure whether ...
1
vote
1answer
28 views

Discrete and Continuous Time Markov Properties

$\newcommand{\indep}{\perp\!\!\!\perp}$ In discrete time, the Markov property is $$P[X_{n+1}\in A\mid X_n=s_n,X_{n-1}=s_{n-1}\dots ]=P[X_{n+1}\in A\mid X_n=s_n]$$ On the other hand, the "general ...
1
vote
1answer
50 views

Q-matrix vs. P-matrix description of a Markov chain

Consider a continuous time Markov chain $(X_t)_{t \geq 0}$ on some state space $S$ with transition matrix (P-matrix) $p_t(x,y)$, the probability density of jumping from $x$ to $y$ in time $t$. The ...
2
votes
2answers
28 views

Probability of returning to a given state after n transitions-Markov chains

Let us denote $f_j^{(n)}$ denote the probability of the first return to state $j $after n transitions. Let $p_{jj}^{(n)}$ be the probability of returning to the state $j$ after $n$ transitions when ...
1
vote
1answer
52 views

Property of Wiener process sample path

What is a mean of time, when the trajectory of wiener process $W_t$ is over the line $y=t$? We need to find $\mathbb{E}\tau$, where $\tau=\sum\limits_{a,b:\forall t\in(a,b) ; ...
1
vote
1answer
45 views

Symmetric random walk and the distribution of the visits of some state

I need help with this problem: Let $(S_n)_n$ a symmetric random walk in $\mathbb{Z}$ i.e $S_n=X_1 + \cdots + X_n$ with $(X_n)$ iid $\mathbb{P}(X_n=1)=\mathbb{P}(X_n=-1)=\frac{1}{2}$. Let $m \in ...
1
vote
0answers
27 views

Branching Brownian Motion and KPP equation

I have troubles understanding the proof of the connection between BBM and KPP equation. I mean the proof of the next lemma from the lecture notes of Anton Bovier about BBM, link. This is almost whole ...
0
votes
0answers
47 views

Can ergodic Markov chains be periodic?

I found a statement in one of my notes which said If a state is persistent, aperiodic and not null the it is said to be ergodic Is it necessary that it should be aperiodic? This statement ...
1
vote
1answer
26 views

Markov chains: An issue in classification of states

I recently came across a lemma which goes as follows. Suppose a Markov chain has N states. Let i and j be pair of states. Then j can be reached from i iff there is an integer $ 0 ≤n< N$ such ...
0
votes
2answers
84 views

Expected number of steps to absorbtion - Markov chain

I want to calculate the expected number of steps (transitions) needed for absorbtion. So the transition probability matrix $P$ has exactly one (lets say it is the first one) column with a $1$ and the ...