0
votes
0answers
14 views

Strong Markov property of continous time Markov process

In the book "Applied probability and queues" which is available here http://books.google.de/books?id=BeYaTxesKy0C&pg=PA32&hl=de&source=gbs_toc_r&cad=3#v=onepage&q&f=false , ...
0
votes
1answer
14 views

Recurrence of states in a function of a Markov chain

Suppose $X$ is a Markov chain (or process, for that matter) and suppose further $f(X)$ is also a Markov chain. Let $s$ be a recurrent state in $X$. Is there a general way to determine the recurrence ...
0
votes
1answer
28 views

Probability transition matrix for maximum of iid random variables

I have a homework problem that goes as follows: Let $\xi_i, \ i=0,1,2,\ldots$ be i.i.d. random variables of discrete type. The distribution of $\xi_0$ is given by: $$\mathbb{P}\{\xi_0=i\} = a_i, \ ...
0
votes
1answer
25 views

Show that this Markov chain is recurrent

So I have a Markov chain on the nonnegative integers such that, starting from $x$, the chain goes to $x+1$ with probability $p$, $0<p<1$, and goes to state $0$ with probability $1-p$. I'm ...
3
votes
2answers
29 views

Interpretation for the determinant of a stochastic matrix?

Is there a probabilistic interpretation for the determinant of a stochastic matrix (i.e. an $n \times n$ matrix whose columns sum to unity)?
1
vote
0answers
13 views

Simple random walk conditioning on non-return

Consider a simple symmetric random walk on $\mathbb{Z}$, $(S_t)_{t \geq 0}$, with $S_0=0$. Let $P_{k,j}$ be the probability that the walker hits the point $k$ without returning to the origin in ...
1
vote
1answer
92 views

A Markov Chain Problem.(Change the color of ball)

There are $n$ different color balls in a box. Take two balls in turns, and change color of the second ball to the first. (This is one operation). Let $k$ be the (random) number of operations needed to ...
0
votes
0answers
31 views

Parental Markov Condition Example

I'm currently reading a text on Bayesian networks and the text is giving some very crude interpretations of what appear to be some of the most important foundations of the subject. It states the ...
0
votes
0answers
23 views

How to solve “So Who's Counting” problem using Markov Decision Process?

In Martin Puterman's book Markov Decision Processes, one of the problems he gives is "So Who's Counting". In that problem, 5 random digits are generated. After each digit is generated, it is placed in ...
0
votes
0answers
16 views

Probabilities in a Markov Model

I am reading a paper on Markov Models and I am trying to figure out how to compute the probabilities for the $\alpha$-pass. I am given an $N\times N$ matrix $A$, that has the probabilities of ...
0
votes
0answers
34 views

Exact probability distribution for hitting time of simple random walk

Consider simple random walk on the line starting from the site $y \in \mathbb{N}$. With probability $p$ the walker moves to the right and with probability $1-p$ to the left. Call $\tau$ the first time ...
0
votes
1answer
45 views

Mean absorption time for pure birth process

Let $\xi_t$, $t\geq0$, be a pure birth process, with $P\{\xi_{t+h} = i +1 | \xi_t = i\} = \lambda^ih + o(\lambda)$, as $h \downarrow 0$. At $t=0$, $\xi_0 =1$. Let $\tau = \min\{t ~|~ \xi_t = N\}$. ...
0
votes
1answer
31 views

Proving that a process has the Markov property

Let $X_t=xe^{ct+aB_t}$ where $B_t$ is one dimensional Brownian motion. How would I prove this is a Markov process using the expectation definition of a Markov process, i.e., ...
0
votes
0answers
24 views

Stopped strong Markov process again strong Markov?

Following setting: I have a right-continuous strong Markov process X in a right-continuous filtration >$\mathbb{F}=(F_t)$ and a P-a.s. finite stopping time $\tau$. My question is: Is the ...
0
votes
1answer
41 views

Board Game Markov Process - Transient Probabilities

I need to write an essay on the Game of Life board game, and so I studied up on Markov Chains to help me calculate the probabilities and average payoffs for the spaces; however I'm not sure whether ...
1
vote
1answer
47 views

Property of Wiener process sample path

What is a mean of time, when the trajectory of wiener process $W_t$ is over the line $y=t$? We need to find $\mathbb{E}\tau$, where $\tau=\sum\limits_{a,b:\forall t\in(a,b) ; ...
1
vote
1answer
37 views

Symmetric random walk and the distribution of the visits of some state

I need help with this problem: Let $(S_n)_n$ a symmetric random walk in $\mathbb{Z}$ i.e $S_n=X_1 + \cdots + X_n$ with $(X_n)$ iid $\mathbb{P}(X_n=1)=\mathbb{P}(X_n=-1)=\frac{1}{2}$. Let $m \in ...
0
votes
2answers
55 views

Expected number of steps to absorbtion - Markov chain

I want to calculate the expected number of steps (transitions) needed for absorbtion. So the transition probability matrix $P$ has exactly one (lets say it is the first one) column with a $1$ and the ...
1
vote
1answer
60 views

random walk with sticky barriers

Consider a random walk on the line 1,...,d. You start at point 1. At each step you flip a coin: heads means go left, tails means go right. If you're at 1 and get a heads, just stay where you are ...
0
votes
1answer
35 views

Markov’s inequality

The annual return, R, of a certain stock is a random variable with mean 10. Use Markov’s inequality to obtain a bound for the probability of the stock return being at least 20. Assuming now that R ...
0
votes
1answer
55 views

Understanding detailed balance equations

I'm trying to understand how the equilibrium distribution satisfy the detailed balance equation. To my understanding, I only understand that a detailed balance equation would only be satisfied if ...
2
votes
1answer
64 views

Monotonicity and Convexity of Stochastic Matrices

The definition of stochastic monotonicity and convexity is given by "Stochastic Orders and Their Applications" by Moshe Shaked and George Shanthikumar (1994) as: Let $P = \{p_{i,j} \}$ be a ...
0
votes
0answers
88 views

Application of Markov Chain to Game of Life Board Game

I need to calculate the expected outcomes for the Game of Life. I believe that if I multiply the probability of landing on a particular square with the payoff of said square and add up all these ...
0
votes
0answers
28 views

Poisson process different type of events

Suppose that it arrives people to a store according to a poisson process with rate $\lambda = 6$/hour , females arrive with probability $0.6$ and male with $0.4$. What is the probability that there ...
0
votes
0answers
17 views

A doubt on markov decision process

Given that a policy is a function from a state action pair to probabilities, the set of policies for a MDP forms a POSET (the partial order is due to value function for a policy). Why there should be ...
1
vote
1answer
33 views

Reflection Principle interpretation

Given a standard Brownian motion $(\Omega,\mathcal{F},(\mathcal{F}_t)_t,\mathbb{P},(B_t)_t)$ (the standard filtration $(\mathcal{F}_t)_t$), we define $$\forall t\ge 0: M_t:=\max_{0\le s\le t} B_s$$ ...
4
votes
1answer
85 views

How is the Chapman-Kolmogorov Equation not a fancy name for matrix multiplication?

The Chapman-Kolmogorov Equation: $$p^{m+n}(i,j)=\sum_kp^m(i,k)p^n(k,j)$$ Matrix Multiplication (with $[A]_{i,j}=a_{i,j}$ where $A$ is a linear map "" for B) $$[AB]_{i,j}=\sum_ka_{i,k}b_{k,j}$$ In ...
0
votes
0answers
16 views

Policy Adjustment in Markov Decision Process

I was using MDP on my work to make optimal decision. I used discrete time, finite state MDP. I assumed that I will have an initial parameters, like the Reward/Cost, state transition probabilities and ...
0
votes
2answers
52 views

Prove that something is a Markov chain

Let $\xi_0, \xi_1, \xi_2, ...$be independent, identically distributed, integer valued random variables. Define $Y_n$ = max{$\xi_i: 0 \leq i \leq n$}. Show that $(Y_{n)n\geq0}$ is a Markov chain and ...
1
vote
1answer
63 views

General birth and death process

hi i need some help to understand the following (from the general birth and death process).I'll give some context first , then i ask questions. Consider general birth and death process with birth ...
2
votes
0answers
51 views

Markov Chain with Normal Transition Matrix

Consider a (sub)-stochastic matrix $P$, and the associated Markov chain $X$ with \begin{align*} \mathbf P [X_n =y | X_0 = x] = P_{xy}^n. \end{align*} Suppose we have the condition $P^T P = P P^T$, ...
1
vote
0answers
26 views

Analysis of Steady State Probability for Markov Process

I have a balance equation, representing a Markov Chain, which yields $$ (K - z) \pi(Z_c = z) = (\lambda_c + (z+1)x) \pi(Z_c = z+1) $$ where K is the maximum state of the server. The term $\lambda_c$ ...
1
vote
1answer
31 views

Continuous-time Markov Question

I have a question about a continuous-time Markov process on the discrete space. I am given the generator and asked for find the expected time the Markov process needs to get back to state 3, given ...
0
votes
2answers
59 views

Find the generator of Markov Process

Homework question: Consider the Markov process $X_t=B_t-t^2+t$ where $B_t$ is the Brownian motion. Find the generator $Q$ of this process. I am completely confused how to find the generator for ...
0
votes
1answer
27 views

Steady state of a $4 \times 4$ transition matrix

Normally I just take $q(M_{m\times n} - I_{m\times n})$ to workout the steady state, but here I have: $$\left(\begin{array}{rrrr} 0 & 0 & .8 & .2 \\ .4 & .6 & 0 & 0 \\ .2 ...
0
votes
0answers
53 views

Stationary distribution of a “birth-death model” that does not have Markov property

A typical birth-death process is defined such as the probability of going from any state $j$ to any state $i$ is given by: $$ p_{ij}= \begin{cases} b_i & \text {if $j = i+1$} \\ ...
3
votes
1answer
97 views

What is the value of this game?

We have 3 black and 2 red balls in an urn. If we pick a black ball, we lose 1 USD. If we pick a red ball we win 1 USD. We can chose to start the game or not. If we start the game we can stop after ...
3
votes
1answer
208 views

Elementary proof of geometric / negative binomial distribution in birth-death processes

The birth-death process concerns a population of $n_0$ individuals, each of which reproduce and die at a constant rate as time $t$ increases from $t=0$. Each individual splits into two individuals ...
1
vote
2answers
100 views

Expected number of steps between states in a Markov Chain

Suppose I am given a state space $S=\{0,1,2,3\}$ with transition probability matrix $\mathbf{P}= \begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 \\[0.3em] \frac{2}{3} ...
0
votes
1answer
74 views

Exercise on Markov chain

Prove, or give an explicit counterexample to refute, the following assertion: if $\{X_n\}$ is a Markov chain, then $\{X_n^2\}$ is also a Markov chain. It's easy to show that ...
3
votes
1answer
71 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is ...
3
votes
1answer
86 views

Strong Markov property of Brownian motion

I was able to understand Brownian Motion $\{B(t):t\geq0\}$ has Strong Markov Property i.e. For any stopping time $\tau$, $P(B(t+\tau)\leq y | \mathcal{F}_{\tau})=P(B(t+\tau)\leq y|B(\tau))$ a.s. , $y ...
3
votes
1answer
73 views

Random walk where increment depend on current position

Consider the following stochastic process, $$b(i+1) = b(i) + \xi_i (b_i),$$ where $\xi_i(b_i) \in \{-1, k \}$ are the independent increments having the following distribution: $$\begin{align} P (\xi ...
1
vote
1answer
67 views

Library chain stationary distribution

This is an exercise 1.47 from Richard Durrett's Essentials of Stochastic Processes p.85 (doi: 10.1007/978-1-4614-3615-7_1 or Google Books). On each request the ith of the $n$ possible books is the ...
1
vote
1answer
98 views

Equivalence Classes of a Markov Chain with Transition Matrix

I have the following transition probability matrix for a markov chain with state space S={0,1,2,3,4,5,6}: $\begin{bmatrix} \frac13 & \frac13 &0 & 0 & \frac16 & 0 & \frac16\\ ...
4
votes
1answer
80 views

random walk in a certain environment

Consider the following random walk in one dimension, starting from $r(0)=0$. $$ r(i+1) = r(i) + \xi, $$ where $\xi(i, r(i))$ is an increment with distribution $P(\xi=1) = \frac{c^{r(i)}}{i-r(i)+1}$ ...
3
votes
1answer
49 views

Probability of a sequence of events in a Poisson process.

I am starting to study Poisson processes and I came up with this question: Let there be two Poisson processes with rates $\lambda$ and $\mu$ respectively, monitoring the occurrence of events (e.g. ...
2
votes
1answer
184 views

Markov chains: is “aperiodic + irreducible” equivalent to “regular”?

I have two books on stochastic processes. In one book, it says that the limiting matrix is possible to find if the matrix is regular, that is, if for some $n$ $P^n$ has only positive values. The ...
0
votes
0answers
71 views

MDP problem - How is the expected cost calculated?

I have been stuck with a problem for a while regarding Markov Decision Processes for a Policy improvement algorithm. Assume that I have probabilities for certain states to evolve the system into, ...
0
votes
0answers
64 views

transition matrix for Markov chain

Can any one help me to solve this home work please? The city of Sacramento recently completed a new light rail system to bring commuters and shoppers into the downtown area and relieve freeway ...