Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
4 views

Markov-Chain with general state space - recurrent sets

I have an irreducible Markov Chain $(z_n )_{n\in \mathbb N } $ with state space $X$ and with transition-probability-kernel $K$, so $K(x,\cdot)$ is a probability measure (on the $\sigma$-Algebra ...
0
votes
0answers
6 views

Plot probability density and calculat a probability from a markov transition matrix

Let's say we have a vector $v_0 = (-10, -1, 0.2, 0.3, 0.7, 1, 1.5, 2, 3)$ where the elements are possible values of a portfolio at time $0$ (denoed $C_0$), and let's say we have a transition matrix ...
0
votes
0answers
8 views

Variance of the number of steps in an absorbing Markov chain

I was trying to derive the variance of the number of steps before absorption in an absorbing Markov chain and I got something like ...
0
votes
0answers
23 views

writing down markov chain transition matrix

Question: An experimental animal can stay in room-A until 1 minute,and it can stay in room-B until 2 minutes. There exist deadly gases in room-C. One room among these three rooms is being randomly ...
0
votes
0answers
13 views

Canonical Construction of a Markov Chain: Intuition

Let $P=(p_{xy})_{x,y \in E}$ be a transition probability matrix over a discrete state space $E$ and $\mu_0$ any distribution over $E$. We proved in the lecture that there is a unique ...
1
vote
0answers
21 views

Time sampling an ordinary poisson process

My questions will be given at the end, let me just give some definitions first. The counting process $\{ N(t), t \geq 0 \} $ is said to be a non homogenous Poisson process with intensity function ...
1
vote
0answers
8 views

Probability Distribution of Infection

I'm currently working on some Markov chain problems and one of the exercises was a question regarding the spread of an infection. Suppose a man has a highly contagious disease and enters a population. ...
0
votes
0answers
11 views

Optimal stopping strategy

I try to solve the following problem : Given a series of random variables : X1,X2,... such that each one can get either -1 or 1 with probability 0.5, give a strategy to maximize the expected value of ...
-3
votes
0answers
26 views

expected first passage time of a simple random walk [on hold]

For a symmetric, simple random walk on $S={0,1,...,k}$ let $T=\min\{n \in \mathbb N\ | \ X_{0}=x\}$ and $a_{x}=E(T|X_{0}=x)$ show that $a_{x}$ satisfies $a_{x}=0.5a_{x-1} + 0.5a_{x+1} + 1$ for $x ...
0
votes
1answer
17 views

conditional probability density function

If the joint probability density function for the waiting times $W_1$ and $W_2$ is given by: $f(w_1,w_2)=\lambda^2$ $exp(-\lambda w_2)$ for $0<w_1<w_2$. How would I determine the conditional ...
3
votes
1answer
25 views
+50

Joint density function Poisson Process

We did an example in class that I'm not sure how we came up with the answer. The problem is: If I let X(t) be a Poisson process of rate $\lambda$. I'm supposed to validate the identity ...
0
votes
1answer
73 views

Markov Chains Question

Markov chains are widely used in modeling several natural and social processes. Consider the following three-state Markov chain modeling the daily weather in Boston. Each day can be sunny, partly ...
3
votes
1answer
32 views

Question about random walk markov chain

For a random walk, let $a$ denote the probability that the markov chain will ever return to state $0$ given that it is currently in state $1$. Because the markov chain will always increase by $1$ with ...
0
votes
0answers
23 views

Independence of random variables derived from a Random walk

Let $w=(w_x)_{x \in \mathbb Z}$ be i.i.d random variables taking values in $(0,1)$. Let $(X_n)_{n \in \mathbb{N}_0} (\mathbb{N} \cup {0})$ be a Markov chain (more specifically a simple random walk ...
0
votes
0answers
21 views

Bayes: markov chain, serial connection, marginalization

Goal is to check if p(a) is unconditionally independent to p(c) in the markov chain - serial connection. $$ p(a,b,c) = p(a) p(b|a) p(c|b) $$ $$ p(a,c) = \sum_b p(a) p(b|a) p(c|b) = p(a) p(c|a) \neq ...
0
votes
1answer
19 views

Probability that Markov chain process has particular state after n steps

If we have a Markov chain X with four discrete states, and we want to find the probability the process is in a certain state (one of the four) n iterations later, would we raise X to the nth power and ...
1
vote
1answer
23 views

Conditional Probabilities Poisson Process

If I let ${X(t); t>=0}$ be a Poisson process having rate parameter $\lambda = 2$. I'm supposed to determine the probability: Pr{${X(1)>=2 | X(1) >=1}$} My solution: I looked at this as ...
1
vote
1answer
20 views

Conditional Distribution Poisson Process

In class, our professor told us to verify this solution on our own time. The problem is: Let $\left\{X(t),t \geq 0 \right\}$ be a Poisson process of rate $\lambda$. For $s,t >0$, determine the ...
1
vote
1answer
31 views

Modelling a continious-time queue which behaves differently when there are more or less people being served.

For my research I am trying to model a continuous-time queue which behaves differently when there are more or less people being served. The arrival rate in the queue is constant, however the departure ...
-3
votes
0answers
15 views

Conditional Expectation discrete random variables [closed]

$E\left[U|V=v,\space W=n\right]$=$\sum$ $up_{U|\left(V,W\right)}\left(u|v,n\right)$ How do I show that this conditional equation is $\mu _{ij}$$\space$=$\space$1+$\sum$$\space$$\mu _{kj}$$p ...
2
votes
0answers
22 views

$T_n$ stopping time, is $\{X_{T_n}\}$ markov chain

Let $\{X_n\}$ be a Markov Chain with finite state space $S$. Let $T_n$ be the $n$-th hitting time of $A \subset S$ i.e. $n$-th time it hits some state from the set $A$. Is $\{X_{T_n}\}$ a Markov chain ...
4
votes
1answer
38 views

How do stochastic matrices really converge?

We are given the matrix $A=\begin{bmatrix}0.9&0.5\\0.1&0.5\end{bmatrix}$ and any initial vector $X^{(0)}=\begin{bmatrix}a\\b\end{bmatrix}$. The matrix $A$ has the following eigensystem: ...
0
votes
0answers
27 views

mean recurrence time

$E\left[T_j |X_0=i,X_1=k\right]$ \left\ space{\begin{matrix} 1+U_{kj} \space\ k\neq j & \\ 1 \space\ k=j &\end{matrix}\right. Does this mean that the number of steps it takes to get back to ...
1
vote
2answers
22 views

Something about Markov chains

We check $P(X_{n+1}\in B|\mathcal{F}_n)=P(X_{n+1}\in B|X_n)$ when we want to prove $X_n,n=1,2,\dots$ is a Markov chain. Through this equation it seems that $X_n$ is a Markov chain if $X_{n+1}$ is ...
0
votes
1answer
32 views

Proving that the Markov chain is recurrent - Confusion/Help

Giving the following transition matrix [ 0.9 0.1 ] [ 0.8 .2 ] Classify the states From drawing the graph I know that both stats are recurrent. However I'm really failing to prove mathematically ...
0
votes
0answers
21 views

Markov Chain's irreducibility and closeness

Problem: Give an example of a Markov chain with state space $S$ and subsets B, $C ⊆ S$ such that $B$ is irreducible but not closed and $C$ is closed but not irreducible.
1
vote
0answers
8 views

Is the steady state of a uniform markov chain always a vector of proportions?

Given that all edges in a markov chain are bi-directional (though not necessarily equally weighted), and each edge for a given node has equal probability, does the steady state always converge to a ...
1
vote
1answer
16 views

Definition of limiting distribution in a Markov chain — why do we condition on the initial state?

Given a Markov chain $\{X_n \mid n \in \{0, 1, \ldots\}\}$ with states $\{0, \ldots, N\}$, define the limiting distribution as $$ \pi = (\pi_0, \ldots, \pi_N) $$ where $$ \pi_j = \lim_{n \to +\infty} ...
0
votes
1answer
24 views

Expected time to absorption

I have been trying to solve the following problem for quite a while now, but not with much luck. The Question Let $P$ be the TPM(Transition Probability Matrix) of a DTMC with state space ...
0
votes
0answers
36 views

I am stuck on this Probability Question. Please help.

The Problem: Let a Markov Chain have R states. Show that if j is recurrent, then there exists $0\leq x\leq 1$ such that for $n > r$ the probability that the first return from state j occurs after ...
0
votes
1answer
26 views

Infinite$-$state absorbing Markov chains

Could someone provide a good reference/book about infinite$-$state absorbing Markov chains? Most of what I've found so far deals only with the finite$-$state case.
0
votes
0answers
35 views

Help with this Markov Chain Proof please

Problem: Consider a finite Markov Chain with N states $(1,2,...,N)$. Let $P(n) = [P_{i,j} (n)]$, be an n-step transition matrix. Suppose that $lim_{n\to\infty} P_{i,j} (n) = \pi_{j} $ for any $1 ...
1
vote
0answers
11 views

Conditional independence for a dynamic random field

Let $X = \left\{ {{X^{\left( \alpha \right)}}:\alpha \in {\mathbb{N}_0}} \right\}$ be a dynamic random field with a set of places $V$ and a phase space $\Lambda $ such that $\left| V \right| < ...
-1
votes
0answers
19 views

Monte Carlo Markov Chain

Is it possible to solve Poisson equation with Monte Carlo Markov Chain method, and how? Is there any book for problem like this? Thank you.
0
votes
0answers
56 views

When is $A^{+} P^{\top} A$ non-negative?

$P$ is a $n \times n$ stochastic matrix (non-negative, rows sum to one). $A \in \mathbb{R}^{n \times k}$ with $k < n$ has non-negative entries and independent columns. Denote by $A^+ \in ...
0
votes
1answer
25 views

Understanding the proof of stationary distribution of a markov chain

I am reading the proof of existence of stationary distribution in an irreducible markov chain from the book Markov Chains and Mixing Times by P. D. A. Levin, Y. Peres, E. L. Wilmer, and I have the ...
0
votes
0answers
14 views

Repair Chain (Markov Chain Sample Model)

A machine has $3$ critical parts that are subject to failure, but can function as long as two of these parts are working. When two are broken, they are replaced and the machine is back to working ...
0
votes
0answers
16 views

Show if $p^n(i,j)\rightarrow \pi(j)$ as $n\rightarrow\infty$ then $\pi(j)$ is a stationary measure..

Suppose $p(i,j)$ is a transition kernel on $S$ for a countable state markov chain $X_n$ with $$p^n(i,j)\rightarrow \pi(j)$$ as $n\rightarrow\infty$ for all $i,j\in S$. want to verify that $\pi$ is a ...
0
votes
1answer
27 views

Proof of aperiodic Markov Convergence Theorem for null recurrent case.

Status quo: We consider a irreducible, aperiodic Markov chain $(X_n)_{n\in\mathbb{N}}$ on a countable set $S$ with tranistion function $p(\cdot,\cdot)$. Now we want to examine ...
1
vote
1answer
25 views

The lower bound for the smallest eigenvalue given the condition

In a paper, i saw a statement that the smallest eigenvalue of $P$($P$ is reversible Markov chain with stationary distribution $\pi$) is greater than $2 \beta - 1$ with the condition, $P \geq \beta I$. ...
1
vote
0answers
30 views

Does Markov Chain converge in Variance Norm?

Assume the chain $\{X_n\}_{n\in\mathbb{N}}$ on the statespace $(S,\mathcal{F})$ is aperiodic, irreducible and positive recurrent. We denote with $\pi$ its (unique) stationary distribution. Is it true ...
0
votes
1answer
16 views

How would I solve for long run average profit?

I was looking at a problem, and I was wondering how I would set this up. Any help would be welcome. Thank you! A store stocks a particular item. The demand for the product each day is 1 item with ...
0
votes
1answer
63 views

How do I compute the variance of expected number of fair coin flips for HTH sequence using linear system of equations?

Assuming fair coin flips, I know how to compute the expected number of coin flips to see HTH sequence by writing out the linear system of equations from the state transition diagram below. ...
0
votes
0answers
8 views

Proof of the “Markovian property” for the LERW?

I'm trying to understand this proof by Werner of the Markovian property of the Loop-erased random walk http://arxiv.org/pdf/math/0303354v1.pdf (page 10). The first part I see but the second "again, ...
2
votes
0answers
17 views

Formal argument on independence of consecutive hitting times of a Markov chain.

I'm refering to the question: Differences of consecutive hitting times. I'm interested in the independence of consequtive hitting times of certain values of a Markov chain. And I do "understand" the ...
1
vote
1answer
32 views

Can a reducible Markov chain have an unique stationary distribution? [closed]

I know for irreducible and positive recurrent Markov Chain there exists an unique stationary distribution. For Markov Chain with several communication classes (example C1, C2) there exist stationary ...
1
vote
2answers
55 views

How to compute the variance of number of coin flips to see HTH sequence using linear system of equations.

Assuming fair coin flips, I know how to compute the expected number of coin flips to see HTH sequence by writing out the linear system of equations from the state transition diagram below. Define ...
1
vote
1answer
43 views

What is the probability there will be no failures?

"A machine has 4 components and the machine cannot operate when any one of these components fail. At the beginning of each day, the machine starts running. During any day component $i$ fails with ...
3
votes
0answers
18 views

$ X_n = 2 Y_n + Y_{n+1} $ (non)Markov Chain

Let $Y_1,Y_2,\dots$ be iid random variables with $P(Y_n=0)=1-p,\; P(Y_n=1)=p$ where $p\in(0,1)$. Define $$ X_n = 2 Y_n + Y_{n+1} $$ The question is, whether $\{X_n\}$ is a Markov chain or not. ...
0
votes
1answer
28 views

Calculating the information per symbol of a markov chain source

I have a 4-state 2nd order markov chain source with symbols 0 and 1. I have all the transition probabilities and have worked out the probabilities of each state. How do I go about finding the amount ...