Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
105 views

How do I compute the variance of expected number of fair coin flips for HTH sequence using linear system of equations?

Assuming fair coin flips, I know how to compute the expected number of coin flips to see HTH sequence by writing out the linear system of equations from the state transition diagram below. ...
0
votes
1answer
36 views

How would I solve for long run average profit?

I was looking at a problem, and I was wondering how I would set this up. Any help would be welcome. Thank you! A store stocks a particular item. The demand for the product each day is 1 item with ...
0
votes
0answers
11 views

Proof of the “Markovian property” for the LERW?

I'm trying to understand this proof by Werner of the Markovian property of the Loop-erased random walk http://arxiv.org/pdf/math/0303354v1.pdf (page 10). The first part I see but the second "again, ...
0
votes
0answers
54 views

Transition probability matrix for $X_1 = \# heads$, *flip heads* $X_2 = \# tails$ * flip tails* $X_3 = \# heads$

Three fair coins are tossed, and we let $X_1$ denote the number of heads that appear. Those coins that were heads on the first trial (there were $X_1$ of them) we pick up and toss again, and now we ...
0
votes
0answers
32 views

Decide whether a class is recurrent or transient (Example)

Consider the Markov chain $(X_n)_{n\geqslant 0}$ with state space $E=\left\{1,2,3,4\right\}$ and transition matrix $$ T=\begin{pmatrix}0 & 1/3 & 1/3 & 1/3\\0 & 0 & 1 ...
0
votes
0answers
21 views

Explicit Probability for Markov Chain on Power Set

A have a Markov chain $F_t$ in discrete time on the power set of a finite totally ordered set $A$. Its probably easiest to explain the transition probabilities in a small example, since they are easy ...
0
votes
0answers
15 views

Is Markov Chain sampled at stopping times a Markov chain?

Given a Markov hain $\{X_n\}$ and $T_n$ be an increasing sequence of stopping times, is $\{X_{T_n}\}$ Markov chain ?
0
votes
0answers
60 views

emails arriving in a Poisson process

Emails arrive according to a Poisson process with rate $λ=2/hour$. You check your inbox (instantly reading all new emails) at time $t=5$ hours and also at some uniformly distributed random time ...
0
votes
0answers
32 views

Mean and Variance of an offspring

If I have that the number of offspring of an individual in a population is $0$, $1$, or $2$ with respective probabilities $a>0$, $b>0$ and $c>0$, where $a+b+c=1$, how would I express the mean ...
0
votes
0answers
35 views

Multiclass Markov process

There are two car M/M/1 queues Q1 and Q2. Arrival rate of Red car and Green car in Q1 is $\lambda_{1R}$ and $\lambda_{1G}$ respectively. Similarly arrival rate of red car and green car in Q2 is ...
0
votes
1answer
38 views

Measurability of a stopping time in a Markov chain

Suppose you have a finite-state continuous-time inhomogeneous Markov chain with transition rate $Q(t)$. Further, let us suppose that $Q(t)$ is a piecewise continuous function of $t$. Two questions: ...
0
votes
1answer
26 views

Strategy for Unbalnaced Gamber Ruin

A gambler plays the following game: A fair coin is tossed until getting three times continuously head. When that happens the Gambler gets 20$\$$. Each round costs the gambler 1$ (even if he won the ...
0
votes
0answers
54 views

Practical Way to Detect a Markov Chain is Regular Given the Transition Matrix

I understand that a Markov Chain is reducible if, given its transition matrix $P$, there exists $n$ such that every element of $P^n$ is greater than 0. However, I am wondering that if there is an ...
0
votes
0answers
49 views

Show that this Markov chain is recurrent or transient

Consider the Markov chain $(X_n)_{n\geq 0}$ with state space $E=\left\{1,2,3,4,5\right\}$ and transition matrix $$ T=\frac{1}{2}\begin{pmatrix}0 & 1 & 1 & 0 & 0\\0 & 0 ...
0
votes
0answers
35 views

diffusion- stuck

In a round room of radius R, a large number of coins N of diameter d are randomly dispersed upon the floor. A ladybird starts from the centre of the room, crawling at speed v. Suppose that every time ...
0
votes
3answers
53 views

How to solve $h(i) = \frac{i^2}{(n-i)^2+i^2}h(i-1) + \frac{(n-i)^2}{(n-i)^2+i^2}h(i+1)$

$h(i) = $P(reach n eventually| the initial state = i) $h(0) = 0$ $h(n) = 1$ 0 and n are stopping time. For $ 0 < i < n$, $$h(i) = \frac{i^2}{(n-i)^2+i^2}h(i-1) + ...
0
votes
1answer
29 views

Show $\sum^n_{j=0} P_{ij} = \sum^n_{j=0} P(X_1 =j | X_0 = i) = 1 $

Let $P$ be the one step transition matrix of a Markov chain with states {$0,1,...,n$}. Show $\sum^n_{j=0} P_{ij} = \sum^n_{j=0} P(X_1 =j | X_0 = i) = 1 $ I understand that this is the row sum, but ...
0
votes
0answers
20 views

Changing the index of the sums when changing the sums - why this way?

Let $(X_n)_{n\in\mathbb{N}_0}$ be a Markov chain with state space $E$. For $i,j\in E$ set $$ h_i(j):=\mathbb{P}_i(H(j)<\infty):=\mathbb{P}(H(j)<\infty|X_o=i), $$ where $H(j)\colon ...
0
votes
0answers
23 views

expected value - last $k$ flips of coin are same

we flip a normal coin $n$ times. We mark $k=0.5log(n)$ and we mark the $i$'th value in $Xi$. $Y$ will be the number of times where the last $k$ flips were the same. What is $E[Y]$? I think this has ...
0
votes
0answers
28 views

Markov Chain - last $0.5log(n)$ Tosses of Coin

We toss a coin $n$ times and we mark $k=0.5log(n)$. $Y$ is the number of times where the last $k$ tosses were the same. What is $E(Y)$? I'm pretty sure I need to use Markov Chain but I'm not sure ...
0
votes
0answers
87 views

Gibbs sampling from a 2D Gaussian

Hi I have the to do the next problem and I am kind of lost, if someone could give a litte hint of where to start I would really appreciate it. Thanks in advance! Suppose $x$~$ N(\mu;\sigma)$ where ...
0
votes
0answers
47 views

Specifying transition probabilities for a Markov Chain

If I have a queueing model and I suppose at most a single customer arrives during a single period, but that the service time of a customer is a random variable Z with geometric probability ...
0
votes
0answers
20 views

Why so complicated to show that $P_j(t(i)<\infty)=1$?

Let $(X_n)_{n\in\mathbb{N}_0}$ be an irreducible and recurrent Markov chain with state space $E$ and transition matrix $P$. For an $i\in E$ let $t(i)$ denote the random variable ...
0
votes
0answers
53 views

Define a maximization problem as an optimal stopping problem

We work over $\mathbb{R}_+^L$. Let $V$ be the set of vectors whose coordinates take values $0$ or $1$. Let $\mathbf{w}(t)$ (in $\mathbb{R}_+^L$) a vector that changes each time slot. To each vector ...
0
votes
0answers
33 views

Identifying a Markov chain

This is a very basic question in the theory of Markov chains and I'm just not sure how to prove it mathematically. Say we have random variables $X, Y$ that are correlated and we have a possibly ...
0
votes
0answers
21 views

Is there any way to find probability of marcov chain when the time is same?

I am just wondering if I am given P(Xn=1 given that Xn=0) (Usual One Marcov chain) Can you find this probability using transition matrix?
0
votes
1answer
81 views

Markov Chain, finding the steady state vector

Suppose that if it is sunny today, there is a 60% chance that it will be sunny tomorrow, a 30% chance that it will be partly cloudy and a 10% chance that it will be completely cloudy. If it is partly ...
0
votes
1answer
30 views

Probability of getting disease and Markov chain

I am studying marcov chain. The question is . There are 5 people ( 4 diseased / 1 healthy) Two people are selected randomly and assumed to interact. If one is diseased and the other is healthy, ...
0
votes
0answers
16 views

How to estimate a hidden model for an unstationary Markov process?

I have a problem that is very similar to the one solved by the Baum–Welch algorithm. I have a process that is very similar to a hidden Markov process. The only difference is that I have an observable ...
0
votes
0answers
22 views

Null recurrence to positive recurrence in DTMC

What are some examples of null recurrent DTMC whose jump chain is positive recurrent? Specifically, for this null recurrent DTMC, removing self loops and normalizing the other outgoing edges from each ...
0
votes
0answers
42 views

recurrence/transience on random walk

Let $X_n$ be a markov chain, $p>\frac{1}{2}$ and $E=\{0,1,2,...\}$ its state space. Let $\Pi$ be its transition matrix with $\Pi(0,0)=p$, $\Pi(i+1,i)=p$, $\Pi(i,i+1)=1-p$ , $i\ge0$. ...
0
votes
0answers
23 views

Reference on Discrete Markov Chains

I am essentially looking for reference books on Discrete Markov Chains. You can see our full syllabus here.
0
votes
0answers
22 views

Show recurrence of a class

I am a little bit confused with the definition of recurrence with respect to Markov chains. For example consider the transition matrix $$ P=\frac{1}{2}\begin{pmatrix}0 & 1 & 1 & 0 & ...
0
votes
0answers
62 views

Dynamics of birth-death process with discouraged arrivals (alternatively, M/M/1 queue with balking customers)

Take a continuous-time birth-death process, where $k \in \{0,1,\ldots\}$ is the count and the arrival rate of death is $\mu \geq 0$ for $k = 1, 2, \ldots$ the arrival rate of births is $\alpha_k ...
0
votes
0answers
26 views

Does a state which is passed at least 3 times had to be passed 5 times in Markov chain

Prove of disprove: Let $\{X_n\}_n$ be homogenous Markov chain. if we start from state $i$, there is a positive probability that we pass at least 3 times at state $j$. Does it follows that exists ...
0
votes
0answers
22 views

continuous markov chain generator

I am trying to learn Markov process with my own. I am a little confused about the generator of markov process. I understand that Markov process consists of embbedded Markov chain matrix and the ...
0
votes
0answers
134 views

Proof of “strong law of large numbers” in Markov Chains

I have been given a theorem stating an analogue of the strong law of large numbers for Markov chains. It states that if $X=(X_n)_{n\in\mathbb{N}}$ is a Markov chain with transition matrix $p$ and ...
0
votes
1answer
75 views

Markov chain doesn't sum up to 1

Let $\{X_n\}$ be a Markov chain on $S=\{1,2,3,4,5,6\}$ with the matrix suppose we define a new sequence $\{Y_n\}$ by $$Y_n=\cases{1\quad X_n=1\vee X_n=2\\2\quad X_n=3\vee X_n=4\\3\quad ...
0
votes
0answers
41 views

Proof of theorem on markov chains

If $X=(X_n)_{n\in\mathbb N}$ is a Markov chain on a space $E$, it has an initial distribution $(\lambda_i)_{i\in E}$ such that $\sum\lambda_i=1$ and a transition matrix $(p_{ij})_{i,j\in E}$ such that ...
0
votes
0answers
43 views

non-integer powers of transition matrices with complex eigenvalues and resulting negative probabilities

I am currently working on a Markov Chain model for transition probabilities of a certain set of states. I am trying to figure out how to scale my transition matrix to arbitrary time periods by raising ...
0
votes
0answers
25 views

Memory less property of a Markov chain- Validation methods

Are there any tests to check the memory less property of a discrete time homogeneous Markov chain? I found a chi squared test to verify the time homogeneity of a Markov chain constructed from a set of ...
0
votes
0answers
5 views

Hastings algorithm

Let $Q=\begin{pmatrix} 0 & 1 & 0 & 0 & 0\\0.5 & 0 & 0.5 & 0 & 0\\ 0 & 0.5 & 0 & 0.5 & 0\\ 0 & 0 & 0.5 & 0 & 0.5\\ 0 & 0 & 0 ...
0
votes
0answers
23 views

Markov chain general help

If I have an absorbing state markov chain (with 2 absorbing states, graduate and dropout), and I know how many people I have in each state (say total for all states is 1000), how would I work out what ...
0
votes
1answer
79 views

Proof that there exists a non-negative eigenvector corresponding to eigenvalue 1 of stochastic matrix

Let $P \in [0,1]^{n \times n}$ be a [irreducible or reducible] stochastic matrix where its rows sum to 1 i.e. $$ \forall i \in \{ 1 , \dots n \} \quad \sum_{j=1}^{n} P_{ij} = 1 $$ It is easy to show ...
0
votes
0answers
49 views

Create a Martingale out of a Markov Chain.

Consider a homogeneous finite state Markov chain $\{X_n\}$ with transition matrix $P$ and state set $S$ consisting of real numbers. How to choose the elements of $S$ so that $\{X_n\}$ be a ...
0
votes
0answers
6 views

Condition under which Markov Chain remains in a compact set a.s.

Let $\{Y_n\}$ be a Marov chain. Is it good question to ask under what conditions this chain will take values from a compact set a.s. ?
0
votes
0answers
12 views

Given stochastic matrices D and K, under what conditions can I find stochastic matrices that satisfy a given equality?

Let $D \in \Re^{n \times m}$ and $K \in \Re^{m \times n}$ be two stochastic matrices, with $n > m$. The problem is to determine under which conditions there exist stochastic matrices $P \in \Re^{m ...
0
votes
1answer
47 views

Simple Random Walk; Proof hitting theorem; Ballot Theorem

Suppose that $(X_{n}:n\in\mathbb{N})$ is a $\pm1\mbox{-valued sequence.}$ Let $p\in(0,1)$ and $p=\mathbb{P}(X_{i}=1)\mbox{ and}\mathbb{P}(X_{i}=-1)=1-p=q$ . Define the simple random walk ...
0
votes
0answers
37 views

Stationary solution of a binary Markov chain of order m

Let $X$ be a binary Markov chain of order m. What is the stationary solution of X? In other words, find $\lim_{n\to \infty} P( (X_{n-m+1},X_{n-m},...,X_{n}) =(a_1,a_2,...,a_m))$, for arbitrary values ...
0
votes
0answers
29 views

Steady state of a specific Markov martirx

Let $n = 2^L$ for an arbitrary integer $L>0$ and let $A=(a_{i,j})$ be an $n \times n$ matrix with the following structure: For $1\leq i \leq \frac{n}{2}$, $a_{i,2i-1} = p_i$, $a_{i,2i} = 1- ...