Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

4
votes
3answers
101 views

A recursive formula to approximate $e$. Prove or disprove.

Let the sequence $\{x_n, n=1,2,...\}$ be defined as follows: Let $x_2=x_1=1$ and for $n>2$ let $$x_{n+1}=x_n-\frac{1}{n}x_{n-1}.$$ This sequence, generated by the recursion above, tends to zero ...
0
votes
0answers
45 views

emails arriving in a Poisson process

Emails arrive according to a Poisson process with rate $λ=2/hour$. You check your inbox (instantly reading all new emails) at time $t=5$ hours and also at some uniformly distributed random time ...
0
votes
0answers
14 views

a reference for problems and exercises in markov chains with solutions

i've started studying markov chains from the book of Lawler,i've solved its exercise but i want a solution for Lawler in order to check my answers. can any one help me finding? any other resource ...
0
votes
0answers
23 views

Mean and Variance of an offspring

If I have that the number of offspring of an individual in a population is $0$, $1$, or $2$ with respective probabilities $a>0$, $b>0$ and $c>0$, where $a+b+c=1$, how would I express the mean ...
1
vote
1answer
32 views

Finding mean and variance of a population problem

A population beings with a single individual. In each generation, each individual in the population dies with probability $1/2$ or doubles with probability $1/2$. If I let $X_n$ denote the number of ...
4
votes
1answer
65 views

Find the Stationary Distribution of an infinite state Markov chain

A Markov Chain on states 0,1,..... has transition probabilities $P_{ij}=1/(i+2)$ for j=0,1,....,i,i+1. I'm supposed to find the stationary distribution. So do I take the limit as n goes to ...
1
vote
2answers
40 views

Limiting Distribution of a Markov Chain

I'm having trouble understanding how to find a limiting distribution. If I have a Markov Chain whose transition probability matrix is: $$ \mathbf{P} = \matrix{~ & 0 & 1 & 2 & 3 & ...
1
vote
1answer
37 views

Electrical components failing in a Poisson process

A machine has infinitely many identical components. They fail according to a Poisson process with rate $\lambda = 4$/hour. A repairman arrives at time $t$ and instantly repairs all of the broken ...
2
votes
1answer
68 views

Prove that the absolute value of the difference of two invariant distributions on a Markov chain is invariant

If we have $a(x)$, $b(x)$ which are invariant distributions on a Markov chain $X_n$ with state space $S$, how can I prove that $|a(x)-b(x)|$ is also invariant? I know that I must show that: ...
0
votes
0answers
31 views

Multiclass Markov process

There are two car M/M/1 queues Q1 and Q2. Arrival rate of Red car and Green car in Q1 is $\lambda_{1R}$ and $\lambda_{1G}$ respectively. Similarly arrival rate of red car and green car in Q2 is ...
1
vote
0answers
16 views

Converse of Perron Frobenius Theorem: Necessary and Sufficient Conditions for positivity (or non negativity)

As I understand, Perron Frobenius theorem asserts only in one direction, i.e. if Matrix A is positive then there is a perron eigenvalue, eigenvector etc. What I wanted to know is what are the ...
0
votes
1answer
31 views

Measurability of a stopping time in a Markov chain

Suppose you have a finite-state continuous-time inhomogeneous Markov chain with transition rate $Q(t)$. Further, let us suppose that $Q(t)$ is a piecewise continuous function of $t$. Two questions: ...
3
votes
1answer
70 views

Markov Property Confusion

I feel like I'm being very dense/employing some sort of circular reasoning, but I'm having trouble understanding the Markov Property. According to Durrett (ISBN-10:1461436141), $X_n$ is a Markov chain ...
0
votes
1answer
22 views

Strategy for Unbalnaced Gamber Ruin

A gambler plays the following game: A fair coin is tossed until getting three times continuously head. When that happens the Gambler gets 20$\$$. Each round costs the gambler 1$ (even if he won the ...
0
votes
0answers
38 views

Practical Way to Detect a Markov Chain is Regular Given the Transition Matrix

I understand that a Markov Chain is reducible if, given its transition matrix $P$, there exists $n$ such that every element of $P^n$ is greater than 0. However, I am wondering that if there is an ...
1
vote
1answer
62 views

Eigenvalue range of $P+P^T$ (P is a transition matrix)

$P$ is a transition matrix of dimension $N\times N$. I know $\lambda_1=1$ and $|\lambda_i|<1, 2\leq i \leq N$. I want to know the eigenvalue range of $P+P^T$. Because $P$ is not symmetric, so I ...
2
votes
2answers
26 views

why are the recurrent classes closed?

i've recently started studying about markov chain, we call a communication class a recurrent one in a markov chain if by starting from that class we infinitely return to it with probability 1,with ...
0
votes
1answer
27 views

Definition of Stationary Distributions of a Markov Chain

I'm having a lot of trouble understanding the definition of the stationary distribution of a Markov Chain from Hoel, Port, Stone's Introduction to Stochastic Processes. They define the stationary ...
1
vote
0answers
13 views

Estimating Markov transition matrix for regularization

Suppose that I have a sequence of discrete distributions: $$ p_j = (p_{1j},...,p_{Cj}), \: j=1...D,\\ p_{ij}>0 \:\: \forall i,j,\: \sum_{k=1}^Cp_{kj}=1\:\:\forall j. $$ I suppose that these ...
2
votes
0answers
17 views

Showing which classes are recurrent and which are transient

If I have a Markov chain on states {0,1,2,3,4,5} $$ \mathbf{a} = \matrix{~ & 0 & 1 & 2 & 3 & 4 & 5 \\ 0 & 1/3 & 0 & 2/3 & 0 & 0 & 0 \\ ...
0
votes
1answer
57 views

Markov chain for two players with two coins [closed]

Two players A and B toss two fair coins independently. Whoever gets the smaller number of heads will pay that many dollars to the other player. For example, if player A tosses two coins and gets 2 ...
0
votes
0answers
38 views

Show that this Markov chain is recurrent or transient

Consider the Markov chain $(X_n)_{n\geq 0}$ with state space $E=\left\{1,2,3,4,5\right\}$ and transition matrix $$ T=\frac{1}{2}\begin{pmatrix}0 & 1 & 1 & 0 & 0\\0 & 0 ...
1
vote
1answer
50 views

Markov chain - Can anyone explain me why this is the solution?

Customers arrive according to a Poisson process at a rate of four customers per hour. A customer who finds four other customers in already waiting gives up and leaves. Some clients in the 3rd ...
0
votes
0answers
30 views

diffusion- stuck

In a round room of radius R, a large number of coins N of diameter d are randomly dispersed upon the floor. A ladybird starts from the centre of the room, crawling at speed v. Suppose that every time ...
1
vote
1answer
25 views

Show that the process created from taking kth steps of a markov chain is markov.

Suppose $(X_n)_{n\geq0}$ is a Markov chain with transition probability matrix $P$ and initial distribution $\lambda$. Show that the process $Y_n = (X_{kn})_{n\geq0}$ with $k$ fixed is Markov with ...
0
votes
3answers
53 views

How to solve $h(i) = \frac{i^2}{(n-i)^2+i^2}h(i-1) + \frac{(n-i)^2}{(n-i)^2+i^2}h(i+1)$

$h(i) = $P(reach n eventually| the initial state = i) $h(0) = 0$ $h(n) = 1$ 0 and n are stopping time. For $ 0 < i < n$, $$h(i) = \frac{i^2}{(n-i)^2+i^2}h(i-1) + ...
0
votes
1answer
29 views

Show $\sum^n_{j=0} P_{ij} = \sum^n_{j=0} P(X_1 =j | X_0 = i) = 1 $

Let $P$ be the one step transition matrix of a Markov chain with states {$0,1,...,n$}. Show $\sum^n_{j=0} P_{ij} = \sum^n_{j=0} P(X_1 =j | X_0 = i) = 1 $ I understand that this is the row sum, but ...
0
votes
0answers
18 views

Changing the index of the sums when changing the sums - why this way?

Let $(X_n)_{n\in\mathbb{N}_0}$ be a Markov chain with state space $E$. For $i,j\in E$ set $$ h_i(j):=\mathbb{P}_i(H(j)<\infty):=\mathbb{P}(H(j)<\infty|X_o=i), $$ where $H(j)\colon ...
2
votes
1answer
25 views

Adding distances/weights to absorbing markov chain

in presence of an absorbing state, I want to calculate mean/expected 'distance' from any state to that absorbing state. What I mean by distance is that I want to give different lengths from one ...
0
votes
0answers
21 views

expected value - last $k$ flips of coin are same

we flip a normal coin $n$ times. We mark $k=0.5log(n)$ and we mark the $i$'th value in $Xi$. $Y$ will be the number of times where the last $k$ flips were the same. What is $E[Y]$? I think this has ...
0
votes
0answers
28 views

Markov Chain - last $0.5log(n)$ Tosses of Coin

We toss a coin $n$ times and we mark $k=0.5log(n)$. $Y$ is the number of times where the last $k$ tosses were the same. What is $E(Y)$? I'm pretty sure I need to use Markov Chain but I'm not sure ...
4
votes
2answers
71 views

Expected value of money left from a coin flipping game

Say we were to play a game. We started off with \$100 and kept flipping a fair coin. If it turned out heads, we won \$1, else our money got inverted. For example, if on the first flip we got heads, ...
0
votes
0answers
59 views

Gibbs sampling from a 2D Gaussian

Hi I have the to do the next problem and I am kind of lost, if someone could give a litte hint of where to start I would really appreciate it. Thanks in advance! Suppose $x$~$ N(\mu;\sigma)$ where ...
2
votes
2answers
37 views

How can i interpret this absorbing markov chain to solve a probability question?

I try to solve a simple question; if I toss a coin and repeat it until a tails come up, what is the mean number of steps? (I want to solve another question but it is just a complicated version of ...
1
vote
0answers
25 views

Survival function of birth-death process

There is a linear birth-death process with $N$ states + an absorbing state $0$, with $$\Pr[X_{t+1}=0|X_{t}=0]=1, \\ \Pr[X_{t+1}=i+1|X_{t}=i]=\Pr[X_{t+1}=i-1|X_{t}=i]=q_i, i\in [1..N-1],$$ and ...
0
votes
1answer
55 views

Can you help me with this Markov Chain question?

The Problem: Prove that if the number of States in a Markov Chain is M, and that state j can be reached from state i, then it can be reached in M steps or less. The work: I assumed by contradiction ...
1
vote
0answers
15 views

No null recurrent state in finite state space from definition.

Let $\{X_n\}$ be a markov chain on finite state space $I$, with stationary transition probabilities. Let us denote $f^n(i,i):=P(X_n=i,X_{n-1}\neq i,\ldots X_1\neq i\mid X_0=i)$. We say $i$ is ...
1
vote
1answer
41 views

Forward Algorithm Hidden Markov Model matrix help [Discrete]!

So this may seem like a bioinformatics question but it is the math part that is giving me trouble. I'm using a Python package called YAHMM to model DNA sequences. I created a model with two states ...
1
vote
0answers
38 views

Markov chain in continuous time (transition probabilities)

Let $(X_t)_{t \geq 0}$ be a markov chain in continuous time with state space $\mathbb{N}_0$. I want to express $\mathbb{P}(X_t = 2| X_0 = 1, X_{3t} = 1)$ and $\mathbb{P}(X_t = 2 | X_0 = 1, X_{3t} = ...
6
votes
3answers
170 views

Probability of going into an absorbing state

If I have a random walk Markov chain whose transition probability matrix is given by $$ \mathbf{P} = \matrix{~ & 0 & 1 & 2 & 3 \\ 0 & 1 & 0 & 0 & 0 \\ ...
1
vote
2answers
21 views

How can I prove that $ p_{x,y}^{(n)}=P(X_n=y|X_0=x)$?

Let $(\Omega,\mathcal{A},P)$ a probability space. Let $E$ be a countable set and $\Bbb P:=(p_{x,y})_{x,y\in E}$ a stochastic matrix (i.e. $p_{x,y}\ge0$ and $\sum_{y\in E}p_{x,y}=1$) and $\mu$ a ...
2
votes
1answer
41 views

How do I show that this game played on a Markov chain has a unique Nash equilibrium?

There are $k$ stages in this game, and each stage is worth one unit of utility to a player (of which there are $n$). Each player $i$ finishes stages at a rate $\lambda_i$ (in a continuous time Markov ...
0
votes
1answer
31 views

Expected number of transition events to complete multiple synchronized Markov chains

Assume the expected number of transitions (events) it takes until a Markov chain with $G+1$ states ranging from $s=0$ to $s=G$ is completed is $M$. Suppose we have $K$ independent instances of this ...
1
vote
1answer
49 views

Markov Chain with heterogeneous transitions

I have a Markov chain as follows: $G+1$ finite states, it begins from $s=G$ and completes at $s=0$ A transition ($s\to s-1$) occurs in case if event $A$ happens. No other form of transition is ...
0
votes
0answers
29 views

Specifying transition probabilities for a Markov Chain

If I have a queueing model and I suppose at most a single customer arrives during a single period, but that the service time of a customer is a random variable Z with geometric probability ...
2
votes
1answer
149 views

The expected time until reaching a specified set in a Markov chain

I am reading an article in which they discuss a specific Markov chain in an example, and it turns out I need to sharpen up my Markov knowledge. First the setup. I have a continuous time Markov chain ...
0
votes
1answer
47 views

Urn Problem-Determining the Transition Probability Matrix

I have two urns A and B containing a total of N balls. An experiment is performed where a ball is selected at random (all selections equally likely) at time t(t=1,2,...) from the totality of N balls. ...
-1
votes
1answer
23 views

Limiting Distributions

Let P be the transition matrix $$ P = \begin{bmatrix} 0 & 0.2 & 0.2 & 0.2& 0.2 & 0.2 \\ 0.2 & 0 & 0.2 & 0.2 & 0.2 &0.2 \\ 0.2 & 0.2 & 0 & 0.2 & ...
0
votes
0answers
20 views

Why so complicated to show that $P_j(t(i)<\infty)=1$?

Let $(X_n)_{n\in\mathbb{N}_0}$ be an irreducible and recurrent Markov chain with state space $E$ and transition matrix $P$. For an $i\in E$ let $t(i)$ denote the random variable ...
0
votes
0answers
45 views

Define a maximization problem as an optimal stopping problem

We work over $\mathbb{R}_+^L$. Let $V$ be the set of vectors whose coordinates take values $0$ or $1$. Let $\mathbf{w}(t)$ (in $\mathbb{R}_+^L$) a vector that changes each time slot. To each vector ...