Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

2
votes
1answer
55 views

Given two biased coins, the probability of obtaining heads on the $i^\text{{th}}$ toss using the following strategy?

We are given two coins: A and B with probability of obtaining heads being: $\alpha$ and $\beta$ respectively. The following sampling rule is used for i=1,2,...: If the $i^{\text{th}}$ toss results in ...
1
vote
1answer
21 views

Why does this MCMC algorithm to estimate parameters of a linear equation not converge to the posterior distribution?

As a kind of proof of principle I'm trying to estimate the parameters of a linear equation (before moving on to ODEs) using Markov Chain Monte Carlo sampling. The post that I am following can be found ...
2
votes
1answer
91 views

Showing a queueing system is a Markov Chain

I generally understand how to do this but I'm having trouble with a formal proof. "Consider an $M/M/1/m+1$ queue with exponential arrivals rate $\lambda$, exponential service rate $\mu$, and finite ...
0
votes
1answer
119 views

Markov chain: molecules in urns

I am struggling to get started on this question. I think I am confused at what the transition matrix is suppose to represent. So I know the matrix is going to have this form: $$ \begin{vmatrix} P_{...
0
votes
1answer
101 views

Markov Chain Snakes and Ladders

I am really stuck on the following question: So first I need to work out the transition matrix. But I am not sure how? Lets say I am at square 0 and I want to square 1, is the probability of moving ...
0
votes
1answer
35 views

If $X$ to $Y$ to $Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$

If $X \to Y \to Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$. I said the statement was true, and from $I(X;Y)\ge I(X;Z)$ by definition, thus $H(X) - H(X\mid Y) \ge H(X)-H(X\mid ...
0
votes
0answers
45 views

Normalisation of an invariant measure

Is there an example of a Markov chain with invariant measure $\pi$ and $\sum_{i \in I}\pi(i) = \infty$ that can be normalised so that we can consider an invariant distribution instead? This is a ...
0
votes
0answers
25 views

Most visited vertex in a random walk with a place dependent drift

Consider the following Markov chain on $\mathbb{Z}$: $$ P(x,x+1)=1-P(x,x-1)=\frac{1}{2}+e^{-|x|}\cdot \mathbf{1}_{\{x\neq 0\}} $$ Do there exist constants $c,C>0$ such that $$ c\cdot P^t(z,z) \...
1
vote
1answer
293 views

Gambler's Ruin and Markov Chains

Suppose that on each play of a certain game, a person will either win one dollar with the probability of $\frac{2}{3}$ or lose one dollar with probability $\frac{1}{3}$. Suppose also that the person's ...
1
vote
1answer
57 views

Markov DTMC and CTMC: How to build a $Q$ generating matrix using given $2$ state $q$ probabilities? Find Steady State Probabilities

I know that $$q_{(i,j)(i+1,j)}=\lambda_1$$ $$q_{(i,j)(i-1,j+1)}=\lambda_2$$ $$q_{(i,j)(i+1,j-1)}=0.5\lambda_3$$ $$q_{(i,j)(i,j-1)}=0.5\lambda_3$$ where $\lambda_1 , \lambda_2 , \lambda_3 $ are rates ...
4
votes
1answer
33 views

Getting all positive integer solution (All possible states of a chemical system) to undertermined linear system (Conservation law from stoichiometry)

Let a chemical system be defined as $${A<=>B<=>C}$$ Then the stoichiometry is given as $$S=\begin{bmatrix} -1& 1& 0& 0\\ 1& -1& -1 & 1\\ 0 & 0 & 1 &-...
0
votes
1answer
44 views

How many stationary distributions does a time homogeneous Markov chain have?

I've been given the following definition: For a THMC with one step transition matrix $\mathbf{P}$, the row vector $\mathbf{\pi}$ with elements $(\pi_{i})_{i \in S}$ (where $S$ is the state space) ...
0
votes
1answer
28 views

Distribution of Markov Chain with transition matrix

An optional challenge assignment: Given a stationary Markov chain $\mathbf X=(X_k)^\infty_{k=1}$ where $X_k$ takes values in {0,1,2}. Let it have a probability transition matrix $P=[P_{ij}]=Pr(X_{k+1}=...
1
vote
0answers
30 views

Probability that sample path ends at a high.

I am trying to get the probability that a sample path ends at a high. To formulate the problem, let sequence $\{S_n\}$ be a random walk, with $S_0 = 0$, defined by $$ S_n = \sum_{k=1}^n X_k$$ Where $...
1
vote
1answer
35 views

Invariant measure nomenclature

I'm looking through my notes and I've come across the following line: If $\sum_{i \in I}\pi(i) = \infty$ then we (usually) say that the Markov chain doesn't have an invariant distribution. My ...
0
votes
0answers
18 views

DTMC and CMTC , First Passage Probabilities and Expectation.

Let the sample space $S=(0,1,2)$. Let $Q=\begin{pmatrix} -4 & 2 & 2 \\ 3 & -5 & 2 \\ 0 & 3 & -3 \\ \end{pmatrix}$ be the generator of our CTMC. ...
0
votes
0answers
37 views

Markov chain matrix multiplication - Finding all path probability in a graph

I realize there are many Markov Chain questions on this site. I have reviewed all relevant questions, the closest to my question are: Finding the probability from a markov chain with transition matrix ...
0
votes
1answer
41 views

Markov Chains: Finding the Embedded DTMC $P(t)$ from generator matrix $Q$

Markov Chains: Finding the Embedded DTMC (transition probability matrix) $P(t)$ from generator matrix $Q$ where the sample space $S=(0,1,2)$ $Q=\begin{pmatrix} -4 & 2 & 2 \\ ...
0
votes
1answer
16 views

How many stationary measures does a random walk with absorbing barriers have?

Suppose I have a markov chain with finite state space $0,\ldots, N$. At each state $1, \ldots, N-1$, we have that the probability of going up and down one state is of probability $\frac{1}{2}$. Now, ...
1
vote
1answer
22 views

What is an example of a Markov Chain with two stationary measures?

I am trying to come up with a transition matrix for a Markov Chain with two stationary measures, but am not able to construct it. Would anyone have an example? Thanks.
0
votes
1answer
21 views

Stochastic Kernel almost surely determined by semidirect product?

Given a measurable space $(\Omega, \mathcal{F})$ with two probability measures $\mathbb{P}_1$, $\mathbb{P}_2$ and a second measurable space $(X,\mathcal{A})$ with two stochastic kernels $\mu_1, \mu_2$ ...
1
vote
0answers
18 views

marginalised markov chain

If there is a Markov chain for the joint variable $z=(x,y)$, the marginal process $x$ is not, in general, Markovian itself. However, if we consider the probability of a two time step process $$p(z_0,...
1
vote
0answers
42 views

What is the invariance principle of Random Walks?

Several papers I have read allude to the fact that a random walk is invariant; however, I have been unable to find any reference to support this fact. Could anyone explain why random walks are ...
1
vote
0answers
52 views

Finite irreducible Markov chain

The question I have is stated as follows: Show that for any finite-state irreducible Markov chain $$\max_{i,j}\mathbb E_iT_j\le C$$where the constant $C$ only depends on the number of states and $\...
0
votes
1answer
34 views

Create a transition table

I am trying to create a transition table for a markov chain but I have difficulties. Consider a game, where each player (of two, lets call them A and B) has a fixed given probability of scoring 3 ...
0
votes
1answer
48 views

Sampling uniform equilibrium distribution with Markov Chain Monte Carlo

I'm wanting to sample the discrete uniform distribution over $n = 10$ integers using MCMC. My question concerns the transition probability matrix, $P$. As I understand it, any symmetric, irreducible ...
1
vote
1answer
59 views

Please can someone help me to understand stationary distributions of Markov Chains?

I'm currently trying to understand (intuitively) what a stationary distribution of a Markov Chain is? In our lecture notes, we're given the following definition: This was of little benefit to my ...
2
votes
1answer
61 views

Find when a given Markov chain is transient

Let $T$ be a tree with countably many nodes so that each node has $n$ neighbors. Let a Markov chain be defined by starting at some random vertex of $T$ and then move by traveling to any of the $n$ ...
1
vote
1answer
24 views

What does this notation, used in Markov Chains, mean?

In my module on Markov processes, the following notation is used: $$ p_{ij}^{(m,n)} = P(X_n = j \mid X_m = i) \quad \text{where } \: m<n \\ p_j^{(n)} = P(X_n = j) \\ p_{ij}^{(k)} = \: ??? $$ Does ...
0
votes
1answer
33 views

Queueing theory M/M/k - probability of number of busy servers seen by next arrival process

Consider a $n$ server parallel queueing system, need to calculate the probability of $1$ busy server as seen by next arrival process. $\lambda$$=$$arrival$ $rate$ $of$ $processes$ ; $\mu$$=$$service$ ...
0
votes
0answers
21 views

Limit transition probability

I would like to prove the following: Let $p$ be the increment distribution of a discrete time random walk in $\mathbb{Z}^2$ which we assume to be irreducible, symmetric and of finite range, so $$S_n=...
0
votes
0answers
30 views

An Enquiry Concerning the “reward function” for a Markov Chain

The Statement of the Problem: Consider a Markov chain with state space $$ S = \{1, 2, 3 \} $$ and probability transition matrix $$ P = \left( \begin{matrix} .3 & .7 & 0 \\ ...
7
votes
1answer
103 views

If $A$ is the generator of $(P_t)$, then $A+f$ is the generator of $(P_t^f)$

Let $X=(X_t)_{t\geq0}$ be a Markov process on a state space $\Gamma$ (a Hausdorff topological vector space), let $A$ be the infinitesimal generator of $X$ and let $\mathcal C(\Gamma)$ the space of ...
0
votes
1answer
36 views

Strong Markov Property Clarification

I see that there have been many questions on the strong Markov property, including Strong Markov property - Durrett and Two definitions of the strong Markov property. I am still slightly confused ...
1
vote
0answers
80 views

Deduce the definition of a harmonic function in the context of a Markov Chain

We know from PDE that a harmonic function $f$ satisfies the mean value property, namely, $f(x)$ = $\frac{1}{\vert{B_r(x)}\vert}\int_{B_r(x)}f(y)dy$ where $B_r(x)$ is the ball about $x$ with radius $r$....
0
votes
1answer
27 views

If I have a two-state Markov Chain, and I start a chain at state 1, and another at state 2, what is the expected time before they hit?

I have a two-state Markov Chain that looks like: $$ P= \left(\begin{matrix} 0.4 &0.6 \\ 0.7 & 0.3 \end{matrix} \right). $$ From this, suppose I define $X_t$ and $Y_t$, where $X_t$ starts ...
0
votes
1answer
47 views

Why are the $n$-step transition probabilities well defined?

I was reading a proof for the Chapman-Kolmogorov equations and now I understand why it is the case that for a discrete-time homogeneous Markov Chain $X=(X_n) _{n\geq 0}$ (with state space $S$) the ...
0
votes
1answer
45 views

Non absorbing markov chain. Average state occurrence

If P is a $3\times 3$ transition matrix. Every state has a chance of going to every other state including itself. Therefore this is not an absorbing markov chain. What I want to be able to calculate ...
2
votes
1answer
93 views

Markov Chain. Time between customers arrival

The times between successive customer arrivals at a facility are independent and identically distributed random variables with the following PMF: $$p(k) = 0.2(k = 1)$$ $$p(k) = 0.3(k = 3)$$ $$p(k) = ...
0
votes
0answers
34 views

Characterisations of Markov Processes: SDE's , Generators,…

There are different characterisations of a Markov Process: Probability Semigroups, Generators, even in some cases by Jumps Chains and Holding Times... And I know that, in "real life", the only thing ...
1
vote
1answer
42 views

Random walk around $n$-dimensional objects

Suppose you are only allowed to move along the edges of a square. At each vertex, you have an equal probability of picking any of the available routes (including doubling back on yourself). Is the ...
3
votes
2answers
48 views

Determine periodicity from transition matrix?

I have a two part question. Let's say we have a transition matrix T: \begin{bmatrix} 0 & 0.2 & 0.8 & 0 & 0 \\ 0.7 & 0 & 0.3 & 0 & 0 \\ 0.6 & 0.4 &...
2
votes
1answer
104 views

Avg Value of Dependent Events

If I have 26 bins and on a given "turn" each bin can take on one of many values, or no value at all (null) with probability that varies by bin. Let's call the average of the values that can occur A-Z, ...
0
votes
1answer
31 views

DTMC: Stationary Distribution with Recurrent Classes

I want to calculate the stationary probability, $\pi_j$ for a DTMC that contains two irreducible classes such as, $$ P_{ij} = \begin{pmatrix} 0 & 0 & 0 & 1 & 0 & 0\\ 0 & 0 &...
0
votes
0answers
28 views

Does there exist a collection of random variables satisfying given conditions?

Suppose $X,-Y$ and $Z$ are i.i.d random variables. I am trying to investigate whether there exists such random variables satisfying the following conditions: \begin{align} &a)\ X \rightarrow (X-...
1
vote
1answer
37 views

A box contains 4 piece of papers, each paper marked with A,B,C, and D respectively.

A box contains 4 piece of papers, each paper marked with A,B,C, and D respectively. A person draws a paper and observes its letter and puts it pack. Papers are now drawn repeatedly without ...
0
votes
1answer
29 views

Up-to-date or Behind - [Markov Chain]

Alex is taking a bioinformatics class and in each week he can be either up-to-date or he may have fallen behind. If he is up-to-date in a given week, the probability that he will be up-to-date (or ...
3
votes
1answer
53 views

Need some help with proving the Erdos-Feller-Pollard theorem

I am working on an analytic proof of Erdos-Feller-Pollard theorem. The exercise basically tells me to prove some steps in order to prove the theorem. First, a few definitions: Let $\{X_n\}$ be a ...
0
votes
0answers
48 views

Basic questions on Markov Chain

I'm a beginner of Markov processes and I have some basic questions. Consider two sequences of real-valued random variables $\{X_t\}_t, \{Y_t\}_t$ where $t$ is a discrete time index, $t=0,1,...$, all ...
0
votes
1answer
54 views

Mickey mouse travels in a maze with nine $3 × 3$ cells. Markov Chain involved?

Mickey mouse travels in a maze with nine $3 × 3$ cells. The cells are numbered as $0, 1, ..., 8$ from left to right and top down. Each step Mickey travels from where it is to one of the surrounding ...