Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
1answer
29 views

Irreducible and recurrent Markov chain - theorem notation question

In [J. R. Norris] Markov Chains (Cambridge Series in Statistical and Probabilistic Mathematics) (2009), page 35, Theorem 1.7.5 says: In (ii), does it mean $\gamma^k$ is notation for $\gamma^k_i$ ...
1
vote
1answer
104 views

Stochastic Markov Chain Application: Rat in the maze problem, a modification

I am really new to Stochastic processes, and this is one of the supplementary practice questions that I stumbled across whilst studying: Modify the situation as described in ...
0
votes
0answers
11 views

Alternated Ehrenfest Chain (Welfare Distribution)

Consider a simple wealth distribution model with two trading agents. Let N denote their total wealth (represented by balls of two colors, black and white). At each time the agents may trade, i.e. ...
1
vote
0answers
19 views

Two Markov chains $X_t$, $Y_t$ have the same transition matrix $P$, show $\Bbb P(\tau_c\le t_0) = \Bbb P(\tau_c\le 2t_0|\tau_c> t_0)$

Given two Markov chains $X_t$, $Y_t$ characterized by the same transition matrix $P$, let $\tau_c$ be the first time the two chains have the same state, i.e. $\tau_c = \min\{t:X_t=Y_t\}$. The ...
1
vote
1answer
25 views

Symmetric Simple Random Walk - Definition Clarification

I'm finding conflicting answers everywhere, including in my own notes. In the phrase "symmetric simple random walk", which part, "symmetric" or "simple" refers to having a probability of $0.5$ to go ...
1
vote
1answer
37 views

regular Discrete Time Markov Chains

I have a transition matrix $P$. I know that $P$ is regular if all $p^{(n)}_{ij}>0$ for some $n \geq 1$. Is there an algorithm that can help me to verify whether $P$ is regular without calculating ...
1
vote
0answers
19 views

Is this the same thing as some simple discrete probability distribution?

I want to count the number of trials until one success in a sequence where the success probability is increased with each failure. For each trial, the success probability is ...
2
votes
1answer
49 views

Given two biased coins, the probability of obtaining heads on the $i^\text{{th}}$ toss using the following strategy?

We are given two coins: A and B with probability of obtaining heads being: $\alpha$ and $\beta$ respectively. The following sampling rule is used for i=1,2,...: If the $i^{\text{th}}$ toss results in ...
1
vote
1answer
21 views

Why does this MCMC algorithm to estimate parameters of a linear equation not converge to the posterior distribution?

As a kind of proof of principle I'm trying to estimate the parameters of a linear equation (before moving on to ODEs) using Markov Chain Monte Carlo sampling. The post that I am following can be found ...
2
votes
1answer
88 views

Showing a queueing system is a Markov Chain

I generally understand how to do this but I'm having trouble with a formal proof. "Consider an $M/M/1/m+1$ queue with exponential arrivals rate $\lambda$, exponential service rate $\mu$, and finite ...
0
votes
1answer
116 views

Markov chain: molecules in urns

I am struggling to get started on this question. I think I am confused at what the transition matrix is suppose to represent. So I know the matrix is going to have this form: $$ \begin{vmatrix} ...
0
votes
1answer
94 views

Markov Chain Snakes and Ladders

I am really stuck on the following question: So first I need to work out the transition matrix. But I am not sure how? Lets say I am at square 0 and I want to square 1, is the probability of moving ...
0
votes
1answer
34 views

If $X$ to $Y$ to $Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$

If $X \to Y \to Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$. I said the statement was true, and from $I(X;Y)\ge I(X;Z)$ by definition, thus $H(X) - H(X\mid Y) \ge H(X)-H(X\mid ...
1
vote
0answers
41 views

Normalisation of an invariant measure

Is there an example of a Markov chain with invariant measure $\pi$ and $\sum_{i \in I}\pi(i) = \infty$ that can be normalised so that we can consider an invariant distribution instead? This is a ...
0
votes
0answers
25 views

Most visited vertex in a random walk with a place dependent drift

Consider the following Markov chain on $\mathbb{Z}$: $$ P(x,x+1)=1-P(x,x-1)=\frac{1}{2}+e^{-|x|}\cdot \mathbf{1}_{\{x\neq 0\}} $$ Do there exist constants $c,C>0$ such that $$ c\cdot P^t(z,z) ...
1
vote
1answer
263 views

Gambler's Ruin and Markov Chains

Suppose that on each play of a certain game, a person will either win one dollar with the probability of $\frac{2}{3}$ or lose one dollar with probability $\frac{1}{3}$. Suppose also that the person's ...
1
vote
1answer
40 views

Markov DTMC and CTMC: How to build a $Q$ generating matrix using given $2$ state $q$ probabilities? Find Steady State Probabilities

I know that $$q_{(i,j)(i+1,j)}=\lambda_1$$ $$q_{(i,j)(i-1,j+1)}=\lambda_2$$ $$q_{(i,j)(i+1,j-1)}=0.5\lambda_3$$ $$q_{(i,j)(i,j-1)}=0.5\lambda_3$$ where $\lambda_1 , \lambda_2 , \lambda_3 $ are rates ...
4
votes
1answer
32 views

Getting all positive integer solution (All possible states of a chemical system) to undertermined linear system (Conservation law from stoichiometry)

Let a chemical system be defined as $${A<=>B<=>C}$$ Then the stoichiometry is given as $$S=\begin{bmatrix} -1& 1& 0& 0\\ 1& -1& -1 & 1\\ 0 & 0 & 1 ...
0
votes
1answer
33 views

How many stationary distributions does a time homogeneous Markov chain have?

I've been given the following definition: For a THMC with one step transition matrix $\mathbf{P}$, the row vector $\mathbf{\pi}$ with elements $(\pi_{i})_{i \in S}$ (where $S$ is the state space) ...
0
votes
1answer
27 views

Distribution of Markov Chain with transition matrix

An optional challenge assignment: Given a stationary Markov chain $\mathbf X=(X_k)^\infty_{k=1}$ where $X_k$ takes values in {0,1,2}. Let it have a probability transition matrix ...
1
vote
0answers
28 views

Probability that sample path ends at a high.

I am trying to get the probability that a sample path ends at a high. To formulate the problem, let sequence $\{S_n\}$ be a random walk, with $S_0 = 0$, defined by $$ S_n = \sum_{k=1}^n X_k$$ Where ...
1
vote
1answer
34 views

Invariant measure nomenclature

I'm looking through my notes and I've come across the following line: If $\sum_{i \in I}\pi(i) = \infty$ then we (usually) say that the Markov chain doesn't have an invariant distribution. My ...
0
votes
0answers
16 views

DTMC and CMTC , First Passage Probabilities and Expectation.

Let the sample space $S=(0,1,2)$. Let $Q=\begin{pmatrix} -4 & 2 & 2 \\ 3 & -5 & 2 \\ 0 & 3 & -3 \\ \end{pmatrix}$ be the generator of our ...
0
votes
0answers
31 views

Markov chain matrix multiplication - Finding all path probability in a graph

I realize there are many Markov Chain questions on this site. I have reviewed all relevant questions, the closest to my question are: Finding the probability from a markov chain with transition matrix ...
0
votes
1answer
36 views

Markov Chains: Finding the Embedded DTMC $P(t)$ from generator matrix $Q$

Markov Chains: Finding the Embedded DTMC (transition probability matrix) $P(t)$ from generator matrix $Q$ where the sample space $S=(0,1,2)$ $Q=\begin{pmatrix} -4 & 2 & 2 \\ ...
0
votes
1answer
14 views

How many stationary measures does a random walk with absorbing barriers have?

Suppose I have a markov chain with finite state space $0,\ldots, N$. At each state $1, \ldots, N-1$, we have that the probability of going up and down one state is of probability $\frac{1}{2}$. Now, ...
1
vote
1answer
20 views

What is an example of a Markov Chain with two stationary measures?

I am trying to come up with a transition matrix for a Markov Chain with two stationary measures, but am not able to construct it. Would anyone have an example? Thanks.
0
votes
1answer
19 views

Stochastic Kernel almost surely determined by semidirect product?

Given a measurable space $(\Omega, \mathcal{F})$ with two probability measures $\mathbb{P}_1$, $\mathbb{P}_2$ and a second measurable space $(X,\mathcal{A})$ with two stochastic kernels $\mu_1, \mu_2$ ...
1
vote
0answers
14 views

marginalised markov chain

If there is a Markov chain for the joint variable $z=(x,y)$, the marginal process $x$ is not, in general, Markovian itself. However, if we consider the probability of a two time step process ...
1
vote
0answers
41 views

What is the invariance principle of Random Walks?

Several papers I have read allude to the fact that a random walk is invariant; however, I have been unable to find any reference to support this fact. Could anyone explain why random walks are ...
1
vote
0answers
43 views

Finite irreducible Markov chain

The question I have is stated as follows: Show that for any finite-state irreducible Markov chain $$\max_{i,j}\mathbb E_iT_j\le C$$where the constant $C$ only depends on the number of states and ...
0
votes
1answer
34 views

Create a transition table

I am trying to create a transition table for a markov chain but I have difficulties. Consider a game, where each player (of two, lets call them A and B) has a fixed given probability of scoring 3 ...
0
votes
1answer
32 views

Sampling uniform equilibrium distribution with Markov Chain Monte Carlo

I'm wanting to sample the discrete uniform distribution over $n = 10$ integers using MCMC. My question concerns the transition probability matrix, $P$. As I understand it, any symmetric, irreducible ...
1
vote
1answer
57 views

Please can someone help me to understand stationary distributions of Markov Chains?

I'm currently trying to understand (intuitively) what a stationary distribution of a Markov Chain is? In our lecture notes, we're given the following definition: This was of little benefit to my ...
2
votes
1answer
56 views

Find when a given Markov chain is transient

Let $T$ be a tree with countably many nodes so that each node has $n$ neighbors. Let a Markov chain be defined by starting at some random vertex of $T$ and then move by traveling to any of the $n$ ...
1
vote
1answer
24 views

What does this notation, used in Markov Chains, mean?

In my module on Markov processes, the following notation is used: $$ p_{ij}^{(m,n)} = P(X_n = j \mid X_m = i) \quad \text{where } \: m<n \\ p_j^{(n)} = P(X_n = j) \\ p_{ij}^{(k)} = \: ??? $$ Does ...
0
votes
1answer
31 views

Queueing theory M/M/k - probability of number of busy servers seen by next arrival process

Consider a $n$ server parallel queueing system, need to calculate the probability of $1$ busy server as seen by next arrival process. $\lambda$$=$$arrival$ $rate$ $of$ $processes$ ; $\mu$$=$$service$ ...
0
votes
0answers
21 views

Limit transition probability

I would like to prove the following: Let $p$ be the increment distribution of a discrete time random walk in $\mathbb{Z}^2$ which we assume to be irreducible, symmetric and of finite range, so ...
0
votes
0answers
30 views

An Enquiry Concerning the “reward function” for a Markov Chain

The Statement of the Problem: Consider a Markov chain with state space $$ S = \{1, 2, 3 \} $$ and probability transition matrix $$ P = \left( \begin{matrix} .3 & .7 & 0 \\ ...
7
votes
1answer
102 views

If $A$ is the generator of $(P_t)$, then $A+f$ is the generator of $(P_t^f)$

Let $X=(X_t)_{t\geq0}$ be a Markov process on a state space $\Gamma$ (a Hausdorff topological vector space), let $A$ be the infinitesimal generator of $X$ and let $\mathcal C(\Gamma)$ the space of ...
0
votes
1answer
31 views

Strong Markov Property Clarification

I see that there have been many questions on the strong Markov property, including Strong Markov property - Durrett and Two definitions of the strong Markov property. I am still slightly confused ...
1
vote
0answers
78 views

Deduce the definition of a harmonic function in the context of a Markov Chain

We know from PDE that a harmonic function $f$ satisfies the mean value property, namely, $f(x)$ = $\frac{1}{\vert{B_r(x)}\vert}\int_{B_r(x)}f(y)dy$ where $B_r(x)$ is the ball about $x$ with radius ...
0
votes
1answer
24 views

If I have a two-state Markov Chain, and I start a chain at state 1, and another at state 2, what is the expected time before they hit?

I have a two-state Markov Chain that looks like: $$ P= \left(\begin{matrix} 0.4 &0.6 \\ 0.7 & 0.3 \end{matrix} \right). $$ From this, suppose I define $X_t$ and $Y_t$, where $X_t$ starts ...
0
votes
1answer
43 views

Why are the $n$-step transition probabilities well defined?

I was reading a proof for the Chapman-Kolmogorov equations and now I understand why it is the case that for a discrete-time homogeneous Markov Chain $X=(X_n) _{n\geq 0}$ (with state space $S$) the ...
0
votes
1answer
41 views

Non absorbing markov chain. Average state occurrence

If P is a $3\times 3$ transition matrix. Every state has a chance of going to every other state including itself. Therefore this is not an absorbing markov chain. What I want to be able to calculate ...
2
votes
1answer
87 views

Markov Chain. Time between customers arrival

The times between successive customer arrivals at a facility are independent and identically distributed random variables with the following PMF: $$p(k) = 0.2(k = 1)$$ $$p(k) = 0.3(k = 3)$$ $$p(k) = ...
0
votes
0answers
31 views

Characterisations of Markov Processes: SDE's , Generators,…

There are different characterisations of a Markov Process: Probability Semigroups, Generators, even in some cases by Jumps Chains and Holding Times... And I know that, in "real life", the only thing ...
1
vote
1answer
39 views

Random walk around $n$-dimensional objects

Suppose you are only allowed to move along the edges of a square. At each vertex, you have an equal probability of picking any of the available routes (including doubling back on yourself). Is the ...
3
votes
2answers
40 views

Determine periodicity from transition matrix?

I have a two part question. Let's say we have a transition matrix T: \begin{bmatrix} 0 & 0.2 & 0.8 & 0 & 0 \\ 0.7 & 0 & 0.3 & 0 & 0 \\ 0.6 & 0.4 ...
2
votes
1answer
100 views

Avg Value of Dependent Events

If I have 26 bins and on a given "turn" each bin can take on one of many values, or no value at all (null) with probability that varies by bin. Let's call the average of the values that can occur A-Z, ...