Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
7 views

The convergence time of a Markov Chain

Sorry, since I just encountered this problem in practice not from literature, I don't know the correct terminology of such a problem. The senario is: consider there is a Markov Chain of state 0, 1, 2, ...
0
votes
1answer
42 views

Pointless probability

POINTLESS is a BBC game show. Each night, four teams compete. If a team does not win, it comes back for a second night; but not a third night. Each night has 1, 2, 3 or 4 new teams. There are ...
2
votes
2answers
38 views

Proving a Trick to More Quickly Calculate N-Step Transition Probabilities

So, I have been working on a homework problem all day that asks me to prove that: $P^n= \Pi +Q^n$ where P is the transition matrix of a finite-state regular Markov Chain, $\Pi$ is a matrix whose rows ...
1
vote
1answer
38 views

Independence of time intervals between visits of a state $x$ on a Markov chain

The question is like the following, Let $X_0,X_1,...,X_n,...$ be an irreducible Markov chain with finite state space. Define $τ_{x,0}^+=0$, and $τ_{x,k}^+=\min\{t:t>τ_{x,k-1}^+,X_t=x\}$. In ...
1
vote
0answers
9 views

Rigorous Derivation of Metropolis-Hastings Transition Density

The Metropolis-Hastings MCMC algorithm is as follows. Set $X_0$ to some initial value in the support of the target density $f$ and choose a proposal density $q(y \mid x)$; a density in $y$ for each ...
1
vote
1answer
33 views

Function of a markov chain $f(x)=x^3$

I have given a Markov Chain $X_n$ with the state space $\{0,1,2\}$ and the transition Matrix $$P= \begin{Bmatrix} 0.3 & 0.2 & 0.5 \\ 0.5 & 0 & 0.5 \\ 0.2 & 0.1 & 0.7 ...
0
votes
0answers
32 views

How to solve system of equilibrium probability state equations

I have started studying markov chains where i have these statistical equilibrium probability state equations.These equations are solved for a particular case $s_1=4,a_1=5,s_2=2, a_2=1$ and a 15*15 ...
6
votes
0answers
38 views

Can Continuous Time Markov Chains be used as a reasonable voting system?

I just compared a couple of example elections, as given on Wikipedia to show how Condorcet-methods differ from non-Condorcet ones, to what happens if you just interpret the underlying preference ...
5
votes
0answers
35 views

markov chain: 2 state chain

I have a machine. It has two states, broken or working. If it is working, then it will be broken with probability $q=0.1$. If the machine is working, I will make \$1000 dollar a day. If it is broken, ...
0
votes
2answers
31 views

How to find the limit of a matrix $P^n = UD^nU^{-1}$ where $D$ is a diagonal matrix of eigenvalues and $U$ a matrix of eigenvectors?

If we have a matrix where $P = UDU^{-1}$, where $D$ is a diagonal matrix of real eigenvalues that are less than or equal to 1, and $U$ is the corresponding matrix of eigenvectors, how can we show that ...
0
votes
0answers
32 views

How to solve the Probability Markov chain system of equations

I have this system of equations from a 2-D Markov chain (see the figure. How can i calculate the coefficient matrix, state probability vector and the constant vector from this system of equations. ...
0
votes
0answers
9 views

Unconditional COvariance in markov switching model

I'm trying to do a portfolio optimisation within a Markov switching framework for some risky asset returns. My utility function ideally is CRRA (power) utility. However maximising a linear sum of two ...
0
votes
1answer
23 views

Ehrenfest chain

In the Ehrenfest model, let $X_n$ denotes the number of balls in the left urn. And there are $N$ balls total. When we calculate $P(X_{n+1}=i+1|X_n=i, X_{n-1}=i_{n-1},...,X_0=i_0)$, why don't we take ...
1
vote
0answers
18 views

What are some modern books on Markov Chains with plenty of good exercises?

I would like to know what books people currently like in Markov Chains (with syllabus comprising discrete MC, stationary distributions, etc.), that contain many good exercises. Some such book on ...
1
vote
0answers
28 views

The “how many pieces do you have buy on average” problem, a markov problem?

I recently discovered a problem similar to this one in a book about Markov chains: Assume you can buy $n-$ different set of cards in a store, but you do not know which one you'll buy: What is the ...
0
votes
0answers
34 views

Are there different definitions of a continuous time Markov chain, condition on a finite or infinite number of earlier values?

My book defines a continuous time Markov chain like this. Let $\{X_t\}, t \in T$ be a stochastic process on $(\Omega, \mathcal{A}, P)$, with a countable state space $S$. The process is a Markov ...
1
vote
1answer
18 views

can two Markov kernels be close in total variation and differ in their ergodicity properties?

This is a question inspired by a recent MCMC paper and linked to an earlier question of Sam Livingstone that did not get any answer. Given two Markov kernels $\mathfrak{K}$ and $\mathfrak{H}$ such ...
3
votes
0answers
35 views

Markov Chain: Steady State Distribution.

A total of $M$ balls are divided between two urns A and B. A ball is chosen uniformly at random. If it is chosen from urn A then it is placed in urn B with probability $b$ and otherwise it is returned ...
0
votes
1answer
23 views

If $P$ is a transition matrix, and $m_{ij}$ the mean return time, how to show $m_{ij} = 1+ \sum_{k \neq j}P_{ik}m_{kj}$?

If $P$ is a transition probability matrix of a finite state regular Markov Chain, and $m_{ij}$ is the mean return time, how can I show that $m_{ij} = 1+ \sum_{k \neq j}P_{ik}m_{kj}$? It seems rather ...
0
votes
0answers
24 views

How to determine the limiting distribution of a Markov Chain which can only increment up or down a state at every stage?

I have a random walk Markov chain that has states from $0$ to $N$. The conditions are that when the chain is at $0$, the chain will go to state $1$ with probability $1$. When the chain is at state ...
1
vote
2answers
38 views

Probability of a fair sequence of tosses ending on two successive tails given the first toss was a head?

Suppose a coin is tossed repeatedly until either two successive heads appear or two successive tails appear. Then, assume that the first coin toss results in a head. I would like to find the ...
4
votes
1answer
22 views

Ehrenfest Chain: stationary distribution

In the Ehrenfest Chain model: There are M balls which are divided between urn A and urn B. At each stage, if a ball is chosen, then it would be moved into a different urn. Let $X_n$ be the # of ...
2
votes
0answers
89 views
+200

The sums over RVs between two return times are independent for a Markov chain

Let $X_0,X_1,...,X_n,...$ be an irreducible Markov chain with finite state space. Define $τ_{x,0}^+=0$, and $τ_{x,k}^+=\min\{t:t>τ_{x,k-1}^+,X_t=x\}$. In plain words, $τ_{x,k}^+$ is the time of the ...
0
votes
1answer
21 views

What's the period of this matrix?

Consider the matrix $$ A = \begin{pmatrix} 0.1 & 0.3 & 0.4 & 0.2 \\ 0.2 & 0.4 & 0.0 & 0.4 \\ 0.0 & 0.3 & 0.5 & 0.2 \\ 0.5 & 0.3 & 0.2 & 0.0 ...
0
votes
0answers
28 views

Explicit formula for return probability of simple random walk

Is there an explicit formula for the probability that a simple symmetric random walk on $\mathbb{Z}$ starting from $1$ will not hit $0$ before time $t$?
2
votes
0answers
58 views

Proof of the existence of a reversible stationary distribution

$p$ is a finite Markov chain where $p(i,j)>0$ for all $i,j$. Prove a reversible stationary distribution exists for $p$ if $p(i,j)p(j,k)p(k,i)=p(i,k)p(k,j)p(j,i)$ for all $i,j,k$ This question is ...
0
votes
0answers
30 views

Calculate the discrete density of the variables of a Markov chain

$X$ and $Y$ are independent random variables of Bernoulli with parameter $\frac{2}{3}$. $Z=X+Y$ $\{X_n\}_{n \in \mathbb{N}}$ with values in {0,1,2} having $Z$ such as initial law and the transition ...
0
votes
2answers
45 views

Markov chains steady-state distribution

Ok so we are given a Markov chain $X_n$, $P=P(ij)$ as the transition matrix and the $(\pi_1,\pi_2,\pi_3,...,\pi_n)$ as steady-state distribution of the chain. We are asked to prove that for every $i$: ...
1
vote
2answers
57 views

Defining the states when we roll one single die repeatedly

We roll a single die and the game stops as soon as the sum of two successive rolls is either 5 or 7. We want to find the probability that the game stops at a sum of 5. It seems like Markov ...
0
votes
0answers
7 views

Recurrent Markov chain with $p_{i,i+2} = p$ , $p_{i,i} = r$ , $p_{i,i−1} = 1−p−r$

Let $Xn$ a Markov chain on $\mathbb{Z}$ with the following transition matrix: $p_{i,i+2} = p$ , $p_{i,i} = r$ , $p_{i,i−1} = 1−p−r$ Find p and q such that the cain is recurrent. I'm tring to ...
0
votes
0answers
10 views

How to calculate mean of busy-cylce

Let $\{X(t)\}$ be birth-death process on a finite state {0,1,2} with non negative birth rates $(\lambda_0,\lambda_1)$and death rates $(\mu_1,\mu_2)$. Suppose $\mathbb{P}(X(0)=0)=1$ and $s_0=\inf ...
-1
votes
0answers
11 views

How to find expectation of birth-death process [closed]

Let $\{X(t)\}$ be birth-death process on two-state space {0,1}. Let birth rate $\lambda=2$ and death rate $\mu=12$. How to calculate $\lim_{t\to\infty} \mathbb{E}[X(t)]$?
0
votes
1answer
13 views

Biased voter model survival

I have a biased voter on $\mathbb{Z}^d,$ where $d>0$ (I am mostly interested in the cases where $d>1$) with the bias parameter $\lambda$. In other words, let us have a process $X=(X_t)_{t \ge ...
0
votes
0answers
11 views

Inequality problem for Markov Process

Is there any upper bound available for the following quantity $$E[\max_{1 \leq k \leq n} X_k]$$ where $\{X_n\}$ is a Markov chain.
2
votes
0answers
77 views

Norris exercise: Showing $P_0[\text{no return to}\ 0]=6/\pi^2$

Consider exercise 1.3.4 of Norris' Markov Chains. The question is as follows: Let $\{X_n\}_{n\geq 0}$ be a Markov Chain with state space $S=\{0,1,2,\dots\}$. Suppose the transition probabilities ...
1
vote
1answer
22 views

An example of a reversible but reducible Markov chain

The reversibility of a Markov chain is defined in the following way with some basic propositions. Unfortunately all examples of reversible Markov chains shown in my textbook so far are irreducible, ...
2
votes
1answer
28 views

An example of a reducible random walk on groups?

Random walk on group is defined in the following way as a Markov chain. A theorem says the uniform distribution is stationary for all random walk on groups. If the random walk is ...
4
votes
0answers
30 views

Find a Markov chain transition kernel

Let $f_{X}$ be a density we would like to sample from. For some reasons, $f_{X}$ may be analytically intractable or expensive to evaluate. A solution consists in considering a density $(x,y) \in X ...
0
votes
0answers
32 views

Identity for return times in continuous Markov chain

I need help with this problem about return times in continuous time Markov chains: We are given a continuous time Markov chain $(X_t)$. Define $T_i=\inf{\{t>0; X_t=i, X_{t-} \neq i\}}$, which ...
0
votes
0answers
28 views

First return times and continuous markov chains.

We are given a generator matrix $Q$ (Q-matrix) for a continuous time Markov chain $(X_t)_t. We want to calculate the probabilities of: returning to State 3 before State 1, while starting at State 3: ...
0
votes
1answer
20 views

Does such a Markov chain exist?

Suppose it has finite state space $S$, and $\lim\limits_{n\to \infty}p_{ij}^{(n)}=0$ for all $i,j\in S$. But guess is there isn't, since for a finite transition matrix, it is unlikely to have ...
0
votes
0answers
23 views

stationary distribution of outputs in Markov chain

consider a hidden Markov model with two states, with following transition/observation matrices: $T = \left( \begin{array}{cc} 0.9 & 0.1 \\ 0.1 & 0.9 \end{array} \right), O = \left( ...
1
vote
1answer
41 views

Discrete Time Markov Chain question

Let $\{X_n : n \ge 0 \}$ be a Markov chain with state space $ \{0, 1, 2, 3\} $ and transition matrix $$P=\begin{pmatrix} \frac{1}{4} & 0 & \frac{1}{2} & \frac{1}{4}\\ 0 & \frac{1}{5} ...
0
votes
1answer
21 views

Detailed balance implies time reversibility, how about the converse?

Given a Markov chain (finite state space) $X_1,X_2,...$ with transition matrix $P$ and initial distribution $\pi$, if they satisfy $\pi(x)P(x,y)=\pi(y)P(y,x)$, we say they satisfy detailed balance. ...
-1
votes
0answers
59 views

Markov Chain Probability Limit [closed]

Show that the Markov chain on the state space $S=\{0,1,...,n \} $ with transition matrix: $$ P(k,l) = \binom{n}{l} \left(\frac{k}{n}\right)^{l} \left(\frac{n-k}{n}\right)^{n-l} $$ is such that, ...
3
votes
1answer
42 views

Rigorous argument of the Markov property used in discrete-time Markov chains

I am reading an example related to discrete-time Markov chains which I do not really understand rigorously. Suppose that $\{ X_n \}_{n \in \mathbb{N} }$ is a time-homogeneous discrete-time Markov ...
1
vote
2answers
34 views

Random Walk Definition

I have just begun studying this script about Random Walks, but I'm having trouble with a definition that is given there right at the beginning (page 10). We're looking at Random Walks on the square ...
0
votes
2answers
51 views

Transition Probability Matrix of Tossing Three coins

Three fair coins are tossed, and we let $X1$ denote the number of heads that appear. Those coins that were heads on the first trial (there were $X1$ of them) we pick up and toss again, and now we let ...
1
vote
0answers
28 views

An example of Markov chain with no closed class?

What is an example of Markov chain with no closed communicating class? Closed class means that once we are in that class, there would be no escape from it. I am thinking that an example would be ...
0
votes
1answer
13 views

How to solve for steady state matrix symbolically?

I'm trying to understand this solution to a question related finding the steady state matrix $s$ for a regular markov chain. Specifically I'm having trouble understanding how my textbook got $$ ...