Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

2
votes
1answer
100 views

Avg Value of Dependent Events

If I have 26 bins and on a given "turn" each bin can take on one of many values, or no value at all (null) with probability that varies by bin. Let's call the average of the values that can occur A-Z, ...
0
votes
1answer
31 views

DTMC: Stationary Distribution with Recurrent Classes

I want to calculate the stationary probability, $\pi_j$ for a DTMC that contains two irreducible classes such as, $$ P_{ij} = \begin{pmatrix} 0 & 0 & 0 & 1 & 0 & 0\\ 0 & 0 ...
0
votes
0answers
28 views

Does there exist a collection of random variables satisfying given conditions?

Suppose $X,-Y$ and $Z$ are i.i.d random variables. I am trying to investigate whether there exists such random variables satisfying the following conditions: \begin{align} &a)\ X \rightarrow ...
1
vote
1answer
37 views

A box contains 4 piece of papers, each paper marked with A,B,C, and D respectively.

A box contains 4 piece of papers, each paper marked with A,B,C, and D respectively. A person draws a paper and observes its letter and puts it pack. Papers are now drawn repeatedly without ...
0
votes
1answer
29 views

Up-to-date or Behind - [Markov Chain]

Alex is taking a bioinformatics class and in each week he can be either up-to-date or he may have fallen behind. If he is up-to-date in a given week, the probability that he will be up-to-date (or ...
3
votes
1answer
53 views

Need some help with proving the Erdos-Feller-Pollard theorem

I am working on an analytic proof of Erdos-Feller-Pollard theorem. The exercise basically tells me to prove some steps in order to prove the theorem. First, a few definitions: Let $\{X_n\}$ be a ...
0
votes
0answers
48 views

Basic questions on Markov Chain

I'm a beginner of Markov processes and I have some basic questions. Consider two sequences of real-valued random variables $\{X_t\}_t, \{Y_t\}_t$ where $t$ is a discrete time index, $t=0,1,...$, all ...
0
votes
1answer
47 views

Mickey mouse travels in a maze with nine $3 × 3$ cells. Markov Chain involved?

Mickey mouse travels in a maze with nine $3 × 3$ cells. The cells are numbered as $0, 1, ..., 8$ from left to right and top down. Each step Mickey travels from where it is to one of the surrounding ...
0
votes
0answers
32 views

$ (X_n : n = 0, 1, 2, …) $ is a Markov chain with state space $(1,2,…,10)$.

$ (X_n : n = 0, 1, 2, ...) $ is a Markov chain with state space $(1,2,...,10)$. Then which of the following is the correct answer? ($X_n+X_{n-1} : n = 1, 2, ...$) is a Markov chain. ($X_n-X_{n-1} : ...
0
votes
1answer
26 views

Random Early Discard Markov Chain

I'm trying to sketch the Markov chain for a Random Early Discard queueing policy where customers arrive to the queue of infinite size according to a Poisson process with rate $\lambda$. Customers that ...
0
votes
1answer
53 views

Starting from state 0, compute the mean number of visits of state 1 from one-step transition probability matrix.

A Markov chain has one-step transition probability matrix $$ \mathrm P= \begin{pmatrix} 1/4 & 1/4 & 1/4 & 1/4 \\ 1/4 & 1/4 & 1/2 & 0 \\ 0 & ...
0
votes
1answer
27 views

Balance Equations for M/M/1/m Queue

I think I found the solution for this problem, but they don't show all the steps. I was wondering if someone could explain to me how they get from the three balance equations to the solution.
3
votes
0answers
42 views

Randomized Chess [duplicate]

In chess, a rook can move either horizontally within its row (left or right) or vertically within its column (up or down) any number of squares. In an $8\times 8$ chess board, imagine a rook that ...
1
vote
0answers
28 views

Irreducible and positive recurrent CTMC — first passage times are finite?

Consider a continuous-time Markov chain (CTMC) $X$ on a countably infinite state space $S$. The CTMC is irreducible and all the states are positive recurrent. Let $T(i,j)$ be the first passage time to ...
0
votes
0answers
24 views

Powers of transition matrix are eventually positive

Given an irreducible, finite state, aperiodic markov chain, one can get $k$ such that $p_{ij}^{(k)}>0$ for all $i,j$. Now, I have proved this using the fact that if a set $S\subseteq \mathbb{N}$ ...
1
vote
1answer
71 views

Expected absorption time in modified Markov Chain

Consider a Markov Chain $\{X_n\}$ on $S=\{0,1,\dots,d\}$ where $0,d$ are absorbing states and all other are transient. Also consider any transient state leads to every state. Let the transition ...
2
votes
0answers
33 views

Irreducible and positive recurrent CTMC: $\sum_{i \in S} \pi(i) c(i) < \infty$?

Suppose we have a continuous-time Markov chain $X$ on the countably infinite state space $S$. The Markov chain is irreducible and all states are positive recurrent. The transition rates are given by ...
0
votes
0answers
21 views

Distribution of right censored observation in an absorbing Markov Chain

Consider a $3$ state Continuous time Process $\{X_t\}_{t \geq0}$ with state space $\mathcal{S} = \{0,1,2\}$ where state $0$ denotes the absorbing state. Let the generator of this process be: $$Q = ...
0
votes
2answers
61 views

Starting from state 0, the mean number of visits of state 2 before coming back to state 0 is?

The transition probability matrix is$$ \mathrm P= \begin{pmatrix} 0 & 2/3 & 1/3 \\ 1/2 & 0 & 1/2 \\ 1/2 & 1/2 & 0 \\ \end{pmatrix} $$ ...
-1
votes
2answers
53 views

$ (X_n : n = 1, 2, …) $ is a Markov chain with state space $(-1, 0, 1)$. [duplicate]

$ (X_n : n = 1, 2, ...) $ is a Markov chain with state space $(-1, 0, 1)$. Then which of the following is the correct answer? $(sin(X_n) : n = 1, 2, ...$) is a Markov chain. $(cos(X_n) : n = 1, 2, ...
0
votes
1answer
64 views

2 Questions about Markov chain

The first question: $ (X_n : n = 1, 2, ...) $ is a Markov chain with state space $(-1, 0, 1)$. $(sin(X_n) : n = 1, 2, ...$) is a Markov chain. $(cos(X_n) : n = 1, 2, ...$) is a Markov chain. $(|X_n| ...
0
votes
0answers
39 views

Geometrically bounded transition probability in Markov Chains

Consider a Markov Chain with finitely many transition states. Show there is $M>0$ and $\alpha<1$ such that $p_{ij}^{(n)}\leq M\alpha^n$ for all $n$, whenever $i,j$ are transient states. I was ...
0
votes
1answer
35 views

How long on average does it take for the ladybugs meet at the same vertex? - Discrete Markov Chains

We consider a regular pentagon which the vertices are numbered from $1$ to $5$ in the direction of clockwise. Initially (i.e. at time $0$), two ladybugs are placed at the vertices $1$ and $3$. ...
0
votes
0answers
50 views

Absorbing markov chain and step probability

I have been using an absorbing markov chain to calculate the expected number of steps to transition to my absorbing state. I have calculated this correctly by following the wiki guide ...
1
vote
2answers
40 views

Stochastic Processes - Determining probability and expectation on tennis player question

Two tennis players, $A$ and $B$, are rendered equal in a game. It takes two point lead for the winner. A player who has one point is said to benefit. Assuming that $A$ has a probability $p$ of ...
0
votes
1answer
25 views

Just a little confusion with recurrence in Markov Chains

Is it possible that in a Markov Chain one can go to a null recurrent state from a positive recurrent state? Note I assume the state space to be infinite otherwise the question makes no sense. If so ...
0
votes
1answer
57 views

Equality for transient Markov chains

Let $S_n$ be a transient, irreducible random walk starting from $0$. Then I want to prove that $$\sum_{n\geq0}{p_n(0,x)}=P_0[S_n=x\text{ for some }n\geq0]\sum_{n\geq0}{p_n(0,0)}$$ where $p_n$ is the ...
1
vote
1answer
35 views

Finding the expected time for a stock to go from $\$25$ to $\$18$ given there is a support level at $20 with upward and downward biases.

This problem is adapted from Stochastic Calculus and Financial Applicationsby J. Michael Steele, Springer, New York, 2001, Chapter 1, Section 1.6, page 9. Consider a naive model ...
0
votes
0answers
13 views

The convergence time of a Markov Chain

Sorry, since I just encountered this problem in practice not from literature, I don't know the correct terminology of such a problem. The senario is: consider there is a Markov Chain of state 0, 1, 2, ...
0
votes
2answers
73 views

Pointless probability

POINTLESS is a BBC game show. Each night, four teams compete. If a team does not win, it comes back for a second night; but not a third night. Each night has 1, 2, 3 or 4 new teams. There are ...
2
votes
2answers
85 views

Proving a Trick to More Quickly Calculate N-Step Transition Probabilities

So, I have been working on a homework problem all day that asks me to prove that: $P^n= \Pi +Q^n$ where P is the transition matrix of a finite-state regular Markov Chain, $\Pi$ is a matrix whose rows ...
2
votes
1answer
51 views

Independence of time intervals between visits of a state $x$ on a Markov chain

The question is like the following, Let $X_0,X_1,...,X_n,...$ be an irreducible Markov chain with finite state space. Define $τ_{x,0}^+=0$, and $τ_{x,k}^+=\min\{t:t>τ_{x,k-1}^+,X_t=x\}$. In ...
1
vote
0answers
13 views

Rigorous Derivation of Metropolis-Hastings Transition Density

The Metropolis-Hastings MCMC algorithm is as follows. Set $X_0$ to some initial value in the support of the target density $f$ and choose a proposal density $q(y \mid x)$; a density in $y$ for each ...
1
vote
1answer
39 views

Function of a markov chain $f(x)=x^3$

I have given a Markov Chain $X_n$ with the state space $\{0,1,2\}$ and the transition Matrix $$P= \begin{Bmatrix} 0.3 & 0.2 & 0.5 \\ 0.5 & 0 & 0.5 \\ 0.2 & 0.1 & 0.7 ...
7
votes
0answers
62 views

Can Continuous Time Markov Chains be used as a reasonable voting system?

I just compared a couple of example elections, as given on Wikipedia to show how Condorcet-methods differ from non-Condorcet ones, to what happens if you just interpret the underlying preference ...
5
votes
0answers
58 views

markov chain: 2 state chain

I have a machine. It has two states, broken or working. If it is working, then it will be broken with probability $q=0.1$. If the machine is working, I will make \$1000 dollar a day. If it is broken, ...
0
votes
2answers
36 views

How to find the limit of a matrix $P^n = UD^nU^{-1}$ where $D$ is a diagonal matrix of eigenvalues and $U$ a matrix of eigenvectors?

If we have a matrix where $P = UDU^{-1}$, where $D$ is a diagonal matrix of real eigenvalues that are less than or equal to 1, and $U$ is the corresponding matrix of eigenvectors, how can we show that ...
-1
votes
0answers
49 views

How to solve the Probability Markov chain system of equations

I have this system of equations from a 2-D Markov chain (see the figure. How can i calculate the coefficient matrix, state probability vector and the constant vector from this system of equations. ...
0
votes
0answers
10 views

Unconditional COvariance in markov switching model

I'm trying to do a portfolio optimisation within a Markov switching framework for some risky asset returns. My utility function ideally is CRRA (power) utility. However maximising a linear sum of two ...
0
votes
1answer
39 views

Ehrenfest chain

In the Ehrenfest model, let $X_n$ denotes the number of balls in the left urn. And there are $N$ balls total. When we calculate $P(X_{n+1}=i+1|X_n=i, X_{n-1}=i_{n-1},...,X_0=i_0)$, why don't we take ...
7
votes
1answer
115 views

What are some modern books on Markov Chains with plenty of good exercises?

I would like to know what books people currently like in Markov Chains (with syllabus comprising discrete MC, stationary distributions, etc.), that contain many good exercises. Some such book on ...
1
vote
0answers
31 views

The “how many pieces do you have buy on average” problem, a markov problem?

I recently discovered a problem similar to this one in a book about Markov chains: Assume you can buy $n-$ different set of cards in a store, but you do not know which one you'll buy: What is the ...
0
votes
0answers
35 views

Are there different definitions of a continuous time Markov chain, condition on a finite or infinite number of earlier values?

My book defines a continuous time Markov chain like this. Let $\{X_t\}, t \in T$ be a stochastic process on $(\Omega, \mathcal{A}, P)$, with a countable state space $S$. The process is a Markov ...
1
vote
1answer
21 views

can two Markov kernels be close in total variation and differ in their ergodicity properties?

This is a question inspired by a recent MCMC paper and linked to an earlier question of Sam Livingstone that did not get any answer. Given two Markov kernels $\mathfrak{K}$ and $\mathfrak{H}$ such ...
3
votes
0answers
46 views

Markov Chain: Steady State Distribution.

A total of $M$ balls are divided between two urns A and B. A ball is chosen uniformly at random. If it is chosen from urn A then it is placed in urn B with probability $b$ and otherwise it is returned ...
1
vote
1answer
30 views

If $P$ is a transition matrix, and $m_{ij}$ the mean return time, how to show $m_{ij} = 1+ \sum_{k \neq j}P_{ik}m_{kj}$?

If $P$ is a transition probability matrix of a finite state regular Markov Chain, and $m_{ij}$ is the mean return time, how can I show that $m_{ij} = 1+ \sum_{k \neq j}P_{ik}m_{kj}$? It seems rather ...
0
votes
0answers
27 views

How to determine the limiting distribution of a Markov Chain which can only increment up or down a state at every stage?

I have a random walk Markov chain that has states from $0$ to $N$. The conditions are that when the chain is at $0$, the chain will go to state $1$ with probability $1$. When the chain is at state ...
1
vote
2answers
53 views

Probability of a fair sequence of tosses ending on two successive tails given the first toss was a head?

Suppose a coin is tossed repeatedly until either two successive heads appear or two successive tails appear. Then, assume that the first coin toss results in a head. I would like to find the ...
4
votes
1answer
63 views

Ehrenfest Chain: stationary distribution

In the Ehrenfest Chain model: There are M balls which are divided between urn A and urn B. At each stage, if a ball is chosen, then it would be moved into a different urn. Let $X_n$ be the # of ...
2
votes
1answer
135 views

The sums over RVs between two return times are independent for a Markov chain

Let $X_0,X_1,...,X_n,...$ be an irreducible Markov chain with finite state space. Define $τ_{x,0}^+=0$, and $τ_{x,k}^+=\min\{t:t>τ_{x,k-1}^+,X_t=x\}$. In plain words, $τ_{x,k}^+$ is the time of the ...