Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
62 views

Finding invariant probability of discrete time Markov Chain

Suppose that $\alpha$ gives a rate for an irreducible cont. time Markov chain on a finite state space. Then suppose the invariant probability measure is $\pi$. Then let $p(x,y)=\alpha(x,y)/\alpha(x)$ ...
0
votes
0answers
24 views

Steady state state distributions.

I am looking for a less "proofy" explanation of how a finite, irreducible, aperiodic Markov chain has a unique steady state $\pi$. No need define terms or include proofs of Bezout's lemma or number ...
0
votes
0answers
35 views

HMM walk through for backward algorithm with given example

This pdf file is a resource that walk through a simple HMM algorithm of two states http://www.indiana.edu/~iulg/moss/hmmcalculations.pdf, I have question in step 4.1 of the algorithm Specifically ...
0
votes
0answers
29 views

Prove or disprove: A statement about generating functions of Markov chains

For a given Markov chain $(X,E,P)$ ($X=(X_n)_{n\in\mathbb{N}_0}$, $E$ is the state space, $P=(p_{x,y})_{x,y\in E}$ the transition matrix), prove or give a counterexample to the following ...
0
votes
2answers
24 views

Does the function of a random variable have the same transition matrix as the variable itself?

If I have a variable X, that follows a Markov Chain with a transition density $\rho(X)$ does a function of that variable f(X) have the same density or is there a one to one mapping to the density of ...
0
votes
0answers
41 views

Mean time for the renewal process

The system is as below. The energy arrival process is $Y_{k}$ with a constant rate of $\rho$. Node has files of size exponential(λ) to be transmitted with fixed rate of transmission $r$. Hence the ...
0
votes
0answers
28 views

What is the limiting distribution of this Markov Chain?

Take a Markov Chain with state space $\left\{ 0, 1, \dots, 20 \right\}$. Then we have the rule that given $X_n$: Compute $Z = X_n + 1$ or $Z = X_n - 1$ with probability $\frac{1}{2}$ each (if the ...
0
votes
1answer
131 views

Markov Chain Expected value

Let $(X_n)_{n\in\mathbb{N}}$ be a Markov chain with State space $E=\{1,2,3\}$ and transission matrix $$P=\begin{bmatrix} 0 & 1/3 & 2/3 \\ 1/4 & 3/4 & 0 \\ 2/5 & 0 & ...
0
votes
1answer
31 views

Markov Chain Expected Value notation.

I have question to answer regarding $X_n$ where $X_n$ is a Markov chain, $n$=$0$,$1$,$2$,... I am loking for What I don't understand is what this $3$ on $X_{n+1}$ is! Any ideas?
0
votes
0answers
9 views

Is there a general formula for determining this distribution in a Markov chain?

Let $C$ be an irreducible Markov chain with state set $S$, $\left| S\right| = n$, transition matrix $T$, starting at state $s_0 \in S$, and yielding the states $s_0, s_1, s_2, \ldots$ during a random ...
0
votes
1answer
58 views

Understanding steady state distribution

I need some help verifying that my understanding of steady state distribution is indeed correct. I have a transition diagram (model). With around 100 states and 6 variables. I have used a software ...
0
votes
0answers
36 views

Monte Carlo Markov Chain simulation

I am going to post the python code logic we used however I want someone to look at the number that are printing out. The Markov chain is uniformly distributed across all $50x50$ matrices with entries ...
0
votes
0answers
78 views

Random Walk, Markov Process

I'm stuck on a homework question and am wondering if anyone can offer some hints. Suppose we have some straight line graph G over the set $ V = \{1, 2, 3, ... , n\} $ of vertices, with an edge between ...
0
votes
0answers
16 views

Am I right that $P_0(t(0)=2k)=\frac{1}{k}\binom{2(k-1)}{k-1}\left(\frac{1}{2}\right)^{2k-1}$?

Consider a Markov chain $(X_n)_{n\in\mathbb{N}_0}$ with state space $E$ containing $0$ and $$ p_{00}^{(2n)}=\binom{2n}{n}\left(\frac{1}{2}\right)^{2n-1}~~~\text{ and ...
0
votes
0answers
43 views

Did I show correctly that $0$ is null recurrent or did I produce nonsense?

Suppose that you have a Markov chain with state space $E$ containing $0$. Assume that $$ p_{00}^{(2n)}=\binom{2n}{n}\left(\frac{1}{2}\right)^{2n-1}~~~\text{ and ...
0
votes
1answer
39 views

Recurrence Equation and Markov Chain: How to get the base case

I established the reccurence equation for a Markov Chain but are not able to finde the base cases. We are interested in whether the sum of $t$ throws of a fair die is divisible by $k$ for some $k ...
0
votes
1answer
45 views

Likelihood of a function of different types of random variables

Is there a general way of expressing the likelihood of some known, but non-trivial function of several random varaibles. For example, suppose that we need to calculate the parameters of a process ...
0
votes
0answers
60 views

Is $0$ transient, positive recurrent or null recurrent?

Suppose that you have a Markov chain with state space $E$ containing $0$. Assume that $$ p_{00}^{(2n)}=\binom{2n}{n}\left(\frac{1}{2}\right)^{2n-1}~~~\text{ and ...
0
votes
0answers
257 views

Random walk on infinite binary tree (recurrence, transience)

Consider a random walk on the infinite binary tree with root $x$ which has the following transition probabilities. $$ p_{x,0}=p_{x,1}=\frac{1}{2},~~~p_{y,y0}=p,~~~p_{y,y1}=q,~~\text{and ...
0
votes
0answers
63 views

Show that every finite closed class is positive recurrent

Let $C$ be a finite closed class. Prove or disprove that $C$ is positive recurrent. Note 1: In our lecture we proved that every finite closed class is recurrent. Note 2: (Positive) recurrence is ...
0
votes
1answer
56 views

Computer failure with Markov chains and n-step transition matrix

Hi I am struggling with a Markov Chain question: A computer network has two servers, only one of which is in operation at any given time. A server may break down on any given day with probability p. ...
0
votes
0answers
18 views

Decomposition of a communicating class with countably infinite state space

Do you know an easy example of a Markov chain with countably infinite state space $E$, that has a communicating class $C$, that can be decomposed into a disjoint union of sets ...
0
votes
0answers
43 views

Transition probabilities do not sum to $1$

I have a set of weekly probabilities, and in order to convert to monthly probabilities, I have firstly convert the weekly probabilities into rates, $$r = -\frac{\ln (1-p)}{ t} ,\quad t=1\text{ ...
0
votes
0answers
50 views

Can I reconstruct Penney's game win probabilities from dominant strategy odds?

The probabilities of each strategy (row in the table below) in Penney's game (assuming the basic version played with a penny — no relation — and strategies consisting of a pattern the outcome of three ...
0
votes
1answer
92 views

Non-stationary Markov Chain Explanation

I am interested in creating a model in R, where I can implement a non-stationary Markov process. I would like to create a matrix of probabilities of going from one state to the next during a one year ...
0
votes
1answer
36 views

Finding a condition for which it is $P(\exists n\in\mathbb{N}: N_n=0)=0$

Let $X, X_{n,k}$ for $k,n\in\mathbb{N}$ denote independent random variables with values in $\mathbb{N}_0$. Define $N_0:=1$ and for $n\in\mathbb{N}$ set $$ N_n:=\begin{cases}0, & \text{ ...
0
votes
0answers
13 views

Identity how many classes in the transient prob matrix

I have searched for the existing questions in the section, but don't find any related result. So given the transient probability matrix of a certain Markov Chain, if I am required to identify how ...
0
votes
2answers
43 views

Does from $P^n$ irreducible for all $n\in\mathbb{N}$ follow that $P$ is aperiodic?

let $P$ be a transition matrix of a Markov chain with state space E, that is finite. Does from $P^n$ irreducible for all $n\in\mathbb{N}$ follow that $P$ is irreducible and aperiodic? ...
0
votes
0answers
24 views

Find all communicating classes and their properties

Indicate all the communicating classes together with their partial ordering for the stochastic matrices $$ P_1=\frac{1}{4}\begin{pmatrix}2 & 2 & 0 & 0 & 0 & 0 & ...
0
votes
0answers
32 views

Calculate the mixing time of a continuous time markov chain

I have Markov Rate Matrix Q for a continuous time Markov chain, that is irreducible. I would like to calculate the mixing time of the matrix - how can I do so? Note that the methods in Markov Chains ...
0
votes
1answer
31 views

Show: $i$ is aperiodic $\implies~\exists n_0: p_{ii}^{(n)}>0~\forall~n\geqslant n_0$

Let $(X_n)$ be a Markov chain with state space E. Show: If a state $i$ is aperiodic then there exists a $n_0\in\mathbb{N}$ so that $p_{ii}^{(n)}>0~\forall~n\geqslant n_0$. I know ...
0
votes
1answer
35 views

Is this statement about periodic communicating classes an equivalence statement?

In our reading about Markov chains we had the following theorem (including proof): Let $(X_n)_{n\in\mathbb{N}_0}$ denote a Markov chain with state space $E$. A periodic communicating class ...
0
votes
0answers
51 views

Finding the number of non empty urns after 9 steps

I'm trying to understand this example given in the book and am having trouble. The example states. Suppose that balls are successively distributed among 8 urns, with each ball being equally likely to ...
0
votes
0answers
50 views

Markov chain, cut point, hitting times

This question refers to this one: Hitting time $h_i(k)\geqslant h_i(j)\cdot h_j(k)$. The preliminaries are again these: Let $(X_n)_{n\in\mathbb{N}_0}$ be a Markov chain with state space $E$. ...
0
votes
0answers
20 views

Show that the closed classes are the maximal elements of the partial order

In the lecture we defined a partial order $\leq$ on the communicating classes associated to a Markov chain. Now it is to show that the maximal elements of the partial order $\leq$ are the ...
0
votes
1answer
24 views

Transition matrix to graph

Is there a program which can given a transition matrix $P$ draw a graph from a it? The transition matrix is also known as stochastic matrix and probability matrix see ...
0
votes
0answers
38 views

statements of matrix analysis

Let $y$ be fixed value. Let $A=a(x,y)$ be a matrix and $f_{t}(x)=\frac{\sum_{n=0}^{\infty}{a^{(n)}(x,y)(\frac{1}{t})^n}}{\sum_{n=0}^{\infty}a^{(n)}(y,y)(\frac{1}{t})^n}$ Show that ...
0
votes
1answer
47 views

How do we compute the mean time spent in transient states of a Markov Chain?

Let $X=\{X_n\}$ be a finite state Markov Chain with the state space $S = \{0,1,2,...,N\}$ such that all the states are transient. The following is the transition matrix. $$ P = \left[\begin{matrix} ...
0
votes
0answers
13 views

Simulating a continous time, inhomogenous Markov chain

What algorithms are used to simulate a time-continous, inhomogenous Markov chain? For the homogenous case, I've found (among others) this reference, which contains a few exact and approximative ...
0
votes
0answers
34 views

Problem involving periodic Markov Chains — probability of being in a given state at time $n$

I'm working on the following problem: I believe that the simplest possible irreducible periodic Markov Chain would be one with two states and no self-loops? Does this seem correct? However, I'm ...
0
votes
0answers
43 views

Why is this class recurrent?

In our reading we had the following example for a Markov chain. I cite from the reading: Here we have three communicating classes: $\left\{0\right\}, \left\{1,2,3\right\}$ and ...
0
votes
0answers
47 views

Stationary VS. limiting probability

I'm just wondering what the difference between stationary probability and limiting probability is. And, if any of you know: What does it mean that some elements exist and some elements doesn't, when ...
0
votes
1answer
157 views

Covariance of states of a finite Markov chain

I know it is possible to construct a covariance matrix for states of a Markov chain but I cannot seem to find a proper way to compute it. I will attach some theories I found from Kemeny and Snell's ...
0
votes
0answers
20 views

A question about Markov chain

Let $(\Omega,\mathscr{F},P)$ be a probability space. Assume $\{X_t\}_{t\geq 0}$ is a Markov chain with finite state space $S$. Assume $u:S\rightarrow\mathbb{R}$. Is it true that the limit ...
0
votes
0answers
37 views

Dealing with conditional OR statements (simple Markov Chain question)

I have a Markov chain with three states, X, Y and Z, and the following transition matrix; $$ P =\pmatrix{0.5 & 0.5 & 0 \\ 0.4 & 0.4 & 0.2 \\ 0 & 0.5 & 0.5 \\}$$ Now, what I ...
0
votes
0answers
55 views

Strong Markov property of continous time Markov process

In the book "Applied probability and queues" which is available here http://books.google.de/books?id=BeYaTxesKy0C&pg=PA32&hl=de&source=gbs_toc_r&cad=3#v=onepage&q&f=false , ...
0
votes
0answers
97 views

Solving the matrix equation $D P = D P A D$ for stochastic matrices.

Here, I call any real matrix with positive entries with rows summing to one a stochastic matrix (it need not be square). $D,A,P$ are stochastic. $P$ of size $n \times n$ is given. $D$ of size $k ...
0
votes
0answers
30 views

Memoryless Property of Markov Chains

I'm trying to understand Markov Chains and have across the following in a book: $ \sum\limits_{y=0,1,....m−1}p(x,y)P(T_A<T_B|X_0=x,X_1=y) $ which then becomes the following, under the Markov ...
0
votes
1answer
110 views

Probability transition matrix for maximum of iid random variables

I have a homework problem that goes as follows: Let $\xi_i, \ i=0,1,2,\ldots$ be i.i.d. random variables of discrete type. The distribution of $\xi_0$ is given by: $$\mathbb{P}\{\xi_0=i\} = a_i, \ ...
0
votes
0answers
34 views

“Taking expectation” to yield conditional probability

This argument is taken from Resnicks Adventures in stochastic processes and let $T _{\infty } < \infty $ denote that an infinite number of transitions in a continuous time markov chain has occurd ...