Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
44 views

Umbrella Markov chain problem

A man has an umbrella, and he commutes from his house to work and back. If it is raining, and he has an umbrella, he takes his umbrella. If it is not raining, or he does not have an umbrella, he does ...
1
vote
2answers
52 views

Reversible probability for a Markov Chain.

Considering the price of a share $X_n$ that evolves every $n$ day that increases of one euro with probability $0<p<1$ or decreases of one euro of probability $q=1-p$. Assume that $X_0=10$ and ...
1
vote
1answer
28 views

$X_n$ Markov Chain, show Show that: $\mathbb{E}^{\mathbb{P}_x}[\tau_x] \geq \mathbb{P}_x(\tau_y < \tau_x) \mathbb{E}^{\mathbb{P}_y}[\tau_x]$

Let $X_n$ be a Markov Chain on a countable, irreducible state space. Assume that the state $x$ is recurrent and that $\pi(x,y)$ >0, where $\pi(\cdot,\cdot)$ is the one-step transition probability. ...
1
vote
1answer
27 views

Irreducible Markov chain with period $d>1$ (Extension of a previous question)

I would like to extend the question asked from the following past post, which proved that a finite irreducible Markov chain of period $d>1$ has exactly $d$ eigenvalues (counting multiplicity) which ...
0
votes
1answer
39 views

Doubt about the transition intensities

A company has $N$ different items which are used for transactions. A transaction needs $m$ items with probability $1/M$, $1 \leq m \leq M$, for certain $M \leq N$. Given that a transaction needs ...
-1
votes
2answers
31 views

Markov chain stationary distribution

Let $(X_n : n \in \mathbb{N}_0)$ be a Markov chain on finite state space $S = (1,2,3,4,5)$ with the transition probability matrix $P = (p_{ij})_{i,j \in S} $ satisfying $$\sum_{i \in S}p_{ij} = 1$$ ...
0
votes
1answer
8 views

Show that $\lambda_1 S_1$ is exponential of parameter 1.

In Norris his book about Markov Chains the following question pops up: Let $S_1, S_2\dots$ be independent exponential random variables with parameters $\lambda_1, \lambda_2 \dots$ respectively. ...
2
votes
0answers
65 views

Equalizing Geometric means of Graph Cycles

Consider a strongly connected directed graph $G$. I have been stuck on the following question: can you assign real numbers to each edge of $G$ so that the geometric mean of all cycles are equal? ...
2
votes
0answers
24 views

Calculating integral with antithetic variables

Use simulation with antithetic variables and find $$\int_{-\infty}^\infty \int_0^\infty \sin(x+y)e^{-x^2+4x-y} \, dx \, dy.$$ so, my question and doubt is how struggle with the infinite limit ? It ...
3
votes
1answer
187 views

Show $S_N = \sum\limits_{n=1}^{N} \text{sign}(Y-X_n)$ is Markov, $(X_n),Y $ iid Uniform(0,1)

Let $(X_n)$ and Y be i.i.d. Uniform$(0,1)$ random variables and let $$S_N = \sum\limits_{n=1}^{N} \text{sign}(Y-X_n)$$ Show that $S_n$ is a Markov Chain and find its transition probabilities. Any ...
0
votes
0answers
22 views

finite state markov chain stationary distribution existence

Here is a fact from Adventures in Stochastic Processes by Prof. Sidney I. Resnick: A finite state, irreducible, aperiodic Markov chain is always positive recurrent and the stationary distribution ...
1
vote
0answers
18 views

convergence of markov chain, find distribution of limit

let $T=\mathbb{N}$ and $(X_t)_{t \in T}$ be a markov chain with state space $E=\{a,b\}$. the one step transition matrices are given by \begin{equation*} P(t,t+1)= \begin{pmatrix} 0,2+0,8f(t) ...
0
votes
1answer
31 views

Galton Watson Process: Probability that the population will be died [closed]

Let be $\{Z_n\} $ a Galton Watson process. If the distribution of the quantity of the descendants $B(2,p),p>0.5$ and the distribution of $Z_0$ is Pois($\lambda$), how can I calculate the ...
1
vote
0answers
16 views

Alternate Markvov Chain Model.

In class we are working on DTMCs for past couple of week, and in our last lecture we did an example. Question was: In a city of nowhere it only rains if there are clouds for successive m-days. Any ...
0
votes
0answers
16 views

Communication classes for Markov Chain with multiple nodes connecting

I understand the basic meaning to communication classes however I am not 100% sure when the nodes match up slightly differently to the basic idea. State Space $S = {1,2,3,4,5}$ $\left( ...
0
votes
0answers
49 views

A stationary distribution of Markov chain

For a irreducible finite Markov chain, I know that the definition of a stationary distribution is as follows: $\pi P = \pi$, where $P$ is a transition matrix and $\pi$ is a stationary distribution ...
0
votes
1answer
17 views

Time Inhomogeneous but still irreducible

Consider a family of random variables $X_t,X_{t+1},...$ on a finite countable space $I$. The transition probability of moving from state $i$ at time $t$ to state $j$ is given by ...
0
votes
0answers
42 views

Gambler's ruin stopping time

I'm trying to show that the expected stopping time of the Gambler's Ruin game is $x(n-x)$, where the gambler starts with \$$x$ and the game stops at \$0 or \$$n$. The probabilities of gaining and ...
0
votes
0answers
16 views

How to find a recurrence relation to find somebody's probability to ruin someone with the following game?

Let be two players $A$ and $B$ coin tossing with a biased coin ($p$ probability for tail and $q-1$ for head) and a global purse of $S$ €. At each stage of the game, if A wins (for instance tail) ...
1
vote
0answers
36 views

Expected number of steps for a random walk- robot

A robot is located at the top-left corner of a m x n grid The robot is trying to reach the bottom-right corner of the grid, he can move randomly in any of the directions: up, down, left, right. ...
1
vote
0answers
15 views

Irreducible finite-state Markov Chain

Could anyone help me with this? Let $X_n$ be an irreducible Markov chain on the state space $S=\{1,\ldots,N\}$. Show that there exist $C<\infty$ and $\rho <1$ such that for any states $i,j$, ...
0
votes
0answers
16 views

Derivation of backward probabilities $\beta_i(s_i)$ of a Hidden Markov Model (message passing). Any help in completing it?

I am trying to formulate in a recursive manner the backwards probabilities $\beta$ of a Hidden Markov Model where $w_i$ are the observed symbols and $s_i$ are the latent states. Is the following ...
0
votes
0answers
28 views

Birth-and-Death process in CTMC

Consider a birth-and-death process with birth rate $\lambda=2$ and $\mu=2$, and the possible states are $(0,1,2)$. a)Find the stationary distribution b)Write the Forward Kolmogorov ...
1
vote
2answers
42 views

What is the expected number of distinct drink types a group of $k$ people will order if there are $n$ choices?

Assuming each of the $n$ choices is equally likely and everybody orders one. I tried answering the question - what's the probability that it's exactly equal to some $i \in [1,n]$ and I get this ...
1
vote
1answer
63 views

Properties of a random walk [closed]

First of all, I know nothing about Markov chains, and I'd like to prove the following without using the theory around them. Let $(M_{n})_{n\geq 1}$ be a random walk over $\mathbb{Z}$, starting at ...
0
votes
1answer
21 views

How to solve for the n-step state probability vector for the Markov chain

Could you help me to solve for the $n$-step state probability vector for the Markov chain given below. Assume that the system starts in $S_1$ (the other two are $S_2$ and $S_3$). $$ ...
0
votes
0answers
25 views

Expected hitting times of continuous time Markov chain

I have a continuous time Markov chain with Q-matrix $$\begin{bmatrix} -3 & 2 & 0 & 0 &1\\ 0 & -3 & 3 & 0 & 0 \\ 0 & 5 & -5 & 0 & 0\\ 0 & 0 ...
1
vote
1answer
28 views

conditional probability markov chain

A man has three shirts. Each day he chooses one to wear at random from those that he was not wearing the previous day. The shirts are labelled by 1, 2 and 3 and X$_n$ is the label of the shirt he ...
0
votes
0answers
15 views

Asymptotic variance for Markov chain

Let $\hat{\mu_t}(f)=\frac{1}{t} \sum_{i=1}^{t} f(X_i)$ is an estimator for a finite, irreducible Markov chain $\{X_t\}$ with its stationary distribution $\pi$. In addition, assume the estimator is ...
1
vote
1answer
48 views

Markov chains transition identity proof

Let $X = (X_{n})_{n \geq 0}$ a Markov chain proof that $$\mathbb{P}(X_{n + 2} = j \mid X_{n} = i) = \sum_{l \in \mathcal{S}} \mathbb{P}(X_{n + 2} = j, X_{n + 1} = l \mid X_{n} = i)$$ Note: ...
0
votes
0answers
21 views

Why is detailed balance condition called like that?

I'm studying Markov's chain and a very important condition is the detailed balance condition.I'm a bit curious and my question is: Why is detailed balance condition called like that? Anyone explains ...
1
vote
1answer
13 views

Interpretation for squared Markov kernel?

Assume a Markov chain on a measurable state space $(E,\Sigma)$ is given, denoted by $(X_n)_{n\in \mathbb{N}}$ with Markov kernel $p$ and stationary measure $\mu$. In this case, we have $$ ...
0
votes
0answers
16 views

Absorption probabilities in birth and death process

For a birth and death process with parameters $\lambda_i$ and $\mu_i$ representing the birth and death rates respectively, define $u_m$ to be the absorption of probability of the process into state ...
2
votes
1answer
47 views

Why is this true with random walks?

Let $S_n=\sum_n{X_n}$ a random walk with $P[X=1]=p=1-P[X=-1]$. Prove that for any $k \in Z$ $P[\cup_{n \geq1} \{S_n\geq k\}]= (P[\cup_{n \geq1} \{S_n\geq 1\}])^k$ I do not understand why is this ...
0
votes
0answers
21 views

Continuous-time markov chain: Distributional properties of first-Passage time

Consider a continuous-time markov chain with finite state space $S$ and irreducible $Q$. Fix the state $i$ at $t=0$. Let $T$ be its first passage time in an element $j \neq i $ of $S$. Is there any ...
1
vote
1answer
43 views

Markov Chain Application - flipping coin heads, tails, finished

Suppose that we are flipping coins iteratively, until we get tails two in a row. Define three states: Heads, Tails, and Finished. Suppose that the probability of getting a head is $p$, and ...
0
votes
0answers
38 views

continuous Markov chain modle

Question:A computer lab has three laser printers, two that are hooked to the network and one that is used as a spare. A working printer will function for an exponential amount of time with mean 20 ...
-1
votes
1answer
29 views

Unsure what to do with this notation concerning Markov Chain

So I was going over some Markov chain problems and came across a question where you have some standard Markov chain, and you find the stationary distribution $\pi = [\pi_{1}, \pi_{2}, \pi_{3}]$. Then ...
1
vote
1answer
40 views

Why is $R(i)=r(i)+\mathbb N_0 d(i)$?

Let $R(i):=\{k\in\mathbb N:p_{k}(i|i)>0\}$ then the period is $d(i)=\gcd R(i)$. Denote also $r(i)$ be the minimum of $R(i)$. Why is it then that $R(i)=r(i)+\mathbb N_0 d(i)$ ? It is clear that ...
2
votes
1answer
32 views

Showing irreducible matrix

Let $A$ be a matrix which is irreducible, so there exists no permutation matrix $P$ such that $P^{T}AP$ is upper block triangular. Let $B$ a matrix for which $A_{ij}\leq B_{ij} \leq 0$ for $i \neq j$ ...
0
votes
1answer
42 views

Showing that a Markov-Chain has this property.

So I have to show that: Let $\{X_n\}$ be a Markov chain. Show that the property $P(X_{n+1}=i|X_n=j_n,X_0=j_0)=P(X_{n+1}=i|X_n=j_n)$ holds. Hint: use the Markov-property. The Markov Property being: ...
1
vote
1answer
81 views

Finite State Space (Markov Chain): Show $\alpha_{m+n}\leq\alpha_{n}\alpha_{m}$

It is a question in Durrett's book probability: theory and examples, Chapter 6, Exercise 6.6.3. Assume the state space is finite. For any transition matrix $p$, define $\alpha_n = \sup_{i,j} ...
0
votes
0answers
22 views

Given a one step transition matrix, find $E[X_n | X_0 = 0 ], $as $n -> \infty $

Or even more generally, how would you find $E[X_n | X_0 = i ], $ as $ n -> \infty $ Given the one-step transition matrix for a markov chain. I can't find anything helpful online. My first ...
2
votes
1answer
81 views

How can I analyse a Markov chain whose transition matrix has repeated eigenvalues?

Consider the following stochastic matrix: $$M = \left(\begin{array}{ccc} \frac{1}{2} & \frac{1}{6} & \frac{1}{6}\\ \frac{1}{2} & \frac{1}{2} & \frac{1}{6}\\ 0 & \frac{1}{3} ...
1
vote
0answers
46 views

Wald's Identity for non-i.i.d. Case

I am looking for Wald's Identity for non-i.i.d. case as discussed in the following links: https://en.wikipedia.org/wiki/Wald%27s_equation What are the assumptions for applying Wald's equation ...
1
vote
1answer
31 views

What does it mean to “marginalise out” something?

Especially in machine learning one often reads the phrase "to marginalise out" something, and while I understand that this means to integrate over a property, I cannot quite grasp the larger ...
0
votes
0answers
18 views

What does the transition matrice mean in Markovian processes?

A diploma is organised by the College of Hogwarts on two years: $year1$ and $year2$. Each year an exam is organized in order to go to the upper level or be graduatie. Student has the probability ...
0
votes
0answers
8 views

How to find the probability vectors of an invariant Markov chain?

Let an homogeneous Markov chain be $\{X_n\}_{n \inℕ}$ with three states $a,b,c$ \begin{pmatrix} \alpha & 0.5 & 0.3 \\ 0.1 & \beta & 0.8 \\ 0.5 & 0.2 & \delta ...
0
votes
1answer
49 views

Problem in solving the long run behavior of a Markov chain. (Exercise 1.3 Georgy F.Lawler )

Exercise 1.3 Introduction to Stochastic Processes Georgy.F Lawler : Consider a Markov chain with state space {1,2,3} and transition matrix $$ P= \begin{pmatrix} .4 & .2 & .4 \\ .6 ...
1
vote
0answers
33 views

A time reversible Markov chain problem on urns

Question: (Ross Probability Models, Ch. 4, Ex. 70) A total of $m$ white balls and $m$ black balls are distributed into two urns such that each urn contains $m$ balls. At each stage, a ball is selected ...