Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
3 views

Hitting time $h_i(k)\leqslant h_i(j)\cdot h_j(k)$

Let $(X_n)_{n\in\mathbb{N}_0}$ be a Markov chain with state space $E$. The hitting time of a set $A\subseteq E$ is a RV $$ ...
0
votes
1answer
33 views

Back to square 1…

A friend of mine was telling me about one of the problems, which he described thus: As you can see, the answer to the toy problem presented here is reportedly 13. However, I don't understand how ...
0
votes
0answers
7 views

Show that the closed classes are the maximal elements of the partial order

In the lecture we defined a partial order $\leq$ on the communicating classes associated to a Markov chain. Now it is to show that the maximal elements of the partial order $\leq$ are the ...
1
vote
0answers
10 views

Closed communicating class and stochastic matrix

Let $(X_n)_{n\in\mathbb{N}_0}$ be a Markov chain with state space $E$ and transition matrix $(p_{ij})_{i,j\in E}$. Let $C\subseteq E$ be a closed communicating class. Show that $$ ...
0
votes
1answer
12 views

Why does $(p_{04}^{(n)})_{n\in\mathbb{N}}$ not converge?

Consider a Markov chain with the states 0,1,2,3,4,5,6 and transition matrix $$ P=\begin{pmatrix}\frac{1}{5} & \frac{3}{5}& 0 & 0 & \frac{1}{5} & 0 & 0\\0 & 0 &1 & 0 ...
0
votes
0answers
9 views

Comparing frequencies in stationary distribution

Do there exist theorems for comparing frequencies in the stationary distribution of a (say) aperiodic, positive recurrent Markov chain? i.e. given the transition probability matrix $\mathbf{P}$ with ...
0
votes
1answer
14 views

Transition matrix to graph

Is there a program which can given a transition matrix $P$ draw a graph from a it? The transition matrix is also known as stochastic matrix and probability matrix see ...
1
vote
1answer
17 views

Show that $\mathbb{P}((X_{n+1},…,X_N)\in F|X_n\in A, (X_{n-1},…,X_0)\in G)=\mathbb{P}((X_{n+1,}…,X_N)\in F|X_n\in A)$

In our reading we had the following Theorem concerning Markov chains: Take $0<n<N$ and $(X_n)_{n\in\mathbb{N}}$ a Markov chain. Then for all $a_n\in E$ (where $E$ is the state space) ...
1
vote
1answer
41 views

Which of the following processes are Markov chains?

A dice is thrown an infinite number of times. Which of the following procsses are Markov chains or not? Justify your answer. For those processes that are Markov chains give the transition ...
1
vote
1answer
20 views

What is the expected number of flips that are needed?

Suppose we flip a fair coin repeatedly until we have flipped four consecutive heads. What is the expected number of flips that are needed? The hint is given is as follows: Consider a Markov chain ...
0
votes
0answers
12 views

Markov chains: identifying a nonregular transitional matrix

I am currently TA'ing for a course in which the students are soon to learn about Markov chains and stochastic matrices. During the sections, it refers to the possible existence of a stable state and ...
0
votes
1answer
10 views

How to properly determine observations related to a Hidden Markov Model alike problem?

I got a an exercise problem which should be seen as a HMM scenario and argument some statements. However I'm quite confused about how to properly solve and argument my solutions. Problem tells: ...
0
votes
0answers
30 views

statements of matrix analysis

Let $y$ be fixed value. Let $A=a(x,y)$ be a matrix and $f_{t}(x)=\frac{\sum_{n=0}^{\infty}{a^{(n)}(x,y)(\frac{1}{t})^n}}{\sum_{n=0}^{\infty}a^{(n)}(y,y)(\frac{1}{t})^n}$ Show that ...
0
votes
0answers
10 views

Calculating the hitting probability using the strong markov property

We have the following Markov chain. $X_n=(F_{n-1},F_n)$ where $F_0=0, F_1=1$ and with probability 1/2 $F_{n+1}$ is the difference of $F_{n-1}$ and $F_{n}$ and with probability 1/2 the sum. I have to ...
0
votes
0answers
29 views

general state space markov chain limit problem

Suppose that the general state space $\chi$ is partitioned as $S$ and $S^c$ and $P(x, S^c)>0$ for any $x\in S$. How can one show that $P_x(\tau_{S^c}<\infty)=1$? I know how to show it when ...
0
votes
1answer
16 views

Irreducibility of markov chains

Let $A=(a(x,y))_{x,y\in X}$ be a finite irreduzibel nonnegative matrix. Let $b,c >0$, and $\alpha=a^{(b+c)}(x,x)>0$. So $a^{(n(b+c))}(x,x)\ge \alpha^n$. And therefore $\lim ...
1
vote
0answers
19 views

Is there a single matrix norm such that for all stochastic $P$, $\| P \| = 1$?

By a stochastic matrix I mean a square real non-negative matrix with rows summing to one. Denote the set of all such matrices $\mathcal{S}$. By matrix norm I mean a norm in the vector space of ...
0
votes
1answer
33 views

How do we establish the existence of fundamental matrix of a Markov chain?

Let $X=\{X_n\}$ be a finite state Markov Chain with the state space $S = \{0,1,2,...,N\}$ such that $0$ is the single absorbing state and all the rest states are transient. The following is the ...
0
votes
1answer
22 views

How do we compute the mean time spent in transient states of a Markov Chain?

Let $X=\{X_n\}$ be a finite state Markov Chain with the state space $S = \{0,1,2,...,N\}$ such that all the states are transient. The following is the transition matrix. $$ P = \left[\begin{matrix} ...
1
vote
2answers
20 views

Symmetric Random walk on $\mathbb {Z}^d$

Consider the symmetric random walk on $\mathbb{Z}^d $. Symmetric means that the probability of going into any of the $2^d$ directions is $1/2^d$. Starting in 0, what is the probability of ...
0
votes
1answer
19 views

Random walk on $\mathbb{Z}$ (probability to be again in the starting point after n steps)

Consider the random walk on $\mathbb{Z}$, where the probability of going one step to the right from any given state shall be $p\in (0,1)$. Starting in 0, what is the probability of returning ...
0
votes
0answers
9 views

Simulating a continous time, inhomogenous Markov chain

What algorithms are used to simulate a time-continous, inhomogenous Markov chain? For the homogenous case, I've found (among others) this reference, which contains a few exact and approximative ...
1
vote
0answers
17 views

Problem involving periodic Markov Chains — probability of being in a given state at time $n$

I'm working on the following problem: I believe that the simplest possible irreducible periodic Markov Chain would be one with two states and no self-loops? Does this seem correct? However, I'm ...
0
votes
0answers
27 views

Smallest irreducible periodic Markov chain

What would be the smallest periodic Markov chain?
-1
votes
0answers
23 views

Stationary distribution of a birth-death model where a parameter follows a uniform distribution.

I asked this question about some type a markov process I was interested in. @Did offers an answer but I fail to understand how to apply his answer to a concrete example. I am therefore seeking for an ...
0
votes
0answers
16 views

finite state space and geometric ergodicity proof

If the state space of is finite, then all irreducible and aperiodic Markov chains are geometrically ergodic. How can one show this fact?
1
vote
1answer
19 views

Discrete-time random process is Markov iif… (Proving a theorem)

First some background: We say that $(X_n)_{n\geq 0}$ is a Markov chain with initial distribution $\lambda$ and transition matrix $P$ if (i) $X_0$ has distribution $\lambda$; (ii) for $n\geq ...
0
votes
0answers
19 views

Markov chains (which book can be recommended?)

This semester I am learning about Markov chains, mainly including basic definitions & properties Recurrence & Transience Perron-Frobenius Theory equilibrium states convergence to ...
0
votes
1answer
25 views

Why is $1/4$ the probability of hitting 6, starting in 0?

We had the following Markov chain: I cannot see the following statement: Starting in 0, the probability of hitting 6 is $1/4$. I do not see because what does this mean "hitting 6"? In ...
0
votes
0answers
17 views

Why is this class recurrent?

In our reading we had the following example for a Markov chain. I cite from the reading: Here we have three communicating classes: $\left\{0\right\}, \left\{1,2,3\right\}$ and ...
1
vote
0answers
17 views

Random Walk and strong law

I want to prove that a Random Walk in 1 dimension is transient when $p\neq\frac{1}{2}$ but i want to prove it by the strong law of large numbers, so i have this: Define a random variable $$X_i = ...
1
vote
1answer
12 views

Construct the Transition matrix for the Markov chain that models this situation?

I'm given this figure and I need to find transition matrix for this. Thep problem says that the robots have been programmed to traverse the maze and at each junction randomly choose which way to go. ...
0
votes
2answers
32 views

About random walk 1D

I just don't understand why is betha expressed in this way. I don't understand the "conditioning on the initial transition" . Hope you help me thanks
1
vote
0answers
48 views

Can we find a correlation between states of a Markov chain?

I have a fair bit of knowledge on Markov chains but I recently wondered if there is a way to find out a correlation between the states of a finite Markov chain. I could not find any material on this. ...
0
votes
1answer
39 views

A way to check the accuracy of a Markov chain?

I am not sure whether I should post this question on MSE or SSE. I will post it here 1st to see if I can get some feedback. Say I have a finite discrete Markov chain constructed maybe using some data ...
1
vote
0answers
43 views

Linear Filtering Problem (Keynman Fac/Particle Model)

$lienar Filtering Problem $$X_n^1 = X_{n-1}^1 + \epsilon_n *W_n $$ $$X_n^2 = (1-\alpha* \delta) X_{n-1}^2 + \beta*\delta X_n^1 $$ $$X_n^3 = X_{n-1}^3 + \delta*X_n^2$$ above is $$\approx$$ $$dX_n^1 ...
0
votes
1answer
21 views

Ergodic behaviour for bounded random dynamical system

Considering an iterated system described by $$ X_n =\gamma_nX_{n-1} , $$ where $\gamma_i$ are non-negative i.i.d. variables. It is easy to show that the expectation will grow unbounded for $\mathbb ...
0
votes
1answer
40 views

Proofs in Stochastic Processes

Let $$X_{n}$$ be an irreducible Markov chain on the state space {1,...,N}. Show that there exists $$C < \infty$$ and $$\rho < 1$$ such that for any states i,j, $$\mathbb{P} [ X_{m}\neq j , m=0 ...
0
votes
0answers
17 views

Stationary VS. limiting probability

I'm just wondering what the difference between stationary probability and limiting probability is. And, if any of you know: What does it mean that some elements exist and some elements doesn't, when ...
1
vote
0answers
28 views

Stochastic process using Markov chain (thief on the run!!)

I'm given an exercise where we are to simulate a thief escaping from an officer. The thief (let's call him/her T for simplicity) and an officer (O) have four cities to be in. Let's call the cities A, ...
0
votes
0answers
25 views

Covariance of states of a finite Markov chain

I know it is possible to construct a covariance matrix for states of a Markov chain but I cannot seem to find a proper way to compute it. I will attach some theories I found from Kemeny and Snell's ...
0
votes
1answer
17 views

Markov chains and boundary theory

In the next semester there is a reading called "Markov chains and boundary theory". I have at least an imagination what a Markov chain is, but what is meant with boundary theory in this context? ...
0
votes
0answers
15 views

A question about Markov chain

Let $(\Omega,\mathscr{F},P)$ be a probability space. Assume $\{X_t\}_{t\geq 0}$ is a Markov chain with finite state space $S$. Assume $u:S\rightarrow\mathbb{R}$. Is it true that the limit ...
0
votes
0answers
30 views

Dealing with conditional OR statements (simple Markov Chain question)

I have a Markov chain with three states, X, Y and Z, and the following transition matrix; $$ P =\pmatrix{0.5 & 0.5 & 0 \\ 0.4 & 0.4 & 0.2 \\ 0 & 0.5 & 0.5 \\}$$ Now, what I ...
1
vote
1answer
23 views

“Almost-Absorbing” Markov Chain with Closed Communicating Classes

I am trying to model the dynamics of a game as an (Absorbing) Markov Chain. There are a bunch of probabilistic transitions between states (as usual) and three "terminal" outcomes: Winning the game ...
1
vote
1answer
68 views

Calculating the expected winner of a Penney's Game using a Markov Chain.

I am trying to calculate the probability that one sequence of coin tosses is more likely to win than the other in a game of Penney's. The sequences are: HTHT and THTT. So far I've come up with the ...
1
vote
1answer
23 views

How to interpret an event of a Markov chain?

Suppose $\{X_n\}_{n=0}^{\infty}$ is a Markov chain with state space $S = \{0,1,2,...,N\}$ with $$ P(X_1=0|X_0=0)=1 \\ P(X_1=N|X_0=N)=1 $$ then why the following result is true? $$ ...
0
votes
0answers
24 views

Strong Markov property of continous time Markov process

In the book "Applied probability and queues" which is available here http://books.google.de/books?id=BeYaTxesKy0C&pg=PA32&hl=de&source=gbs_toc_r&cad=3#v=onepage&q&f=false , ...
0
votes
0answers
21 views

Solving the matrix equation $D P = D P A D$ for stochastic matrices.

Here, I call any real matrix with positive entries with rows summing to one a stochastic matrix (it need not be square). $D,A,P$ are stochastic. $P$ of size $n \times n$ is given. $D$ of size $k ...
3
votes
0answers
39 views

Why a positive recurrent Markov chain implies positive limiting probability?

Let $X=\{X_0,X_1,\ldots\}$ be an irreducible, positive recurrent, and aperiodic Markov chain with the state space $S=\{0,1,2,\ldots\}$ then how do we show that the probability $$ ...