Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
24 views

Which of the following processes are Markov chains?

A dice is thrown an infinite number of times. Which of the following procsses are Markov chains or not? Justify your answer. For those processes that are Markov chains give the transition ...
1
vote
1answer
20 views

What is the expected number of flips that are needed?

Suppose we flip a fair coin repeatedly until we have flipped four consecutive heads. What is the expected number of flips that are needed? The hint is given is as follows: Consider a Markov chain ...
0
votes
0answers
11 views

Markov chains: identifying a nonregular transitional matrix

I am currently TA'ing for a course in which the students are soon to learn about Markov chains and stochastic matrices. During the sections, it refers to the possible existence of a stable state and ...
0
votes
1answer
10 views

How to properly determine observations related to a Hidden Markov Model alike problem?

I got a an exercise problem which should be seen as a HMM scenario and argument some statements. However I'm quite confused about how to properly solve and argument my solutions. Problem tells: ...
0
votes
0answers
30 views

statements of matrix analysis

Let $y$ be fixed value. Let $A=a(x,y)$ be a matrix and $f_{t}(x)=\frac{\sum_{n=0}^{\infty}{a^{(n)}(x,y)(\frac{1}{t})^n}}{\sum_{n=0}^{\infty}a^{(n)}(y,y)(\frac{1}{t})^n}$ Show that ...
0
votes
0answers
10 views

Calculating the hitting probability using the strong markov property

We have the following Markov chain. $X_n=(F_{n-1},F_n)$ where $F_0=0, F_1=1$ and with probability 1/2 $F_{n+1}$ is the difference of $F_{n-1}$ and $F_{n}$ and with probability 1/2 the sum. I have to ...
0
votes
0answers
29 views

general state space markov chain limit problem

Suppose that the general state space $\chi$ is partitioned as $S$ and $S^c$ and $P(x, S^c)>0$ for any $x\in S$. How can one show that $P_x(\tau_{S^c}<\infty)=1$? I know how to show it when ...
0
votes
1answer
16 views

Irreducibility of markov chains

Let $A=(a(x,y))_{x,y\in X}$ be a finite irreduzibel nonnegative matrix. Let $b,c >0$, and $\alpha=a^{(b+c)}(x,x)>0$. So $a^{(n(b+c))}(x,x)\ge \alpha^n$. And therefore $\lim ...
1
vote
0answers
19 views

Is there a single matrix norm such that for all stochastic $P$, $\| P \| = 1$?

By a stochastic matrix I mean a square real non-negative matrix with rows summing to one. Denote the set of all such matrices $\mathcal{S}$. By matrix norm I mean a norm in the vector space of ...
0
votes
1answer
33 views

How do we establish the existence of fundamental matrix of a Markov chain?

Let $X=\{X_n\}$ be a finite state Markov Chain with the state space $S = \{0,1,2,...,N\}$ such that $0$ is the single absorbing state and all the rest states are transient. The following is the ...
0
votes
1answer
22 views

How do we compute the mean time spent in transient states of a Markov Chain?

Let $X=\{X_n\}$ be a finite state Markov Chain with the state space $S = \{0,1,2,...,N\}$ such that all the states are transient. The following is the transition matrix. $$ P = \left[\begin{matrix} ...
1
vote
2answers
20 views

Symmetric Random walk on $\mathbb {Z}^d$

Consider the symmetric random walk on $\mathbb{Z}^d $. Symmetric means that the probability of going into any of the $2^d$ directions is $1/2^d$. Starting in 0, what is the probability of ...
0
votes
1answer
19 views

Random walk on $\mathbb{Z}$ (probability to be again in the starting point after n steps)

Consider the random walk on $\mathbb{Z}$, where the probability of going one step to the right from any given state shall be $p\in (0,1)$. Starting in 0, what is the probability of returning ...
0
votes
0answers
9 views

Simulating a continous time, inhomogenous Markov chain

What algorithms are used to simulate a time-continous, inhomogenous Markov chain? For the homogenous case, I've found (among others) this reference, which contains a few exact and approximative ...
1
vote
0answers
17 views

Problem involving periodic Markov Chains — probability of being in a given state at time $n$

I'm working on the following problem: I believe that the simplest possible irreducible periodic Markov Chain would be one with two states and no self-loops? Does this seem correct? However, I'm ...
0
votes
0answers
27 views

Smallest irreducible periodic Markov chain

What would be the smallest periodic Markov chain?
-1
votes
0answers
23 views

Stationary distribution of a birth-death model where a parameter follows a uniform distribution.

I asked this question about some type a markov process I was interested in. @Did offers an answer but I fail to understand how to apply his answer to a concrete example. I am therefore seeking for an ...
0
votes
0answers
16 views

finite state space and geometric ergodicity proof

If the state space of is finite, then all irreducible and aperiodic Markov chains are geometrically ergodic. How can one show this fact?
1
vote
1answer
19 views

Discrete-time random process is Markov iif… (Proving a theorem)

First some background: We say that $(X_n)_{n\geq 0}$ is a Markov chain with initial distribution $\lambda$ and transition matrix $P$ if (i) $X_0$ has distribution $\lambda$; (ii) for $n\geq ...
0
votes
0answers
19 views

Markov chains (which book can be recommended?)

This semester I am learning about Markov chains, mainly including basic definitions & properties Recurrence & Transience Perron-Frobenius Theory equilibrium states convergence to ...
0
votes
1answer
25 views

Why is $1/4$ the probability of hitting 6, starting in 0?

We had the following Markov chain: I cannot see the following statement: Starting in 0, the probability of hitting 6 is $1/4$. I do not see because what does this mean "hitting 6"? In ...
0
votes
0answers
17 views

Why is this class recurrent?

In our reading we had the following example for a Markov chain. I cite from the reading: Here we have three communicating classes: $\left\{0\right\}, \left\{1,2,3\right\}$ and ...
1
vote
0answers
17 views

Random Walk and strong law

I want to prove that a Random Walk in 1 dimension is transient when $p\neq\frac{1}{2}$ but i want to prove it by the strong law of large numbers, so i have this: Define a random variable $$X_i = ...
1
vote
1answer
11 views

Construct the Transition matrix for the Markov chain that models this situation?

I'm given this figure and I need to find transition matrix for this. Thep problem says that the robots have been programmed to traverse the maze and at each junction randomly choose which way to go. ...
0
votes
2answers
32 views

About random walk 1D

I just don't understand why is betha expressed in this way. I don't understand the "conditioning on the initial transition" . Hope you help me thanks
1
vote
0answers
48 views

Can we find a correlation between states of a Markov chain?

I have a fair bit of knowledge on Markov chains but I recently wondered if there is a way to find out a correlation between the states of a finite Markov chain. I could not find any material on this. ...
0
votes
1answer
37 views

A way to check the accuracy of a Markov chain?

I am not sure whether I should post this question on MSE or SSE. I will post it here 1st to see if I can get some feedback. Say I have a finite discrete Markov chain constructed maybe using some data ...
1
vote
0answers
43 views

Linear Filtering Problem (Keynman Fac/Particle Model)

$lienar Filtering Problem $$X_n^1 = X_{n-1}^1 + \epsilon_n *W_n $$ $$X_n^2 = (1-\alpha* \delta) X_{n-1}^2 + \beta*\delta X_n^1 $$ $$X_n^3 = X_{n-1}^3 + \delta*X_n^2$$ above is $$\approx$$ $$dX_n^1 ...
0
votes
1answer
21 views

Ergodic behaviour for bounded random dynamical system

Considering an iterated system described by $$ X_n =\gamma_nX_{n-1} , $$ where $\gamma_i$ are non-negative i.i.d. variables. It is easy to show that the expectation will grow unbounded for $\mathbb ...
0
votes
1answer
40 views

Proofs in Stochastic Processes

Let $$X_{n}$$ be an irreducible Markov chain on the state space {1,...,N}. Show that there exists $$C < \infty$$ and $$\rho < 1$$ such that for any states i,j, $$\mathbb{P} [ X_{m}\neq j , m=0 ...
0
votes
0answers
17 views

Stationary VS. limiting probability

I'm just wondering what the difference between stationary probability and limiting probability is. And, if any of you know: What does it mean that some elements exist and some elements doesn't, when ...
1
vote
0answers
28 views

Stochastic process using Markov chain (thief on the run!!)

I'm given an exercise where we are to simulate a thief escaping from an officer. The thief (let's call him/her T for simplicity) and an officer (O) have four cities to be in. Let's call the cities A, ...
0
votes
0answers
25 views

Covariance of states of a finite Markov chain

I know it is possible to construct a covariance matrix for states of a Markov chain but I cannot seem to find a proper way to compute it. I will attach some theories I found from Kemeny and Snell's ...
0
votes
1answer
17 views

Markov chains and boundary theory

In the next semester there is a reading called "Markov chains and boundary theory". I have at least an imagination what a Markov chain is, but what is meant with boundary theory in this context? ...
0
votes
0answers
15 views

A question about Markov chain

Let $(\Omega,\mathscr{F},P)$ be a probability space. Assume $\{X_t\}_{t\geq 0}$ is a Markov chain with finite state space $S$. Assume $u:S\rightarrow\mathbb{R}$. Is it true that the limit ...
0
votes
0answers
30 views

Dealing with conditional OR statements (simple Markov Chain question)

I have a Markov chain with three states, X, Y and Z, and the following transition matrix; $$ P =\pmatrix{0.5 & 0.5 & 0 \\ 0.4 & 0.4 & 0.2 \\ 0 & 0.5 & 0.5 \\}$$ Now, what I ...
1
vote
1answer
23 views

“Almost-Absorbing” Markov Chain with Closed Communicating Classes

I am trying to model the dynamics of a game as an (Absorbing) Markov Chain. There are a bunch of probabilistic transitions between states (as usual) and three "terminal" outcomes: Winning the game ...
1
vote
1answer
68 views

Calculating the expected winner of a Penney's Game using a Markov Chain.

I am trying to calculate the probability that one sequence of coin tosses is more likely to win than the other in a game of Penney's. The sequences are: HTHT and THTT. So far I've come up with the ...
1
vote
1answer
23 views

How to interpret an event of a Markov chain?

Suppose $\{X_n\}_{n=0}^{\infty}$ is a Markov chain with state space $S = \{0,1,2,...,N\}$ with $$ P(X_1=0|X_0=0)=1 \\ P(X_1=N|X_0=N)=1 $$ then why the following result is true? $$ ...
0
votes
0answers
24 views

Strong Markov property of continous time Markov process

In the book "Applied probability and queues" which is available here http://books.google.de/books?id=BeYaTxesKy0C&pg=PA32&hl=de&source=gbs_toc_r&cad=3#v=onepage&q&f=false , ...
0
votes
0answers
21 views

Solving the matrix equation $D P = D P A D$ for stochastic matrices.

Here, I call any real matrix with positive entries with rows summing to one a stochastic matrix (it need not be square). $D,A,P$ are stochastic. $P$ of size $n \times n$ is given. $D$ of size $k ...
3
votes
0answers
39 views

Why a positive recurrent Markov chain implies positive limiting probability?

Let $X=\{X_0,X_1,\ldots\}$ be an irreducible, positive recurrent, and aperiodic Markov chain with the state space $S=\{0,1,2,\ldots\}$ then how do we show that the probability $$ ...
0
votes
1answer
55 views

Stochastic process gambler's ruin [closed]

This is a gambler's ruin problem I would appreciate if anyone can give me a hint about how to solve it. So A, B play this game by tossing a coin. If H shows then B gets 1 dollar from A and if T shows ...
1
vote
2answers
135 views

Thinking about a probability question using Markov chains

The problem is part (b): 1.4.7. A pair of dice is cast until either the sum of seven or eigh appears.  (a) Show that the probability of a seven before an eight is 6/11.  (b) Next, this ...
1
vote
1answer
29 views

Literature request on Markov Chains which state transition probability matrix evolves over time

I want to know is there any literature on markov chains who's state transition probability matrix evolves over time? For instance, I have 2 states, 1 and 2. With $$P = \begin{bmatrix} p_{11} & ...
2
votes
1answer
51 views

Is every PMF on the set of non-negative integers the stationary distribution of some birth-death process?

Let $f(.)$ be a probability mass function on the non-negative integers such that $0<f(n)<1$ and $f(0)+f(1)+...=1$. Then does there exist an irreducible birth-death process with stationary ...
3
votes
1answer
38 views

Gambler's ruin: Distribution of the maximum fortune along the game conditioned to lose

I having troubles with this problem: Let $(X_n)$ a gambler's ruin Markov chain on $\{0,\dots,N\}$ i.e. a Markov chain with state set $E=\{0,\dots,N\}$ and probability transitions $$p(k,k+1)= ...
0
votes
1answer
14 views

Law of Total Probability in Markov Chains

I'm reading about Markov Chains and have come across the following: $ P_x (X_2 = y) = \sum\limits_{z\in \mathbb S} P_x (X_1 = z).P_x(X_2 = y|X_1 = z) $ where $ P_x (X_1 = z) = p(X_1 = z|X_0 = x) $ ...
2
votes
2answers
44 views

finding the generating function $\phi(s) = \mathbb{E}(s^{H_0})$.

i just started the course of markov chains and i'm having a few problems with one of the excercises. Let $Y_1,Y_2, \dots$ be i.i.d random variables with: $\mathbb{P}(Y_1 = 1) = \mathbb{P}(Y_1 = -1) ...
0
votes
0answers
18 views

Memoryless Property of Markov Chains

I'm trying to understand Markov Chains and have across the following in a book: $ \sum\limits_{y=0,1,....m−1}p(x,y)P(T_A<T_B|X_0=x,X_1=y) $ which then becomes the following, under the Markov ...