1
vote
0answers
16 views

Problem involving periodic Markov Chains — probability of being in a given state at time $n$

I'm working on the following problem: I believe that the simplest possible irreducible periodic Markov Chain would be one with two states and no self-loops? Does this seem correct? However, I'm ...
0
votes
0answers
24 views

Smallest irreducible periodic Markov chain

What would be the smallest periodic Markov chain?
-1
votes
0answers
22 views

Stationary distribution of a birth-death model where a parameter follows a uniform distribution.

I asked this question about some type a markov process I was interested in. @Did offers an answer but I fail to understand how to apply his answer to a concrete example. I am therefore seeking for an ...
0
votes
1answer
36 views

A way to check the accuracy of a Markov chain?

I am not sure whether I should post this question on MSE or SSE. I will post it here 1st to see if I can get some feedback. Say I have a finite discrete Markov chain constructed maybe using some data ...
1
vote
0answers
42 views

Linear Filtering Problem (Keynman Fac/Particle Model)

$lienar Filtering Problem $$X_n^1 = X_{n-1}^1 + \epsilon_n *W_n $$ $$X_n^2 = (1-\alpha* \delta) X_{n-1}^2 + \beta*\delta X_n^1 $$ $$X_n^3 = X_{n-1}^3 + \delta*X_n^2$$ above is $$\approx$$ $$dX_n^1 ...
1
vote
0answers
27 views

Stochastic process using Markov chain (thief on the run!!)

I'm given an exercise where we are to simulate a thief escaping from an officer. The thief (let's call him/her T for simplicity) and an officer (O) have four cities to be in. Let's call the cities A, ...
0
votes
0answers
29 views

Dealing with conditional OR statements (simple Markov Chain question)

I have a Markov chain with three states, X, Y and Z, and the following transition matrix; $$ P =\pmatrix{0.5 & 0.5 & 0 \\ 0.4 & 0.4 & 0.2 \\ 0 & 0.5 & 0.5 \\}$$ Now, what I ...
1
vote
1answer
68 views

Calculating the expected winner of a Penney's Game using a Markov Chain.

I am trying to calculate the probability that one sequence of coin tosses is more likely to win than the other in a game of Penney's. The sequences are: HTHT and THTT. So far I've come up with the ...
1
vote
1answer
23 views

How to interpret an event of a Markov chain?

Suppose $\{X_n\}_{n=0}^{\infty}$ is a Markov chain with state space $S = \{0,1,2,...,N\}$ with $$ P(X_1=0|X_0=0)=1 \\ P(X_1=N|X_0=N)=1 $$ then why the following result is true? $$ ...
0
votes
0answers
24 views

Strong Markov property of continous time Markov process

In the book "Applied probability and queues" which is available here http://books.google.de/books?id=BeYaTxesKy0C&pg=PA32&hl=de&source=gbs_toc_r&cad=3#v=onepage&q&f=false , ...
0
votes
1answer
54 views

Stochastic process gambler's ruin [closed]

This is a gambler's ruin problem I would appreciate if anyone can give me a hint about how to solve it. So A, B play this game by tossing a coin. If H shows then B gets 1 dollar from A and if T shows ...
1
vote
2answers
135 views

Thinking about a probability question using Markov chains

The problem is part (b): 1.4.7. A pair of dice is cast until either the sum of seven or eigh appears.  (a) Show that the probability of a seven before an eight is 6/11.  (b) Next, this ...
3
votes
1answer
38 views

Gambler's ruin: Distribution of the maximum fortune along the game conditioned to lose

I having troubles with this problem: Let $(X_n)$ a gambler's ruin Markov chain on $\{0,\dots,N\}$ i.e. a Markov chain with state set $E=\{0,\dots,N\}$ and probability transitions $$p(k,k+1)= ...
0
votes
1answer
15 views

Recurrence of states in a function of a Markov chain

Suppose $X$ is a Markov chain (or process, for that matter) and suppose further $f(X)$ is also a Markov chain. Let $s$ be a recurrent state in $X$. Is there a general way to determine the recurrence ...
0
votes
1answer
31 views

Probability transition matrix for maximum of iid random variables

I have a homework problem that goes as follows: Let $\xi_i, \ i=0,1,2,\ldots$ be i.i.d. random variables of discrete type. The distribution of $\xi_0$ is given by: $$\mathbb{P}\{\xi_0=i\} = a_i, \ ...
0
votes
0answers
24 views

“Taking expectation” to yield conditional probability

This argument is taken from Resnicks Adventures in stochastic processes and let $T _{\infty } < \infty $ denote that an infinite number of transitions in a continuous time markov chain has occurd ...
0
votes
1answer
28 views

Show that this Markov chain is recurrent

So I have a Markov chain on the nonnegative integers such that, starting from $x$, the chain goes to $x+1$ with probability $p$, $0<p<1$, and goes to state $0$ with probability $1-p$. I'm ...
0
votes
1answer
17 views

$n$-step transition probability of a Markov chain

Let $(X_t)_{t\in\mathbb{N}_0}$ be a time-homogenous Markov chain over a finite state space $\left\{1,\dots,m\right\}$, so that we've got $$\Pr(X_{t+1}=j\mid X_t=i_t,\dots,X_0=i_0)=\Pr(X_{t+1}=j\mid ...
3
votes
2answers
34 views

Interpretation for the determinant of a stochastic matrix?

Is there a probabilistic interpretation for the determinant of a stochastic matrix (i.e. an $n \times n$ matrix whose columns sum to unity)?
2
votes
1answer
87 views

Probability of a substring occurring in a string

Consider a random string of length $n<\infty$ where each digit can be between 0-9 with equal probability and a substring of length $k<n$ consisting of only zeros. What is the probability of ...
2
votes
0answers
13 views

Proving that an inductively defined function is a Markov chain

Let $X_0$ be a random variable with values in a countable set $I$. Let $Y_1,Y_2,\ldots$ be a sequence of independent random variables, uniformly distributed on $[0,1]$. Suppose we are given a function ...
0
votes
1answer
22 views

Expectation of staying in same state for a simple MC

Consider a simple dicrete-time Markov Chain $X_t$ with finite state $\Omega = \{1,2,3\}$. At time 0 the chain is with probability 1 in state 1 $\mathbb{P}(X_0 = 1) =1$. Then the transition probability ...
1
vote
1answer
97 views

A Markov Chain Problem.(Change the color of ball)

There are $n$ different color balls in a box. Take two balls in turns, and change color of the second ball to the first. (This is one operation). Let $k$ be the (random) number of operations needed to ...
1
vote
1answer
28 views

Markov property for discrete Markov chains. A question about “adjacent random variables”

Consider a discrete Markov chain (with values in $\mathbb R$) $\{X_n:\, n\in\mathbb N\}$: namely the state space $S$ is a countable subset of $\mathbb R$ and the random variables are $X_0, X_1, ...
1
vote
1answer
28 views

Branching process: Why does the population die or explode?

Consider a population such that each member, independently from other members, at a certain instant of time is replaced by its offspring. Lets denote with $X_n$ $({n\ge 1})$ the amount of the ...
1
vote
1answer
15 views

Limit distribution is invariant

Consider a homogeneous Markov chain $\{X_n\}_{n\in\mathbb N}$ with a countable (but not necessarily finite) state space $S$. Suppose that there exists a limit distribution $\pi$, namely: ...
1
vote
1answer
21 views

About homogeneous Markov chains

Consider a homogeneous Markov chain $\{X_n\}_{n\in\mathbb N}$ with discrete state space $S$. Now consider the map $$T_{ij}=\text{min}\{n\in\mathbb N\,:\, X_n=j\mid X_0=i\}$$ where $T_{ij}$ is defined ...
0
votes
1answer
29 views

Supply the transition matrix for these (possible) Markov chains

Reading Grimmet, Stirzaker: Probability and Random Processes, which unfortunately doesn't have solutions. Trying to make sure I understand Markov chains. A die is rolled repeatedly. Which of these ...
0
votes
0answers
18 views

Probabilities in a Markov Model

I am reading a paper on Markov Models and I am trying to figure out how to compute the probabilities for the $\alpha$-pass. I am given an $N\times N$ matrix $A$, that has the probabilities of ...
1
vote
0answers
30 views

Conditional return time of simple random walk

Consider a simple random walk on $\mathbb{Z}$, $(S_t)_{t \geq 0}$, with $S_0 = 0$. The probability to jump to the right neighbour is $p \geq \frac{1}{2}$. Call $\tau_k = \min\{t \in \mathbb{N}\, : \, ...
0
votes
0answers
38 views

Exact probability distribution for hitting time of simple random walk

Consider simple random walk on the line starting from the site $y \in \mathbb{N}$. With probability $p$ the walker moves to the right and with probability $1-p$ to the left. Call $\tau$ the first time ...
0
votes
0answers
41 views

Proof of Hammersley and Clifford theorem in Besag's paper

I am reading Besag's paper on Spatial Interaction and the Statistical Analysis of Lattice Systems, see http://www.cise.ufl.edu/~anand/fa11/Besag_Spatial_interaction.pdf. In section 3, it introduces ...
0
votes
1answer
44 views

Board Game Markov Process - Transient Probabilities

I need to write an essay on the Game of Life board game, and so I studied up on Markov Chains to help me calculate the probabilities and average payoffs for the spaces; however I'm not sure whether ...
1
vote
1answer
41 views

Symmetric random walk and the distribution of the visits of some state

I need help with this problem: Let $(S_n)_n$ a symmetric random walk in $\mathbb{Z}$ i.e $S_n=X_1 + \cdots + X_n$ with $(X_n)$ iid $\mathbb{P}(X_n=1)=\mathbb{P}(X_n=-1)=\frac{1}{2}$. Let $m \in ...
0
votes
1answer
76 views

How to make sense out of this: Ergodic theorem for Markov chains

We had the ergodic theorem for Markov chains, stating that: For a state space $S \subset \mathbb{N}$ and all functions $f \in L^1$ (meaning that $\sum_{s \in S} |f(s)|\pi(s) < \infty$) and an ...
3
votes
1answer
75 views

2D random walk variation

If a point on a 2D lattice is allowed to take a random walk by taking a unit step either up, down, left or right, there is probability $1$ of reaching any point (including the starting point) as the ...
1
vote
1answer
66 views

Need help with a basic exercise about Markov chains

Suppose $\left\{ X_{n}\right\} _{n=1}^{\infty}$ is a Markov Chain taking real values. Are the following Markov Chains? $$Y_{n}=\sum_{i=1}^{n}X_{i} , Z_{n}=\left(X_{n},X_{n-1}\right)$$ Edit1 I ...
0
votes
2answers
61 views

Expected number of steps to absorbtion - Markov chain

I want to calculate the expected number of steps (transitions) needed for absorbtion. So the transition probability matrix $P$ has exactly one (lets say it is the first one) column with a $1$ and the ...
0
votes
2answers
45 views

Given a Markov-chain, what is the probability of being at a given state?

Given a Markov-chain, what is the probability of being at a given state? I drew the diagram below just as an example, there is nothing special about it but it would be nice if your answer used it as ...
0
votes
1answer
37 views

How do you find the probability of a certain state in Markov Chain?

This question appears without answer in an old exam I found (not a homework question) Suppose messages that enter a system need to be processed by two servers. They arrive at the system at a ...
10
votes
1answer
172 views

Can we qualitatively predict the strategy of the German and US teams in today's World Cup soccer match?

In today's World Cup soccer match between Germany and the US, both teams only need a draw to advance to the next round. There's been speculation about possible collusion, especially given the friendly ...
2
votes
2answers
123 views

Using Markov - Chain to find average and probability

Suppose a computer generate a random vector of n positions where each position appears on of the numbers from 1 to n. The generation is performed uniformly on the $n!$ possibilities. In the problem we ...
1
vote
1answer
48 views

Proving Product of Transition Matrices is again a Transition Matrix.

Let $P = [p_{ij}]$ be an $n\times n$ transition matrix for an $n$-state markov chain. How do you prove that $P^2$, or even better, that $P^n$ is again a transition matrix? My approach leaves me ...
6
votes
2answers
344 views

In a tournament $n$ players take part in a series of duels

I've recently been thinking about this problem and I think I solved it correctly. However, I was using a rather peculiar method with lots of algebra. I'll post my solution as an answer below. Is there ...
1
vote
0answers
55 views

Use Hasting-Metropolis to generate a random element from a large complicated combinatorial set L

Let $P$ the set of all permutations $(x_1, \ldots, x_n)$ of numbers $(1, \ldots,10)$ and $L$ the subset of $P$ where $\sum_{j=1}^{n}jx_j> a$. To use Hasting-Metropolis algorithm I followed the ...
0
votes
0answers
16 views

Skew and Kurtosis of Absorbing Markov Chains

An absorbing Markov chain $P$ can be put in canonical form: $$ P = \left( \begin{array}{cc} Q & R\\ \mathbf{0} & I_r \end{array} \right), $$ where $Q$ is a t-by-t matrix, $R$ is a nonzero ...
2
votes
1answer
24 views

Applying transition matrix to a probability vector seems to ruin its normalization

I had a little bit about stochastic processes during my "Statistical Physics" course and on my exam I got a problem with a Markov chain. My solution seems to be without computational mistakes (checked ...
2
votes
1answer
30 views

Markov chains by hand

If I have a starting point: $A_T=[0,1]$ at $T=1$ and a one step transition matrix of: $B=\left[ \begin{align} &\frac34 & \frac14& \\& \frac1{20}& \frac {19}{20} &\end{align} ...
0
votes
1answer
30 views

Why does this hold for the mean hitting time?

Let $X$ be a Markov chain and $T_A$ the hitting time. My text uses this in a proof: $$\mathbb E[T_A \ | \ X_0=k ] = \sum_{l\in S} \mathbb P(X_1=l \ | X_0=k\ )(1+\mathbb E[T_A \ | \ X_0=l ])$$ and I ...
0
votes
1answer
33 views

Integration with respect to conditional measure?

Let $(X_n)$ be a Markov chain. For $i\in S$ my text defines $$N_i:=\sum_{n=0}^\infty \mathbf 1_{\{ X_n=i \}}$$ and then, as a part of a larger proof, claims that $$\mathbb E_i(N_i)=\sum_{n=0}^\infty ...