Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
0answers
19 views

Markov's chain. Worm and apple. [on hold]

An apple is located at vertex $A$ of pentagon $ABCDE$, and a worm is located two vertices away, at $C$. Every day the worm crawls with equal probability to one of the two adjacent vertices. Thus after ...
0
votes
0answers
17 views

“Simple” proof about expected number of visits

Let $X_n$ be a markov chain with state space $\Omega$. Let $G(x,A)$ denote the expected number of visits to $x \in A$ before exiting a subset $A \subset \Omega$. Prove that for all $x,y$ and A, ...
3
votes
1answer
16 views

non-stationary Markov chain n-step

When I search for the long term behaviour of a stationary markov chain I just multiply the transition matrix with itself for the number of steps: P(n) = P(0)^n. But how do you go about doing it ...
0
votes
0answers
11 views

Book recommendation needed: asymptotic behavior of non-stationary Markov chain

Is there any stochastic process textbook which covers some standard results for non-stationary Markov chain? For my purpose, countable state space is enough. Thanks!
0
votes
1answer
21 views

How to prove that the column sum for a markov matrix is 1?

As is the topic, it is obvious and easy to explain in non-math language but how do I mathematically prove it?
0
votes
0answers
6 views

Efficient random sample from Markov chain with known states at two times

Assume a 2-state Markov chain with known transition matrix. Suppose I know, for example, that the chain is in state 1 at time 1, and is also in state 0 at time 10. I want to sample randomly from the ...
0
votes
2answers
48 views

Does there exist a steady state vector of this Markov Matrix?

Does there exist a steady state vector of Markov Matrix $$P=\begin{bmatrix} \frac{1}{2} & \frac{1}{3}\\ \frac{1}{2} & \frac{2}{3} \end{bmatrix}$$ Initially I was not sure whether to answer ...
1
vote
1answer
16 views

Converting second order Markov chain into a first order Markov chain

I'm having some trouble converting a second order Markov chain into a first order Markov chain, namely I want to define some new random variables $Y_i$, that have the property ...
10
votes
1answer
330 views

What happens to a random walk when we increase the probabilities of going right?

Consider a random walk on the integers where the probability of transitioning from $n$ to $n+1$ is $p_n$ (and of course, the probability of transitioning from $n$ to $n-1$ is $1-p_n$); we assume all ...
0
votes
1answer
33 views

Beginner's questions to Hidden Markov Models

I have started reading about Hidden Markov Models, and have some (more or less) minor questions about things I am not sure I understood correctly. I hope asking here is fine: (1) Assumption about the ...
0
votes
0answers
5 views

Finding a One Step Transition Matrix for a Markov Process? (Gambling Application)

I need help finding what a one step transition matrix would look like for the following gambling scenario: Using the bold strategy, say you have a certain amount of money x at any time and you're ...
1
vote
0answers
58 views

How to calculate this integral?

I got confused with this Markov Chain problem: suppose the kernel $Q$ is $Q_x=N(cx,1)$, $c$ is a fixed constant with $|c|<1$ and the stationary distribution is $\pi=N(0,\frac{1}{1-c^2})$. I want ...
0
votes
0answers
46 views

Is this backwards process a Markov chain?

Consider a regular Markov chain which starts at a probability mass vector which is not an equilibrium vector. Is the random process running backward in time from the mass vector of the original ...
1
vote
0answers
25 views

Expected time of reaching 0 of a simple symmetric random walk

Consider the symmetric, simple random walk on $S = \{0, 1, \ldots , k\}$ for $k \in \mathbb N$. Let $$T = \min \{ n \in \mathbb N_0|X_n = 0\}$$ be the first time where the process reaches $0$ and ...
4
votes
1answer
89 views

$P^n$ transition matrix of a Markov chain

The setup: We have an unlimited supply of balls and $k$ boxes. In every step, we randomly (all of them have the same probability) choose a box and put a ball in it. Let $X_n$ be the number of ...
2
votes
1answer
51 views

Show that a Markov Chain is ergodic

Let $Y_n$ be iid random variables with values 1,2,3..n so that $P[Y_i=j]=p_j>0$, where $i\leq1$ and $1\leq j\leq n$. I think I managed to show that $Y_n$ is a Markov chain using the definition, ...
2
votes
1answer
28 views

Stochastic kernel as linear operator

Let $K$ be a stochastic kernel for a set $S$ equipped with a countably generated $\sigma$-Algebra $B(S)$, i.e. $K:S\times B(S)\rightarrow [0,1]$ such that $K(\cdot,A)$ is a measurable function for ...
1
vote
1answer
35 views

Stronger version of Markov Chain

I have just started looking into the concept of Markov chains and I was wondering if anyone could help me with this problem. Let $X_1, X_2, ...$ be a Markov chain with the state space $S$. I need ...
0
votes
2answers
24 views

Aperiodicity of Markov chain

If a markov chain which has many states but only one state has a self-loop edge, then does it mean that the markov chain is aperiodic? Or every state in the markov chain has to have self-loop? For ...
1
vote
1answer
527 views

Expected state of a Markov chain

Let's start with a slightly trivial Markov chain defined as follows: the beginning state is called $1$ and the set of states is $\mathbb{N}$. At each step, when the current state is $n$, the ...
0
votes
1answer
67 views

Probability of extinction in branching process

Consider a branching process where the offspring distribution is given by $$P(X = k) = \frac{1}{2^{k+1}}$$ what is the probability that the process becomes extinct at exactly at the nth generation? ...
0
votes
0answers
26 views

Two-state Markov Chains

If I have a two-state Markov chian $V(t)$ with transition probabilities: $P_{00}(t)=(1-\pi) + \pi e^{-\tau t}$ $P_{01}(t)= \pi - \pi e^{-\tau t}$ $P_{10}(t)=(1-\pi) - (1-\pi)e^{-\tau t}$ ...
1
vote
0answers
10 views

Random DFS properties

Have there been any work analyzing some properties of random DFS walks? By that I mean a DFS search, which chooses the next node to visit with uniform probability. i.e, it still refrains from visiting ...
0
votes
1answer
34 views

Can You Help Me With This Continuous Markov Chain Question?

Consider 2 machines, both of which have an exponential lifetime with mean $\frac{1}{\lambda}$. There is a single repairman that can service machines at an exponential rate $\mu$. Set up the ...
2
votes
1answer
55 views

Random walks : Hitting and recurrence Times relation

I have trouble understanding that how $$E\left[T_0|X_{0} = 0\right] = 1 + E[H_0|X_0=1] $$ where $T_0 = \inf\{n \geq 1:X_n = 0 \}$ and $H_A =\inf\{ n\geq 0: X_n \in A \}$. In other words $T_0$ is the ...
0
votes
0answers
30 views

how to determine transient and recurrent state from transition matrix

I wonder how can I determine the transient and recurrent state from transition matrix ? I mean if I have 10 states It would be very hard to draw diagram for them so how to analyse the matrix? For ...
1
vote
0answers
37 views

Limiting probability of a successful bid

I'm having trouble completing the above question, as my working knowledge of "limiting probabilities" is not very good. For the 1-step transition matrix, I have $$P= \begin{pmatrix} 0.0 & 0.0 ...
0
votes
1answer
52 views

PageRank (power iteration method) convergence rate?

I could not get my head around the idea that the second eigenvalue is the convergence rate. Since the matrix in this application is a Markov matrix (rows/columns sum to one), the largest eigenvalue ...
1
vote
1answer
350 views

Renewal Processes for Uniform and exponential Distributions

Suppose the lifetime of a component $T_i$ in hours is uniformly distributed on $[100, 200]$. Components are replaced as soon as one fails and assume that this process has been going on long enough to ...
0
votes
1answer
24 views

Trying to find the markov chain and adjacency matrix of this graph?

This is graph of the problem: Suppose animal x is at node 3 of the graph. It chooses small path labelled s with 2 times probability then long path l. If length is same then probability is same ...
1
vote
1answer
19 views

Period of a Markov Chain: Why is this one aperiodic?

Here is the problem from a stochastic processes book: Consider a Markov Chain on {0,1,2} having transition matrix 0 1 2 0| 0 0 1| 1| 1 0 0| 2|.5 .5 0| ...
1
vote
1answer
950 views

Example of a Markov chain transition matrix that is not diagonalizable?

It is well-known that every detailed-balance Markov chain has a diagonalizable transition matrix. I am looking for an example of a Markov chain whose transition matrix is not diagonalizable. That is: ...
0
votes
0answers
11 views

Computing smoothed state distribution in HMM

Suppose we have an HMM with two states: $s_1$ and $s_2$. The transitional model is as follows: $P(s_1|s_1) = 0.5$, and $P(s1|s2) = 0.25$. There are two observations: $P(a|s_1) = 0.25$ and $P(a|s_2) = ...
1
vote
0answers
23 views

Learning about Markov Chains

I am trying to learn about how to use markov chains for complicated probability problems. I have been looking for different materials to learn these but haven't had much luck. Does anyone have any ...
0
votes
1answer
40 views

Left eigenvector of stochastic matrices with eigenvalue 1

I am only talking about matrices for finite number of states. By the existence of unique equilibrium distribution, this surely means there can only be one of such eigenvector (i.e. the eigenvalue 1 ...
1
vote
1answer
41 views

Question about HMM

I have this HMM model that I need to solve. Unfortunately, my textbook isn't the best and only describes general cases which I have difficulty working with. Consider an HMM with two states: s1 and ...
1
vote
0answers
12 views

Mixing time analysis of time inhomogeneous markov chaons

There are common methods to characterize mixing times of time homogeneous Markov chains through coupling, conductance and strongly stationary times. However, suppose there is a time-inhomogeneous ...
0
votes
1answer
25 views

Markov Chain--starting states

How do we define the starting states in a Markov Chain. For example if we are asked to calculate the transition matrix for different starting states, what does that mean? I am ultimately asked to ...
-1
votes
0answers
25 views

Conditions for a Markov process to have independent increments [duplicate]

I consistently see "Let $\{X(t)\}$ be a stochastic process with independent increments..." in various texts, though I have yet to find any conditions under which we can guarantee a process to have ...
1
vote
1answer
22 views

Proving irreducibility of Markov chain

I have a Markov chain: state: a permutation of n cards transition: taking the top-most card and randomly choose one of the n possible positions for the card I know it is obviously irreducible ...
2
votes
1answer
36 views

Time sampling an ordinary poisson process

My questions will be given at the end, let me just give some definitions first. The counting process $\{ N(t), t \geq 0 \} $ is said to be a non homogenous Poisson process with intensity function ...
0
votes
0answers
19 views

DTMC: repairing the machine

A machine works for $Y_0$ time then fails and takes $X_1$ time to repair. Then again works for $Y_1$ time and then fails and takes $X_2$ time to repair and so on. All the $X_n$'s and $Y_n$'s ...
0
votes
1answer
20 views

Markov-Chain with general state space - recurrent sets

I have an irreducible Markov Chain $(z_n )_{n\in \mathbb N } $ with state space $X$ and with transition-probability-kernel $K$, so $K(x,\cdot)$ is a probability measure (on the $\sigma$-Algebra ...
1
vote
2answers
359 views

What is the difference between the forward and backward equations in a CTMC?

Given that the Forward equation in a CTMC (Continuous Time Markov Chain) is: $P'(t)=P_t G$, and the Backward equation is: $P'(t)=G P_t$, which equations should I use of the two depending on the case I ...
1
vote
1answer
575 views

Long run probability of going to a state from another

We consider the following transition matrix for a markov chain with state space {A,B,C,D,E} : $P= \left( \begin{array}{ccccc} \frac{1}{2} & 0 & 0 &0 &\frac{1}{2} \\ 0 & ...
1
vote
0answers
24 views

Getting stuck in a loop or the probability of hitting all points in a random walk around a circle.

Suppose you are walking around a circular path made up of $n$ tiles. Each tile $i$ is assigned a distinct value $r_i$ by a random variable uniformly distributed on the set of integers $\{1,...,k\}$ ...
2
votes
1answer
54 views

Random walk : probability of reaching value $i$ without passing by negative value $j$

This is just some question that popped out of nowhere while starting studying random walks, and I don't really know how to approach this. Say I have a random walk that starts at zero, and goes up or ...
0
votes
1answer
25 views

Expected success of trial with conditions

Assume that $n$ people want to achieve a task T. One person can try, and is successful with probability $p$. But when a person try all the other have to do an other trial to have the right to ...
0
votes
1answer
298 views

Best martingale for sequence of “dozen” bets at roulette game

Jim goes the Casino to play roulette. He only makes “dozen” bets at each spin ; his probability of winning is therefore $\frac{1}{3}$ every time (to simplify, we neglect the effect of the zeros in ...
1
vote
0answers
7 views

Dwell times of an absorbing markov chain conditional on reaching specific absorbing state

The fundamental matrix of a discrete time markov chain with absorbing states dictates the expected amount of time spent in each state $j$, given that you started in state $i$. The equation is $$S = ...