Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
60 views

Markov Chain probabilities

I'm having trouble with this problem from Resnick's Adventures in Stochastic Processes: Consider a Markov Chain on states {0,1,2} with transition matrix $ \left( \begin{array}{ccc} 0.3 & 0.3 ...
0
votes
1answer
73 views

Average time of permanence in a state of a Markov-chain

I know that in a Markov-chain the mean permanence time in a state is a random variable distributed accordingly to: Geometric distribution for Time Discrete Markov Chains Exponential distribution for ...
0
votes
0answers
108 views

Markov renewal process vs Markov Jump process

The venerable wikipedia for "Markov renewal process" says that: "a Markov renewal process is a random process that generalizes the notion of Markov jump processes". So what's the definition of a ...
0
votes
0answers
85 views

Continuous time Markov chain

I would like to know if I am on a right track? Continuous time Markov chain on Wikipedia A very new European “Rapid Reaction Force for Fire” has been created today and begins operation between three ...
0
votes
0answers
50 views

Approximating a Markov process by differential equations

I have a system of states, $m_S = 1, 0, -1$. After performing a certain manipulation (it can be assumed to be instantaneous), a transition can happen with probability p. However, not all states can ...
0
votes
1answer
127 views

Markov chains example

Your exam could be marked with a range of possible grades, simplified as on the following state diagram: To begin with the chances are that you will pass with a standard result. Each 45 minutes ...
0
votes
0answers
37 views

Convergence of sequence of stationary distributions of Markov chains

I have a sequence of finite, discrete-time ergodic Markov chains indexed by a parameter $N$, and I want to prove that their stationary distributions are converging to a well-defined limit as $N\to ...
0
votes
0answers
38 views

References for Hidden Markov Chains

I'm looking for some nice introductions to Hidden Markov Chains. Preferably some that begin from the basic definitions. I would like some of these references to be papers published in journals. Any ...
0
votes
1answer
46 views

how to reformulate general markov property in discrete case

I read the wiki article on the markov property http://en.wikipedia.org/wiki/Markov_property#Definition and wondered how to work out this reformulation. It seems intuitively but I can not work it out. ...
0
votes
0answers
84 views

transition matrix for Markov chain

Can any one help me to solve this home work please? The city of Sacramento recently completed a new light rail system to bring commuters and shoppers into the downtown area and relieve freeway ...
0
votes
0answers
50 views

Does time homogeneity imply strong Markov property in a Markovian process

Does a time homogeneous Markovian process necessarily have strong Markovian property? Does continuity in state space, time, or path make a difference? What are the examples if it does not?
0
votes
0answers
81 views

Factor graphs: HMM

A friend of mine and me are struggling for a while now on how to start with this example that we have to work out. There is a 10x10 map which an agent is randomly placed on. The single tiles of ...
0
votes
0answers
23 views

Set of points visited by a Markov chain issued from a point x

I am looking for examples of Markov chains for each of the following conditions: The set of points visited by the chain issued from a point x is not a.s. constant. The set of points visited by the ...
0
votes
1answer
105 views

Markov Chain: I don't understand this solution to a conditional probability problem after n state transitions…

What is the probability of being in state 4 after two steps, given that one is in state 5 after 8 steps? Markov Chain is at top of link. The sample solution is part (f). I have no idea what the ...
0
votes
1answer
21 views

What is the difference between a communicating class and a closed communicating class?

I checked the definition on Wikipedia http://en.wikipedia.org/wiki/Markov_chain#Properties I couldn't see any difference.
0
votes
1answer
143 views

Transition probability in Continuous Time Markov Chain (CTMC)

I know that for a CTMC, the transition matrix $P(t)=e^{tQ}$, where $Q$ is the infinitesimal generator matrix of the irriducible CTMC. My question is how do I deal with situations or problems that ask ...
0
votes
1answer
51 views

Coupon collector's problem as Random Walk

I see in a book the following as coupon collector's problem. We have $N$ coupons labelled $1,2,\dots,N$ from which we pick with replacement. I could not understand what is the random walk here.
0
votes
1answer
88 views

Decision theory question about selling a house

I have a real world problem and I was wondering if you guys have any nice insight on the best way to solve it mathematically. I'm not sure there is a decisive solution, but it would be nice to have a ...
0
votes
0answers
145 views

Markov chains, limiting distribution and periodicity

My textbook on Markov chains has theorems on when a chain has a unique limiting distribution, but not the other way around, i.e. when a chain does not have a limiting distribution. My question is the ...
0
votes
1answer
61 views

Stochastic matrix with structure

Let $P \in [0,1]^{(n \times n)}$ be a stochastic matrix i.e $P_{ij} > 0 ~ \forall i,j$ and $\sum_{j}P_{ij} = 1~ \forall i$. Now let us impose additional structure on $P$ by saying that $P_{ij} + ...
0
votes
1answer
232 views

What values of $p$ make this a transient chain?

Suppose we have a Markov chain with state space $S = \{0, 1, 2, \dots \}$ and probabilities $p(x, x + 2) = p$, $p(x, x - 1) = 1-p$ for $x > 0$ and $p(0, 2) = p$ and $p(0, 0) = 1 - p$ I would ...
0
votes
1answer
167 views

Markov Chain Monte Carlo in plain English

I barely know what a markov chain is (I had a terrible teacher) and I probably have an idea of what a stationary distribution is... but I don't know how a Monte Carlo method works and I don't know how ...
0
votes
1answer
71 views

Markov chain - clique

Is there a special name (or case) for a finite Markov chain which all states are reachable from any state with positive probability? Does anyone familiar with a problem modeled by this kind of chain?
0
votes
1answer
72 views

Markov Model transition probability

Hy, i have a little doubt about a Markov model problem. The problem requests to find a transition probability matrix for a situation with two statistically independent person that can be in 4 ...
0
votes
1answer
66 views

Stationary probabilities of markov chain

I am confused in which conditions the stationary probabilities of both discrete and continuous Markov chain donot exist. If it is due to periodic chain then is it for both discrete and continuous. ...
0
votes
1answer
68 views

question about the transformation of a Markov process

I have a question about Markov Process: Let $X_t=(X_t^1, X_t^2,..., X_t^n)$ be a Markov process with regard to the filtration $\mathcal{F}_t$, let $Y_t:=\max_{1\leq k\leq n}X_t^k$, then is $Y_t$ a ...
0
votes
0answers
45 views

When random walk Markov matrix of the graph is normal?

I'm considering random walk on undirected graph $G$. At every time step, walker moves to a random neighbour, with all neighbours being equally likely. With adjacency matrix $A$ the random walk Markov ...
0
votes
0answers
81 views

Markov Chain : Montonicity of Sample Mean

Let $\{X_n\}_{n\geq1}$ be an irreducible, ergodic Markov chain with discrete state-space $S$, transition probability matrix $P$ and steady state distribution $\pi = \{\pi_j\}_{j\in S}$. Let $f$ be a ...
0
votes
1answer
54 views

when would next integer $n+1$ be a prime in a given range of $p_*< p< p_*^2$?

Conjecture that along the sequence of natural numbers $n\in\Bbb N$, if walking upwards $1,2,3,4,\ldots,n,n+1,\ldots,$ from every integer to the next (starting with $n=1$), the probability $\phi_p$ ...
0
votes
1answer
272 views

Coin toss probability calculation

A gambler bets on coin flips. With each flip, he wins $1$ dollar with probability $p$, and loses $1$ dollar with probability $1-p$. He starts with $2$ dollar and stops when he reaches either $0$ or ...
0
votes
2answers
91 views

Markov Chain Transition Probability [duplicate]

When dealing with markov chains, say I am in state 0 on day 1, is the probability that I will be in state 0 on day 4 equal to the probability that I will be in state 0 on all of day 2, day 3 and day ...
0
votes
0answers
33 views

Markov proof that a state is either transitive or ergotic - can it be so simple?

This is the chart associated with a Markov matrix The equivalence(communication) classes are: {1,2,3,4} - transitive {5,6,7} - transitive {8} - ergotic My teacher said that "all equivalence ...
0
votes
1answer
42 views

Can an absorbing CTMC be reversible?

Can a CTMC with an absorbing state be reversible? I guess not, as the product of rates through any loop cannot be equal when the loop involves the absorbing state (Kolmogorov criterion). Is my ...
0
votes
1answer
172 views

three-state Markov chain

a male and a female go to a 2-table restaurant on the same day. each day the male sits at one or the other of the 2 tables, starting at the table 1, with a Markov chain transition matrix: ...
0
votes
1answer
175 views

Markov chain problem in Ross's Introduction to probability models

It is example 4.10 and the problem states that a pensioners receives 2 at the beginning of the each month. The amount of money he needs to spend is independent of the amount he has and is equal to $i$ ...
0
votes
0answers
54 views

Question about Infinite Markov chains

Do 2 Markov chains $\left\{X_n\right\}^\inf_{n=0} $ and $\left\{Y_n\right\}^\inf_{n=0} $ with all of these properties exist so that the probability for infinite n values to maintain $X_n=Y_n$ is 0? ...
0
votes
0answers
97 views

Markov Chains Worked Example (Stirzaker)

I have a Markov Chain with state space the non-negative integers. The rules of the M.C. are that when it is in state $i \neq 0$, it moves to one of {${0,1,2,\ldots,i+1}$} with probability $1/(i+2)$ ...
0
votes
0answers
111 views

Markov Chain: Turn a problem into a markov chain

We have 2 machines, which are working in 0.75 precent if it wasn't out o order the day before. When a machine goes out of order, it takes 2 days to fix her up. let Xn be the num of working machines in ...
0
votes
1answer
145 views

Calculating probabilities in genetic sequences

I am working with certain recurring sequences in genetics and try to calculate certain probabilities: Let for instance $$\langle g_i\rangle :=\{1,1,1,6,1,1,1,6,...,1,1,1,6\}$$ and $$\langle ...
0
votes
1answer
114 views

Metropolis Hastings definition - Proving $\pi(x)$ is the invariant density of our transition matrix

I'm currently working through the proof of the Metropolis-Hastings algorithm, and using two sources: page 328, section 3 page 1704-1705 I have a good understanding of most of the proof until ...
0
votes
1answer
641 views

Transition Probability Matrix and Stationary Distribution

Suppose that a communications network transmits binary digits, $0$ or $1$. A message may pass through several links on its way from source to destination, with the possibility of a transmission error ...
0
votes
1answer
166 views

Example of continuous transient Markov chain in detailed balance?

I have been thinking of such a chain but I've found none. I thought about random walk on $\mathbb{N}$ with probability p to go to right and $q=1-p$ to go back(i.e. this is the transition probabilities ...
0
votes
1answer
55 views

stationary Markov chain without starting in stationary distribution?

What would be a concrete example (i.e. the transition Matrix $P$) for a discrete time stationary Markov chain, i.e. $(X(t_{1}+t),t_{2}+t),...,t_{n}+t))$ does not depend on $t$, $\forall n\geq 1, ...
0
votes
1answer
146 views

Calculate the determinant of the matrix ad hence prove

By computing the determinant of $\lambda I-L$ where $L$ is the Leslie matrix, derive the Euler Lotka equation. $$L= \begin{bmatrix} b_{1} & b_{2} & \ldots & b_{w-1} & b_{w}\\ s_{1} ...
0
votes
1answer
477 views

How to show that a stochastic process is Markov

How can I prove that a given stochastic process is a Markov chain. Assume the following process: Joey is walking in the woods. at every turn: if at the previous turn Joey turned left then he will ...
0
votes
0answers
257 views

Infinite Doubly stochastic matrix questions

I have the following question about a Markov chain ${(X_n)}_{n \geq 0}$ with infinite irreducible doubly stochastic matrix $P$. We have the state space $\{1,2,...\}$ . Determine the stationary ...
0
votes
0answers
89 views

Markov Chain and cryptanalysis

Where I will be able to found papers to read the state-art of the use that Markov chain in cryptanalysis. I founded this Canteaut, A. and Chabaud, F. (1998). A new algorithm for finding ...
0
votes
0answers
80 views

Expected time spent in $i$, assymetric random walk on $\mathbb{Z}$

This is exercise 1.7.4 in Norris' Markov Chains textbook. I'm having difficulty calculating a simple looking expectation. Let $(X_n)_{n\geq0}$ be a simple random walk on $\mathbb{Z}$ with transition ...
0
votes
2answers
82 views

Invariant Distribution in a special case of Markov Chains

Suppose I have a finite set say $C=\{c_0,c_1,\ldots,c_R\}$ which are some disjoint subsets of the space say $\{0,1\}^m$. The cardinality of the union of these subsets is much less than $2^m$ say. If ...
0
votes
1answer
135 views

Equilibrium Distribution of Reducible Markov Chain

Let's say a Markov chain has the state-space $S = \{A,B,C,D\}$ Transition Matrix as follows: $$ \begin{bmatrix} 0 & 0 & 0 & 1 \\ 1 & 0 & 0 & 0 \\ 0 ...