Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

1
vote
1answer
34 views

Equality involving a sequence of independent exponentially distributed variables

I'm trying to prove the following statement: Let $\left( {{T_n}:n \geqslant 1} \right)$ be a sequence of independent, exponentially distributed random variables with ${T_n} \sim Exp\left( {{q_n}} ...
1
vote
0answers
15 views

Birkhoff-Neumann like result for stochastic matrices?

during my research I came along a nice lemma which looks like a Birkhoff-Neumann-theorem result, but in a version for stochastic matrices. Namely, I have: Lemma. Let $M$ be a stochastic matrix, then ...
2
votes
1answer
49 views

probability that a game finishes at $n$th step

A coin is flipped sequentially. The game finishes when the sequence TTH is formed(player X wins) or the sequence HTT is formed(player Y wins). I can find the expected time until absorption by X or Y ...
1
vote
1answer
61 views

Markov: Expected time of first visit to a state starting from that state.

Question: Calculate the expected time of first visit to state 2 given we start in state 2. Is the answer to this the mean recurrence time of 2 or simply zero? I at first thought that the answer ...
0
votes
0answers
26 views

The second eigenvalue of a reducible stochastic matrix

The magnitude of the second dominant eigenvalue of a reducible matrix, as I know, is supposed to be 1, why it's not the case for this matrix : $$ \begin{matrix} 0 & 1 & 0 ...
1
vote
1answer
36 views

Markov processes limiting probability questions

I am going over previous mock exams in preparation for an upcoming exam and am having problems with parts (ii) and (iv) and was looking for some guidance. For part (ii), my thinking was that the ...
2
votes
1answer
129 views

Safe small wins vs. risky large wins at roulette

Short statement of problem : Two players play roulette at a casino. They both start with the same initial amount. Each player always plays his favorite bet each time, and stops playing as soon as he ...
5
votes
2answers
236 views

Generalization of the Jordan form for infinite matrices

Under what conditions is it the case that for a matrix $M$ whose rows and columns are indexed by a countably infinite set $S$ one has a Hamel basis consisting of generalized eigenvectors (i.e. $v \in ...
0
votes
1answer
57 views

How to use symmetry of transition rate matrix in a continuous-time Markov chain?

This is part of a bigger question, so I have to change the question a bit to focus on the point. We have a continuous- time Markov chain with the following transition rate matrix: $$Q= \begin{pmatrix} ...
0
votes
0answers
53 views

G/G/1 Queues - Book with Discrete Time Markov Chain examples

Need some book recommendation or links which have examples how to solve G/G/1 queues with detailed Discrete Time Markov Chain drawn and how to get the steady state distribution, the average number of ...
1
vote
1answer
154 views

Probability of a trajectory in Markov processes

I need help with a simple formula! (My question is taken from here, pag 26 eq 1.112. ) Consider a Markov Process with associated Master Equation: \begin{equation*} ...
0
votes
1answer
50 views

Calculating probability from Markov Chain

I have a Markov Chain with states {1,2,3,4,5} which has the following transition matrix below: $$P= \begin{bmatrix} 0.3 & 0 & 0.7 & 0 & 0\\ 0 & 1 & 0 & 0 & 0\\ 0.5 ...
1
vote
1answer
26 views

Finding $P_{11}(n)$ in Markov Chains

Calculate: $P_{11}(n)=P(X_n=1|X_0=1)$ where the transition matrix is of the form: $$\left[\begin{matrix}0 & 1 &0 \\ 0 & \dfrac{1}{2} & \dfrac{1}{2} \\ \dfrac{1}{2} & 0 & ...
2
votes
1answer
58 views

Proving that a HMC state is recurrent or transient?

Looking at the HMC $$\begin{bmatrix} 1-\alpha & \alpha \\ 0 & 1 \end{bmatrix} $$ How do I prove that the state 2 is recurrent and that state 1 is transient? What does it actually mean by ...
0
votes
1answer
37 views

Continuous time Markov chain. proportion of time spent in state i

If a question asks for the proportion of time spent in a specific state is this the same as the stationary distribution or something else? For continuous time Markov chain with finite state space.
0
votes
2answers
42 views

Proof of Markov Property

I'm trying to understand a simple proof for the markov property which states that: "$A_1$, $A_3$ are conditionally independent given $A_2$ iff $P(A_3 | A_1 \cap A_2)=P(A_3|A_2)$" The Proof begins as ...
0
votes
2answers
104 views

Continuous Markov chains, arriving pairs

I have been trying to sort out this exercise but really stuck on this. Preparing myself for exams and found many exercise on continuous Markov chains but I am always stuck when it comes to transition ...
1
vote
0answers
33 views

Changes in the transition matrix of a Markov chain

In most or all Markov chain theories that I know of assumes that the transition matrix does not change over time. But what if certain changes are expected to occur at certain times in the transition ...
1
vote
1answer
37 views

Optimization of a Sum of Variables

Let there be variables $A$, $B$, $C$, $D$, and $E$ such that a total of $N$ points is allocated among the variables: $A$+$B$+$C$+$D$+$E$=$N$, $N$∈$ℝ$. Let the corresponding point values returned by ...
0
votes
1answer
334 views

Best martingale for sequence of “dozen” bets at roulette game

Jim goes the Casino to play roulette. He only makes “dozen” bets at each spin ; his probability of winning is therefore $\frac{1}{3}$ every time (to simplify, we neglect the effect of the zeros in ...
0
votes
1answer
21 views

A problem on Markov process

Suppose, $\Pi_{\theta}$ be the transition probability function of a Markov chain. For any function $f$ define $$\Pi_{\theta}f_{\theta}(x) = \int f(y,\theta)\Pi_{\theta}(x,dy).$$ Is there any ...
5
votes
3answers
190 views

Expected value of number of draws

We have $5$ number in a bag: $(1,3,5,7,9)$. We draw one from the bag and then put it back. We do this until the sum of the numbers can be divided by $3$. Whats the expected value of the number of ...
1
vote
0answers
76 views

Problem with stochastic processes book - should I switch.

I've been reading "Essentials of Stochastic Processes" (second edition) by "Richard Durrett" and I quite liked it, it's a nice size book and it's very easy to read. However, and this is quite a big ...
0
votes
0answers
155 views

Markov chains, Essential, Inessential, Transient and Recurrent states - redundant definitions (what is the difference)

I've searched my books and gone through a tonne of lecture notes, I am now very sure that we have some redundant definitions. this question is about the difference between essential and recurrent, I ...
1
vote
1answer
96 views

PRobability Markov chain, system of equations

I'm looking for techniques or tricks to solve a system of linear equations you get where you want to find the limiting probabilities. The system is this: $\pi_0 = 0.7\pi_0 + 0.2\pi_1 + 0.1\pi_2$ ( ...
1
vote
0answers
51 views

iterates of generalized matrix system

This may be somewhat of an underspecified question but I'll nonetheless give it a try. In the context of applied work, I've recently come across systems of the form $$ ...
0
votes
1answer
49 views

Stopping time for circular random walk.

This is preparation for an exam I have coming up, not an assignment. Hope you won't mind helping. I've got a random walk, $Y_m, m = 0,1,2, \dots$ on $S = \{0,1,2,\dots,N\}$ with periodic boundaries ...
1
vote
1answer
82 views

Continuous-time Markov Chain forward/backward equations and MLE

I have two questions: 1) Using Kolmogorov's forward and backward equations, show that $p_{11}(t) + p_{21}(t) + p_{31}(t) = 1$ and $p_{21}(t) = p_{31}(t)$ where $p_{ij}(t) = P(X(t) = j | X(0) = i)$. ...
1
vote
1answer
49 views

Markov chain property

I would like to make clarification and show my curiosity about markov process. I will show some part of definition related to markov process from here. The Markov property is the dependence ...
0
votes
1answer
53 views

Practical differences between a PRNG and a Markov chains

In computer programming you can easily find people describing both a PRNG, like a Mersenne Twister, and a Markov / Stochastic process as "pseudo random generators". I honestly never liked this ...
0
votes
1answer
119 views

Markov chain notation

In a book of stochastic approximation, in the convergence of the ODE method chapter I see the following notation : the state vector of a system $X_n$ has a dynamic representation controlled by ...
1
vote
1answer
237 views

Ehrenfest urn model expectation question

Consider the Ehrenfest urn model in which $M$ molecules are distributed between two urns, and at each time point one of the molecules is chosen at random and is then removed from its urn and placed in ...
5
votes
1answer
167 views

How is the Chapman-Kolmogorov Equation not a fancy name for matrix multiplication?

The Chapman-Kolmogorov Equation: $$p^{m+n}(i,j)=\sum_kp^m(i,k)p^n(k,j)$$ Matrix Multiplication (with $[A]_{i,j}=a_{i,j}$ where $A$ is a linear map "" for B) $$[AB]_{i,j}=\sum_ka_{i,k}b_{k,j}$$ In ...
0
votes
1answer
190 views

Convergence rate of PageRank, the problem when the second eigenvalue is complex

As far as I know the Google matrix used to calculate the PageRank is not symetric, that means that some eigenvalues can be complex, furthermore, we know that the second eigenvalue is equal to the ...
3
votes
1answer
108 views

What does a customer see when it begins to be served in $M/M/1$ queue?

In queueing theory, the PASTA (Poisson Arrivals See Time Averages) principle [wiki] justifies $a_n = P_n$ where $$a_n = \text{proportion of customers that find } n \text{ customers in the system when ...
0
votes
2answers
106 views

Prove that something is a Markov chain

Let $\xi_0, \xi_1, \xi_2, ...$be independent, identically distributed, integer valued random variables. Define $Y_n$ = max{$\xi_i: 0 \leq i \leq n$}. Show that $(Y_{n)n\geq0}$ is a Markov chain and ...
0
votes
0answers
72 views

How to prove stationary distribution of a particular MC

I am reading a paper since afternoon and having searched extensively im not yet clear how does the author derive the equation. The paper is [1] Its 'Learning Random Walk Models for Inducing Word ...
2
votes
1answer
245 views

Inverse of a regular stochastic matrix

Is it true that the inverse of a regular stochastic matrix is also regular? Are there any other interesting features that the inverse may have of a regular stochastic matrix? Hope someone could answer ...
1
vote
1answer
99 views

General birth and death process

hi i need some help to understand the following (from the general birth and death process).I'll give some context first , then i ask questions. Consider general birth and death process with birth ...
1
vote
0answers
74 views

References for time-inhomogeneous Markov jump processes?

In some central models in life insurance mathematics, the state of the insured is modeled using a continuous-time time-inhomogeneous Markov process with finitely many states. While many results for ...
0
votes
0answers
42 views

relations between properties of stochastic processess

If we have an integer valued stochastic process, are these implications correct? independent increments $\rightarrow$ Markov property Markov property $\nrightarrow$ independent increments stationary ...
2
votes
1answer
117 views

time-homogeneous continuous time Markov chain

I have a question about the continuous time Markov chain. In the Poisson process we have independent and stationary increments. Do we have this in a continuous time Markov chain that is ...
2
votes
0answers
79 views

Powers of (large) lower triangular matrix

Consider the following "game" of chance. Each time the player pushes a button he is awarded a random (finite, integer, non-negative) number of points. The probability of receiving any particular score ...
2
votes
1answer
57 views

Is the following Markov Chain a martingale?

Say I have a finite, ergodic Markov chain with states ${0,1,2,3}$ and with the following transition matrix: $$\begin{bmatrix} \frac{7}{10} & \frac{3}{10} & 0 &0\\ \frac{1}{10} & ...
0
votes
0answers
15 views

Markov Model of Exponential State Transitions

I am trying to derive the stationary distribution of a system with 2 states, in which transitions between the two states occur as Poisson Processes. That is, if the only two edges are 0 -> 1 with ...
3
votes
1answer
162 views

A linear growth model with immigration

Ill give some background first before asking questions.(the text below is straight out of the book) Each individual in the population is assumed give birth at an exponential rate of $\lambda$ in ...
1
vote
0answers
80 views

Continuous transition kernel for Markov Chains

I am trying to show that the stationary distribution for a Markov Chain on a continuous state space can be obtained by building a transition density kernel, which obeys the detailed balance rule where ...
0
votes
1answer
28 views

“Number of passages from the state $i$”: a strange equality.

Consider a homogenenous Markov chain $\{X_n\,:\, n\in \mathbb N\}$ ($0\in\mathbb N$). The state space is $S$ with $|S|\le |\mathbb N|$ and $i\in S$. Consider moreover the function ...
0
votes
1answer
152 views

M/M/1 Queuing Theory Question

Lets say I have packets arrive to a terminal at Poisson rate $\lambda$ per hour and my terminal has an exponential service rate $\mu$ per hour (so the mean service time is $\frac{1}{\mu}$). So this is ...
2
votes
0answers
97 views

Markov Chain with Normal Transition Matrix

Consider a (sub)-stochastic matrix $P$, and the associated Markov chain $X$ with \begin{align*} \mathbf P [X_n =y | X_0 = x] = P_{xy}^n. \end{align*} Suppose we have the condition $P^T P = P P^T$, ...