A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

2
votes
0answers
13 views

Building a hidden markov model with an absorbing state.

I'm working on trying to implement a hidden markov model to model the affect of a specific protein that can cut an RNA when the ribosome is translating the RNA slowly. Some brief background: The ...
1
vote
0answers
31 views

Write out the explicit Kolmogorov forward differential equation

Let $(X_t)$ be a continuous-time Markov process with two states, as shown below. Assume that there are two positive numbers $a$ and $b$ such that for all times $t\geq 0$ and $h>0$, $P(X_{t+h} = 2 ...
1
vote
0answers
15 views

Kolmogorov forward and Backwards equation interpretation

Let $\lambda_i$ be the sojourn rate of state i, $q_{ij}$ be the transition rate form i to j, and $p_{ij}$ be the transition probability from i to j. The Kolmogorov Forward and backwards equation are ...
0
votes
1answer
26 views

Simple Random Walk - Why are these two events the same?

Let $S = (S_n)_{n \geq 1}$ be a simple random walk. We denote the hitting time of a point $b$ by $\tau_b = \min \{i \geq 1 : S_i \geq b\}$. My text says that the events $\displaystyle\{\max_{k \leq ...
1
vote
1answer
19 views

Are queues CTMC?

The $M/M/1$ queue have all the properties of the countable state continuous time markov chain. Is any general queue also a countable state CTMC?
0
votes
0answers
13 views

Markovian Model: scheduling jobs to servers

I have the following problem. I tried to look at queuing theory, but it probably fits better as a scheduling problem. I have a set of $C$ servers: each one can perform 1 job. Processes arrive ...
0
votes
0answers
9 views

Identification of Infinite Dimensional State in Hidden Markov Model

Consider a hidden markov model (HMM) where the state, $X_t(\alpha)$, is a stochastic distribution over $\alpha \in \mathbb{R}_+$ and one observes a signal $Y_t$, which is simply a moment of this ...
-1
votes
0answers
17 views

Few Questions on Markov Chains [on hold]

Markov Chain In the picture I have a Markov chain, call it $X(n)$. I have a few questions about this Markov chain. First, is it aperiodic? Second, what is the value of ...
0
votes
0answers
3 views

How to make Markov Chain model from sequence of data in MATLAB?

Markov Chain model considers only 1-step transition probabilities i.e. probability distribution of next state depends only on current state and not on previous state. I have a sequence and from that I ...
0
votes
0answers
9 views

Predicting Nash equilibrium after one player enters or leaves

Suppose I have a game with $N$ players, and that the Nash equilibrium can be calculated. If one player enters or leaves the game, is it possible to predict or quickly calculate the resulting Nash ...
1
vote
0answers
11 views

Markov Chains that preserve an ordering of the state space

Suppose $X = (X_k)_{k=0}^\infty$ is a homogeneous Markov chain/process (for example on the state space $E = \lbrace 1, \dots, m\rbrace$). We can interpret the elements of the state space as "values". ...
1
vote
2answers
47 views

Dice Game with 1 die and Payoff Function

Imagine a dice game where you may repeatedly roll a die until you either decide to stop, or roll a 1, with the following payoff function (where k is the number on the die), $f(k) = 0$ when $k=1$ ...
0
votes
1answer
13 views

Does an embedded discrete-time Markov chain preserve its properties in continuous time?

Given a discrete-time Markov chain without independent increments, is the embedding of it into a continuous time Markov chain (i.e. via the use of exponential waiting times) an example of a continuous ...
3
votes
1answer
41 views

Example of a Continuous-Time Markov Process which does NOT have Independent Increments

1. Given a discrete-time Markov chain without independent increments, is the embedding of it into a continuous time Markov chain (i.e. via the use of exponential waiting times) an example of a ...
0
votes
2answers
32 views

Markov chain - Stationnary distribution - Unique

Consider the following respective infinitesimal generators of Markov chains in continuous time: \begin{equation} A=\begin{bmatrix} -4 & 1 & 3 \\ 3 & -5 & 2 \\\ 0 & 3 ...
3
votes
1answer
26 views

Show that a matrix satisfying certain conditions is non-singular

I have a square matrix $A$ satisfying the following conditions: The elements on the diagonal are negative; All other elements are non-negative; All row sums are less than or equal to $0$; There is ...
0
votes
0answers
6 views

What is the probability of this Markov Jump process remaining in this state?

Suppose you had a time homogeneous Markov jump processed defined by the following transition diagram I'm assuming that this means that the process remains in state $0$ for time $t$ with probability ...
0
votes
1answer
19 views

Period of an irreducible Markov Chain is given by the number of eigenvalues with unit modulus

Suppose $\{X_n\}$ is an irreducible Markov Chain on finite state space $S$. Then, the number of eigenvalues of the transition matrix with unit modulus is precisely equal to the period of the chain. ...
1
vote
1answer
11 views

What does the notation $P_{\overline{MM}}(t)$ mean in this context?

The notation $P_{\overline{MM}}(t)$ is used in part (iii) of the following question: I'm unsure of exactly what this notation represents. My guess would be that it represents the probability that a ...
0
votes
0answers
12 views

Pure jump process

I'm having touble understand the pat of the solution that I have underlined in green for b)
1
vote
1answer
32 views

Markov Process graphical representation

I don't understand how the picture has been constructed. Specifically how $\mu^{11}=-(\mu^{12}+\mu^{13}+\mu^{14})$ and $\mu^{44}=-\mu^{43}$ has been graphically represented. Here $\mu^{ij}$ is the ...
0
votes
1answer
19 views

Calculating the variance of the time until a Markov process jumps to a specific state from a starting state?

A Markov process on $E = {1, 2}$ is constructed according to holding time parameters $λ_1 = 2$ and $λ_2 = 4$; the defining Markov chain has transition probabilities $p_{11} = p_{12} = 0.5$ and ...
0
votes
0answers
14 views

Deciding whether a maximum asset price process is a markov process

I understand how Mn has been drawn. For the second computing part, after computing, I have no idea how to decide if Mn is a markov process I don't understand the solution at all, don't know what ...
1
vote
0answers
22 views

When can an embedded Markov chain X for a Markov process Y be reducible?

It's pretty widely documented that a Markov process Y is reducible/irreducible if and only if the embedded Markov chain X is reducible/irreducible. However I'm not sure this works in reverse. I'm ...
1
vote
1answer
36 views

How to use the Markov property of Brownian motion

This is a problem from Durrett's probability with examples, exercise 8.2.1. It is not homework. The exercise states: Let $T_0 = \inf\{s > 0 : B_s = 0\}$ and let $R = \inf\{t > 1 : B_t = 0\}$. ...
0
votes
1answer
10 views

emmission probabilities in a hidden markov model with 2 states and an alphabet of 4 characters

I'm reading through a text that is describing how to use use hidden markov models to identify areas of biological sequences that correspond to specific biological features. It starts with a simple ...
5
votes
0answers
41 views

Poisson: Conditional Probability on Pizza order

I am not sure about my answer. In particular, part b of the following question. Pizza orders arrive according to a Poisson process of rate 20 per hour. Orders are independently for a vegetarian ...
3
votes
0answers
41 views

This is a Markov Chain?

Consider two irreducible ergodic Markov chains with the same state space $\{0, 1, . . . , N\}$, with transition matrices $P$ and $Q$ and respective stationary distributions $\pi$ and $\rho$. We ...
0
votes
1answer
15 views

Distribution of the first exit time of a one-dimensional diffusiom/ Brownian motion

I have a one-dimensional diffusion on $[0,1]$ and I need to calculate the distribution of the first exit time of the interval $(-\epsilon,\epsilon)$ for an $\epsilon > 0$. A good first step would ...
0
votes
0answers
14 views

Question about marked poisson process [migrated]

Let's say I have a Poisson point process on $\left[0,T\right]$ with rate $\lambda\left(t\right)=2t^2$. Suppose I attach a mark $m_t$ to each point $t$ of the process such that $m_t\sim ...
0
votes
1answer
31 views

Prove that $\text{lim}_{\Delta t} \rightarrow 0$ of the transition PDF of a std Weiner process is 0

The transition probability density function of the standard Wiener process is: $$ f(x_2,t_2|x_1,t_1) = \frac{1}{\sqrt{2 \pi (t_2-t_1)}}e^{-\frac{(x_2-x_1)^2}{2(t_2-t_1)^2}} $$ I know that if Markov ...
3
votes
1answer
33 views

Random walk on the positive integers with reflecting boundary

Consider a Markov chain $X$ on the positive integers where for each $n$: $$n\longrightarrow 1,\;2,\;3\;\dots \;n,\;n+1$$ with equal probability, and $n\longrightarrow m$ with zero probability if ...
1
vote
1answer
28 views

Constructing transition graph from transition matrix

Ok so for this question I'm having trouble understanding how the transition graph has been drawn from the given transition matrix. This is what I understand and hopefully someone can correct the ...
0
votes
2answers
29 views

Soft Question - book recommendation - Stochastic Processes

My mother language book on stochastic processes is pretty much complete(~500 pages) but would like another one in English, to have in my library. I'm looking for a similar book containing the ...
0
votes
0answers
27 views

Is it correct to say the expected waiting time is infinite?

Say I have some process that starts in a state in the set of states $S$, each possible starting state having non-zero probability. Some (most) of the starting states eventually result in reaching a ...
1
vote
1answer
59 views

Prove that $Y_n=X_{n-1}X_n$ is a markov chain

Let $\{X_n\}_{n=0}^\infty$ a sequence of discrete random variables independent identically distributed. Let $Y_n$ such that $Y_n=X_{n-1}X_n$ for all $n\ge 1$ Is $\{Y_n\}_{n=0}^\infty$ a markov ...
0
votes
1answer
22 views

What is the (practical) difference between a stationary distribution and an equilibrium distribution of a MC?

I know that, for a Markov Chain, a stationary distribution is the (row) vector $\pi$ such that $\pi \cdot P = \pi$, where $P$ is the one-step transition matrix for the MC. Intuitively, I assume that ...
0
votes
1answer
66 views

prove homogeneous markov chain

$Y_0, Y_1,Y_2,\dots$ are independent and identically distributed random non-negative integer outcomes. Let X_0 = Y_0 $ Let $X_0 = Y_0$ and $X_n = X_{n-1} - Y_n$ if $X_{n-1}>0$, else $X_n = ...
1
vote
1answer
22 views

How to interpret 'arbitrary customer'

My question is about how to interpret 'arbitrary customer' in the following scenario (see question 2. listed below): "At a single server service station two types of arrivals occur. According to a ...
0
votes
0answers
29 views

Defective Markov transition matrix and relation to its limiting distributions

Im trying to come to grips with what the physical interpretation of a non diagonalisable Markov Matrix means in terms of what we can deduce about it having a limiting distribution/ what potential ...
-1
votes
0answers
25 views

How have these probabilities been derived (using 'symmetry')?

I'm looking at the following section of my lecture notes, where it is discussing the probability that a simple random walk, which starts at the state $X_{0}=0$, will return to the origin. I ...
2
votes
1answer
41 views

Theory of Queueing

There's in a banc two identical queues and totally separated : these are two queues of type $M/M/1$. For each of them, the arrivals are separated by exponential times of parameter $\nu$, the ...
2
votes
2answers
58 views

Renewal process problem, where $X_i$'s are i.i.d. with exponential distribution.

A room is lit by $2$ bulbs. Bulbs are replaced only when both bulbs burn out. Lifetimes of bulb's are i.i.d exponentially distributed with parameter $λ=1$. What fraction of the time is the room only ...
1
vote
2answers
57 views

Prove that $i$ is an accesible state in a markov chain

Let $i$ be a recurrent state of an homogeneous markov chain such that the state $j$ is accesible from $i$ (that is $\exists$ $k\ge 1$ such that $p_{ij}(k)>0$) Prove that $i$ is accesible from $j$ ...
0
votes
0answers
40 views

Computing hitting times from the stationary distribution

My question is whether it is possible to compute hitting times from the previously calculated stationary distribution $\pi$ of a continuous-time Markov process $(X_t)_{t \geq 0}$. I know that, from ...
2
votes
2answers
38 views

Mean number of tosses of a fair dice to get a sum of outcomes being a multiple of $5$

Let $S_n$ denote the sum of the outcomes of the $n$ tosses of a fair dice. Let $T=\inf\{n>0: S_n$ is a multiple of $5\}$. Compute $E(T)$ (by means of markov chains). Attempt. Instead of ...
0
votes
1answer
21 views

Mean number of steps needed to reach a recurrent state in a finite irreducible Markov chain

Let $\mathbb{X}$ be a finite state space of an irreducible markov chain $\{X_n\}$ and let $T_x=\inf\{k\geq 0\mid X_k=x\}$ be the number of steps until $\{X_n\}$ reaches state $x\in \mathbb{X}$. ...
1
vote
1answer
36 views

Markov chains and queues

I do not understand how may I use the Markov Chain $Y$ and and describe the system $X$ using the states that the exercise suggest. I was searching queue's examples and -i understand this is a M/M/1 ...
1
vote
1answer
40 views

Markov process and conditional probability

Given a Markov Process, that is $$\mathbb{P}(X_{n+1}=x_{n+1}\mid X_{n}=x_{n},X_{n-1}=x_{n-1}, \dots, X_0=x_0)=\mathbb{P}(X_{n+1}=x_{n+1}\mid X_{n}=x_{n})$$I Need to prove that ...
0
votes
0answers
21 views

Binary Hidden Markov Model

Consider a binary HMM with 2 observed variables $O_n \in \{0,1\} \; \forall n \in \mathbb{N}$. Suppose that the hidden Markov process $X_n$ is characterised by a known transition probability matrix ...