A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
0answers
7 views

How to handle Finite-state-machine with correlated inputs?

My system can be represented by the following state-diagram. The inputs to this FSM are correlated. This implies that I can no longer make "independent input" assumption. My question is: How ...
0
votes
1answer
22 views

$\sum_{i=1}^{n-k}\frac{i}{n-k}\binom{2n-2k}{n-k+i}\frac{1}{2}^{2(n-k)}=\frac{1}{2}^{2n-2k}\binom{2(n-k)-1}{n-k}$

2$\sum_{i=1}^{n-k}\frac{i}{n-k}\binom{2n-2k}{n-k+i}\frac{1}{2}^{2(n-k)}=2\frac{1}{2}^{2n-2k}\binom{2(n-k)-1}{n-k}$. This is an identity in a note for a class in Markov Processes, but I can't ...
1
vote
1answer
20 views

Strong Markov Property for Markov Chains - Statement Verification

I suspect that my handwritten lecture notes for the Strong Markov Property are wrong. I'd appreciate corrections to them. We first define the following: A random variable $\tau$ is called a ...
4
votes
0answers
21 views

Exploiting the Markov property

I've encountered the following problem when dealing with short-rate models in finance and applying the Feynman-Kac theorem to relate conditional expectations to PDEs. Let ...
0
votes
1answer
17 views

Finding Initial state vector with given values

I am not sure how to use a given values to form a initial state vector. There are 30% of customers from Company A switch to company B every month, and 35% customers from Company B switch to company A ...
-4
votes
0answers
64 views

Markov Decision Processes in Python using PyMDPToolbox [on hold]

Consider a die with N sides (where N is an integer greater than 1 and less than 30) and a nonempty set B of integers between 1 and N (inclusive). The rules of the game are: 1 You start with 0 ...
0
votes
1answer
74 views

Distribution of a process dependent on a Markov chain's states

Consider a Markov chain $X_t$ with state space $\{0,1\}$, initial distribution $$ \begin{array}{l} \mathbf{P}(X_0=1)=\lambda \\ \mathbf{P}(X_0=0)=1-\lambda \end{array} $$ and transition ...
-1
votes
0answers
30 views

Stochastics Process [closed]

There are two transatlantic cables each of which can handle one telegraph message at a time. The time-to-breakdown for each has the same exponential distribution with parameter λ. The time to repair ...
0
votes
0answers
20 views

how to define probability in Markov chain

I am wondering if probability of the state transition in Markov chain is a pre-defined known number. if so, does it mean that to sovle every problem using Markov process decision, the probability of ...
0
votes
2answers
30 views

$\mathcal{P}$ stochastic matrix. If there is $k > 0$ st $\mathcal{P}^k(j, i) > 0$, then there is $r \leq (n-1)$ st $\mathcal{P}^r(j, i) > 0$

Let $\mathcal{P}$ be stochastic matrix of order n. If there is $k > 0$ such that $\mathcal{P}^k(j, i) > 0$, then there is $r \leq (n-1)$ such that $\mathcal{P}^r(j, i) > 0$. My attempt: ...
3
votes
1answer
40 views

Probability that two random walks on $\mathbb{Z}^2$ meet at the origin

Suppose $X,Y$ are symmetric, independent random walks on the lattice $\mathbb{Z}^2$. I am trying to find the probability: $$\mathbb{P}\big(X_n=Y_n=(0,0)\;\text{for ...
2
votes
1answer
737 views

Expected state of a Markov chain

Let's start with a slightly trivial Markov chain defined as follows: the beginning state is called $1$ and the set of states is $\mathbb{N}$. At each step, when the current state is $n$, the ...
1
vote
1answer
30 views

Why do we have these probability functions for this Markov Chain?

The following shows one of the questions we were given in lectures a while back: We have been given the following solutions to this question: I'm rather confused by these. Take, for example, the ...
0
votes
2answers
22 views

What is an example of a second-order markov chain? [closed]

I'd like to see an example of a second-order markov chain. Haven't found one over google or in any of my textbooks
2
votes
1answer
22 views

Building a hidden markov model with an absorbing state.

I'm working on trying to implement a hidden markov model to model the affect of a specific protein that can cut an RNA when the ribosome is translating the RNA slowly. Some brief background: The ...
2
votes
1answer
28 views

Sufficient condition for a measure to be invariant

Given a Polish metric space $H$ and a Borel probability measure $\pi$. Let $\mathcal B_b(H)$ be the set of bounded measurable functions on $H$, and $L^2(H, \pi)$ be the set of square integrable ...
0
votes
1answer
32 views

Expectation in reversible Markov chain

Let $X$ be a Markov chain with transition matrix: $$\mathbf{P}=\begin{pmatrix} 0 & \frac{3}{5} & \frac{2}{5} \\ \frac{3}{4} & 0 & \frac{1}{4} \\ \frac{2}{3} & \frac{1}{3} & ...
0
votes
0answers
12 views

Spectral Density of an ARMA process.

For an upcoming Stochastic Processes exam, we have had a sudden brief email about Spectral Density as the lecturer had forgotten to mention it in classes. He states, For an ARMA process with ...
0
votes
0answers
20 views

Transition functions induced by Markov processes

Let $(\Omega,\mathcal{F},\mathbb{P})$ be a probability space and denote by $(X_t,\mathcal{F}_t)_{t\geq 0}$ a time-continuous Markov process with values in $(E,\mathcal{E})$. For $s<t\in ...
0
votes
2answers
40 views

Simple Random Walk - Why are these two events the same?

Let $S = (S_n)_{n \geq 1}$ be a simple random walk. We denote the hitting time of a point $b$ by $\tau_b = \min \{i \geq 1 : S_i \geq b\}$. My text says that the events $\displaystyle\{\max_{k \leq ...
2
votes
0answers
21 views

Strategy for selling/buying a stock by average reward value iteration

At beginning of any day $t$, I may own $0$ or $1$ share. The price of the share follows the Markov chain in the table below. At the beginning of a day where I own a share, I may either sell at today’s ...
0
votes
0answers
5 views

Recurrent states - proof of claim

I want to prove: If $x↔y$, then $x$ is recurrent iff $y$ is recurrent. $i\in S$ is recurrent if $P(T_i<\infty)=1$ How can I properly prove this? I don't know where to start from. Thanks
2
votes
2answers
30 views

Markovian Gaussian stationary process with continuous paths

Could you, please, help me figure out the following problem. We call a stationary Gaussian process $\xi_t$ (with continuous paths) an Ornstein-Uhlenbeck process if its correlation function ...
0
votes
0answers
20 views

How to Prove that a (Centered) Gaussian Process is Markov if and only if this Equation Holds?

A centered Gaussian process is Markov if and only if its covariance function $\Gamma: \mathbb{R}\times\mathbb{R} \to \mathbb{R}$ satisfies the equality: ...
0
votes
0answers
9 views

Inference on deterministic HMM

I have a Discrete HMM with hidden Markovian signals of the form $\{X_n\} \in \{ 1,2,3,4\}$ and observed outputs of the form $\{Y_n\} \in \{ 1,2\}$. I have a transition probability matrix ...
1
vote
1answer
51 views

How to Express the Probabilities Associated with a Third Variable in a Hidden Markov Model?

Suppose I have an observation $Y_t$ that is conditionally dependent on $X_t$. (More specifically, Y is a series of observations emitted by an underlying hidden Markov state sequence X.) I can ...
1
vote
1answer
377 views

HMM as special case of MRF

I have learned that any Hidden Markov Model (HMM) can be described as a special case of a Markov Random Field (MRF) model. However, AFAIK, the dependencies in a HMM are directed, while the ...
1
vote
1answer
14 views

Invariant distributions: Applications in the real World

I'm studying about invariant distributions for Markov processes; say in the context of dynamics of Random Neural Networks (biological Networks). I can't fully understand what does an invariant ...
1
vote
0answers
36 views

Write out the explicit Kolmogorov forward differential equation

Let $(X_t)$ be a continuous-time Markov process with two states, as shown below. Assume that there are two positive numbers $a$ and $b$ such that for all times $t\geq 0$ and $h>0$, $P(X_{t+h} = 2 ...
1
vote
0answers
18 views

Kolmogorov forward and Backwards equation interpretation

Let $\lambda_i$ be the sojourn rate of state i, $q_{ij}$ be the transition rate form i to j, and $p_{ij}$ be the transition probability from i to j. The Kolmogorov Forward and backwards equation are ...
2
votes
1answer
71 views

Amount of information a hidden state can convey (HMM)

In this paper (Products of Hidden Markov Models, http://www.cs.toronto.edu/~hinton/absps/aistats_2001.pdf), the authors say that: The hidden state of a single HMM can only convey log K bits of ...
1
vote
1answer
43 views

How long does it take two identical hidden Markov models run on same observations to forget their initial distributions (if ever)?

Let $H_1$ and $H_2$ be two instances of a finite Hidden Markov Model (HMM) $H$. That is, $H_1$ and $H_2$ have identical state spaces $Q$ as well as identical transition $A$ and emission probabilities ...
1
vote
0answers
317 views

Hidden Markov Model, transition probabilities which are modeled with an exponential distribution

I'm looking at implementing an algorithm described in a paper, but I'm having trouble understanding how the transition probabilities for a Hidden Markov Model are defined. In the first sections, I ...
0
votes
0answers
10 views

Identification of Infinite Dimensional State in Hidden Markov Model

Consider a hidden markov model (HMM) where the state, $X_t(\alpha)$, is a stochastic distribution over $\alpha \in \mathbb{R}_+$ and one observes a signal $Y_t$, which is simply a moment of this ...
0
votes
1answer
12 views

emmission probabilities in a hidden markov model with 2 states and an alphabet of 4 characters

I'm reading through a text that is describing how to use use hidden markov models to identify areas of biological sequences that correspond to specific biological features. It starts with a simple ...
1
vote
0answers
54 views

Hidden Markov Model Transition Probability

I am doing my assignment and I am asked to derive transition probability of a HMM. There are Three states. H, E and T. They initially gave me the information as follow. E is followed by an H 40% of ...
0
votes
0answers
22 views

Binary Hidden Markov Model

Consider a binary HMM with 2 observed variables $O_n \in \{0,1\} \; \forall n \in \mathbb{N}$. Suppose that the hidden Markov process $X_n$ is characterised by a known transition probability matrix ...
1
vote
1answer
23 views

Are queues CTMC?

The $M/M/1$ queue have all the properties of the countable state continuous time markov chain. Is any general queue also a countable state CTMC?
0
votes
1answer
40 views

How to understand this kind of Markov chain?

There are two unidirectionally coupled processes $X_t$ and $Y_t$. The coupling is $Y_t=g(X_{t-\delta},\dots)$. The Bayesian network of the two process is described in the following figure: Now this ...
0
votes
2answers
143 views

Bayesian Network vs Markov Decision Process

I am wondering if somebody can tell me anything about the practical differences between using Markov Decision Processes and and Bayesian Networks in reasoning about probabilistic processes?
0
votes
1answer
26 views

Symmetric Simple Random Walk - Definition Clarification

I'm finding conflicting answers everywhere, including in my own notes. In the phrase "symmetric simple random walk", which part, "symmetric" or "simple" refers to having a probability of $0.5$ to go ...
0
votes
0answers
15 views

Markovian Model: scheduling jobs to servers

I have the following problem. I tried to look at queuing theory, but it probably fits better as a scheduling problem. I have a set of $C$ servers: each one can perform 1 job. Processes arrive ...
0
votes
1answer
14 views

Does an embedded discrete-time Markov chain preserve its properties in continuous time?

Given a discrete-time Markov chain without independent increments, is the embedding of it into a continuous time Markov chain (i.e. via the use of exponential waiting times) an example of a continuous ...
0
votes
0answers
6 views

How to make Markov Chain model from sequence of data in MATLAB?

Markov Chain model considers only 1-step transition probabilities i.e. probability distribution of next state depends only on current state and not on previous state. I have a sequence and from that I ...
0
votes
0answers
9 views

Predicting Nash equilibrium after one player enters or leaves

Suppose I have a game with $N$ players, and that the Nash equilibrium can be calculated. If one player enters or leaves the game, is it possible to predict or quickly calculate the resulting Nash ...
1
vote
0answers
11 views

Markov Chains that preserve an ordering of the state space

Suppose $X = (X_k)_{k=0}^\infty$ is a homogeneous Markov chain/process (for example on the state space $E = \lbrace 1, \dots, m\rbrace$). We can interpret the elements of the state space as "values". ...
1
vote
2answers
49 views

Dice Game with 1 die and Payoff Function

Imagine a dice game where you may repeatedly roll a die until you either decide to stop, or roll a 1, with the following payoff function (where k is the number on the die), $f(k) = 0$ when $k=1$ ...
3
votes
1answer
53 views

Example of a Continuous-Time Markov Process which does NOT have Independent Increments

1. Given a discrete-time Markov chain without independent increments, is the embedding of it into a continuous time Markov chain (i.e. via the use of exponential waiting times) an example of a ...
1
vote
1answer
32 views

Markov Process graphical representation

I don't understand how the picture has been constructed. Specifically how $\mu^{11}=-(\mu^{12}+\mu^{13}+\mu^{14})$ and $\mu^{44}=-\mu^{43}$ has been graphically represented. Here $\mu^{ij}$ is the ...
0
votes
2answers
33 views

Markov chain - Stationnary distribution - Unique

Consider the following respective infinitesimal generators of Markov chains in continuous time: \begin{equation} A=\begin{bmatrix} -4 & 1 & 3 \\ 3 & -5 & 2 \\\ 0 & 3 ...