A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

1
vote
1answer
70 views

A Markov Chain Problem.(Change the color of ball)

There are $n$ different color balls in a box. Take two balls in turns, and change color of the second ball to the first. (This is one operation). Let $k$ be the (random) number of operations needed to ...
0
votes
0answers
27 views

Parental Markov Condition Example

I'm currently reading a text on Bayesian networks and the text is giving some very crude interpretations of what appear to be some of the most important foundations of the subject. It states the ...
0
votes
3answers
66 views

Transition Matrix of M/M/1 Queue

We know that for an M/M/1 queue the state space is $S=\{0,1,2,... \}$. Further the probability to go from state $i$ to $i+1$ is $\lambda$ for all $i$ in $S$. Moreover, to go from $i$ to $i-1$ is the ...
0
votes
0answers
19 views

Ross probability models questions [closed]

I am studying for a course and have no professors to talk to live, so I hope some members here can be kind enough to help me. Rather than writing everything out, and splitting it up into different ...
0
votes
0answers
48 views

Follow-up on solution to markov process equation

I asked a question here about solving a system related to an absorbing markov chain. I now have a variation where there are $m$ types (of student, job seeker, etc) each of which applies to ...
0
votes
0answers
14 views

Markov Models and Applications

I am looking for resources in Markov models and its applications. I'm looking for tutorials, videos, books etc which provide the following Explain Markov chains in layperson terms and provide ...
0
votes
1answer
44 views

Is there a solution to this system for the diagonal matrix?

I'm trying to find a solution to a system of equations, but its quite different from anything I've come across before. I believe there is a solution, but I could be wrong. $\mathbf{A} = ...
0
votes
1answer
18 views

Markov property for a stochastic process with discrete state space.

Consider a stochastic process $\{X_s\}_{s\in\mathcal S\subseteq\mathbb R}$ with value in $(\mathbb R,\mathcal B(\mathbb R))$ adapted to a filtration $\{\mathcal F_s\}$ (we can suppose that ...
0
votes
0answers
10 views

What are the techniques to solve finite-horizon Markov decision process with large yet finite state and action space?

I formulate a problem into a Markov decision process with finite horizon and plan to solve it with the Backward Induction algorithm. However, both the state and action space are large (on the order of ...
0
votes
0answers
20 views

How to solve “So Who's Counting” problem using Markov Decision Process?

In Martin Puterman's book Markov Decision Processes, one of the problems he gives is "So Who's Counting". In that problem, 5 random digits are generated. After each digit is generated, it is placed in ...
0
votes
0answers
15 views

Probabilities in a Markov Model

I am reading a paper on Markov Models and I am trying to figure out how to compute the probabilities for the $\alpha$-pass. I am given an $N\times N$ matrix $A$, that has the probabilities of ...
0
votes
0answers
29 views

Treatment of Markov process with absolute states

In the standard treatment of a markov process, the state vector is a probability vector, whose elements can be between zero and one. But I have a need to constrain the state vector to zeros or ones. ...
0
votes
0answers
28 views

Exact probability distribution for hitting time of simple random walk

Consider simple random walk on the line starting from the site $y \in \mathbb{N}$. With probability $p$ the walker moves to the right and with probability $1-p$ to the left. Call $\tau$ the first time ...
0
votes
1answer
43 views

Mean absorption time for pure birth process

Let $\xi_t$, $t\geq0$, be a pure birth process, with $P\{\xi_{t+h} = i +1 | \xi_t = i\} = \lambda^ih + o(\lambda)$, as $h \downarrow 0$. At $t=0$, $\xi_0 =1$. Let $\tau = \min\{t ~|~ \xi_t = N\}$. ...
0
votes
1answer
30 views

Proving that a process has the Markov property

Let $X_t=xe^{ct+aB_t}$ where $B_t$ is one dimensional Brownian motion. How would I prove this is a Markov process using the expectation definition of a Markov process, i.e., ...
1
vote
1answer
28 views

A question about Markov process

Let a stochastic process $(x(t),\theta(t))$ be given by $$ \dot{x}(t)=f(x(t),\theta(t)) $$ for a well defined continuous function $f(\cdot,\cdot)$. Let $\mathcal{F}_t$ denote the natural filtration ...
0
votes
1answer
62 views

Continuous time markov chains, is this step by step example correct

I have some questions regarding CTMC... and most importantly whether the step-by-step example I provide below is correct. My main sources about CTMC are: ([1], and [2]). Let's assume 3 possible ...
0
votes
1answer
38 views

Can the transition probabilities of an inhomogeneous Markov chain be written as an exponential?

If $Z_t$ is a homogeneous continuous-time Markov chain with finite state space $E=\{1,\ldots,p\}$, transition matrices $(P(t))$ and intensity matrix $Q$, it holds that $$ P(t) = \exp(tQ), $$ see for ...
0
votes
0answers
23 views

Stopped strong Markov process again strong Markov?

Following setting: I have a right-continuous strong Markov process X in a right-continuous filtration >$\mathbb{F}=(F_t)$ and a P-a.s. finite stopping time $\tau$. My question is: Is the ...
0
votes
1answer
24 views

Markov process states adding.

I got a question from my exam paper. In its third (c) question the transition lime and orange is combined to a single drink called Li-Ora. How can I add the transition probabilities in this case? Will ...
3
votes
2answers
37 views

Always null recurrence at the boundary between positive recurrence and transience?

I have the following theorem: Let $\rho$ be the traffic intensity. a) If $\rho<1$, then $X$ is positive recurrent. b) If $\rho>1$, then $X$ is transient. c) If $\rho=1$, ...
0
votes
1answer
34 views

Board Game Markov Process - Transient Probabilities

I need to write an essay on the Game of Life board game, and so I studied up on Markov Chains to help me calculate the probabilities and average payoffs for the spaces; however I'm not sure whether ...
1
vote
1answer
24 views

Discrete and Continuous Time Markov Properties

$\newcommand{\indep}{\perp\!\!\!\perp}$ In discrete time, the Markov property is $$P[X_{n+1}\in A\mid X_n=s_n,X_{n-1}=s_{n-1}\dots ]=P[X_{n+1}\in A\mid X_n=s_n]$$ On the other hand, the "general ...
0
votes
1answer
28 views

Q-matrix vs. P-matrix description of a Markov chain

Consider a continuous time Markov chain $(X_t)_{t \geq 0}$ on some state space $S$ with transition matrix (P-matrix) $p_t(x,y)$, the probability density of jumping from $x$ to $y$ in time $t$. The ...
2
votes
2answers
23 views

Probability of returning to a given state after n transitions-Markov chains

Let us denote $f_j^{(n)}$ denote the probability of the first return to state $j $after n transitions. Let $p_{jj}^{(n)}$ be the probability of returning to the state $j$ after $n$ transitions when ...
1
vote
1answer
43 views

Property of Wiener process sample path

What is a mean of time, when the trajectory of wiener process $W_t$ is over the line $y=t$? We need to find $\mathbb{E}\tau$, where $\tau=\sum\limits_{a,b:\forall t\in(a,b) ; ...
1
vote
1answer
32 views

Symmetric random walk and the distribution of the visits of some state

I need help with this problem: Let $(S_n)_n$ a symmetric random walk in $\mathbb{Z}$ i.e $S_n=X_1 + \cdots + X_n$ with $(X_n)$ iid $\mathbb{P}(X_n=1)=\mathbb{P}(X_n=-1)=\frac{1}{2}$. Let $m \in ...
1
vote
0answers
13 views

Branching Brownian Motion and KPP equation

I have troubles understanding the proof of the connection between BBM and KPP equation. I mean the proof of the next lemma from the lecture notes of Anton Bovier about BBM, link. This is almost whole ...
0
votes
0answers
40 views

Can ergodic Markov chains be periodic?

I found a statement in one of my notes which said If a state is persistent, aperiodic and not null the it is said to be ergodic Is it necessary that it should be aperiodic? This statement ...
1
vote
1answer
25 views

Markov chains: An issue in classification of states

I recently came across a lemma which goes as follows. Suppose a Markov chain has N states. Let i and j be pair of states. Then j can be reached from i iff there is an integer $ 0 ≤n< N$ such ...
0
votes
2answers
53 views

Expected number of steps to absorbtion - Markov chain

I want to calculate the expected number of steps (transitions) needed for absorbtion. So the transition probability matrix $P$ has exactly one (lets say it is the first one) column with a $1$ and the ...
1
vote
1answer
49 views

Poisson process of sending e-mails

Suppose a person receives and sends e-mails according to a Poisson Process with independent intensities mu and lambda respectively. He reads a received e-mail with probability p and deletes the mail ...
0
votes
0answers
16 views

A question in a textbook about Blumenthal 0-1 law for a general Markov process

This question came up as a result of reading this question . Here is the Blumenthal 0-1 law in the book Stochastic Processes by Richard F. Bass. Proposition 20.8 Let $(X_t , \Bbb{P}^x)$ be a ...
0
votes
1answer
25 views

Writing down the transition matrix of a discrete Markov chain

Please consider the following scenario: One person is walking along a discrete circle induced by $\mathbb{Z}/n\mathbb{Z}$ In each round we roll a dice with $w\in\left\{2,\ldots n\right\}$ sides If ...
0
votes
1answer
30 views

Inferring transition rates from continuous markov chain question

A house has 2 rooms of similar sizes with identical air conditioners equipped with thermostats which turn on and off as needed to maintain the temperature in each room to a desired level of 22 ...
1
vote
1answer
59 views

random walk with sticky barriers

Consider a random walk on the line 1,...,d. You start at point 1. At each step you flip a coin: heads means go left, tails means go right. If you're at 1 and get a heads, just stay where you are ...
0
votes
1answer
20 views

Injections of Markov Processes

In office hours, a professor mentioned that an injective transformation of a Markov process remains Markov. Intuitively, this makes sense to me as you can "recover" the original Markov process from ...
0
votes
0answers
22 views

Transition matrix in left-right hidden semi-Markov model

I'm developing a hidden semi-Markov model left-right . In a left-right model a sequence of $M$ states starts in state $1$ and ends in state $M$, with no repetition of states. Since the model is ...
1
vote
2answers
23 views

Can the ergodic theorem for Markov chains be proved with linear algebra?

This theorem is in my book, let me just say that it is for discrete-time Markov chains, that are time-homogeneous. Ergodic is defined in the book as being positive recurrent and aperiodic. The ...
0
votes
0answers
30 views

Probability of not reaching completion in Markov process

This question is supposed to be easy but is very hard for me. The Norwegian Skating Association has mass produced certain "collectors' cards" with all $N$ speedskaters (Norwegian as well as ...
0
votes
1answer
36 views

Markov chain exercise

Hello i have this Markov chain exercise: Basically we can always move up 1 step, but there is always a possibility that we will go down to the first state 0, the Markov chain consists of N states. ...
0
votes
1answer
34 views

Markov’s inequality

The annual return, R, of a certain stock is a random variable with mean 10. Use Markov’s inequality to obtain a bound for the probability of the stock return being at least 20. Assuming now that R ...
0
votes
1answer
60 views

Tips on how I would find the transition matrix for the following phenomenon?

how would I go about finding the transition matrix for the following phenomenon (which can be modeled as a Markov process)? Any hints or advice is appreciated! During a study break, a student's ...
2
votes
1answer
38 views

Question regarding Notes on Strong Markov Property

I wrote the following notes from a lecture a couple of weeks ago and I don't understand a particular line. Suppose $B_t$ is a Brownian Motion. Now look at $B^x_t = x + B_t$ which is a BM starting ...
0
votes
1answer
50 views

Understanding detailed balance equations

I'm trying to understand how the equilibrium distribution satisfy the detailed balance equation. To my understanding, I only understand that a detailed balance equation would only be satisfied if ...
2
votes
1answer
63 views

Monotonicity and Convexity of Stochastic Matrices

The definition of stochastic monotonicity and convexity is given by "Stochastic Orders and Their Applications" by Moshe Shaked and George Shanthikumar (1994) as: Let $P = \{p_{i,j} \}$ be a ...
0
votes
3answers
89 views

How fast does this Markov chain converge?

Observe the above a Markov chain and limiting matrix of it. Finding the limiting matrix if it exists is easy but I am curious as to how fast this given matrix converges to its limiting matrix. Is ...
0
votes
0answers
73 views

Application of Markov Chain to Game of Life Board Game

I need to calculate the expected outcomes for the Game of Life. I believe that if I multiply the probability of landing on a particular square with the payoff of said square and add up all these ...
0
votes
1answer
18 views

Markov Processes: How to show $\int P(X_t\in B\mid X_0)dPX_0^{-1}=P(X_t\in B)$?

Let $\left\{X_t \right\}_{t\in T}$ be a time homogeneous Markov process with state space $S$. How do I formally demonstrate$$P(X_t\in B)=\int_S P(X_t\in B\mid X_0)dPX_0^{-1}$$(here $PX_0^{-1}$ is the ...
1
vote
1answer
27 views

Markov: Expected time of first visit to a state starting from that state.

Question: Calculate the expected time of first visit to state 2 given we start in state 2. Is the answer to this the mean recurrence time of 2 or simply zero? I at first thought that the answer ...