A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

learn more… | top users | synonyms

0
votes
0answers
8 views

Deducing results about continuous time random walks from the corresponding discrete time result

Is there any standard way to prove results about continuous time random walks from the corresponding results for discrete time random walks? Specifically, my problem is that I was reading Lawler and ...
1
vote
1answer
27 views

Intuition behind Stopping Times

I'm attending a stocahstic processes course. I have some trouble with the intuition behind a stopping time. I will consider the discrete case to make it simpler. a stopping time is given by ...
2
votes
1answer
26 views

Can two nodes in a Markov chain have transitions that don't total 1?

In all the Markov diagrams I see, the transitions from state A to B always total to one. Just one of many examples, this image ...
3
votes
1answer
25 views

Using Markov Property in solving PDE/SDE

I am solving the PDE I used Feynman-Kac and eventually arrived at $F(t,x)$ $ = E[X_T^2|X_t = x]$ $ = E[(X_t \pm \sigma (W_T -W_t))^2|X_t = x]$ (iirc) So, I try to evaluate $E[(X_t \pm \sigma ...
0
votes
0answers
19 views

Does a state which is passed at least 3 times had to be passed 5 times in Markov chain

Prove of disprove: Let $\{X_n\}_n$ be homogenous Markov chain. if we start from state $i$, there is a positive probability that we pass at least 3 times at state $j$. Does it follows that exists ...
0
votes
0answers
20 views

Proving criterion for a transient state in Markov Chain

Let $\{X_n\}_n$ be a homogenous Markov chain. Prove that if exist a connected subset of states (means set of states which exist positive probability to move between them), $S$ which is not closed, ...
0
votes
1answer
53 views

Markov chain doesn't sum up to 1

Let $\{X_n\}$ be a Markov chain on $S=\{1,2,3,4,5,6\}$ with the matrix suppose we define a new sequence $\{Y_n\}$ by $$Y_n=\cases{1\quad X_n=1\vee X_n=2\\2\quad X_n=3\vee X_n=4\\3\quad ...
0
votes
0answers
26 views

Operator semigroups (lecture notes)

Can you recommend good lecture notes (or a book) about this topic? Basically I would like something which covers more or less the first chapter of the book "Markov Processes" by Ethier and Kurtz, but ...
1
vote
0answers
45 views

I have to show that the following stochastic process is a Markov process

I don't understand how to show that some stochastic processes have the Markov property. For example, if I have the following process: $$(\Omega, \mathcal{F}, (X_t)_{t \geq 0}, P^y)$$ where $\Omega = ...
0
votes
0answers
11 views

Is a Markov Decision Process finite horizon or infinite when the length of horizon depends on actions taken?

I want to model a problem as a Markov decision process but I'm not able to classify it as finite or infinite horizon. In my problem, the time for which the process will continue is a function of ...
1
vote
1answer
29 views

Uniform convergence of $A^n/n!$

In a proof regarding finite space Markov Jump Processes in which the function $P(t)=e^{tG}$ is a solution to both the backward and forward Chapman-Kolmogrov equations, one of the steps assumes that ...
1
vote
1answer
46 views

Probability of a time-dependent set of states in Markov chain

Consider a Markov matrix $P$ defining $m$ states. For each time $n$, define a set of states $S_n$ that contains a predefined subset of the states $\left\{ {1,...,m} \right\}$. For time $n=k$, I would ...
-1
votes
0answers
63 views

Conditional independence for Markov processes

Suppose we have the following, $P_i(Z_t^1=j)=P(X_t^1=j|X_{t-1}^1=i)$ and $P_i(Z_t^2=j)=P(X_t^2=j|X_{t-1}^2=i)$ where $\{X_t^1\}_{t \geq1}$ and $\{X_t^2\}_{t \geq1}$ are 2 discrete-time homogeneous ...
1
vote
0answers
29 views

Construction of pure birth process

I am considering a Markov chain $\lbrace X(t) \rbrace_{t≥0}$ in continunous time on the countable state space $S=\lbrace 0 \rbrace\cup \lbrace (i,j) \mid i \in \mathcal{A} , j \in \mathbb{N} \rbrace, ...
1
vote
0answers
22 views

Invariant Distribution of Two Dependent $\cdot/M/\infty$ Queues Running in Parallel

This is in preparation for an exam I have coming up. We have two $\cdot / M / \infty$ queues with external arrivals occurring according to a Poisson Process of rate $\lambda$. Service occurs with ...
1
vote
1answer
32 views

First time markov process spends tau units in certain state

Consider a continuous time Markov process $\{X(t)\}_{t≥0}$ on the state space $\{0, 1, 2, . . .\}$ with stationary probabilities $\{π_0, π_1, π_2, . . .\}$. Suppose that, when currently in state $i$, ...
3
votes
2answers
85 views

Solving recursive integral equation from Markov transition probability

How do I solve something like: $$f(x) = \frac{1}{\sqrt{2 \pi}} \int_{-\infty}^\infty e^{\frac{-(y - x/2)^2}{2}}f(y)\:\mathrm{d}y$$ for $f(x)$? Is there also a general formula that this falls under? ...
0
votes
0answers
22 views

How to prove that the rational thief problem model is monotone.

A thief goes out stealing every day and has a chance of $p_k$ of stealing a sum $k$ with $0\leq k \leq N$. But there's also a chance $q$ of getting caught, in which case he loses everything he got ...
2
votes
2answers
61 views

Equilibrium distribution of Ehrenfest's urn

(I'll post my own answer to this, but others may be of interest, so post your own if you have one.) (PS: In reply to comments posted below: Stackexchange encourages posting an answer to one's own ...
0
votes
0answers
23 views

What is the state space of this markov chain?

Consider a system where two persons sit at a table and share three books. At any point in time both are reading a book, and one book is left on the table. When a person finishes reading his/her ...
1
vote
0answers
55 views

Semigroup of operators: weak continuity at 0+ implies weak continuity at any t > 0

Let ($E$, $d$) be a metric space. Consider the semigroup $\{P(t)\}_{t\geq 0}$ of bounded linear operators on the Banach space $\hat{C}(E)$ of continuous real functions on ($E$, $d$) vanishing at ...
0
votes
0answers
37 views

Solving simple decision-making model over multiple periods

Consider the following model. Each period t=0,1,..., an agent makes an effort $x\in R_+$ to solve a problem. The value from solving the problem is $V>0$. The relationship between effort and ...
0
votes
2answers
45 views

Markov chain steady state existence

Is it possible for a Markov chain to have no steady state solution ? What is an example of such system ?
1
vote
1answer
22 views

Understanding the proof of an Ergodic theorem for Markov chains

An ergodic theorem for Markov chains is as follows. If a Markov chain $(X_n)_{n \ge 0}$ is irreducible and has an invariant distribution $\pi$, then $$\frac{1}{n} \sum_{k=0}^{n-1} f(X_k) \to ...
1
vote
1answer
25 views

Robustness of Markov Chains

A Markov Chain on a measurable space $X$ is uniquely determined by a stochastic kernel $P$ on $X$. Let $\mathsf P_x$ denote the probability on paths generated by $P$ and the initial condition $x\in ...
0
votes
0answers
35 views

probability of hitting state $i$ in random walk

We have a random walk on the integers with probability of going to the right is $\lambda$ and to the left is $\mu$. Suppose we start at 0. I want to find the probability of ever hitting a fixed state ...
0
votes
0answers
18 views

markov property in Durrett's textbook

Assume $B_t(\omega)=\omega(t),\omega\in (C,\mathcal{C},\mathbb{P}^x)$ is a B.M.(C is the continuous function space ,$\mathcal{C}$ is generated by the coordinate maps) In Durrett's textbook,he proved ...
0
votes
0answers
8 views

Computing world states from uncomprehensive sensor readings

I have a real world system, which consists of items assuming different locations at different times. The state transitions are controlled by machinery in the real world, which is well understood. ...
1
vote
0answers
44 views

What is a convolution kernel?

What is a convolution kernel? (in measure theory, probability theory) In which book can I read about kernels on measurable spaces and convolution kernels? Thank you!
0
votes
1answer
24 views

Markov Chain Steady State 3x3

I have been learning markov chains for a while now and understand how to produce the steady state given a 2x2 matrix. For example given the matrix, ...
1
vote
0answers
27 views

Generator of a Feller semigroup on a coutable space

Let $E$ be a countable set in the discrete topology. Let $(T_t)_{t \geq 0}$ be a Feller semigroup on $E$, i.e. a strongly continuous semigroup of operators on $\mathcal{C}_0(E)$ (in the topology of ...
1
vote
1answer
31 views

Is $(\int_0^t W_s ds, W_t)$ Markov?

Approximating $I_t = \int_0^t W_s ds$ by Riemann sums I have convinced myself that it is not Markov, but I have been met by the claim that $(I,W)$ is and I cannot figure out why. Do you guys have any ...
0
votes
0answers
26 views

Poison Process Dance Marathon Question

The Dance Marathon is a 30 hour event during which people can make online or cash donations. Assume that 80 percent of the donations are made online and all other donations are made by cash. Donations ...
5
votes
1answer
106 views

Conditions for birth and death process having only finitely many deaths.

Consider a birth and death process on $\mathbb{N}=\left\{0,1,2,\ldots\right\}$, given by the transition probabilities $p(n,n+1)=\lambda_n$ and $p(n,n-1)=\mu_n$ (those are the birth and death rates, ...
0
votes
0answers
32 views

Mean time for the renewal process

The system is as below. The energy arrival process is $Y_{k}$ with a constant rate of $\rho$. Node has files of size exponential(λ) to be transmitted with fixed rate of transmission $r$. Hence the ...
0
votes
0answers
20 views

What is the limiting distribution of this Markov Chain?

Take a Markov Chain with state space $\left\{ 0, 1, \dots, 20 \right\}$. Then we have the rule that given $X_n$: Compute $Z = X_n + 1$ or $Z = X_n - 1$ with probability $\frac{1}{2}$ each (if the ...
0
votes
1answer
27 views

Markov property of Brownian motion

There are two statements about Markov property: $B_t $ is Brownian motion and $\mathcal{F}$ is generated by $B$ If $s>0$ and $Y$ is bounded and measuable, then ...
0
votes
1answer
81 views

Gambler's ruin and Markov Chain, coin toss and stakes

I'm considering a classical problem about Markov Chains: A gambler has $£8$ and wishes to get to $£10$. A coin is tossed repeatedly : if it comes down tails, the gambler loses his stake, and if it ...
4
votes
1answer
37 views

A die whose score cannot be as before (Markov chains)

A die is "fixed" so that each time it is rolled the score cannot be the same as the preceding score, all other scores having probability $1/5$. Given that the first score is 6, what is the probability ...
0
votes
0answers
20 views

HMM - forward algorithm (Part-of-Speech Tagging)

In order to understand the Forward algorithm for Hidden Markov Models, I created a Little example of Part-of-Speech Tagging. Consider the Hidden Markov Model with states $N$ (Noun), $V$ ...
1
vote
0answers
11 views

Proportion of arrivals taking a particular path in a Routing Matrix

I have a routing matrix with Node-0 being the source/sink (outside world) and there are service Nodes 1,2..k in the system. The matrix has entries R_ij = Probability of an arrival at Node-i moving to ...
1
vote
3answers
49 views

an exercise about mean and probability

Let $(\Omega, \mathcal{F}, P)$ be a probability space, $X : \Omega \rightarrow \mathbb{R}$ be a discrete random variable and $$\phi : [0, \infty) \rightarrow (0, \infty)$$ an increasing function (so ...
1
vote
1answer
67 views

Finite state Markov chain

Under what conditions a Markov chain can be considered as finite (and not infinite)? Thank you!
1
vote
0answers
20 views

Stochastic scheduling to maximize the expected number of customers arrived at the root of a Jackson tree

In a Jackson network, organized as a tree rooted at queue r, several customers are queued at time 0 and there is no new customer arrival. The service time of each customer in queue i is geometrically ...
1
vote
1answer
41 views

convergence of nullrecurrent markov chain

Hi guys! At the moment I'm working on this proof. It's in a german book so hopefully you understand everything. I understand everything in the picture without the use of the markov property at ...
1
vote
1answer
45 views

How to connect the deterministic and probabilistic descriptions of the SIR model

I am a 17 year old student and I was reading up on epidemic modelling for a math project, specifically the SIR model and I came across this: "This" refers to the assumptions to which the Markov Chain ...
0
votes
1answer
43 views

Computer failure with Markov chains and n-step transition matrix

Hi I am struggling with a Markov Chain question: A computer network has two servers, only one of which is in operation at any given time. A server may break down on any given day with probability p. ...
0
votes
0answers
32 views

A Counter Example of Doeblin Condition

The question is to prove that the following Markov process doesn't satisfy the Doeblin Condtion. Let $X=\{\ldots,-n,\ldots,-1,0,1,\ldots,n,\ldots \} $, The Markov Transition Matrix $P$ is defined as ...
1
vote
0answers
43 views

Kolmogorov backward equations for Birth-Death

I'm trying to solve the Kolmogorov backward equations for a Birth-Death Markov chain with three states. I have 2 equations: $$P_{00}'(t) = \lambda_0 (P_{10}(t)-P_{00}(t))$$ $$P_{10}'(t) = ...
2
votes
1answer
67 views

Is $(B_t^2)$ Markov where $(B_t)$ is Brownian motion?

I am pretty sure $(B_{t}^{2})$ not Markov because the squared random walk is not. Showing the square of a Markov process is or isn't Markov I guess I can repeat the method since to be Markov it ...