Tagged Questions

0
votes
1answer
22 views

Time Periodic Homogeneous Markov Chain

I want to find a textbook or survey article reference with a treatment of discrete-time, inhomogeneous, yet time periodic, markov chains on finite state spaces. Elaboration: I have an inhomogeneous ...
0
votes
0answers
21 views

Transition matrix after lumping Markov Chain?

I managed to partition the set of states $P=\{A_1,A_2,...\}$ so that they satisfy the lumpability criterion (for all states in $A_i$ the sum of the outgoing rates to the states in the target partition ...
2
votes
0answers
119 views

Boundedness of expected reward Markov chain (may be related to discret $M/M/\infty$ queue)

[EDIT]: I read a bit on $M/M/\infty$ queue and it may not be the right comparison and my notation may be confusing (I'm in discrete time and $\lambda,\mu$ look likes rates when they are probability). ...
1
vote
0answers
36 views

Efficient random number generation for sojourn times in semi-Markov processes

I'm doing a self-study of semi-Markov processes and was wondering if there are efficient methods for generating random numbers for sojourn times. For example, generating a bunch of random numbers from ...
3
votes
0answers
23 views

Name for maximum transition probability

Let $p(x,y)$ denote the transition probability of a markov chain. Similarly, let $p^n(x,y)$ be the n-step transition probability. My question is, is there a formal name for $S(x,y):=\sup_n p^n(x,y)$. ...
1
vote
0answers
24 views

Books about Markov Models

I am looking about books on Markov chains, with recent findings such as autoregressive HMM, HMM with inputs, multiple HMM connected together. Is there anything I can look at?
1
vote
0answers
40 views

Markov chain supplementary litterature

I'm studying markov chains through Durrett and I'm finding it quite hard to read. Does anyone have a good idea to supplementary book, preferably one on his level of generality and which studies some ...
0
votes
1answer
736 views

Markov Models simple introduction

We're studying Markov models (still at the basis: transitory states, periodic states, etc..) but the professor isn't very good at teaching and I feel I'm getting lost soon. I'd love to have a simple ...
2
votes
0answers
130 views

Does every continuous time minimal Markov chain have the Feller property?

Consider a Q-matrix on a countable state space. (A Q-matrix is a matrix whose rows sum up to $0$, with nonpositive finite diagonal entries and nonnegative offdiagonal entries). As explained for ...
1
vote
0answers
45 views

Green kernel of Hunt processes

Let $X$ be a regular Hunt process on $R^+$ with starting at $x$ and $T_y :=\inf\{t>0: X_t=y\}, y\in R^+$. $G_q(\cdot,\cdot), q >0$ denotes Green kernel of the process $X$. We have the following ...
5
votes
0answers
145 views

Potential theory: discrete-time Markov processes

Recently I've found lecture notes on "Analysis on Graphs" where the potential theory methods were used to study discrete-time, time-reversible Markov chains (i.e. the state space is countable). ...
1
vote
1answer
120 views

Steady distribution for the reflected random walk

Let us consider the state space being $0,1,\dots,M$ for some $M\in \mathbb N$ and put there $N$ walkers: $$ X = (X_1,\dots,X_N). $$ Each of the walkers move independently, they can be in the same ...
1
vote
1answer
79 views

Shuffling cards and the horseshoe map

I wonder if there is a connection between the dynamics of repeated cut & shuffle operations on a deck of cards, and topological chaotic maps such as the horseshoe map? I ask this entirely naively. ...
6
votes
2answers
167 views

“Small sets” in Markov chains

I came across a definition for a "small set" (of the state space) $A \subset \Omega$: there exists a $\delta > 0$ and a measure $\mu$ such that $p^{(k)}(x, \cdot) \geq \delta \mu (\cdot)$ for every ...
3
votes
1answer
451 views

First time passage decomposition for continuous time Markov chain

For discrete time finite Markov chain, the first passage time $T_j$ to visit state $j$, is determined from the recurrence equation: $$ p^{(n)}_{ij} = \sum_{k=0}^n f_{ij}^{(k)} p^{(n-k)}_{jj} = ...
4
votes
3answers
199 views

From a deterministic discrete process to a Markov chain: conditions?

When will a probabilistic process obtained by an "abstraction" from a deterministic discrete process satisfy the Markov property? Example #1) Suppose we have some recurrence, e.g., $a_t=a^2_{t-1}$, ...
7
votes
2answers
783 views

Nice references on Markov chains/processes?

I am currently learning about Markov chains and Markov processes, as part of my study on stochastic processes. I feel there are so many properties about Markov chain, but the book that I have makes ...
13
votes
5answers
1k views

Good introductory book for Markov processes

Which is a good introductory book for Markov chains and Markov processes? Thank you.