Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

learn more… | top users | synonyms

0
votes
1answer
94 views

Definition of Stationary Distributions of a Markov Chain

I'm having a lot of trouble understanding the definition of the stationary distribution of a Markov Chain from Hoel, Port, Stone's Introduction to Stochastic Processes. They define the stationary ...
0
votes
1answer
98 views

Markov chain for two players with two coins [closed]

Two players A and B toss two fair coins independently. Whoever gets the smaller number of heads will pay that many dollars to the other player. For example, if player A tosses two coins and gets 2 ...
0
votes
1answer
35 views

Card shuffling transition matrix

a short understanding question. Consider a pile of $n$ cards. At every step we choose randomly 2 cards and transpose them. Now $X_n$ should be a Markov chain which describes the order of the pile at ...
0
votes
1answer
213 views

Random Surfer as a Markov Chain

Consider a random surfer who begins at a web page (a node of the web graph) and executes a random walk on the Web as follows. At each time step, the surfer proceeds from his current page A to a ...
0
votes
1answer
36 views

Every finite closed class is recurrent

Let $(X,E,P)$ denote a Markov chain, where $X=(X_n)_{n\in\mathbb{N}_0}$, $E$ is finite state space and $P$ is the transition matrix. Claim: Every finite closed class is recurrent. Here is ...
0
votes
1answer
102 views

Why does from Perron-Frobenius follow that that constant functions are the only harmonic functions here?

in our reading we had the following example for a Markov chain. State Space $E=\left\{1,2,3,4\right\}$ and Transition Matrix $$ P=\frac{1}{3}\begin{pmatrix}0 & 1 & 1 & 1\\1 ...
0
votes
1answer
55 views

Expected number of lines in use in call centre (markov process: queuing theory)

Suppose we have a call centre with infinitely many lines to be able to call to. Calls come in a rate of $\lambda$ and customers are served with rate $\mu$. It is easy to see that the $Q$-matris looks ...
0
votes
1answer
65 views

Markov chain modes of convergence

This is continuation of the question stated here. Let $\left( {{X_\alpha }:\alpha \in A} \right)$ be a finite space Markov chain (discrete or continuous), consisting of only transient and absorbing ...
0
votes
2answers
34 views

Show $P(X(t)=0 | X_{0}=2)= P(X(t)=0 | X_{0}=1)^{2}$

Question: Let $X(t)$ be a continuous-time Markov chain on all non-negative integers with generator matrix $Q$ having for all $i\geq 0:$ $$ q_{i,i}=-i(\lambda +\mu ) \qquad q_{i,i+1}=\lambda i \qquad ...
0
votes
1answer
48 views

Limit for a Markov chain

I am considering a Markov chain on S = {1, . . . , 21} with transition matrix given by: $ 1 =p_{1,2} = p_{2,1}= p_{13,14} = p_{18,14} = p_{15,16} = p_{16,3} = p_{17,16}\\ 1/2 =_{p5,5} =p_{5,7}=p_{7,7} ...
0
votes
1answer
72 views

Calculating the probability in $m$ steps of a Homogeneous Markov Chain

I have the next problem: Consider a homogeneous Markov chain $\{X_n: n = 0,1,2, ... \} $ with state space $E = \{0,1,2, ... \} $, with the following transition probabilities where $ 0 <\theta ...
0
votes
1answer
37 views

Expected number of visits to a state of a Markov chain up to a certain time

Let $P=\{p_{ij}\}$ be a stochastic matrix (with rows and columns indexed by a countable set) and let $p^{(k)}_{ij}$ be the entries of $P^k$. I'm trying to prove that, if the associated Markov chain is ...
0
votes
1answer
26 views

Markov Chains (State transitions)

I was wondering which part I am misunderstanding about the individual-by-individual updating scheme from the book of Jackson M. (Social and Economic Networks, 2008) . The full transition matrix in the ...
0
votes
1answer
54 views

Markov chain: if $X\rightarrow Y\rightarrow Z$, then why is $Z\rightarrow Y\rightarrow X$ true?

in a Markov chain, given three random variables $X,Y,Z$, we have $X\rightarrow Y\rightarrow Z$, which means $p(x,y,z) = p(x)p(y|x)p(z|y)$. The right arrow symbol $\rightarrow$ is used to denote a ...
0
votes
1answer
26 views

Markov Chain (Learning)

If I have a Matrix like the one below, what is the probability $p_t$ that at a certain time $t$, we are still not able to arrive at state $z$ $$ \begin{array}{c|lcr} \text{States} & x & y ...
0
votes
1answer
37 views

Is it possible to compute these probabilities concerning a 6-digit password using theory of Markov chains? [duplicate]

Consider the situation of decoding a 6-digit password that consists of the symbols A to Z and 0 to 9, where all possible combinations are tried randomly and uniformly. Consider the ...
0
votes
2answers
56 views

Why is P irreducible and aperiodic?

Let $P=(p_{ij})_{i,j\in E}$ denote the transition matrix of a Markov chain with finite state space $E$. Why does the following implication hold: $$ \exists n\in\mathbb{N}: ...
0
votes
1answer
17 views

Transformation to achieve unit transition rate in a continuous time Markov chain

I have a continuous time Markov chain (CTMC) defined by a transition matrix $P$ and where all transition times go as a exponential random variables with transition rate $\gamma$. I would like to ...
0
votes
1answer
23 views

Determining the Likelihoods of Different Game States

Suppose a game is played in which Player 1 must gain two points to win and Player 2 must gain five points to win. Both players start with zero points. In any round, Player 1 has a $1/3$ chance of ...
0
votes
1answer
371 views

A Markov Chain problem concerning a flea moving around a triangle

A Flea moves around the vertices of a triangle in the following manner: Whenever it is at vertex i it moves to its clockwise neighbor vertex with probability $p_i$ and to the counterclockwise neighbor ...
0
votes
2answers
76 views

Hitting time $h_i(k)\geqslant h_i(j)\cdot h_j(k)$

Let $(X_n)_{n\in\mathbb{N}_0}$ be a Markov chain with state space $E$. The hitting time of a set $A\subseteq E$ is a RV $$ ...
0
votes
1answer
15 views

Why does $(p_{04}^{(n)})_{n\in\mathbb{N}}$ not converge?

Consider a Markov chain with the states 0,1,2,3,4,5,6 and transition matrix $$ P=\begin{pmatrix}\frac{1}{5} & \frac{3}{5}& 0 & 0 & \frac{1}{5} & 0 & 0\\0 & 0 &1 & 0 ...
0
votes
1answer
18 views

How to properly determine observations related to a Hidden Markov Model alike problem?

I got a an exercise problem which should be seen as a HMM scenario and argument some statements. However I'm quite confused about how to properly solve and argument my solutions. Problem tells: ...
0
votes
1answer
22 views

Irreducibility of markov chains

Let $A=(a(x,y))_{x,y\in X}$ be a finite irreduzibel nonnegative matrix. Let $b,c >0$, and $\alpha=a^{(b+c)}(x,x)>0$. So $a^{(n(b+c))}(x,x)\ge \alpha^n$. And therefore $\lim ...
0
votes
1answer
45 views

Smallest irreducible periodic Markov chain

What would be the smallest periodic Markov chain? We're studying periodic Markov chains in my probability course. I'm just trying to picture the smallest possible one but I can't seem to come up ...
0
votes
1answer
35 views

Why is $1/4$ the probability of hitting 6, starting in 0?

We had the following Markov chain: I cannot see the following statement: Starting in 0, the probability of hitting 6 is $1/4$. I do not see because what does this mean "hitting 6"? In ...
0
votes
2answers
48 views

About random walk 1D

I just don't understand why is betha expressed in this way. I don't understand the "conditioning on the initial transition" . Hope you help me thanks
0
votes
1answer
189 views

A way to check the accuracy of a Markov chain?

I am not sure whether I should post this question on MSE or SSE. I will post it here 1st to see if I can get some feedback. Say I have a finite discrete Markov chain constructed maybe using some data ...
0
votes
1answer
29 views

Ergodic behaviour for bounded random dynamical system

Considering an iterated system described by $$ X_n =\gamma_nX_{n-1} , $$ where $\gamma_i$ are non-negative i.i.d. variables. It is easy to show that the expectation will grow unbounded for $\mathbb ...
0
votes
1answer
70 views

Proofs in Stochastic Processes

Let $X_{n}$ be an irreducible Markov chain on the state space $\{1,\dots,N\}$. Show that there exists $C < \infty$ and $\rho < 1$ such that for any states $i,j$, $$\mathbb{P} [ X_{m}\neq j , ...
0
votes
1answer
19 views

Markov chains and boundary theory

In the next semester there is a reading called "Markov chains and boundary theory". I have at least an imagination what a Markov chain is, but what is meant with boundary theory in this context? ...
0
votes
1answer
82 views

Law of Total Probability in Markov Chains

I'm reading about Markov Chains and have come across the following: $ P_x (X_2 = y) = \sum\limits_{z\in \mathbb S} P_x (X_1 = z).P_x(X_2 = y|X_1 = z) $ where $ P_x (X_1 = z) = p(X_1 = z|X_0 = x) $ ...
0
votes
1answer
31 views

Recurrence of states in a function of a Markov chain

Suppose $X$ is a Markov chain (or process, for that matter) and suppose further $f(X)$ is also a Markov chain. Let $s$ be a recurrent state in $X$. Is there a general way to determine the recurrence ...
0
votes
1answer
367 views

Supply the transition matrix for these (possible) Markov chains

Reading Grimmet, Stirzaker: Probability and Random Processes, which unfortunately doesn't have solutions. Trying to make sure I understand Markov chains. A die is rolled repeatedly. Which of these ...
0
votes
1answer
369 views

Can the transition probabilities of an inhomogeneous Markov chain be written as an exponential?

If $Z_t$ is a homogeneous continuous-time Markov chain with finite state space $E=\{1,\ldots,p\}$, transition matrices $(P(t))$ and intensity matrix $Q$, it holds that $$ P(t) = \exp(tQ), $$ see for ...
0
votes
1answer
26 views

Is there a relationship between the mean matrix and the transition matrix of a multi type branching process?

Let $\mathbf{M}$ be the mean matrix of a multi type branching process $(\mathbf{Z}^{(n)})_{n\geq1}=((Z^{(n)}_1,\ldots,Z^{(n)}_k))_{n\geq1}$. This matrix is defined as follows $$M_{i,j}=\mathbb ...
0
votes
1answer
135 views

Long run behavior of a absorbing markov chain

$$A=\begin{pmatrix}1&0&0&0\\0&1&0&0\\0.2&0&0.6&0.2\\0&0.2&0.2&0.6\end{pmatrix}.$$ In the above matrix how do I calculate the probability that in the ...
0
votes
1answer
142 views

Writing down the transition matrix of a discrete Markov chain

Please consider the following scenario: One person is walking along a discrete circle induced by $\mathbb{Z}/n\mathbb{Z}$ In each round we roll a dice with $w\in\left\{2,\ldots n\right\}$ sides If ...
0
votes
1answer
60 views

Inferring transition rates from continuous markov chain question

A house has 2 rooms of similar sizes with identical air conditioners equipped with thermostats which turn on and off as needed to maintain the temperature in each room to a desired level of 22 ...
0
votes
2answers
131 views

Continuous Markov chains, arriving pairs

I have been trying to sort out this exercise but really stuck on this. Preparing myself for exams and found many exercise on continuous Markov chains but I am always stuck when it comes to transition ...
0
votes
1answer
26 views

A problem on Markov process

Suppose, $\Pi_{\theta}$ be the transition probability function of a Markov chain. For any function $f$ define $$\Pi_{\theta}f_{\theta}(x) = \int f(y,\theta)\Pi_{\theta}(x,dy).$$ Is there any ...
0
votes
1answer
85 views

Stopping time for circular random walk.

This is preparation for an exam I have coming up, not an assignment. Hope you won't mind helping. I've got a random walk, $Y_m, m = 0,1,2, \dots$ on $S = \{0,1,2,\dots,N\}$ with periodic boundaries ...
0
votes
1answer
59 views

Practical differences between a PRNG and a Markov chains

In computer programming you can easily find people describing both a PRNG, like a Mersenne Twister, and a Markov / Stochastic process as "pseudo random generators". I honestly never liked this ...
0
votes
1answer
388 views

Convergence rate of PageRank, the problem when the second eigenvalue is complex

As far as I know the Google matrix used to calculate the PageRank is not symetric, that means that some eigenvalues can be complex, furthermore, we know that the second eigenvalue is equal to the ...
0
votes
1answer
31 views

“Number of passages from the state $i$”: a strange equality.

Consider a homogenenous Markov chain $\{X_n\,:\, n\in \mathbb N\}$ ($0\in\mathbb N$). The state space is $S$ with $|S|\le |\mathbb N|$ and $i\in S$. Consider moreover the function ...
0
votes
1answer
240 views

M/M/1 Queuing Theory Question

Lets say I have packets arrive to a terminal at Poisson rate $\lambda$ per hour and my terminal has an exponential service rate $\mu$ per hour (so the mean service time is $\frac{1}{\mu}$). So this is ...
0
votes
1answer
55 views

Why does markov chains power method converge at the rate of |λ_2/λ_1 |

I'm doing some researches on Markov Chains, and every time I meet this statement, that The rate of convergence of the power method is given by |λ_2/λ_1 |^k→0, when k→inf. And where λ_1 and λ_2 are ...
0
votes
1answer
98 views

M/M/1 queue with probability of new client leaving

I'm looking at a M/M/1 queue system and trying to show that $\{M_t\}_{t\geq}0$, the number of clients in the system, is a birth-death process. In the simplest of cases this is true if $\lambda_i = ...
0
votes
1answer
42 views

Limiting probability of Markov chain(Terminology)

If I am asked to find the limiting probability of a Markov chain, what does this pertain to? $\lim \limits_{n \to \infty} P^n$? Where $P$ is the stepping matrix and $n$ is the number of steps. "What ...
0
votes
1answer
46 views

Markov chains for group decision making

I am new to Markov chains since I am doing my own studying on it recently. I was doing some questions and came across this one that got me stuck. Suppose there are four employees and they need to ...