# Tag Info

Let $\phi(\lambda)=\ln\mathbb{E}[e^{\lambda \xi_1}]$ assuming it exists for $\lambda >0$ And $\phi_n(\lambda)$ defined as $\phi_{1}(\lambda)=\phi(\lambda)$ and $\phi_{n+1}(\lambda)=\phi(\lambda + \phi_{n}(\lambda))$ Then: $$\mathbb{E}\left[e^{\lambda \sum_{i=1}^nW_i}\right]=e^{\phi_n(\lambda)}$$ thus, $$\mathbb{E}\left[e^{\lambda \sum_{i=1}^{\infty}... 2 This is not a whole book, but I recommend the chapter on controlled Markov chains in Kushner, Harold, and Paul G. Dupuis. Numerical methods for stochastic control problems in continuous time. Vol. 24. Springer Science & Business Media, 2013. Another option is Puterman, Martin L. Markov decision processes: discrete stochastic dynamic programming. John ... 2 A. This is pretty straight forward. Imagine T_k as the k-th hitting time of the state j, or the time when the Markov chain hits/reaches state j for the k-th time. Y_n is the time between n-1 and n-th hits. So Y_n=i means all the i-1 states between the n-1-th and n-th hits did not hit the state j, which implies all the states of X ... 1 The notation seems to change somewhat between arxiv versions and journal version. My guess is that the "1/2" is a typo. I agree that example should have been more clearly explained. The idea is to compare to an optimal randomized stationary algorithm, which is one that makes decisions without using queue backlog. It would have been nice for the paper to ... 1 Note that Math1000 observes that exact expressions for moment generating functions of the steady state queue process are available in the special case K=1 (which is what your original question used) and putting the \epsilon_n outside the max. You can also get similar expressions for exact mean and (I think) moment generating functions when the \... 1 Your recurrence for X_{n+1} is a bit off; it should be$$X_{n+1} = (X_n-1)^+ + \varepsilon_{n+1}\tag 1 (where a^+:=\max\{a,0\}). Let a_k=\mathbb P(\varepsilon_1=k), then the transition probabilities are given by \begin{align} P_{ij} &= \mathbb P(X_{n+1}=j\mid X_n=i)\\ &= \mathbb P((i-1)^++\varepsilon_1=j)\\ &= \mathbb P(\varepsilon_1 = ... 1 Assuming a steady state, the long term rate of increase from the arrival variables must be equal to the rate of decrease from subtracting by Q, so: E[X] = QP[C_t \geq Q] \implies P[C_t \geq Q] = \frac{E[X]}{Q} $$You will get such a steady state if E[X] < Q. More formally, define the indicator function 1\{C_t\geq Q\}= 1 if C_t\geq Q, and 0 ... 1 HINT Markov process has future independent of its past. Formally if \mathcal{F}_{s \le t} is the sigma-algebra generated by (X_s)_{s \le t} then$$ \mathbb{P}\left[X_{t+\epsilon} \in A| \mathcal{F}_{s \le t}\right] = \mathbb{P}\left[X_{t+\epsilon} \in A| \mathcal{F}_t \right]  for any event $A$. Can you apply the definitions of the Wiener process to ...
Your formula for the infinitesimal generator of $X$ needs to be adjusted slightly. It should be $Nf(x) ={1\over \sqrt{2\pi}}\int_{\Bbb R} e^{-(x-y)^2/2}[f(y)-f(x)]\,dy$. The most direct way to proceed is to construct a cadlag process with $(e^{tN})_{t\ge 0}$ as transition semigroup. The process will be a compound Poisson process with the standard normal ...