# Can you predict server response time using Queuing Theory? How?

From this lecture slides (page 8), there is an example of Single Server Queuing System. I set up a simple web app to experiment with, tests results sometimes are as expected, most of the times are not.

Notation:

Rate of arrivals $$\lambda=\frac{Transactions Count}{Duration}$$ Average time it takes a server to service a request is (Average Response Time in seconds) $$Ts$$

Average rate of service would be (requests/second) $$\mu=\frac{1}{Ts}$$

The utilization of the system, which is the ratio between the rate of arrivals and the rate of service is: $$\rho=\frac{\lambda}{\mu}$$ In a queuing system, a customer's time is spent either waiting for service or getting service. $$Tr = Tw + Ts$$ Average time waiting in a M/M/1 queue $$Tw= \frac{\rho}{ \mu(1-\rho) }$$

1. Results seem to be within expected range $$\lambda=121/120 \approx 1$$ $$Ts = 0.199$$ $$\mu = \frac{1}{0.0199}=5.02513$$ $$\rho=\frac{\lambda}{\mu}=\frac{1}{5.02513}=0.199$$ \begin{align} Tr & = Tw + Ts \\ & = \frac{\rho}{ \mu(1-\rho) } + 0.199\\ & = \frac{0.199}{ 5.02513(1-0.199) } + 0.199\\ & = 0.248439 \end{align}
2. Results seem not right $$duration=10*60+5$$ $$count=85$$ $$\lambda = count/duration=0.140496$$ $$Ts = 7.117$$ $$\mu = 1/Ts=0.140509$$ $$Tw=78279.9$$ $$Tr=78287$$

Did I do something wrong or Queuing Theory is not designed for this? Is there a better way to model the system?

I got a suggestion to use different methods for heavy traffic approximation.

Consider an M/G/1 queue with arrival rate $\lambda$.

Let the variance of the service time be $var[S]$. The variance of a data set is calculated by taking the arithmetic mean of the squared differences between each value and the mean value, which is also standard deviation squared.

The exact average waiting time in queue in steady state is given by: $$W=\frac {\rho ^{2}+\lambda ^{2} var[S]}{2\lambda (1-\rho )} \tag{1}\label{1}$$

Let $\rho =\frac {\lambda }{\mu }$ denote the traffic intensity.

The corresponding heavy traffic approximation: $$Wh=\frac{ \lambda ( \frac{1}{\lambda ^{2}} + var[S]) }{ 2(1-\rho ) }$$

The relative error of the heavy traffic approximation $$ero=\frac{ 1-\rho ^{2} }{ \rho ^{2}+\lambda ^{2} var[S] }$$ For the same input from example 2: $$\lambda = 0.140496$$ $$\mu=0.140509$$ $$var[S]=3.10817$$ $$\rho=0.999907$$ $$W=40817.9$$ $$Wh=40825$$ $$ero=0.000174367$$

This is better than the last model but still not quite right.

I have changed the test setting to simulate a Poisson arrival process.

It is looking good for light traffic.$\tag{A}\label{A}$

$$duration=2*60$$ $$count=194$$ $$\lambda=1.61667$$ $$Ts = 0.217$$ $$\mu = 4.60829$$ $$Tw=0.117266$$ $$Tr=0.334266$$ But when arrival rate getting close to service rate, it is not as good. Raw test results from JMeter can be viewed here.$\tag{B}\label{B}$ $$duration=10*60+1$$ $$count=87$$ $$\lambda=0.144759$$ $$Ts = 6.509$$ $$\mu = 0.153633$$ $$Tw=106.171$$ $$Tr=112.68$$

If I use Equation $\eqref{1}$ $$var[S]=0.031684$$ $$\rho=0.942235$$ $$W=53.1252$$ $$Wh=59.8337$$ $$ero=0.126278$$

I have been pointed out that my test workload is not Poisson process. So I thought M/G/k queue may be what I need to use.

The formulas:

Average delay/waiting time: $$Eg= \frac{C^2 +1}{2}Em$$ C is the coefficient of variation of the service time distribution $$C=\frac{std}{m}=\frac{std}{Ts}$$ where std is the standard deviation, m is the mean.

I got the formulas for calculating response time in M/M/c queue from Wikipedia. $$Em =\frac{ C(c,\frac{\lambda}{\mu}) }{c\mu-\lambda} + \frac{1}{\mu}$$ $C(c,\frac{\lambda}{\mu})$ is the probability that an arriving customer is forced to join the queue (all servers are occupied), referred to as Erlang's C formula. $$C(c,\frac{\lambda}{\mu}) = \frac {1}{ 1+(1-\rho)\frac {c!}{ (c\rho )^{c} } \sum _{k=0}^{c-1}{\frac { (c\rho )^{k} }{ k! }} }$$ where $\rho$ is the server utilization, $c$ is the number of servers. $$\rho = \frac{\lambda}{ c*\mu }$$ For Test $\ref{A}$ $$\lambda=1.61667$$ $$\mu = 4.60829$$ $$Ts = 0.217$$ $$std=0.124$$ $$c=1$$ What I got is very good: $$\rho=0.350817$$ $$C(c,\frac{\lambda}{\mu}) = 0.350817$$ $$Em = 0.334266$$ $$Eg = 0.221707$$ But for Test $\ref{B}$, it is still not there. $$\lambda=0.144759$$ $$\mu = 0.153633$$ $$Ts = 6.509$$ $$std = 0.178$$ $$c=1$$ $$\rho=0.942235$$ $$C(c,\frac{\lambda}{\mu}) = 0.942235$$ $$Em = 112.68$$ $$Eg = 56.3821$$

Is this the best I can do?

• I think the $0.0199$ in example 1 should be $0.199$, which affects $Tr$ – Dap Sep 29 '17 at 5:34
• Is it correct that the benchmark delivers a load of $1$, i.e. requests only arrive once the previous request has finished (rather than according to a Poisson process)? So actually no time is spent queueing at all. – Dap Sep 29 '17 at 5:37
• Thanks! I have corrected the error with Ts, it should be 0.199. And I have changed my test to model Poisson arrival process. – Miranda Sep 30 '17 at 7:11
• The interarrival-distribution worksheet chart doesn't look at all like an exponential distribution, which is what I'd expect for a Poisson process. – Dap Sep 30 '17 at 8:01