I'm seeking the correct mathematical tools to analyze/model a real world process for the purpose of optimizing resource allocation. I want to analyze consumed throughput of a AWS DynamoDB table and its associated indexes (up to 5) in order to optimize the provisioned throughput. Throughput is measured in operations per second (I'm simplifying slightly). DynamoDB allows one to provision a throughput and you are charged for the provisioned throughput rather than consumed throughput. You can change the provisioned throughput throughout the course of the day with certain limits.
- Reads and writes are independent variables - That is, throughput is measured and provisioned for reads and writes independently. One table could have high reads and low write while another has the opposite.
- Table and each index have independent reads & writes - That is, the table and each index (up to 5) have their own provisioned read and provisioned write throughput. Reading from a table has no effect on the indexes and reading from an index has no effect on the table. You don't write directly to indexes, rather writing to the table causes writes to the indexes, however a write to the table may or may not cause a write to any given index.
- Consumed throughput is sampled in 5 minute windows - Throughput is sampled in 5 minute windows. So in a given 5 minutes we know how many reads/writes were done. Dividing that by the 300 seconds in 5 minutes gives average reads/writes per second in that 5 minutes (which is the same units we are provisioning in).
- Historical data is available for the last 2 weeks - The system keeps historical data on consumed and provision throughput in 5 minute intervals for the past 2 weeks.
- Changing throughput can take 5 to 10 minutes to take effect - When a change of provisioned throughput is requested it takes 5 to 10 minutes for the provisioned throughput to be changed. The new throughput is not available until the end of that time.
- Changing throughput for the table's reads & writes and all indexes' reads and writes is one action - When you request changes to provisioned reads and writes for the table or any indexes you do so as part of a single action allowing you to change them all to whatever value at once (see next bullet for why this is important).
- Only allowed to decrease provisioned throughput 4 times per calendar UTC day - In the 24 hour period from UTC midnight to the next UTC midnight, you get only 4 decreases to provisioned throughput. However, since changing provisioning for the table reads and writes and all index reads and writes is one operation, you can decrease all of those during each of those 4 allowed decrease operations.
- Reads/write beyond the provisioned amount will fail - if the number of reads or writes in a time period exceed the provisioned amount, all of them beyond the provisioned amount will be throttled i.e. rejected and fail. Throttled operations are reported in 5 minute samples as well, but in different units (operations instead of reads or writes).
Graphing provisioned and consumed reads and writes for a table gives us something like:
I think of this as essentially a continually varying load on the system caused by the number of users and what they are using the system for with volatility caused by the random intervals between user actions and the variability in the number of reads/writes caused by a given action. There may or not be a pattern in the load on the system. For example, some systems experience clear peak usage during specific times of day.
Current approaches to the problem try to setup rules. For example, "if consumed is > 75% of provisioned, scale provisioned by 25%." They will also include a minimum provisioned amount to never go below and rules similar to the scale up one given for when to scale down. This approach is particularly flawed for scaling down because it ignores the limit of 4 scale downs a day and all approaches to addressing that generally make it worse. You can see from the graph above that these approaches can often be quite wasteful depending on the rules that are setup.
I had first thought to try to apply tools from finance to this. However, it seems they aren't quite applicable. In particular, the average rate of return of a stock is a meaningful value on which to based volatility because it is presumed stock values should be stable/increasing. However, this data is expected to frequently be cyclical and doesn't have the tendency to increase indefinitely the way a stock might.
When setting provisioned throughput:
- Avoid failed reads and writes - caused when reads/writes exceed the provisioned amount.
- Minimize the area under the provisioned throughput curve - this minimizes our cost
- Is there a way to measure/estimate the volatility of consumed throughput? - The standard deviation does not seem sufficient here because it includes the variation in usage throughout the day. I am thinking of something like the standard deviation from the moving average, though that doesn't seem quite right either
- How to estimate minimum provisioned throughput? - We need to set minimum provisioned throughput to have a low probability that it won't be exceeded going from zero so fast that we don't have a chance to scale and reads/writes fail.
- Heuristics for when to scale down - While it is possible in retrospect of a day to determine what would have been the optimal points to scale down, we need heuristics to predict when it is appropriate to do so in real time. The challenges here are the limits of 4 scale downs per UTC day and that we must consider reads & writes on the table and all indexes for determining whether now is a good time to scale down.
- How to determine when there is an increase in load despite volatility? - In determining when to scale provisioned throughput it seems like it would be helpful to have a way to determine if load is increasing and if so at what rate in order to project how much to scale. However, it could be that the recent increase is simply random fluctuation and doesn't represent a sustained increase in load. I've heard of things like when the data crosses the moving average. Does that approach have validity? If so, how do you know which moving average to use (i.e. how long of a window for your moving average).
- Other approaches - perhaps I am asking the wrong questions and there is a better way to go about thinking of the problem of optimizing provisioned throughput.