Block Storage: Quality of Service
We use Quality of Service to ensure that all our users get predictable performance.
This is enforced on the hypervisor level (using the mechanisms available in libvirt).
| Operation | per-GB-value (burst) | Minimum (burst) | Maximum (burst) |
|---|---|---|---|
| Read IOPS | 2 (50) | 100 (2500) | 5000 (10000) |
| Write IOPS | 10 (50) | 500 (2500) | 2500 (10000) |
The burst length is 60 seconds.
The leaky-bucket-algorithm
The IO budget is calculated using the leaky bucket algorithm. The minimum/maximum/burst-values are set based on the volume capacity. Let's use a volume size of 200G as an example.
Let's use a volume size of 200G as an example.
- Read IOPS per-GB-value = 2, times 200G, means a minumum read IOPS value of 400.
- Read IOPS per-GB-value burst = 50, times 200G, means a burst read IOPS value of 10000.
- Write IOPS per-GB-value = 10, times 200G, means a minimum write IOPS value of 2000.
- Write IOPS per-GB-value burst = 50, times 200G, means a minimum write IOPS value of 10000.
The base budget is calculated by takeing the burst IOPS values, and multiplying them by the burst length (60 seconds).
- Read base budget is 10000 * 60 = 600000
- Write base budget is 10000 * 60 = 600000
The balance is capped at this value, and can't exceed it.
The volume can perform burst IO with this budget, until the balance is depleted.
Every second, the balance is increased by the read/write IOPS calculated earlier (read 400 iops/write 500 iops).
If a server is using its full IO capacity (10000/10000) for 60 seconds against this volume, the balance will be:
- Read: 24000
- Write: 30000
If this continues, the server will soon experience to be throttled on IO level, and be efficiently capped at its minimal capacity (read 400 iops/write 500 iops).
A volume that stays within the minimal capacity on average, will not experience to be throttled.
Would you like to calculate the budget with a different volume size than 200G? Try the slider!
- Read ops: 400 (burst 10000)
- Write ops: 2000 (burst 10000)
Strategies to stay within the budget
One way to reduce the amount of reads can be to increase the amount of memory of the server.
For example, if the server is hosting a database with a large dataset, and the "hot" dataset does not fit in memory, this will usually cause the server to have to frequently re-read data from the disk.
Depending on your workloads, different database tuning can also be done to ensure that writes are done in larger continuous block, thus reducing the amount of IOPS required for the data to be written.