==========================================================================================

[hashd-params result] 2021-06-22 17:23:03 - 17:43:47

System info: kernel="5.6.13-0_fbk16_5756_gdcbe47195163"
             nr_cpus=36 memory=63.9G swap=32.0G swappiness=60 zswap
             mem_profile=16 (avail=57.4G share=12.0G target=11.0G)
             passive=io

IO info: dev=nvme0n1(259:0) model="WDC CL SN720 SDAQNTW-512G-1020" size=477G
         iosched=mq-deadline wbt=off iocost=off other=off

Params: log_bps=1.0M

Result: hash_size=1.2M rps_max=1029 mem_actual=16.1G chunk_pages=25

==========================================================================================

[protection result] "iocost-off" 2021-06-22 19:13:37 - 19:30:25

System info: kernel="5.6.13-0_fbk16_5756_gdcbe47195163"
             nr_cpus=36 memory=63.9G swap=32.0G swappiness=60
             mem_profile=16 (avail=58.3G share=12.0G target=11.0G)
             passive=io

IO info: dev=nvme0n1(259:0) model="WDC CL SN720 SDAQNTW-512G-1020" size=477G
         iosched=mq-deadline wbt=off iocost=off other=off

hashd params: hash_size=1.2M rps_max=1029 mem_actual=16.1G chunk_pages=25


Scenario 1/2 - Memory Hog
=========================

Params: loops=2 load=1 speed=2x

Info: baseline_rps=1029.00 baseline_lat=69.6m:1.7m vrate=0.00:0.00
      io_usage=0.0 io_unused=0.0 hog_io_usage=0.0 hog_io_loss=0.0
      hog_bytes=53.1G hog_lost_bytes=333G

IO Latency Distribution:

READ      min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u 35.0u 55.0u 65.0u  105u  865u  865u 35.0u 25.6u  59.1u
p01      5.0u 35.0u 65.0u 75.0u 95.0u  115u  335u  1.5m  1.5m 65.0u 70.7u   131u
p05      5.0u 75.0u 85.0u  105u  115u  195u  1.5m  3.5m  3.5m 95.0u  125u   299u
p10      5.0u 85.0u 95.0u  115u  165u  285u  2.5m  6.5m  6.5m  105u  188u   589u
p25     75.0u  115u  135u  185u  395u  1.5m  7.5m 19.5m 19.5m  165u  451u   1.8m
p50     95.0u  165u  225u  505u  1.5m  4.5m 24.5m 49.5m 49.5m  495u  1.2m   4.9m
p75      115u  195u  595u  1.5m  4.5m 12.5m 59.5m 99.5m 99.5m  1.5m  3.1m  10.5m
p90      145u  375u  1.5m  3.5m 10.5m 30.5m 96.5m  150m  150m  5.5m  6.1m  16.1m
p95      175u  555u  2.5m  5.5m 22.5m 52.5m 97.5m  250m  250m 10.5m  9.1m  22.0m
p99      315u  895u  5.5m 12.5m 45.5m 96.5m 98.5m  350m  350m 50.5m 16.1m  31.2m
p99.9    375u  945u  7.5m 26.5m 87.5m 97.5m 99.5m  450m  450m  150m 22.6m  40.2m
p99.99   395u  965u  8.5m 27.5m 88.5m 98.5m  150m  550m  550m  350m 25.3m  48.1m
p99.999  435u  995u  9.5m 28.5m 94.5m 99.5m  250m  650m  650m  450m 29.4m  60.7m
max      445u  1.5m 10.5m 31.5m 98.5m  150m  350m  750m  750m  750m 37.7m  80.3m

WRITE     min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u 28.5m 73.5m 73.5m  5.0u  657u   6.2m
p01      5.0u  5.0u  5.0u  5.0u  5.0u 15.0u 50.5m 91.5m 91.5m  5.0u  1.3m   8.9m
p05      5.0u  5.0u  5.0u  5.0u 15.0u 75.0u 62.5m 94.5m 94.5m  5.0u  1.8m  10.2m
p10      5.0u  5.0u  5.0u 15.0u 25.0u  2.5m 69.5m 95.5m 95.5m  5.0u  2.2m  11.4m
p25      5.0u  5.0u 25.0u 35.0u  1.5m 25.5m 91.5m 96.5m 96.5m 25.0u  3.6m  14.5m
p50      5.0u 25.0u 55.0u  135u 20.5m 60.5m 95.5m 99.5m 99.5m 55.0u  7.3m  20.1m
p75      5.0u 35.0u  185u 14.5m 47.5m 93.5m 96.5m  150m  150m 12.5m 14.1m  26.8m
p90      5.0u 45.0u  2.5m 33.5m 78.5m 95.5m 97.5m  250m  250m 45.5m 21.7m  32.5m
p95      5.0u 65.0u  6.5m 45.5m 92.5m 96.5m 98.5m  350m  350m 78.5m 26.7m  37.5m
p99      5.0u  105u 11.5m 59.5m 96.5m 97.5m 99.5m  450m  450m  150m 33.0m  44.9m
p99.9    5.0u  125u 13.5m 66.5m 97.5m 98.5m  150m  550m  550m  350m 36.4m  52.3m
p99.99   5.0u  175u 14.5m 68.5m 98.5m 99.5m  250m  650m  650m  550m 40.2m  63.5m
p99.999  5.0u  205u 15.5m 69.5m 99.5m  150m  350m  750m  750m  750m 46.7m  80.3m
max     15.0u  295u 17.5m 70.5m  150m  250m  450m  850m  850m  850m 60.2m   106m

IO Latency: R p50=1.2m:4.9m/49.5m p90=6.1m:16.1m/150m p99=16.1m:31.2m/350m max=37.7m:80.3m/750m
            W p50=7.3m:20.1m/99.5m p90=21.7m:32.5m/250m p99=33.0m:44.9m/450m max=60.2m:106m/850m

Slice resource stat:

[ROOT]        min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%         9.47 13.16 15.57 18.31 31.93 49.31 70.78 82.19 86.68 91.62 92.50 50.29 22.90 
sys%         2.41  6.79 10.44 12.19 15.17 19.71 25.79 32.60 36.39 50.38 56.26 21.25  8.58 
mem         61.9G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 19.5M 
io%             -     -     -     -     -     -     -     -     -     -     -     -     - 
rbps         279M  314M  399M  432M  536M  653M  787M  908M  965M  1.2G  7.5G  697M  501M 
wbps        55.5M 59.0M  183M  205M  239M  275M  317M  343M  367M  414M  1.1G  278M 81.7M 
cpu-some%    0.28  0.41  0.54  0.63  0.88  1.55  4.21  8.77 13.50 26.92 62.13  3.68  5.91 
mem-some%    4.53  5.09 14.80 18.25 24.31 31.64 46.06 64.37 80.16 89.52 192.1 37.76 21.01 
mem-full%    3.29  3.41 13.50 17.34 22.67 29.45 40.23 61.57 75.86 84.09 181.7 34.63 19.96 
io-some%     1.90  2.03  6.21  7.79 11.22 16.17 27.52 47.90 57.52 80.75 149.9 22.59 18.48 
io-full%     1.09  1.20  4.81  5.59  8.08 12.48 20.31 42.77 48.65 65.95 142.5 18.13 16.44 

[WORKLOAD]    min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%         0.43  0.46  0.86  1.25  2.87 27.78 51.67 58.38 65.84 81.70 82.81 29.91 23.28 
sys%         0.09  0.11  0.22  0.27  0.67  2.80  4.85  6.74  7.39  9.79 10.16  3.19  2.47 
mem          9.1G  9.1G  9.1G  9.1G  9.3G  9.5G  9.7G 10.5G 10.6G 11.4G 11.5G  9.6G  537M 
io%             -     -     -     -     -     -     -     -     -     -     -     -     - 
rbps         1.3M  1.5M  3.6M  5.1M  9.6M 51.5M 71.1M  141M  164M  182M  204M 58.0M 49.1M 
wbps         680K  740K  1.3M  1.8M  3.7M 12.0M 20.7M 34.2M 42.7M 62.4M 67.6M 15.0M 13.5M 
cpu-some%    0.02  0.02  0.03  0.04  0.07  0.16  1.09  4.40  6.82 11.66 19.46  1.28  2.55 
mem-some%    0.57  1.05  2.88  3.66  5.22  8.28 14.35 21.12 28.68 41.71 53.33 11.04  8.82 
mem-full%    0.34  0.49  2.50  3.46  4.94  7.64 13.09 19.08 28.36 41.67 51.48 10.30  8.44 
io-some%     0.44  0.97  1.38  1.91  2.85  4.78 10.25 16.93 24.36 41.93 48.91  8.03  8.40 
io-full%     0.30  0.60  1.38  1.85  2.58  4.25  9.58 16.35 23.96 41.93 48.91  7.44  8.11 

[SYSTEM]      min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%         4.12  4.43  6.50  8.51 10.65 14.15 19.89 24.96 31.29 44.48 48.55 16.07  7.56 
sys%         2.27  2.28  5.15  6.38  8.78 12.46 17.24 24.10 28.95 44.04 46.55 14.16  7.58 
mem          1.5G  1.6G  2.6G  2.7G  3.5G  3.7G  4.0G  4.1G  4.1G  4.2G  4.2G  3.6G  554M 
io%             -     -     -     -     -     -     -     -     -     -     -     -     - 
rbps         125M  249M  342M  385M  480M  598M  714M  830M  912M  1.1G  1.1G  606M  176M 
wbps        33.7M 40.5M  174M  194M  222M  257M  295M  328M  345M  356M  435M  257M 56.6M 
cpu-some%    0.08  0.15  0.20  0.24  0.29  0.55  1.85  4.55  6.79 15.97 21.03  1.76  2.90 
mem-some%    1.33  2.26 11.92 14.22 19.82 26.19 38.61 61.14 71.19 85.38 89.20 32.34 18.79 
mem-full%    0.78  1.58 11.50 13.30 18.71 25.21 36.22 59.76 69.47 83.12 86.71 30.83 18.19 
io-some%     0.81  0.89  5.09  6.98 10.14 14.68 24.43 47.49 53.64 75.58 83.45 20.65 16.54 
io-full%     0.76  0.84  4.69  6.40  9.08 13.30 21.48 43.21 49.84 66.62 79.36 18.41 14.66 

Isolation and Request Latency Impact Distributions:

              min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
isol%           0     0  1.07  1.65  2.82 41.40 63.36 71.33 76.68 95.24 97.28 38.52 28.14 
lat-imp%        0     0     0     0     0 11.27 45.26 87.49 306.3 557.1 882.5 46.35 106.4 

Result: isol=38.52:28.14% lat_imp=46.35%:106.4 work_csv=100.0% missing=0.25%

Scenario 2/2 - Memory Hog
=========================

Params: loops=2 load=0.8 speed=2x

Info: baseline_rps=823.20 baseline_lat=61.7m:565u vrate=0.00:0.00
      io_usage=0.0 io_unused=0.0 hog_io_usage=0.0 hog_io_loss=0.0
      hog_bytes=52.6G hog_lost_bytes=367G

IO Latency Distribution:

READ      min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u 35.0u 35.0u 45.0u 65.0u  275u  275u 35.0u 18.2u  22.0u
p01      5.0u 35.0u 45.0u 65.0u 85.0u 95.0u  115u  675u  675u 65.0u 51.9u  44.6u
p05      5.0u 75.0u 85.0u 95.0u  105u  115u  255u  1.5m  1.5m 95.0u 80.1u  89.7u
p10      5.0u 85.0u 95.0u  105u  115u  155u  425u  2.5m  2.5m  105u  101u   147u
p25     85.0u  105u  115u  155u  235u  385u  1.5m  6.5m  6.5m  155u  198u   433u
p50      105u  115u  245u  425u  625u  1.5m  5.5m 18.5m 18.5m  415u  481u   1.2m
p75      115u  165u  625u  1.5m  2.5m  3.5m 12.5m 47.5m 47.5m  1.5m  1.3m   3.2m
p90      145u  285u  1.5m  3.5m  5.5m  7.5m 27.5m 87.5m 87.5m  4.5m  3.1m   6.7m
p95      175u  545u  2.5m  5.5m  8.5m 12.5m 40.5m 98.5m 98.5m  8.5m  4.8m   9.3m
p99      205u  955u  6.5m  9.5m 21.5m 38.5m 80.5m 99.5m 99.5m 35.5m  9.7m  14.8m
p99.9    255u  975u  8.5m 13.5m 42.5m 74.5m 97.5m  150m  150m  150m 14.9m  22.6m
p99.99   305u  985u  9.5m 15.5m 44.5m 81.5m 98.5m  250m  250m  350m 16.7m  26.0m
p99.999  315u  1.5m 10.5m 16.5m 45.5m 84.5m 99.5m  350m  350m  450m 18.0m  29.6m
max      365u  2.5m 11.5m 18.5m 48.5m 89.5m  150m  450m  450m  750m 22.0m  40.7m

WRITE     min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u 15.0u 33.5m 33.5m  5.0u  161u   2.0m
p01      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u 35.0u 34.5m 34.5m  5.0u  192u   2.2m
p05      5.0u  5.0u  5.0u  5.0u  5.0u 15.0u  6.5m 36.5m 36.5m  5.0u  277u   2.6m
p10      5.0u  5.0u 15.0u 15.0u 15.0u 25.0u 15.5m 41.5m 41.5m 15.0u  422u   3.3m
p25      5.0u 15.0u 25.0u 25.0u 45.0u 75.0u 30.5m 54.5m 54.5m 25.0u  889u   5.4m
p50      5.0u 25.0u 35.0u 65.0u  2.5m 13.5m 50.5m 96.5m 96.5m 55.0u  2.4m   9.8m
p75      5.0u 55.0u 85.0u  4.5m 22.5m 36.5m 65.5m 97.5m 97.5m  4.5m  6.4m  15.0m
p90      5.0u 95.0u  2.5m 15.5m 38.5m 64.5m 94.5m 98.5m 98.5m 27.5m 13.1m  21.7m
p95      5.0u  155u  6.5m 23.5m 52.5m 80.5m 96.5m 99.5m 99.5m 50.5m 17.7m  25.5m
p99      5.0u  345u 13.5m 35.5m 75.5m 96.5m 97.5m  150m  150m  150m 24.4m  29.9m
p99.9    5.0u  375u 15.5m 38.5m 84.5m 97.5m 98.5m  250m  250m  250m 27.0m  32.9m
p99.99   5.0u  435u 16.5m 39.5m 86.5m 98.5m 99.5m  350m  350m  450m 28.0m  35.4m
p99.999  5.0u  485u 17.5m 40.5m 88.5m 99.5m  150m  450m  450m  750m 29.8m  40.3m
max     15.0u  615u 18.5m 42.5m 90.5m  150m  250m  550m  550m  850m 36.1m  55.4m

IO Latency: R p50=481u:1.2m/18.5m p90=3.1m:6.7m/87.5m p99=9.7m:14.8m/99.5m max=22.0m:40.7m/450m
            W p50=2.4m:9.8m/96.5m p90=13.1m:21.7m/98.5m p99=24.4m:29.9m/150m max=36.1m:55.4m/550m

Slice resource stat:

[ROOT]        min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%        16.85 17.80 20.55 24.12 31.76 60.78 68.34 74.09 76.27 82.42 88.19 53.10 19.38 
sys%         2.37  3.68 11.21 12.58 16.77 21.47 25.90 34.91 44.30 62.61 67.02 23.01 10.08 
mem         62.0G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 16.4M 
io%             -     -     -     -     -     -     -     -     -     -     -     -     - 
rbps        29.6M 31.1M  372M  416M  491M  604M  715M  833M  901M  1.0G  4.3G  621M  295M 
wbps         8.4M 14.3M  163M  189M  215M  246M  279M  313M  327M  371M  470M  246M 56.1M 
cpu-some%    0.13  0.14  0.50  0.64  0.90  1.39  3.12  6.61  8.50 13.82 17.73  2.60  2.92 
mem-some%    1.22  1.36 15.32 18.50 22.66 29.76 38.80 52.82 71.40 84.30 116.7 33.05 15.99 
mem-full%    1.20  1.34 14.76 17.33 22.13 28.02 35.02 47.13 63.42 77.88 112.1 30.64 14.44 
io-some%     0.34  0.40  5.03  6.40  8.33 12.26 18.63 28.78 39.24 61.55 92.65 16.01 12.41 
io-full%     0.33  0.37  4.47  5.53  7.07  9.53 14.36 22.38 29.96 48.10 88.28 12.31  9.40 

[WORKLOAD]    min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%         3.94  3.97  5.22  5.38  7.99 38.46 49.50 53.39 55.60 57.50 58.40 31.33 19.88 
sys%         0.50  0.65  0.81  0.97  1.52  3.69  5.42  6.51  6.99  9.82 12.79  3.69  2.27 
mem          9.2G  9.2G  9.2G  9.2G  9.3G  9.5G  9.8G 10.1G 10.3G 10.9G 10.9G  9.6G  377M 
io%             -     -     -     -     -     -     -     -     -     -     -     -     - 
rbps         6.7M  8.8M 11.8M 13.1M 16.8M 45.4M 61.6M  108M  117M  143M  185M 50.0M 35.5M 
wbps         716K  1.4M  2.7M  3.8M  4.8M  7.1M 10.4M 22.7M 29.5M 38.4M 42.4M 10.0M  8.1M 
cpu-some%    0.03  0.04  0.05  0.05  0.07  0.24  0.91  1.74  2.62  6.09  8.16  0.70  1.13 
mem-some%    0.24  0.26  3.07  3.66  5.07  7.09  9.71 13.96 17.17 31.43 47.60  8.24  5.53 
mem-full%    0.21  0.25  3.04  3.59  4.93  6.71  9.08 12.49 15.81 27.56 43.95  7.74  5.00 
io-some%     0.27  0.31  1.34  1.66  2.31  3.47  5.75  8.54 11.34 25.76 43.52  4.78  4.75 
io-full%     0.27  0.30  1.33  1.65  2.25  3.25  5.17  7.57 10.02 22.14 39.10  4.38  4.27 

[SYSTEM]      min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%         3.16  3.27  7.07  8.46 11.70 15.83 20.42 29.88 35.21 55.82 58.77 17.43  9.07 
sys%         1.34  1.35  5.72  6.81  9.83 13.32 18.16 26.64 32.70 54.80 57.01 15.29  8.88 
mem          2.2G  2.2G  2.9G  3.1G  3.4G  3.7G  3.9G  4.0G  4.0G  4.0G  4.0G  3.6G  389M 
io%             -     -     -     -     -     -     -     -     -     -     -     -     - 
rbps         7.2M 15.3M  314M  367M  442M  550M  673M  766M  848M  920M  973M  555M  167M 
wbps         6.7M 11.0M  155M  183M  205M  232M  267M  297M  317M  361M  446M  234M 54.6M 
cpu-some%    0.01  0.01  0.18  0.22  0.31  0.55  1.43  4.15  5.57  8.98 11.72  1.36  1.93 
mem-some%    0.02  0.05 11.27 13.36 18.45 24.77 32.86 46.41 58.97 72.99 81.06 27.60 14.65 
mem-full%    0.01  0.05 10.81 12.94 17.95 23.98 31.11 43.53 57.20 69.76 76.47 26.41 13.70 
io-some%     0.01  0.04  3.82  5.13  7.07 11.24 18.28 28.30 35.64 58.00 69.60 14.41 11.15 
io-full%     0.01  0.04  3.73  4.82  6.68 10.05 15.14 22.35 31.32 46.32 56.32 12.47  8.79 

Isolation and Request Latency Impact Distributions:

              min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
isol%           0  6.92  9.48 10.45 14.46 65.72 76.77 85.03 92.44 100.0 100.0 51.53 31.46 
lat-imp%        0     0     0  3.30 11.61 20.30 37.31 63.29 92.84 317.9 380.3 32.99 47.43 

Result: isol=51.53:31.46% lat_imp=32.99%:47.43 work_csv=100.0% missing=0.28%

Memory Hog Summary
==================

Info: baseline_rps=926.10 baseline_lat=65.7m:1.3m vrate=0.00:0.00
      io_usage=0.0 io_unused=0.0 hog_io_usage=0.0 hog_io_loss=0.0
      hog_bytes=106G hog_lost_bytes=701G

IO Latency Distribution:

READ      min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u 35.0u 45.0u 65.0u 95.0u  515u  865u 35.0u 22.2u  45.9u
p01      5.0u 35.0u 55.0u 75.0u 95.0u 95.0u  265u  1.5m  1.5m 65.0u 61.9u   101u
p05      5.0u 75.0u 85.0u 95.0u  115u  135u  795u  2.5m  3.5m 95.0u  104u   228u
p10      5.0u 85.0u 95.0u  115u  135u  205u  1.5m  5.5m  6.5m  105u  147u   444u
p25     75.0u  105u  125u  175u  275u  565u  4.5m 17.5m 19.5m  155u  333u   1.3m
p50     95.0u  145u  235u  445u  1.5m  2.5m 11.5m 49.5m 49.5m  415u  885u   3.7m
p75      115u  185u  615u  1.5m  3.5m  7.5m 33.5m 97.5m 99.5m  1.5m  2.3m   8.1m
p90      145u  325u  1.5m  3.5m  7.5m 19.5m 71.5m 98.5m  150m  4.5m  4.7m  12.7m
p95      175u  555u  2.5m  5.5m 11.5m 31.5m 97.5m 99.5m  250m  8.5m  7.1m  17.4m
p99      205u  925u  5.5m 10.5m 34.5m 67.5m 98.5m  150m  350m 35.5m 13.1m  25.1m
p99.9    255u  965u  7.5m 17.5m 60.5m 96.5m 99.5m  250m  450m  150m 19.0m  33.4m
p99.99   305u  985u  8.5m 19.5m 67.5m 98.5m  150m  350m  550m  350m 21.3m  39.6m
p99.999  315u  995u  9.5m 20.5m 68.5m 99.5m  250m  450m  650m  450m 24.1m  49.1m
max      365u  1.5m 11.5m 22.5m 73.5m  150m  350m  550m  750m  750m 30.4m  65.4m

WRITE     min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  105u 73.5m 73.5m  5.0u  425u   4.7m
p01      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u 30.5m 91.5m 91.5m  5.0u  805u   6.7m
p05      5.0u  5.0u  5.0u  5.0u 15.0u 25.0u 36.5m 94.5m 94.5m  5.0u  1.1m   7.7m
p10      5.0u  5.0u  5.0u 15.0u 25.0u 45.0u 42.5m 95.5m 95.5m 15.0u  1.4m   8.7m
p25      5.0u 15.0u 25.0u 25.0u 55.0u  8.5m 56.5m 96.5m 96.5m 25.0u  2.3m  11.3m
p50      5.0u 25.0u 45.0u 75.0u 10.5m 36.5m 90.5m 97.5m 99.5m 55.0u  5.0m  16.3m
p75      5.0u 45.0u  115u  9.5m 33.5m 63.5m 96.5m 98.5m  150m  4.5m 10.5m  22.4m
p90      5.0u 75.0u  2.5m 23.5m 61.5m 94.5m 97.5m 99.5m  250m 27.5m 17.6m  28.3m
p95      5.0u  115u  6.5m 35.5m 78.5m 95.5m 98.5m  150m  350m 50.5m 22.5m  32.8m
p99      5.0u  195u 12.5m 47.5m 95.5m 96.5m 99.5m  250m  450m  150m 29.0m  38.8m
p99.9    5.0u  225u 15.5m 53.5m 97.5m 97.5m  150m  350m  550m  250m 32.0m  44.5m
p99.99   5.0u  275u 16.5m 54.5m 98.5m 98.5m  250m  450m  650m  450m 34.5m  52.6m
p99.999  5.0u  325u 17.5m 56.5m 99.5m 99.5m  350m  550m  750m  750m 38.8m  65.3m
max     15.0u  345u 18.5m 57.5m  150m  150m  450m  650m  850m  850m 48.9m  87.0m

IO Latency: R p50=885u:3.7m/49.5m p90=4.7m:12.7m/150m p99=13.1m:25.1m/350m max=30.4m:65.4m/750m
            W p50=5.0m:16.3m/99.5m p90=17.6m:28.3m/250m p99=29.0m:38.8m/450m max=48.9m:87.0m/850m

Slice resource stat:

[ROOT]        min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%         9.47 13.91 18.12 20.93 31.93 54.97 69.47 76.93 82.69 89.97 92.50 51.76 21.15 
sys%         2.37  3.71 10.94 12.32 15.96 20.38 25.87 33.93 37.58 52.14 67.02 22.13  9.45 
mem         61.9G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 17.9M 
io%             -     -     -     -     -     -     -     -     -     -     -     -     - 
rbps        29.6M  279M  378M  427M  508M  627M  748M  874M  940M  1.1G  7.5G  665M  437M 
wbps         8.4M 55.5M  170M  195M  224M  258M  299M  331M  350M  392M  1.1G  262M 71.5M 
cpu-some%    0.13  0.28  0.52  0.64  0.90  1.45  3.45  7.85 10.69 19.25 62.13  3.21  5.04 
mem-some%    1.22  4.53 15.24 18.50 23.69 30.38 41.71 60.83 74.55 86.85 192.1 35.51 19.19 
mem-full%    1.20  3.29 13.91 17.34 22.31 28.41 37.55 56.44 68.76 83.57 181.7 32.71 17.69 
io-some%     0.34  1.90  5.85  6.84  9.41 14.31 22.83 38.36 51.13 76.06 149.9 19.30 16.18 
io-full%     0.33  1.09  4.74  5.56  7.32 10.86 16.95 30.14 43.51 61.59 142.5 15.19 13.65 

[WORKLOAD]    min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%         0.43  0.68  1.27  1.43  7.45 34.71 50.12 55.53 58.38 78.23 82.81 30.72 21.53 
sys%         0.09  0.18  0.27  0.42  1.34  3.13  5.08  6.64  7.39  9.79 12.79  3.44  2.38 
mem          9.1G  9.1G  9.1G  9.1G  9.3G  9.5G  9.7G 10.2G 10.5G 11.1G 11.5G  9.6G  461M 
io%             -     -     -     -     -     -     -     -     -     -     -     -     - 
rbps         1.3M  2.2M  5.1M  7.5M 16.2M 47.5M 70.1M  120M  147M  185M  2.7G 65.7M  162M 
wbps         680K  956K  1.7M  2.5M  4.5M  8.6M 16.3M 28.6M 35.6M 54.6M  150M 12.8M 13.2M 
cpu-some%    0.02  0.03  0.04  0.05  0.07  0.22  0.95  2.73  5.26 10.26 42.95  1.11  2.88 
mem-some%    0.24  0.57  3.07  3.66  5.13  7.31 11.28 18.70 24.29 42.33 62.02  9.83  8.05 
mem-full%    0.21  0.34  2.87  3.55  4.94  7.04 10.50 16.85 22.99 41.67 58.37  9.20  7.55 
io-some%     0.27  0.44  1.38  1.80  2.58  4.08  7.03 13.22 21.22 41.93 65.40  6.59  7.68 
io-full%     0.27  0.37  1.38  1.79  2.43  3.75  6.29 12.17 19.16 39.10 63.91  6.09  7.35 

[SYSTEM]      min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%         1.96  3.27  6.67  8.46 10.89 15.17 20.20 26.94 32.01 45.63 58.77 16.69  8.45 
sys%         1.04  1.52  5.15  6.58  9.28 12.81 17.81 24.61 30.40 44.85 57.01 14.66  8.33 
mem          1.5G  1.8G  2.6G  3.0G  3.5G  3.7G  3.9G  4.1G  4.1G  4.1G  4.2G  3.6G  475M 
io%             -     -     -     -     -     -     -     -     -     -     -     -     - 
rbps         7.2M  125M  335M  373M  450M  573M  689M  811M  898M  1.1G  4.8G  598M  296M 
wbps         6.7M 36.7M  159M  185M  212M  245M  280M  317M  335M  361M  900M  246M 64.0M 
cpu-some%    0.01  0.08  0.19  0.24  0.30  0.55  1.69  4.50  6.42 15.26 32.18  1.67  3.00 
mem-some%    0.02  1.33 11.80 13.72 18.90 25.73 36.43 56.56 69.03 85.38 187.2 30.64 19.49 
mem-full%    0.01  0.78 11.32 13.18 18.26 24.70 34.00 55.21 66.27 83.12 180.5 29.24 18.54 
io-some%     0.01  0.81  4.56  5.83  8.32 13.23 20.34 36.62 51.67 75.58 167.8 18.11 17.03 
io-full%     0.01  0.76  4.01  5.44  7.47 11.84 17.99 31.75 46.20 65.03 165.4 15.99 15.26 

Isolation and Request Latency Impact Distributions:

              min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
isol%           0  0.49  1.65  2.24 13.12 50.90 72.52 82.12 88.56 100.0 100.0 45.50 30.72 
lat-imp%        0     0     0     0  4.69 17.00 40.54 75.06 121.9 380.3 882.5 39.42 81.53 

Result: isol=45.50:30.72% lat_imp=39.42%:81.53 work_csv=100.0% missing=0.26%

==========================================================================================

[iocost-params result] 2021-06-22 19:30:28 - 19:38:52

System info: kernel="5.6.13-0_fbk16_5756_gdcbe47195163"
             nr_cpus=36 memory=63.9G swap=32.0G swappiness=60
             mem_profile=16 (avail=58.3G share=12.0G target=11.0G)

IO info: dev=nvme0n1(259:0) model="WDC CL SN720 SDAQNTW-512G-1020" size=477G
         iosched=mq-deadline wbt=off iocost=off other=off

iocost model: rbps=1348822120 rseqiops=235687 rrandiops=218614
              wbps=601694170 wseqiops=133453 wrandiops=69308
iocost QoS: rpct=95.00 rlat=19562 wpct=95.00 wlat=65667 min=60.00 max=100.00

==========================================================================================

[protection result] "iocost-on" 2021-06-22 19:38:53 - 20:02:27

System info: kernel="5.6.13-0_fbk16_5756_gdcbe47195163"
             nr_cpus=36 memory=63.9G swap=32.0G swappiness=60
             mem_profile=16 (avail=58.3G share=12.0G target=11.0G)

IO info: dev=nvme0n1(259:0) model="WDC CL SN720 SDAQNTW-512G-1020" size=477G
         iosched=mq-deadline wbt=off iocost=on other=off
         iocost model: rbps=1348822120 rseqiops=235687 rrandiops=218614
                       wbps=601694170 wseqiops=133453 wrandiops=69308
         iocost QoS: rpct=95.00 rlat=19562 wpct=95.00 wlat=65667 min=60.00 max=100.00

hashd params: hash_size=1.2M rps_max=1029 mem_actual=16.1G chunk_pages=25


Scenario 1/2 - Memory Hog
=========================

Params: loops=2 load=1 speed=2x

Info: baseline_rps=1029.00 baseline_lat=67.8m:1.1m vrate=99.79:0.56
      io_usage=276.8 io_unused=323.0 hog_io_usage=39.6 hog_io_loss=1121.8
      hog_bytes=22.1G hog_lost_bytes=627G

IO Latency Distribution:

READ      min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u 35.0u 45.0u 65.0u 95.0u 95.0u 95.0u 25.0u 23.0u  22.1u
p01      5.0u 55.0u 75.0u 75.0u 75.0u 85.0u  105u  105u  125u 75.0u 59.9u  26.1u
p05      5.0u 75.0u 85.0u 95.0u 95.0u  105u  125u  135u  145u 85.0u 83.1u  19.3u
p10      5.0u 95.0u 95.0u  105u  105u  115u  145u  155u  155u 95.0u 96.9u  15.7u
p25      5.0u  105u  115u  115u  135u  165u  165u  165u  165u  115u  117u  16.0u
p50     75.0u  145u  165u  185u  205u  235u  365u  415u  415u  165u  170u  41.2u
p75     85.0u  245u  325u  445u  555u  635u  1.5m  1.5m  1.5m  385u  366u   192u
p90      105u  525u  725u  995u  1.5m  2.5m  3.5m  6.5m 17.5m  935u  909u   882u
p95      115u  735u  1.5m  1.5m  2.5m  3.5m  6.5m 13.5m 38.5m  1.5m  1.6m   1.8m
p99      225u  1.5m  2.5m  4.5m  5.5m  6.5m 15.5m 90.5m 97.5m  4.5m  3.4m   5.5m
p99.9    265u  2.5m  4.5m  6.5m  8.5m  9.5m 20.5m 94.5m 98.5m 10.5m  5.1m   6.9m
p99.99   325u  3.5m  5.5m  7.5m  9.5m 11.5m 26.5m 96.5m 99.5m 42.5m  6.2m   7.7m
p99.999  335u  4.5m  6.5m  8.5m 10.5m 12.5m 27.5m 98.5m  150m  150m  7.2m   9.0m
max      445u  5.5m  7.5m  9.5m 12.5m 14.5m 29.5m  150m  250m  250m  8.7m  13.0m

WRITE     min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u      0
p01      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u 15.0u 15.0u 15.0u  5.0u  5.1u   1.1u
p05      5.0u  5.0u  5.0u  5.0u  5.0u 15.0u 25.0u 35.0u 35.0u  5.0u  6.0u   3.6u
p10      5.0u  5.0u 15.0u 15.0u 15.0u 25.0u 35.0u 55.0u  215u 15.0u 13.6u   9.8u
p25      5.0u 15.0u 25.0u 25.0u 35.0u 35.0u  145u  345u  3.5m 25.0u 31.1u   132u
p50      5.0u 35.0u 35.0u 55.0u 85.0u  175u  475u 15.5m 31.5m 35.0u  169u   1.5m
p75      5.0u 65.0u  105u  235u  335u  345u 15.5m 72.5m 94.5m 85.0u  841u   6.0m
p90      5.0u  185u  335u  345u  975u 10.5m 58.5m 76.5m 95.5m  345u  2.1m   8.6m
p95      5.0u  315u  345u  365u  4.5m 15.5m 63.5m 92.5m 96.5m  2.5m  2.9m  10.8m
p99      5.0u  345u  385u  725u 12.5m 22.5m 78.5m 94.5m 97.5m 36.5m  4.3m  13.2m
p99.9    5.0u  365u  445u  1.5m 13.5m 24.5m 87.5m 96.5m 98.5m 99.5m  4.7m  13.8m
p99.99   5.0u  375u  485u  2.5m 14.5m 25.5m 93.5m 97.5m  150m  150m  5.2m  14.8m
p99.999  5.0u  395u  645u  3.5m 15.5m 26.5m 95.5m 99.5m  250m  250m  6.0m  16.8m
max     15.0u  465u  2.5m  5.5m 18.5m 28.5m  150m  150m  350m  350m  7.8m  22.7m

IO Latency: R p50=170u:41.2u/415u p90=909u:882u/17.5m p99=3.4m:5.5m/97.5m max=8.7m:13.0m/250m
            W p50=169u:1.5m/31.5m p90=2.1m:8.6m/95.5m p99=4.3m:13.2m/97.5m max=7.8m:22.7m/350m

Slice resource stat:

[ROOT]        min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%        63.05 73.19 80.89 83.62 85.90 87.64 89.37 90.70 91.59 92.95 93.30 87.20  3.49 
sys%         3.16  3.95  4.84  5.39  6.19  7.72  9.65 12.03 13.53 16.34 20.61  8.23  2.71 
mem         61.8G 61.9G 61.9G 62.0G 62.0G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.0G 55.6M 
io%         12.61 17.39 25.08 27.90 34.95 44.29 54.37 66.79 75.68 99.59 719.6 47.27 31.70 
rbps         131M  161M  211M  238M  295M  383M  490M  604M  718M  986M  6.9G  420M  316M 
wbps        19.3M 22.3M 31.0M 35.9M 46.2M 62.0M 83.3M  103M  121M  165M  817M 68.0M 41.7M 
cpu-some%    0.42  1.00  3.03  3.88  5.65  7.72 10.31 13.28 15.63 19.11 164.8  8.48  7.39 
mem-some%    4.34  5.34  7.52  8.81 11.53 18.78 45.90 74.92 85.55 97.62 264.6 31.58 27.82 
mem-full%    2.35  2.92  4.09  4.75  6.27  8.48 12.73 16.11 18.11 23.11 111.7  9.82  6.19 
io-some%     2.19  3.56  4.43  5.59  7.66 13.40 43.68 74.29 85.50 97.43 195.4 27.99 28.02 
io-full%     1.16  1.68  2.13  2.45  3.00  4.31  7.99 11.62 13.46 16.82 57.45  5.91  4.37 

[WORKLOAD]    min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%        56.74 66.31 74.19 76.32 78.58 80.35 82.02 83.63 84.37 85.54 87.59 79.94  3.40 
sys%         1.58  1.68  1.82  1.88  2.01  2.31  2.92  3.69  4.21  5.35  7.61  2.59  0.83 
mem         10.8G 10.8G 11.0G 11.1G 11.2G 11.3G 11.3G 11.4G 11.4G 11.5G 11.7G 11.3G  141M 
io%          2.20  5.25  6.73  7.40  8.41 10.16 12.21 15.33 18.17 24.31 27.40 10.86  3.59 
rbps        22.8M 48.3M 64.3M 68.3M 76.8M 88.9M  106M  142M  191M  256M  297M 99.5M 39.4M 
wbps         576K  2.6M  4.1M  5.3M  7.5M 11.7M 18.1M 25.6M 29.0M 42.4M 53.1M 13.9M  8.5M 
cpu-some%    0.30  0.61  2.00  2.79  4.16  5.54  7.24  8.84  9.88 12.00 14.05  5.72  2.37 
mem-some%    0.20  0.73  1.21  1.49  2.04  2.80  4.37  6.20  7.68 12.23 20.68  3.53  2.41 
mem-full%    0.12  0.44  0.67  0.93  1.25  1.80  2.83  4.10  5.32  8.53 18.60  2.31  1.81 
io-some%     0.18  0.51  0.88  1.13  1.45  1.90  2.75  4.04  5.04  8.83 19.83  2.38  1.81 
io-full%     0.14  0.42  0.64  0.87  1.11  1.43  1.95  2.83  3.63  6.40 17.94  1.75  1.40 

[SYSTEM]      min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%         0.72  0.82  1.28  1.52  2.33  3.51  5.28  7.18  9.17 11.28 13.49  4.06  2.35 
sys%         0.40  0.53  0.84  1.02  1.45  2.15  3.31  4.75  5.76  7.05  9.34  2.57  1.54 
mem          1.4G  1.5G  1.7G  1.7G  1.8G  1.8G  1.9G  2.1G  2.2G  2.4G  2.4G  1.9G  158M 
io%          9.62 10.46 16.59 19.23 25.35 33.71 42.63 51.26 59.52 75.41 83.50 34.96 13.19 
rbps        72.2M 94.6M  134M  158M  221M  303M  396M  486M  567M  767M  892M  321M  139M 
wbps        12.8M 16.0M 21.2M 25.3M 33.5M 45.8M 63.6M 82.3M 97.7M  142M  179M 51.4M 25.0M 
cpu-some%    0.12  0.33  0.73  1.12  1.68  2.64  4.02  6.12  7.88 10.71 13.68  3.23  2.22 
mem-some%    2.61  4.49  5.93  7.06 11.17 20.92 61.91 88.90 94.33 98.51 146.9 36.12 31.07 
mem-full%    2.43  3.80  5.09  6.17  9.34 18.66 55.76 84.29 89.49 95.80 132.8 33.18 29.77 
io-some%     1.82  3.63  5.13  6.37 10.37 19.52 62.59 89.47 95.45 98.89 145.0 35.84 31.73 
io-full%     1.79  3.35  4.83  5.98  9.28 18.22 57.35 84.61 89.94 95.68 130.7 33.35 30.07 

Isolation and Request Latency Impact Distributions:

              min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
isol%       45.77 84.06 87.66 89.30 91.84 94.46 96.50 98.35 99.22 100.0 100.0 93.89  4.61 
lat-imp%        0     0  2.22  3.77  6.35  9.14 13.15 17.37 20.01 29.63 263.0 10.74 12.70 

Result: isol=93.89:4.61% lat_imp=10.74%:12.70 work_csv=46.15% missing=0.27%

Scenario 2/2 - Memory Hog
=========================

Params: loops=2 load=0.8 speed=2x

Info: baseline_rps=823.20 baseline_lat=61.6m:712u vrate=100.00:0.00
      io_usage=237.8 io_unused=362.2 hog_io_usage=45.7 hog_io_loss=1167.5
      hog_bytes=24.3G hog_lost_bytes=621G

IO Latency Distribution:

READ      min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u 35.0u 35.0u 45.0u 65.0u 65.0u 75.0u 25.0u 19.4u  16.8u
p01      5.0u 45.0u 55.0u 75.0u 75.0u 75.0u 85.0u 95.0u 95.0u 65.0u 50.5u  23.7u
p05      5.0u 85.0u 85.0u 85.0u 85.0u 85.0u 95.0u  105u  105u 85.0u 80.5u  16.1u
p10      5.0u 95.0u 95.0u 95.0u 95.0u 95.0u  105u  115u  115u 95.0u 92.5u  13.3u
p25     95.0u  105u  105u  115u  115u  125u  135u  155u  165u  115u  109u   8.8u
p50      105u  125u  145u  175u  205u  235u  315u  385u  405u  165u  157u  42.3u
p75      115u  225u  315u  455u  585u  655u  1.5m  2.5m  3.5m  385u  372u   245u
p90      135u  525u  725u  995u  1.5m  1.5m  3.5m  6.5m 11.5m  935u  921u   763u
p95      165u  725u  1.5m  1.5m  2.5m  3.5m  5.5m 14.5m 20.5m  1.5m  1.6m   1.4m
p99      265u  1.5m  2.5m  4.5m  6.5m  7.5m 10.5m 32.5m 35.5m  4.5m  3.4m   2.9m
p99.9    615u  2.5m  4.5m  6.5m  8.5m 11.5m 20.5m 71.5m 74.5m 10.5m  5.5m   5.4m
p99.99   645u  4.5m  5.5m  7.5m 10.5m 12.5m 23.5m 72.5m 82.5m 36.5m  6.5m   5.8m
p99.999  665u  5.5m  6.5m  9.5m 11.5m 13.5m 24.5m 73.5m 83.5m  150m  7.6m   5.9m
max      765u  6.5m  7.5m 10.5m 12.5m 15.5m 25.5m 74.5m 84.5m  250m  8.9m   6.1m

WRITE     min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u      0
p01      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u 15.0u  5.0u  5.0u   385n
p05      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u 15.0u 25.0u 25.0u  5.0u  5.5u   2.3u
p10      5.0u  5.0u 15.0u 15.0u 15.0u 15.0u 25.0u 65.0u 85.0u 15.0u 11.7u   6.7u
p25      5.0u 15.0u 25.0u 25.0u 25.0u 35.0u 65.0u  145u 13.5m 25.0u 42.6u   520u
p50      5.0u 25.0u 35.0u 45.0u 65.0u 85.0u  315u 25.5m 41.5m 35.0u  146u   1.9m
p75      5.0u 45.0u 55.0u  105u  315u  345u 14.5m 42.5m 47.5m 65.0u  568u   3.5m
p90      5.0u 95.0u  155u  335u  4.5m 14.5m 53.5m 86.5m 93.5m  315u  2.5m   9.5m
p95      5.0u  155u  255u  365u 12.5m 26.5m 68.5m 94.5m 94.5m  1.5m  4.1m  12.3m
p99      5.0u  335u  375u  1.5m 20.5m 39.5m 79.5m 96.5m 96.5m 29.5m  6.1m  15.3m
p99.9    5.0u  375u  505u  2.5m 23.5m 41.5m 81.5m 97.5m 97.5m 99.5m  6.8m  15.9m
p99.99   5.0u  415u  665u  4.5m 24.5m 42.5m 84.5m 98.5m 98.5m  150m  7.3m  16.2m
p99.999  5.0u  455u  1.5m  5.5m 25.5m 43.5m 85.5m 99.5m  150m  250m  8.1m  17.0m
max     15.0u  585u  3.5m  7.5m 27.5m 45.5m 86.5m  150m  250m  350m  9.8m  20.6m

IO Latency: R p50=157u:42.3u/405u p90=921u:763u/11.5m p99=3.4m:2.9m/35.5m max=8.9m:6.1m/84.5m
            W p50=146u:1.9m/41.5m p90=2.5m:9.5m/93.5m p99=6.1m:15.3m/96.5m max=9.8m:20.6m/250m

Slice resource stat:

[ROOT]        min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%        56.24 57.61 58.70 59.46 60.63 62.65 65.31 69.19 72.69 78.19 79.52 63.67  4.32 
sys%         2.77  3.10  3.79  4.31  5.12  6.34  8.59 11.62 13.53 16.70 22.05  7.22  3.09 
mem         61.8G 61.9G 61.9G 62.0G 62.0G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.0G 56.0M 
io%         11.03 14.25 19.89 22.51 28.41 37.22 48.37 60.92 68.69 82.91 366.9 40.18 20.19 
rbps        84.0M  112M  150M  174M  228M  307M  401M  516M  609M  757M  3.3G  335M  185M 
wbps        17.3M 19.7M 28.7M 34.1M 46.2M 58.3M 77.9M  103M  125M  167M  489M 65.6M 33.9M 
cpu-some%    0.14  0.17  0.20  0.22  0.28  0.44  0.96  2.33  3.70  6.17  9.87  0.94  1.27 
mem-some%    4.10  5.16  6.49  7.39  9.70 13.81 28.98 56.45 76.58 95.37 139.7 23.79 22.37 
mem-full%    3.45  4.45  5.26  6.05  7.79 10.38 17.34 27.76 35.42 41.78 93.12 14.00  9.63 
io-some%     1.47  2.91  3.93  4.79  6.47 10.14 25.17 56.34 74.59 95.27 112.5 20.76 22.76 
io-full%     1.30  2.23  3.22  3.47  4.49  6.24 12.62 23.05 28.85 38.50 65.47 10.11  8.74 

[WORKLOAD]    min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%        44.82 50.31 54.36 54.86 55.75 56.44 57.56 58.50 59.13 61.35 62.45 56.58  1.72 
sys%         1.15  1.18  1.23  1.26  1.34  1.51  1.92  2.62  3.15  4.09  5.65  1.76  0.65 
mem         10.4G 10.5G 10.5G 10.6G 10.6G 10.7G 10.8G 10.9G 10.9G 11.0G 11.6G 10.7G  133M 
io%          2.90  3.70  4.22  4.60  5.24  6.28  7.72 10.01 12.18 15.74 16.95  6.91  2.45 
rbps        16.5M 33.0M 39.3M 41.3M 46.5M 54.1M 67.6M 90.1M  121M  155M  173M 62.0M 25.1M 
wbps         292K  1.0M  2.5M  3.3M  5.5M  8.0M 11.2M 14.2M 17.6M 26.0M 37.8M  8.7M  4.9M 
cpu-some%    0.07  0.08  0.09  0.10  0.11  0.16  0.34  0.83  1.31  2.16  3.79  0.34  0.45 
mem-some%    0.42  0.68  0.85  0.98  1.21  1.75  2.63  4.02  5.37 11.07 25.43  2.31  2.07 
mem-full%    0.39  0.64  0.80  0.91  1.13  1.59  2.39  3.64  4.71  9.18 22.15  2.09  1.79 
io-some%     0.40  0.64  0.80  0.92  1.10  1.45  2.08  3.10  3.67  9.51 22.26  1.86  1.65 
io-full%     0.38  0.63  0.77  0.87  1.05  1.35  1.91  2.77  3.25  8.21 19.20  1.70  1.41 

[SYSTEM]      min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%         0.50  0.60  0.93  1.19  1.74  2.97  4.92  7.92 10.95 15.07 17.02  3.91  3.09 
sys%         0.32  0.35  0.58  0.73  1.06  1.72  3.20  5.21  6.69  9.44 12.25  2.46  2.04 
mem          1.5G  2.0G  2.1G  2.2G  2.3G  2.4G  2.5G  2.6G  2.6G  2.7G  2.7G  2.4G  152M 
io%          7.08  9.77 14.59 17.12 22.62 30.37 40.33 50.61 56.79 68.55 82.79 32.44 13.33 
rbps        47.1M 66.4M  102M  131M  178M  261M  351M  445M  531M  611M  831M  278M  128M 
wbps         8.8M 11.6M 21.2M 26.1M 37.1M 50.0M 66.1M 90.4M  107M  145M  175M 55.0M 26.8M 
cpu-some%    0.04  0.04  0.06  0.07  0.11  0.20  0.52  1.48  2.50  4.43  7.00  0.55  0.91 
mem-some%    2.17  4.18  7.20  8.37 11.01 17.72 36.42 72.26 89.68 97.06 118.9 28.43 25.28 
mem-full%    2.13  4.08  6.89  7.97 10.78 16.97 33.24 69.28 85.34 94.77 116.6 27.37 24.42 
io-some%     0.50  2.93  7.04  8.32 11.06 17.30 38.89 74.14 90.70 97.05 118.8 28.75 25.78 
io-full%     0.49  2.88  6.82  8.10 10.77 16.76 35.36 70.56 84.43 94.69 115.2 27.44 24.59 

Isolation and Request Latency Impact Distributions:

              min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
isol%           0     0 92.57 95.48 98.15 100.0 100.0 100.0 100.0 100.0 100.0 96.44 14.85 
lat-imp%        0     0  0.80  1.72  2.93  4.55  6.66 10.02 12.86 22.62 53.89  5.55  4.99 

Result: isol=96.44:14.85% lat_imp=5.55%:4.99 work_csv=39.64% missing=0.15%

Memory Hog Summary
==================

Info: baseline_rps=926.10 baseline_lat=64.7m:899u vrate=99.90:0.39
      io_usage=514.6 io_unused=685.2 hog_io_usage=85.3 hog_io_loss=2289.2
      hog_bytes=46.4G hog_lost_bytes=1.2P

IO Latency Distribution:

READ      min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u 35.0u 45.0u 55.0u 75.0u 95.0u 95.0u 25.0u 21.3u  19.8u
p01      5.0u 45.0u 65.0u 75.0u 75.0u 75.0u  105u  105u  125u 65.0u 55.4u  25.4u
p05      5.0u 85.0u 85.0u 85.0u 95.0u 95.0u  115u  135u  145u 85.0u 81.8u  17.9u
p10      5.0u 95.0u 95.0u 95.0u  105u  105u  135u  155u  155u 95.0u 94.8u  14.7u
p25      5.0u  105u  115u  115u  125u  135u  165u  165u  165u  115u  113u  13.7u
p50     75.0u  135u  155u  185u  205u  235u  335u  415u  415u  165u  164u  42.2u
p75     85.0u  235u  315u  445u  575u  635u  1.5m  1.5m  3.5m  385u  369u   219u
p90      105u  525u  725u  995u  1.5m  2.5m  3.5m 11.5m 17.5m  935u  915u   827u
p95      115u  725u  1.5m  1.5m  2.5m  3.5m  6.5m 20.5m 38.5m  1.5m  1.6m   1.6m
p99      225u  1.5m  2.5m  4.5m  6.5m  7.5m 13.5m 90.5m 97.5m  4.5m  3.4m   4.5m
p99.9    265u  2.5m  4.5m  6.5m  8.5m 10.5m 20.5m 94.5m 98.5m 10.5m  5.3m   6.2m
p99.99   325u  3.5m  5.5m  7.5m  9.5m 11.5m 25.5m 96.5m 99.5m 36.5m  6.4m   6.9m
p99.999  335u  4.5m  6.5m  8.5m 11.5m 13.5m 27.5m 98.5m  150m  150m  7.4m   7.7m
max      445u  5.5m  7.5m 10.5m 12.5m 14.5m 29.5m  150m  250m  250m  8.8m  10.3m

WRITE     min   p25   p50   p75   p90   p95   p99 p99.9   max   cum  mean  stdev
min      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u      0
p01      5.0u  5.0u  5.0u  5.0u  5.0u  5.0u  5.0u 15.0u 15.0u  5.0u  5.1u   843n
p05      5.0u  5.0u  5.0u  5.0u  5.0u 15.0u 15.0u 35.0u 35.0u  5.0u  5.8u   3.0u
p10      5.0u  5.0u 15.0u 15.0u 15.0u 25.0u 25.0u 65.0u  215u 15.0u 12.7u   8.5u
p25      5.0u 15.0u 25.0u 25.0u 35.0u 35.0u  125u  3.5m 13.5m 25.0u 36.7u   373u
p50      5.0u 25.0u 35.0u 45.0u 65.0u  135u  335u 31.5m 41.5m 35.0u  158u   1.7m
p75      5.0u 55.0u 75.0u  175u  325u  345u 15.5m 72.5m 94.5m 65.0u  710u   4.9m
p90      5.0u  115u  265u  345u  2.5m 12.5m 57.5m 93.5m 95.5m  315u  2.3m   9.1m
p95      5.0u  185u  345u  365u  7.5m 22.5m 68.5m 94.5m 96.5m  1.5m  3.5m  11.6m
p99      5.0u  345u  385u  965u 15.5m 32.5m 90.5m 96.5m 97.5m 29.5m  5.1m  14.3m
p99.9    5.0u  375u  465u  2.5m 17.5m 33.5m 91.5m 97.5m 98.5m 99.5m  5.7m  14.9m
p99.99   5.0u  385u  545u  3.5m 18.5m 34.5m 94.5m 98.5m  150m  150m  6.2m  15.5m
p99.999  5.0u  425u  995u  4.5m 19.5m 38.5m 96.5m  150m  250m  250m  7.0m  16.9m
max     15.0u  495u  2.5m  6.5m 22.5m 42.5m 99.5m  250m  350m  350m  8.8m  21.7m

IO Latency: R p50=164u:42.2u/415u p90=915u:827u/17.5m p99=3.4m:4.5m/97.5m max=8.8m:10.3m/250m
            W p50=158u:1.7m/41.5m p90=2.3m:9.1m/95.5m p99=5.1m:14.3m/97.5m max=8.8m:21.7m/350m

Slice resource stat:

[ROOT]        min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%        56.24 57.90 59.46 60.21 62.65 76.84 87.62 89.68 90.70 92.49 93.30 75.42 12.41 
sys%         2.77  3.29  4.13  4.69  5.57  7.02  9.20 11.79 13.53 16.69 22.05  7.72  2.95 
mem         61.8G 61.9G 61.9G 62.0G 62.0G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.0G 55.8M 
io%         11.03 15.99 21.06 24.74 31.51 40.64 52.19 64.11 73.99 92.16 813.0 44.37 34.78 
rbps        84.0M  124M  170M  200M  262M  345M  442M  572M  670M  930M  7.6G  384M  339M 
wbps        17.3M 21.3M 29.8M 35.1M 46.2M 59.8M 80.5M  104M  123M  167M  878M 67.5M 44.6M 
cpu-some%    0.14  0.18  0.22  0.26  0.44  3.35  7.76 10.85 13.35 17.49 164.8  4.75  6.62 
mem-some%    4.10  5.28  6.75  8.09 10.40 15.96 38.26 69.82 82.89 97.90 264.6 27.86 26.22 
mem-full%    2.35  3.37  4.58  5.31  6.84  9.22 14.38 21.93 28.40 40.74 147.8 12.03  9.22 
io-some%     1.47  3.21  4.22  5.09  6.97 11.74 35.05 68.07 82.10 97.81 195.4 24.50 26.14 
io-full%     1.16  1.77  2.38  2.73  3.63  5.33  9.48 16.67 23.63 36.42 91.78  8.08  7.61 

[WORKLOAD]    min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%        44.82 53.19 54.86 55.50 56.44 61.65 80.35 82.44 83.63 85.07 87.59 68.26 11.99 
sys%         1.15  1.20  1.26  1.31  1.51  1.99  2.55  3.33  3.86  4.88  7.61  2.17  0.85 
mem         10.4G 10.5G 10.6G 10.6G 10.7G 10.9G 11.3G 11.4G 11.4G 11.5G 11.7G 11.0G  310M 
io%          2.20  3.78  4.59  4.97  6.15  8.19 10.75 13.59 16.01 22.22 220.4  9.25  8.71 
rbps        16.5M 35.5M 41.2M 44.9M 53.6M 74.2M 94.3M  125M  155M  252M  1.8G 84.2M 82.9M 
wbps         292K  1.5M  3.1M  4.0M  6.2M  9.4M 14.2M 21.8M 26.9M 40.0M  319M 11.7M 12.0M 
cpu-some%    0.07  0.09  0.10  0.11  0.16  1.54  5.54  7.70  8.88 11.27 127.2  3.16  4.85 
mem-some%    0.20  0.71  0.94  1.09  1.48  2.28  3.55  5.40  7.06 13.07 57.47  3.02  3.15 
mem-full%    0.12  0.52  0.75  0.92  1.18  1.68  2.65  3.89  5.29  9.49 47.47  2.28  2.44 
io-some%     0.18  0.61  0.81  0.95  1.25  1.68  2.42  3.56  4.69 11.20 43.43  2.19  2.33 
io-full%     0.14  0.50  0.73  0.87  1.08  1.38  1.93  2.81  3.56  9.63 37.51  1.79  1.92 

[SYSTEM]      min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
cpu%         0.50  0.74  1.07  1.34  2.01  3.23  5.11  7.57  9.52 13.90 17.02  3.98  2.74 
sys%         0.32  0.43  0.67  0.85  1.22  1.95  3.28  4.86  6.18  8.96 12.25  2.51  1.81 
mem          1.4G  1.6G  1.7G  1.7G  1.8G  2.1G  2.4G  2.5G  2.6G  2.6G  2.7G  2.1G  312M 
io%          7.08 10.29 15.25 18.05 23.97 31.99 41.64 50.95 58.81 75.17 580.9 34.78 26.28 
rbps        47.1M 73.8M  118M  145M  202M  282M  376M  478M  557M  763M  5.9G  310M  265M 
wbps         8.8M 15.6M 21.2M 25.6M 35.2M 48.1M 64.5M 86.1M  106M  150M  657M 54.4M 35.3M 
cpu-some%    0.04  0.05  0.07  0.10  0.20  1.24  2.83  4.78  6.20  9.88 49.45  1.95  2.62 
mem-some%    2.17  4.49  6.36  7.72 11.09 18.52 49.63 84.45 92.73 99.13 517.8 33.30 35.27 
mem-full%    2.13  3.92  5.50  6.91 10.33 17.48 45.36 80.26 89.07 95.97 478.9 31.25 33.69 
io-some%     0.50  3.45  5.92  7.32 10.85 18.01 49.81 85.43 92.84 99.09 499.8 33.29 35.39 
io-full%     0.49  3.32  5.36  7.02 10.33 17.11 46.23 81.15 89.12 96.96 471.7 31.36 33.72 

Isolation and Request Latency Impact Distributions:

              min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
isol%           0     0 88.34 90.57 93.78 97.30 100.0 100.0 100.0 100.0 100.0 95.18 11.06 
lat-imp%        0     0  0.96  2.20  3.79  6.49 10.22 15.63 18.32 29.55 263.0  8.14  9.99 

Result: isol=95.18:11.06% lat_imp=8.14%:9.99 work_csv=42.89% missing=0.21%

