[leo_gateway] Revise the default for large_object.reading_chunked_obj_len

[mocchira]

We've had https://groups.google.com/forum/#!topic/leoproject_leofs/T-NL2ENxRIU recently and the same sort of questions many times in the past so that it might be good to have more conservative default for majority use cases as the current default has been set for high performance cluster.

[mocchira]

Memo: how to decide the new default

• Goal

  • Make the default buffer size as minimum as possible
  • Keep the throughput on the high performance cluster

• Procedure

  • Decrease large_object.reading_chunked_obj_len step by step
  • Benchmark on the high performance cluster to check whether the throughput degradation doen't happen
  • Repeat the above 2 step until the throughput degradation happens

[mocchira]

Get tests done in below cases.

• large_object.reading_chunked_obj_len setting
  • 5MB (current default)
  • 2MB
  • 1MB
  • 512K
  • 256K (stopped improving both throughput/latency here

As large_object.reading_chunked_obj_len gets small, Not only throughput has it improved but latency. Needless to say, the possibility the timeout happens also decreases. that being said, there are no drawbacks at all so we will go with 512K as the new default of large_object.reading_chunked_obj_len.

the result of 512K (the most outperformed one)

• summary.csv

elapsed, window, total, successful, failed
10.000718, 10.000718, 69654, 69654, 0
20.001748, 10.00103, 65901, 65901, 0
30.001704, 9.999956, 55665, 55665, 0
40.001672, 9.999968, 32795, 32795, 0
50.001716, 10.000044, 35531, 35529, 2
60.001772, 10.000056, 49616, 49616, 0
70.001732, 9.99996, 46739, 46739, 0
80.001707, 9.999975, 24455, 24455, 0
90.001715, 10.000008, 26549, 26549, 0
100.001747, 10.000032, 44649, 44649, 0
110.00178, 10.000033, 39796, 39796, 0
120.001741, 9.999961, 36433, 36433, 0
130.001749, 10.000008, 36989, 36988, 1
140.001714, 9.999965, 22703, 22703, 0
150.001745, 10.000031, 37180, 37180, 0
160.001766, 10.000021, 36094, 36094, 0
170.001739, 9.999973, 39254, 39254, 0
180.001745, 10.000006, 39350, 39350, 0
190.001735, 9.99999, 40943, 40943, 0
200.001678, 9.999943, 31856, 31856, 0
210.00169, 10.000012, 36644, 36644, 0
220.001703, 10.000013, 44113, 44113, 0
230.001717, 10.000014, 38927, 38927, 0
240.001766, 10.000049, 32039, 32038, 1
250.001768, 10.000002, 0, 0, 0
260.001705, 9.999937, 39724, 39724, 0
270.001732, 10.000027, 43361, 43361, 0
280.001714, 9.999982, 42905, 42904, 1
290.001734, 10.00002, 45707, 45707, 0
300.001714, 9.99998, 42769, 42769, 0
310.001869, 10.000155, 38359, 38359, 0
320.001751, 9.999882, 45995, 45995, 0
330.001683, 9.999932, 39563, 39562, 1
340.001718, 10.000035, 35128, 35128, 0
350.001727, 10.000009, 37605, 37605, 0
360.001743, 10.000016, 42139, 42139, 0
370.001738, 9.999995, 45217, 45217, 0
380.001654, 9.999916, 41118, 41118, 0
390.001762, 10.000108, 42703, 42703, 0
400.001738, 9.999976, 42366, 42366, 0
410.001774, 10.000036, 32932, 32932, 0
420.001714, 9.99994, 23954, 23954, 0
430.001663, 9.999949, 43450, 43450, 0
440.001739, 10.000076, 42833, 42833, 0
450.001679, 9.99994, 40738, 40738, 0
460.001727, 10.000048, 21816, 21816, 0
470.001765, 10.000038, 46254, 46254, 0
480.001738, 9.999973, 40195, 40195, 0
490.001731, 9.999993, 38963, 38963, 0
500.001734, 10.000003, 37441, 37441, 0
510.001738, 10.000004, 37813, 37813, 0
520.001769, 10.000031, 4848, 4848, 0
530.001748, 9.999979, 38951, 38951, 0
540.001764, 10.000016, 39404, 39403, 1
550.001754, 9.99999, 42948, 42948, 0
560.001721, 9.999967, 40181, 40181, 0
570.001724, 10.000003, 22264, 22264, 0
580.001748, 10.000024, 37963, 37963, 0
590.001756, 10.000008, 36819, 36819, 0
600.001718, 9.999962, 35879, 35879, 0
610.001706, 9.999988, 35785, 35785, 0
620.001798, 10.000092, 34770, 34770, 0
630.00172, 9.999922, 35540, 35540, 0
640.001772, 10.000052, 39021, 39021, 0
650.001726, 9.999954, 37586, 37586, 0
660.001718, 9.999992, 39887, 39887, 0
670.001748, 10.00003, 36636, 36636, 0
680.00176, 10.000012, 34189, 34189, 0
690.001757, 9.999997, 21541, 21541, 0
700.001768, 10.000011, 38200, 38199, 1
710.001737, 9.999969, 36501, 36501, 0
720.00174, 10.000003, 39374, 39374, 0
730.001677, 9.999937, 36384, 36384, 0
740.001672, 9.999995, 38261, 38261, 0
750.001723, 10.000051, 22094, 22094, 0
760.001753, 10.00003, 35192, 35190, 2
770.001873, 10.00012, 38821, 38820, 1
780.001722, 9.999849, 32692, 32692, 0
790.001735, 10.000013, 34722, 34722, 0
800.001899, 10.000164, 21512, 21512, 0
810.001853, 9.999954, 9067, 9067, 0
820.001692, 9.999839, 6785, 6785, 0
825.360989, 5.359297, 1260, 1260, 0

• put_latencies.csv

elapsed, window, n, min, mean, median, 95th, 99th, 99_9th, max, errors
10.000718, 10.000718, 69654, 1449, 9237.3, 6614, 22711, 39631, 164844, 296803, 0
20.001748, 10.00103, 65901, 1550, 9686.1, 6520, 25207, 49855, 169817, 338857, 0
30.001704, 9.999956, 55665, 1412, 11169.4, 5751, 29471, 83401, 512331, 1036833, 0
40.001672, 9.999968, 32795, 1316, 19405.6, 5336, 30763, 505842, 3244651, 3347857, 0
50.001716, 10.000044, 35529, 1550, 22824.3, 6188, 26040, 237842, 3244651, 3347857, 2
60.001772, 10.000056, 49616, 1465, 13826.8, 6397, 29933, 217950, 300540, 2556399, 0
70.001732, 9.99996, 46739, 1523, 12989.7, 6455, 27535, 216811, 482644, 773738, 0
80.001707, 9.999975, 24455, 1568, 27697.9, 6482, 165216, 470430, 721799, 1093348, 0
90.001715, 10.000008, 26549, 1543, 23328.3, 6718, 77943, 309117, 1135515, 1235968, 0
100.001747, 10.000032, 44649, 1471, 14720.0, 6906, 44904, 168486, 416323, 544450, 0
110.00178, 10.000033, 39796, 1364, 16142.2, 7010, 52703, 209971, 354680, 449311, 0
120.001741, 9.999961, 36433, 1364, 17720.2, 7102, 64533, 227083, 291593, 465559, 0
130.001749, 10.000008, 36988, 1512, 16449.3, 6850, 59359, 206940, 366749, 476993, 1
140.001714, 9.999965, 22703, 1575, 25762.4, 6778, 57868, 236753, 4211919, 4225916, 0
150.001745, 10.000031, 37180, 1603, 23536.7, 6753, 65008, 232080, 4204874, 4225916, 0
160.001766, 10.000021, 36094, 1506, 17909.0, 6715, 61783, 180814, 1238541, 1367299, 0
170.001739, 9.999973, 39254, 1474, 15901.3, 7048, 50515, 215454, 290396, 437785, 0
180.001745, 10.000006, 39350, 1523, 16206.4, 6534, 56908, 232821, 297199, 437785, 0
190.001735, 9.99999, 40943, 1558, 16178.8, 6735, 59068, 205205, 300179, 498051, 0
200.001678, 9.999943, 31856, 1539, 19312.2, 6759, 51475, 110238, 3026673, 3296069, 0
210.00169, 10.000012, 36644, 1517, 17308.5, 6974, 62081, 221047, 314683, 440049, 0
220.001703, 10.000013, 44113, 1561, 14549.2, 6901, 52163, 101754, 252906, 298459, 0
230.001717, 10.000014, 38927, 1402, 16068.7, 6320, 67589, 123199, 237158, 335643, 0
240.001766, 10.000049, 32038, 1513, 17159.3, 6085, 74338, 131330, 224735, 306418, 1
250.001768, 10.000002, 0, 0, 0, 0, 0, 0, 0, 0, 0
260.001705, 9.999937, 39724, 1514, 35059.0, 6942, 42605, 200723, 13007000, 13155620, 0
270.001732, 10.000027, 43361, 1426, 14262.2, 7265, 48372, 111044, 304712, 428766, 0
280.001714, 9.999982, 42904, 1506, 15135.1, 7216, 50006, 160306, 290723, 440395, 1
290.001734, 10.00002, 45707, 1446, 14121.2, 7006, 48090, 115761, 274618, 396641, 0
300.001714, 9.99998, 42769, 1399, 14497.7, 6914, 45533, 184901, 320250, 439157, 0
310.001869, 10.000155, 38359, 1495, 16384.4, 6972, 45934, 177601, 1493074, 1717911, 0
320.001751, 9.999882, 45995, 1458, 13771.8, 6920, 45612, 102064, 375926, 579616, 0
330.001683, 9.999932, 39562, 1567, 16338.9, 6888, 61845, 195699, 283174, 419454, 1
340.001718, 10.000035, 35128, 1480, 15519.3, 6764, 53246, 199956, 276082, 423480, 0
350.001727, 10.000009, 37605, 1645, 19413.2, 6754, 47982, 155029, 3024168, 3108859, 0
360.001743, 10.000016, 42139, 1575, 15374.6, 6895, 54445, 161887, 299120, 391339, 0
370.001738, 9.999995, 45217, 1410, 14235.5, 6903, 48118, 107604, 345208, 465303, 0
380.001654, 9.999916, 41118, 1527, 15468.8, 6675, 57047, 162148, 314408, 425202, 0
390.001762, 10.000108, 42703, 1525, 14861.6, 6718, 56588, 126717, 262169, 460371, 0
400.001738, 9.999976, 42366, 1528, 14809.4, 6544, 52526, 173917, 297571, 478693, 0
410.001774, 10.000036, 32932, 1548, 18120.0, 6485, 55726, 126777, 2520913, 3040719, 0
420.001714, 9.99994, 23954, 1587, 31478.5, 6265, 65669, 155849, 4044383, 7147753, 0
430.001663, 9.999949, 43450, 1510, 15043.6, 7063, 53034, 133957, 265693, 453366, 0
440.001739, 10.000076, 42833, 1528, 14832.7, 7131, 48052, 150419, 278482, 419038, 0
450.001679, 9.99994, 40738, 1528, 15518.4, 6996, 53866, 181049, 289605, 410102, 0
460.001727, 10.000048, 21816, 1578, 26257.5, 6761, 59670, 247978, 3041819, 3230459, 0
470.001765, 10.000038, 46254, 1559, 14130.2, 7127, 46106, 125871, 268555, 344972, 0
480.001738, 9.999973, 40195, 1534, 15643.2, 6769, 61906, 116430, 258204, 430171, 0
490.001731, 9.999993, 38963, 1522, 16283.7, 6878, 62640, 144528, 266221, 399295, 0
500.001734, 10.000003, 37441, 1594, 17136.8, 6760, 70147, 176670, 264225, 383684, 0
510.001738, 10.000004, 37813, 1564, 17002.7, 6459, 71330, 137460, 309381, 493259, 0
520.001769, 10.000031, 4848, 1564, 51975.7, 5986, 83168, 369334, 5252436, 5346161, 0
530.001748, 9.999979, 38951, 1484, 23317.9, 6608, 66757, 167156, 3412113, 8778019, 0
540.001764, 10.000016, 39403, 1528, 16599.6, 6544, 65382, 165374, 291838, 348546, 1
550.001754, 9.99999, 42948, 1387, 15384.3, 6706, 58239, 134100, 279149, 383656, 0
560.001721, 9.999967, 40181, 1513, 16009.0, 6602, 62749, 162531, 288405, 493808, 0
570.001724, 10.000003, 22264, 1512, 26209.7, 6257, 63203, 172333, 4535863, 4613947, 0
580.001748, 10.000024, 37963, 1512, 16364.9, 6518, 66862, 127060, 280181, 360902, 0
590.001756, 10.000008, 36819, 1494, 17313.5, 6486, 73182, 115117, 198262, 293118, 0
600.001718, 9.999962, 35879, 1508, 18001.1, 6074, 78759, 147106, 249314, 338876, 0
610.001706, 9.999988, 35785, 1610, 17865.0, 6361, 76247, 141512, 259157, 344412, 0
620.001798, 10.000092, 34770, 1480, 17995.8, 6268, 76138, 166894, 264998, 428244, 0
630.00172, 9.999922, 35540, 1565, 18160.7, 6591, 75613, 150223, 291126, 395220, 0
640.001772, 10.000052, 39021, 1545, 16419.1, 6994, 62739, 118112, 377858, 469054, 0
650.001726, 9.999954, 37586, 1603, 17025.0, 6629, 60728, 206469, 456146, 551462, 0
660.001718, 9.999992, 39887, 1536, 15920.8, 6511, 63390, 164940, 293185, 395461, 0
670.001748, 10.00003, 36636, 1520, 17713.2, 6137, 76175, 203969, 321054, 446741, 0
680.00176, 10.000012, 34189, 1445, 18409.7, 6183, 80612, 179261, 272647, 551559, 0
690.001757, 9.999997, 21541, 1495, 28600.7, 5943, 73900, 166022, 4551367, 4619880, 0
700.001768, 10.000011, 38199, 1474, 16336.6, 6019, 73366, 122056, 231076, 301598, 1
710.001737, 9.999969, 36501, 1525, 16991.9, 6202, 73383, 133891, 234000, 286697, 0
720.00174, 10.000003, 39374, 1507, 16392.8, 6427, 69473, 124580, 254972, 313603, 0
730.001677, 9.999937, 36384, 1456, 17225.1, 6593, 73393, 140416, 286495, 390857, 0
740.001672, 9.999995, 38261, 1478, 16571.6, 6449, 68187, 141680, 365486, 545944, 0
750.001723, 10.000051, 22094, 1623, 26503.5, 6368, 68889, 225399, 3972604, 4011633, 0
760.001753, 10.00003, 35190, 1549, 18170.8, 6323, 75559, 228276, 342140, 403518, 2
770.001873, 10.00012, 38820, 1458, 17256.6, 6416, 71232, 158615, 330093, 436418, 1
780.001722, 9.999849, 32692, 1578, 19298.3, 5929, 84885, 171385, 255736, 309485, 0
790.001735, 10.000013, 34722, 1570, 17858.8, 6504, 78151, 124397, 212589, 328848, 0
800.001899, 10.000164, 21512, 1570, 14027.9, 4722, 68514, 110201, 200271, 337276, 0
810.001853, 9.999954, 9067, 1826, 7401.9, 3905, 20196, 85889, 149618, 228054, 0
820.001692, 9.999839, 6785, 1915, 4834.4, 3687, 8325, 18366, 131215, 279764, 0
825.360989, 5.359297, 1260, 1672, 4179.0, 3355, 7404, 12617, 99031, 279764, 0

[mocchira]

5M

2M

1M

512K

256K

[yosukehara]

As large_object.reading_chunked_obj_len gets small, Not only throughput has it improved but latency. Needless to say, the possibility the timeout happens also decreases. that being said, there are no drawbacks at all so we will go with 512K as the new default of large_object.reading_chunked_obj_len.

We benchmarked the latest of LeoFS, v1.4.2-dev to find the most suitable configuration of large_object.reading_chunked_obj_len setting, and published its report on leo-project/notes. Then we reached the consensus that the suitable value is 512 KB.