Skip to content

Instantly share code, notes, and snippets.

@hugosenari
Last active September 15, 2023 05:12
Show Gist options
  • Save hugosenari/dbc33d833593e3b21eb83299295580b1 to your computer and use it in GitHub Desktop.
Save hugosenari/dbc33d833593e3b21eb83299295580b1 to your computer and use it in GitHub Desktop.
For smallest op how expensive is another thread vs same thread
count mean std min 25% 50% 75% max
Setup 1000 83 36 54 76 78 83 668
Serial 1000 28 5 22 25 26 30 106
Spwn0000E0 - Epoch 1000 7505 5537 2999 4156 6041 10329 111353
Spwn0001E0 - Epoch 1000 8931 5624 3574 5349 8424 11697 112437
Spwn0002E0 - Epoch 1000 9988 5694 3945 6388 9635 12703 113501
Spwn0003E0 - Epoch 1000 10930 5793 4007 7318 10755 13671 113359
Spwn0004E0 - Epoch 1000 12146 6002 5510 8388 12125 14658 118282
Spwn0005E0 - Epoch 1000 14336 6379 6325 10317 14129 16290 119504
Spwn0006E0 - Epoch 1000 16807 7572 8382 12575 16036 18738 125174
Spwn0007E0 - Epoch 1000 19668 7960 10014 16111 18631 21911 126893
Spwn0008E0 - Epoch 1000 21460 8041 10458 17849 20412 23537 129112
Spwn0009E0 - Epoch 1000 23440 8319 12146 19732 22343 25372 127780
Spwn0010E0 - Epoch 1000 25692 8634 13425 21468 24341 27632 129296
Spwn0011E0 - Epoch 1000 27956 9151 14564 23530 26434 29985 130131
Spwn0012E0 - Epoch 1000 30613 9508 14212 25686 28950 33331 131041
Spwn0013E0 - Epoch 1000 32969 9686 15041 27860 31718 36139 135624
Spwn0014E0 - Epoch 1000 35194 10142 18201 30170 34142 38264 150609
Spwn0015E0 - Epoch 1000 37427 10534 19310 32404 36168 40497 153009
Spwn0016E0 - Epoch 1000 39804 12969 21332 34312 38461 42607 264011
Spwn0017E0 - Epoch 1000 42145 13052 21505 36583 40968 44969 265308
Spwn0018E0 - Epoch 1000 44469 13445 21801 38760 42921 47362 266145
Spwn0019E0 - Epoch 1000 46807 14671 24016 40772 45187 49518 267338
Spwn0020E0 - Epoch 1000 48968 14962 25957 42700 47294 51957 267805
Spwn0021E0 - Epoch 1000 51381 16098 27448 45130 49572 54126 268188
Spwn0022E0 - Epoch 1000 53562 16170 28529 47196 51882 56599 269873
Spwn0023E0 - Epoch 1000 56006 17498 30084 49400 53932 58764 273536
Spwn0024E0 - Epoch 1000 58418 17818 30318 51681 56394 61371 282802
Spwn0025E0 - Epoch 1000 60577 18117 30897 53499 58993 63701 284953
Spwn0026E0 - Epoch 1000 62729 18578 32309 55457 60885 65864 287403
Spwn0027E0 - Epoch 1000 65000 18963 34645 57461 63180 68397 287810
Spwn0028E0 - Epoch 1000 67187 19297 34347 59918 65289 70635 288483
Spwn0029E0 - Epoch 1000 69412 19590 37697 61895 67585 73127 289936
Spwn0030E0 - Epoch 1000 71673 19948 38943 63989 69842 75351 291504
Spwn0031E0 - Epoch 1000 73953 20425 39437 66078 71950 77533 298918
Spwn0032E0 - Epoch 1000 76815 26030 44419 68140 74139 80269 580019
Spwn0033E0 - Epoch 1000 78998 26496 44817 70484 76376 82608 592272
Spwn0034E0 - Epoch 1000 81447 26754 46074 72607 78746 85109 592886
Spwn0035E0 - Epoch 1000 83691 27202 46881 74595 80783 87605 593365
Spwn0036E0 - Epoch 1000 85871 27481 48818 76640 83122 89625 593812
Spwn0037E0 - Epoch 1000 88008 27654 47943 78839 85251 91320 594260
Spwn0038E0 - Epoch 1000 90113 27798 48387 80750 87477 93909 594705
Spwn0039E0 - Epoch 1000 92480 28665 48931 82783 89716 96006 595149
Spwn0040E0 - Epoch 1000 94979 29020 50091 84747 91943 98547 595592
Spwn0041E0 - Epoch 1000 97060 29343 50800 87024 93854 100552 596031
Spwn0042E0 - Epoch 1000 99303 29517 51233 89197 96363 103134 599807
Spwn0043E0 - Epoch 1000 101633 29739 56464 90951 98912 105996 606685
Spwn0044E0 - Epoch 1000 104014 30136 57682 93064 101167 108550 600520
Spwn0045E0 - Epoch 1000 106403 30497 59840 95846 103432 110524 600943
Spwn0046E0 - Epoch 1000 108574 30526 61110 98076 105436 112537 601432
Spwn0047E0 - Epoch 1000 110944 30776 63867 100334 107537 114998 601868
Spwn0048E0 - Epoch 1000 113103 31025 66875 102650 109624 117197 602307
Spwn0049E0 - Epoch 1000 115529 32765 68631 104598 111923 119812 695508
Spwn0050E0 - Epoch 1000 117722 33078 70584 106653 114386 122420 706145
Spwn0051E0 - Epoch 1000 119961 33255 73718 108455 116503 124412 699582
Spwn0052E0 - Epoch 1000 122416 33613 75271 110987 118552 126935 700492
Spwn0053E0 - Epoch 1000 124604 33822 76559 112912 120952 129365 701157
Spwn0054E0 - Epoch 1000 126822 35889 78390 114977 122807 131027 701680
Spwn0055E0 - Epoch 1000 129280 36282 80298 117353 125645 133761 702269
Spwn0056E0 - Epoch 1000 131441 36451 79969 119151 127760 135590 702740
Spwn0057E0 - Epoch 1000 133745 36603 83412 121618 130040 137686 704496
Spwn0058E0 - Epoch 1000 136070 37165 85795 123833 131747 140124 709872
Spwn0059E0 - Epoch 1000 138422 39403 85570 126276 134415 142878 834957
Spwn0060E0 - Epoch 1000 140610 39517 88832 128056 136258 145212 835669
Spwn0061E0 - Epoch 1000 142958 39750 90829 129978 138690 147480 836162
Spwn0062E0 - Epoch 1000 145486 42485 92208 132105 141060 150197 963466
Spwn0063E0 - Epoch 1000 147729 43022 98268 134475 143484 151938 973120
Spwn0064E0 - Epoch 1000 149962 43145 99052 136506 145533 154317 967790
Spwn0065E0 - Epoch 1000 152334 46055 101453 138444 147266 156943 1116024
Spwn0066E0 - Epoch 1000 154722 46465 101619 141115 149731 159207 1116838
Spwn0067E0 - Epoch 1000 156923 46881 104816 142916 151862 161289 1117326
Spwn0068E0 - Epoch 1000 159094 46946 106313 144434 154162 163186 1117801
Spwn0069E0 - Epoch 1000 161517 47053 107964 146727 156640 165832 1118204
Spwn0070E0 - Epoch 1000 163734 47288 112930 149233 158619 168302 1118611
Spwn0071E0 - Epoch 1000 166084 49623 113220 151482 160761 170338 1119034
Spwn0072E0 - Epoch 1000 168341 49621 115435 153468 163481 172640 1119433
Spwn0073E0 - Epoch 1000 170579 49981 115613 156079 165366 174681 1119830
Spwn0074E0 - Epoch 1000 172973 50196 117370 158082 167959 177301 1120535
Spwn0075E0 - Epoch 1000 175172 50242 116358 160175 170287 178808 1122189
Spwn0076E0 - Epoch 1000 177474 50653 117983 161974 171917 181354 1122592
Spwn0077E0 - Epoch 1000 179731 51150 119548 163472 174133 184107 1123928
Spwn0078E0 - Epoch 1000 182117 51205 120400 166194 176438 186499 1125759
Spwn0079E0 - Epoch 1000 184437 51470 120980 168626 178460 188938 1126457
Spwn0080E0 - Epoch 1000 186624 51806 122490 170512 180939 191432 1128416
Spwn0081E0 - Epoch 1000 188801 51993 123058 172749 183203 193661 1132071
Spwn0082E0 - Epoch 1000 191092 52208 124110 175036 185422 195312 1130940
Spwn0083E0 - Epoch 1000 193268 52329 122942 177625 187523 197241 1131466
Spwn0084E0 - Epoch 1000 195453 52821 124251 179544 189887 198889 1131887
Spwn0085E0 - Epoch 1000 197832 55310 126503 181883 191934 201855 1266626
Spwn0086E0 - Epoch 1000 200185 55701 133700 183798 194164 204148 1272367
Spwn0087E0 - Epoch 1000 202600 56380 134319 186065 196571 206368 1277251
Spwn0088E0 - Epoch 1000 205137 59029 134388 187674 198443 208718 1400540
Spwn0089E0 - Epoch 1000 207383 59334 137149 190218 200805 211219 1403020
Spwn0090E0 - Epoch 1000 209774 59594 138333 192408 203043 213610 1401203
Spwn0091E0 - Epoch 1000 212073 59741 141217 194512 205079 215978 1401603
Spwn0092E0 - Epoch 1000 214390 60281 141570 196733 207118 218176 1402290
Spwn0093E0 - Epoch 1000 216792 60343 147359 199012 209543 220267 1402884
Spwn0094E0 - Epoch 1000 219049 60594 147802 200780 212301 223001 1403473
Spwn0095E0 - Epoch 1000 221323 60718 149347 203017 214229 224763 1406716
Spwn0096E0 - Epoch 1000 223653 60879 151383 205277 216642 227478 1408364
Spwn0097E0 - Epoch 1000 225961 61305 153799 207455 219221 230119 1423702
Spwn0098E0 - Epoch 1000 228242 61651 155994 209553 221083 232506 1427891
Spwn0099E0 - Epoch 1000 230317 62095 158523 211683 223195 234722 1430427
Spwn0100E0 - Epoch 1000 232468 62345 159729 213268 224852 237080 1431471
Spwn0101E0 - Epoch 1000 234801 62502 162126 215499 227513 239380 1431265
Spwn0102E0 - Epoch 1000 237119 62674 162502 217799 229834 241483 1437065
Spwn0103E0 - Epoch 1000 239648 62869 164762 220379 232150 244597 1439034
Spwn0104E0 - Epoch 1000 242058 63196 168876 222670 234342 246927 1436782
Spwn0105E0 - Epoch 1000 244330 63567 170139 224547 236689 248698 1437662
Spwn0106E0 - Epoch 1000 246697 63744 171898 226801 239121 251159 1441169
Spwn0107E0 - Epoch 1000 249155 66276 174246 228486 241354 253171 1439792
Spwn0108E0 - Epoch 1000 251392 66429 177154 230511 243236 255642 1440342
Spwn0109E0 - Epoch 1000 253607 66720 180993 233268 245524 257920 1440804
Spwn0110E0 - Epoch 1000 255900 67338 184326 235477 247440 260410 1441291
Spwn0111E0 - Epoch 1000 258428 67591 185641 237658 249280 263754 1443401
Spwn0112E0 - Epoch 1000 260834 68259 187435 239713 251820 265571 1444440
Spwn0113E0 - Epoch 1000 263160 68869 187681 241913 254032 267499 1468505
Spwn0114E0 - Epoch 1000 265524 69168 187296 244469 256470 270004 1468102
Spwn0115E0 - Epoch 1000 268021 69630 192533 246680 258933 272861 1470677
Spwn0116E0 - Epoch 1000 270182 69846 193614 248149 261171 274693 1476189
Spwn0117E0 - Epoch 1000 272426 71253 197990 250100 263607 277022 1543566
Spwn0118E0 - Epoch 1000 274658 71657 200251 252104 265390 279950 1543105
Spwn0119E0 - Epoch 1000 276944 71853 201265 254754 267695 282117 1543808
Spwn0120E0 - Epoch 1000 279254 72661 200921 256574 269947 284186 1548066
Spwn0121E0 - Epoch 1000 281580 72750 202883 259501 272046 285997 1549450
Spwn0122E0 - Epoch 1000 283696 72810 204960 261540 274196 288020 1549074
Spwn0123E0 - Epoch 1000 286171 74065 203430 263432 276294 290400 1549614
Spwn0124E0 - Epoch 1000 288489 74321 205592 265785 278624 292235 1553117
Spwn0125E0 - Epoch 1000 290616 74382 208433 268230 280888 294283 1553899
Spwn0126E0 - Epoch 1000 292516 74639 209900 269689 283222 295895 1553669
Spwn0127E0 - Epoch 1000 294265 75054 210819 271428 284350 298072 1558422
Spwn0001E0 - Spwn0000E1 1000 1398 1585 -2482 390 1129 2030 10476
Spwn0002E0 - Spwn0001E1 1000 1028 1248 -2365 283 741 1724 10000
Spwn0003E0 - Spwn0002E1 1000 913 1242 -1940 221 668 1591 19700
Spwn0004E0 - Spwn0003E1 1000 1187 2259 -2004 169 591 1675 33165
Spwn0005E0 - Spwn0004E1 1000 2162 3502 -3165 315 1138 2306 30343
Spwn0006E0 - Spwn0005E1 1000 2442 5063 -3487 328 1104 2403 114010
Spwn0007E0 - Spwn0006E1 1000 2832 4064 -4449 469 1642 4037 67334
Spwn0008E0 - Spwn0007E1 1000 1763 2294 -2796 415 1257 2364 22449
Spwn0009E0 - Spwn0008E1 1000 1951 3457 -3756 303 1182 2505 64798
Spwn0010E0 - Spwn0009E1 1000 2223 3136 -2140 424 1395 2524 32052
Spwn0011E0 - Spwn0010E1 1000 2235 3116 -4483 394 1329 2685 19160
Spwn0012E0 - Spwn0011E1 1000 2628 3396 -2391 510 1465 3044 19840
Spwn0013E0 - Spwn0012E1 1000 2327 2929 -1933 477 1547 2753 25611
Spwn0014E0 - Spwn0013E1 1000 2194 3178 -2946 386 1211 2637 33654
Spwn0015E0 - Spwn0014E1 1000 2204 2925 -2524 471 1448 2704 25128
Spwn0016E0 - Spwn0015E1 1000 2348 7565 -2051 372 1283 2522 221567
Spwn0017E0 - Spwn0016E1 1000 2313 3000 -2122 456 1378 2802 16530
Spwn0018E0 - Spwn0017E1 1000 2295 3026 -1705 516 1493 2715 30674
Spwn0019E0 - Spwn0018E1 1000 2309 4647 -2494 435 1312 2660 113748
Spwn0020E0 - Spwn0019E1 1000 2132 2850 -2172 429 1368 2569 19971
Spwn0021E0 - Spwn0020E1 1000 2384 3797 -3383 526 1547 2731 74349
Spwn0022E0 - Spwn0021E1 1000 2152 2929 -2042 381 1352 2586 15215
Spwn0023E0 - Spwn0022E1 1000 2415 4648 -2151 482 1368 2799 111592
Spwn0024E0 - Spwn0023E1 1000 2383 3125 -2138 522 1557 2906 29416
Spwn0025E0 - Spwn0024E1 1000 2130 3077 -2058 356 1287 2615 30568
Spwn0026E0 - Spwn0025E1 1000 2123 2827 -2285 482 1404 2657 34464
Spwn0027E0 - Spwn0026E1 1000 2242 3056 -2374 419 1350 2644 26369
Spwn0028E0 - Spwn0027E1 1000 2157 2889 -1906 367 1325 2664 17100
Spwn0029E0 - Spwn0028E1 1000 2196 2727 -2771 462 1439 2696 17176
Spwn0030E0 - Spwn0029E1 1000 2233 2930 -2265 409 1270 2875 15382
Spwn0031E0 - Spwn0030E1 1000 2251 3105 -2283 469 1437 2684 25138
Spwn0032E0 - Spwn0031E1 1000 2833 15366 -2123 468 1368 2730 477838
Spwn0033E0 - Spwn0032E1 1000 2155 2906 -2920 427 1446 2696 26149
Spwn0034E0 - Spwn0033E1 1000 2420 3198 -2259 470 1438 2781 20383
Spwn0035E0 - Spwn0034E1 1000 2215 3366 -2534 372 1256 2647 33870
Spwn0036E0 - Spwn0035E1 1000 2151 2886 -2532 428 1319 2564 24086
Spwn0037E0 - Spwn0036E1 1000 2108 2887 -2063 398 1341 2533 26798
Spwn0038E0 - Spwn0037E1 1000 2076 2755 -2210 412 1259 2610 16640
Spwn0039E0 - Spwn0038E1 1000 2338 5032 -2516 467 1429 2703 131149
Spwn0040E0 - Spwn0039E1 1000 2470 3292 -1979 525 1419 2882 31985
Spwn0041E0 - Spwn0040E1 1000 2052 2840 -2658 430 1351 2541 25587
Spwn0042E0 - Spwn0041E1 1000 2213 2833 -3782 450 1468 2736 16612
Spwn0043E0 - Spwn0042E1 1000 2301 2898 -2081 449 1389 2805 15114
Spwn0044E0 - Spwn0043E1 1000 2353 3211 -6268 470 1420 2783 30138
Spwn0045E0 - Spwn0044E1 1000 2360 3185 -2027 456 1390 2795 24848
Spwn0046E0 - Spwn0045E1 1000 2142 2960 -2759 420 1299 2527 17305
Spwn0047E0 - Spwn0046E1 1000 2339 3107 -2882 462 1425 2724 21350
Spwn0048E0 - Spwn0047E1 1000 2130 3004 -2059 383 1334 2569 31145
Spwn0049E0 - Spwn0048E1 1000 2397 4194 -2328 411 1314 2910 93144
Spwn0050E0 - Spwn0049E1 1000 2165 2820 -2229 458 1334 2665 15612
Spwn0051E0 - Spwn0050E1 1000 2210 3049 -6623 377 1339 2727 20687
Spwn0052E0 - Spwn0051E1 1000 2426 3096 -2489 525 1534 2868 19604
Spwn0053E0 - Spwn0052E1 1000 2159 2784 -2535 467 1409 2715 20384
Spwn0054E0 - Spwn0053E1 1000 2189 7065 -2460 369 1226 2391 207354
Spwn0055E0 - Spwn0054E1 1000 2429 3183 -1658 477 1462 2805 27364
Spwn0056E0 - Spwn0055E1 1000 2132 2872 -2508 421 1300 2497 17965
Spwn0057E0 - Spwn0056E1 1000 2275 3091 -3323 367 1363 2824 28547
Spwn0058E0 - Spwn0057E1 1000 2296 2945 -1972 450 1461 2760 18661
Spwn0059E0 - Spwn0058E1 1000 2323 4877 -2357 395 1304 2761 124909
Spwn0060E0 - Spwn0059E1 1000 2160 2824 -2301 409 1344 2641 15177
Spwn0061E0 - Spwn0060E1 1000 2319 2891 -1820 574 1450 2827 16183
Spwn0062E0 - Spwn0061E1 1000 2499 5488 -2056 438 1429 2740 127249
Spwn0063E0 - Spwn0062E1 1000 2214 2921 -3623 471 1428 2548 23631
Spwn0064E0 - Spwn0063E1 1000 2204 2982 -6002 400 1358 2848 21332
Spwn0065E0 - Spwn0064E1 1000 2343 5538 -2178 384 1275 2661 148135
Spwn0066E0 - Spwn0065E1 1000 2359 3047 -2668 520 1540 2755 28100
Spwn0067E0 - Spwn0066E1 1000 2172 3178 -2165 374 1230 2590 29379
Spwn0068E0 - Spwn0067E1 1000 2143 2973 -2946 383 1314 2615 17766
Spwn0069E0 - Spwn0068E1 1000 2394 3034 -1910 493 1433 2843 17017
Spwn0070E0 - Spwn0069E1 1000 2188 2882 -2131 428 1346 2743 21335
Spwn0071E0 - Spwn0070E1 1000 2321 6754 -4011 412 1351 2760 196393
Spwn0072E0 - Spwn0071E1 1000 2228 2896 -4970 458 1420 2677 15857
Spwn0073E0 - Spwn0072E1 1000 2209 3168 -2574 415 1411 2643 27093
Spwn0074E0 - Spwn0073E1 1000 2364 3020 -1996 513 1450 2801 22023
Spwn0075E0 - Spwn0074E1 1000 2170 3001 -2440 374 1307 2623 16809
Spwn0076E0 - Spwn0075E1 1000 2273 3135 -2078 444 1382 2632 29296
Spwn0077E0 - Spwn0076E1 1000 2228 3105 -2626 499 1390 2530 36267
Spwn0078E0 - Spwn0077E1 1000 2357 3051 -3305 481 1418 2684 18148
Spwn0079E0 - Spwn0078E1 1000 2292 2982 -2866 535 1474 2715 26680
Spwn0080E0 - Spwn0079E1 1000 2158 2932 -1914 400 1312 2540 19513
Spwn0081E0 - Spwn0080E1 1000 2148 2885 -2652 386 1345 2548 17041
Spwn0082E0 - Spwn0081E1 1000 2262 2964 -1480 457 1341 2733 17965
Spwn0083E0 - Spwn0082E1 1000 2147 2973 -9122 410 1324 2668 25984
Spwn0084E0 - Spwn0083E1 1000 2156 3011 -2058 425 1347 2636 32515
Spwn0085E0 - Spwn0084E1 1000 2350 5110 -8924 468 1502 2712 134691
Spwn0086E0 - Spwn0085E1 1000 2323 3420 -2133 414 1389 2797 55950
Spwn0087E0 - Spwn0086E1 1000 2386 4269 -2233 495 1471 2726 98802
Spwn0088E0 - Spwn0087E1 1000 2509 5234 -2365 515 1435 2879 123236
Spwn0089E0 - Spwn0088E1 1000 2217 3164 -2217 381 1286 2637 24562
Spwn0090E0 - Spwn0089E1 1000 2362 2949 -1915 495 1555 2769 20889
Spwn0091E0 - Spwn0090E1 1000 2270 3131 -4741 437 1346 2621 16555
Spwn0092E0 - Spwn0091E1 1000 2288 3283 -4662 440 1351 2685 30276
Spwn0093E0 - Spwn0092E1 1000 2373 2958 -2075 512 1460 2908 16208
Spwn0094E0 - Spwn0093E1 1000 2228 3075 -2089 440 1330 2662 34297
Spwn0095E0 - Spwn0094E1 1000 2246 2873 -2710 461 1402 2746 15207
Spwn0096E0 - Spwn0095E1 1000 2301 3098 -2947 422 1283 2652 19947
Spwn0097E0 - Spwn0096E1 1000 2279 3181 -4228 408 1272 2804 26127
Spwn0098E0 - Spwn0097E1 1000 2252 3062 -4021 477 1431 2655 26206
Spwn0099E0 - Spwn0098E1 1000 2047 2693 -2652 368 1350 2628 17975
Spwn0100E0 - Spwn0099E1 1000 2122 2794 -2233 426 1360 2674 27059
Spwn0101E0 - Spwn0100E1 1000 2304 2924 -3152 513 1542 2809 19755
Spwn0102E0 - Spwn0101E1 1000 2289 2927 -2521 484 1396 2787 16206
Spwn0103E0 - Spwn0102E1 1000 2500 3093 -1623 576 1664 2830 21818
Spwn0104E0 - Spwn0103E1 1000 2381 5985 -4538 431 1396 2656 165533
Spwn0105E0 - Spwn0104E1 1000 2244 4163 -2272 381 1325 2619 97972
Spwn0106E0 - Spwn0105E1 1000 2337 2959 -2445 505 1546 2746 27677
Spwn0107E0 - Spwn0106E1 1000 2430 7031 -2466 428 1362 2687 203910
Spwn0108E0 - Spwn0107E1 1000 2208 3669 -2095 420 1293 2685 78926
Spwn0109E0 - Spwn0108E1 1000 2187 2923 -2677 417 1303 2701 21375
Spwn0110E0 - Spwn0109E1 1000 2264 4454 -1845 412 1298 2613 106954
Spwn0111E0 - Spwn0110E1 1000 2498 3264 -2028 538 1509 2893 32068
Spwn0112E0 - Spwn0111E1 1000 2377 4588 -2672 415 1390 2698 108347
Spwn0113E0 - Spwn0112E1 1000 2297 3157 -1921 440 1352 2694 24011
Spwn0114E0 - Spwn0113E1 1000 2335 3090 -2296 484 1415 2807 17318
Spwn0115E0 - Spwn0114E1 1000 2468 3463 -2025 405 1370 2909 30833
Spwn0116E0 - Spwn0115E1 1000 2132 2764 -4978 420 1401 2622 15073
Spwn0117E0 - Spwn0116E1 1000 2214 3479 -1894 463 1380 2646 67327
Spwn0118E0 - Spwn0117E1 1000 2203 3357 -2636 444 1414 2622 63392
Spwn0119E0 - Spwn0118E1 1000 2258 2964 -1996 462 1364 2714 25992
Spwn0120E0 - Spwn0119E1 1000 2281 3502 -1816 398 1337 2658 55080
Spwn0121E0 - Spwn0120E1 1000 2297 3148 -3261 423 1318 2812 27451
Spwn0122E0 - Spwn0121E1 1000 2087 2704 -2204 451 1362 2538 15795
Spwn0123E0 - Spwn0122E1 1000 2447 7896 -2834 412 1335 2526 232011
Spwn0124E0 - Spwn0123E1 1000 2288 4416 -2244 379 1308 2632 105359
Spwn0125E0 - Spwn0124E1 1000 2099 2708 -2049 372 1246 2733 15058
Spwn0126E0 - Spwn0125E1 1000 1871 2538 -1916 305 990 2487 23211
Spwn0127E0 - Spwn0126E1 1000 1720 2144 -1996 280 956 2465 11829

Malebolgia paralgos: parallel vs serial

Disclaimer

Please don't rely on those numbers since they are too tied to test environment. Instead, run this code on an environment similar to production environment.

Objective

Try to understand what is the optimal value for bulkSize, that means, how many operations (OP) would be better have been performed on same thread than executed in other.

Methodology

Since it depends on the size and the time required to execute the operation, we would test it with the fastest operation (OP) we could measure. Get current time and store it in a variable.

First, we do it twice and get delta, to know how much time our OP takes.

We will collect the time before calling parMap and use it as EPOCH, instead of system EPOCH.

We run 5 operations in 3 threads to make sure we have the following scenarios

  • T0E0: First element in the first thread after EPOCH, initial scenario
  • T0E1: Second element in the first thread after EPOCH, serial scenario
  • T1E0: First element in the second thread after EPOCH, parallel scenario
  • T1E1: Second element in the second thread after EPOCH, parallel serial scenario
  • T2E0: First element in the third thread after EPOCH, odd parallel scenario

Results

Negative values

3.7% of times T1E0 outperforms T0E1, while this is good, average being greater than average OP, means our system got stuck in first thread creation.

We couldn't guarantee that this only happens to the first thread, and the percentage is not expressive, so the next observations are on absolute value |T1E0-T0E1|.

Percentile of |T1E0-T0E1|

See file percentil.csv

~75th percentile of T1E0 completed in 1100ns (0.0011ms) after T0E1, T0 could execute our OP 15 times when OP takes 77ns or 55 times when OP takes 20ns.

~50th percentile of T1E0 completed in 0800ns (0.0008ms) after T0E1, T0 could execute our OP 11 times when OP takes 77ns or 40 times when OP takes 20ns.

Notes

For bulkSize have the following scenarios:

  1. len(OPs)/no new thread, serial monothread
  2. len(OPs)/one new thread, serial offload
  3. len(OPs)/two new threads, parallel minimum
  4. len(OPs) new threads, parallel maximum
  5. Something between maximum and minimum

means we would not use parMap at all, what makes sense for 75th percentile with len(OPs) < 40. So if bulkSize < 40, would be better run in serial monothread. Update in another test, 75th10,000 nanoseconds to start first job in first thread, means serial monothread can do len(OPs) < ~250

means we only care that it won't block current thread, not sure if current implementation can take advantage of this. And we didn't collected data to mesure how fast it could be instead of keep in monothread. Update, in another test, 75th~10,000 nanoseconds is the avarage cost of using 1 thread.

taking as input only makes sense after bulkSize > 40.

has the bottleneck of current processor max threads, 16 in test environment, but some production HPC can have 256 threads, anyway, in and we affirm that would not be efficient too much less than 40 ops/threads.

Respecting CPU limits and efficiency limits would be the largest number between len(OPs)/(40 - (len(OPs) mod 40)) + 40 and len(OPs)/(len(cputhreads) - (len(OPs) mod len(cputhreads))) + len(cputhreads).

Still everything depending on the size of OP, there are faster than getMonoTime and there are slower.

PR 19 Update

As PR18 vs PR 19 shows the cost spliting tasks descreased a lot.

PR 21 Update

In PR 21 and data I could conclude a cost of ~400ms, what is ~50% percentile of PR19.

That reduce my recomendation of ~40x to ~20x

OPs

I did this using cost of call getMonoTime ~25ns but there some values for other operations.

Latency Numbers Every Programer Should Know

I would expect math operations to take ~1ns and RAM operations to take ~100ns (currently less), anything 4 times slower than this may benefit from smaller bulkSize.

Thread OPs

In a different benchmark try mesure the actual cost os thread releated operations.

See the results

4 scenarios:

  • 1º news lock/cond operated N times
  • 2º news lock/cond reused N times with sleep to clear cpu cache
  • 3º news lock/cond reused N/10 times
  • 4º news lock/cond reused N/10 times with sleep to clear cpu cache

Values:

  • case: scneario
  • op: max precision in nanoseconds
  • lock: time required to acquire lock
  • rele: time required to release the lock
  • tryT: time required to tryAcquire lock with success
  • tryF: time required to tryAcquire lock with failure
  • sign: time required to send signal

Flaws:

  • We need a less invasive better way to clear cpu cache, but real scenario could really came after sleep ou long wait.
  • Since is a monothread test, it may be a cpu affinity values

I'm not sure if I could mesure wait or locked acquire

But I can say that with cache operations are cheap even for new ones (4º).

import std/monotimes
import std/times
import std/tasks
import std/strutils
import malebolgia/paralgos
proc now(i: var MonoTime) {.inline.} =
i = getMonoTime()
let sep = "\t"
let runs = 1000
echo [
"OP",
"T0E0",
"T0E1",
"T1E0",
"T1E1",
"T2E0",
"T0E1-T0E0",
"T1E0-T0E1",
].join(sep)
var ops = [getMonoTime(), getMonoTime(), getMonoTime(), getMonoTime(), getMonoTime()]
for i in 0..runs:
let bigbang = getMonoTime()
let epoch = getMonoTime()
parMap ops, 2, now
echo [
$inNanoseconds(epoch - bigbang),
$inNanoseconds(ops[0] - epoch),
$inNanoseconds(ops[1] - epoch),
$inNanoseconds(ops[2] - epoch),
$inNanoseconds(ops[3] - epoch),
$inNanoseconds(ops[4] - epoch),
$inNanoseconds(ops[1] - ops[0]),
$inNanoseconds(ops[3] - ops[1]),
].join(sep)
Time Events Percent Percentil
0000<0100ns 008 00.80 00.80
0100<0200ns 033 03.30 04.10
0200<0300ns 078 07.80 11.90
0300<0400ns 103 10.30 22.20
0400<0500ns 078 07.80 30.00
0500<0600ns 090 09.00 39.00
0600<0700ns 072 07.20 46.20
0700<0800ns 066 06.60 52.80
0800<0900ns 086 08.60 61.40
0900<1000ns 061 06.10 67.50
1000<1100ns 046 04.60 72.10
1100<1200ns 030 03.00 75.10
1200<1300ns 046 04.60 79.70
1300<1400ns 022 02.20 81.90
1400<1500ns 022 02.20 84.10
1500<1600ns 022 02.20 86.30
1600<1700ns 013 01.30 87.60
1700<1800ns 014 01.40 89.00
1800<1900ns 010 01.00 90.00
1900<2000ns 011 01.10 91.10
2000<2100ns 014 01.40 92.50
2100<2200ns 007 00.70 93.20
2200<2300ns 003 00.30 93.50
2300<2400ns 004 00.40 93.90
2400<2500ns 005 00.50 94.40
2500<2600ns 004 00.40 94.80
2600<2700ns 003 00.30 95.10
2700<2800ns 003 00.30 95.40
2800<2900ns 006 00.60 96.00
2900<3000ns 006 00.60 96.60
3000<3100ns 003 00.30 96.90
3100<3200ns 002 00.20 97.10
3200<3300ns 002 00.20 97.30
3300<3400ns 003 00.30 97.60
3400<3500ns 003 00.30 97.90
3500<3600ns 002 00.20 98.10
3700<3800ns 001 00.10 98.20
3800<3900ns 002 00.20 98.40
3900<4000ns 002 00.20 98.60
4100<4200ns 001 00.10 98.70
4500<4600ns 001 00.10 98.80
4800<4900ns 001 00.10 98.90
5000<5100ns 001 00.10 99.00
5200<5300ns 001 00.10 99.10
5300<5400ns 001 00.10 99.20
5400<5500ns 001 00.10 99.30
5500<5600ns 001 00.10 99.40
6200<6300ns 001 00.10 99.50
6300<6400ns 001 00.10 99.60
6700<6800ns 001 00.10 99.70
8100<8200ns 001 00.10 99.80
TIME Percentile PR18 Percentile PR19
> 0000 < 0100ns 00.10 00.10
> 0100 < 0200ns 00.90 00.30
> 0200 < 0300ns 02.40 33.10
> 0300 < 0400ns 07.00 49.10
> 0400 < 0500ns 14.50 57.70
> 0500 < 0600ns 22.00 62.60
> 0600 < 0700ns 31.40 66.70
> 0700 < 0800ns 41.00 73.50
> 0800 < 0900ns 53.30 81.90
> 0900 < 1000ns 64.60 86.50
> 1000 < 1100ns 70.50 89.00
> 1100 < 1200ns 76.10 90.30
> 1200 < 1300ns 80.80 91.40
> 1300 < 1400ns 83.40 92.00
> 1400 < 1500ns 85.70 92.40
> 1500 < 1600ns 87.80 93.10
> 1600 < 1700ns 89.50 94.50
> 1700 < 1800ns 91.20 94.80
> 1800 < 1900ns 92.00 95.40
> 1900 < 2000ns 92.80 95.80
> 2000 < 2100ns 94.20 95.90
> 2100 < 2200ns 94.60 96.20
> 2200 < 2300ns 95.30 96.20
> 2300 < 2400ns 95.80 96.40
> 2400 < 2500ns 96.10 96.40
> 2500 < 2600ns 96.40 96.40
> 2600 < 2700ns 97.00 96.90
> 2700 < 2800ns 97.30 97.10
> 2800 < 2900ns 97.40 97.30
> 2900 < 3000ns 97.60 97.40
> 3000 < 3100ns 97.80 97.80
> 3100 < 3200ns 98.00 98.30
> 3200 < 3300ns 98.10 98.50
> 3300 < 3400ns 98.30 98.60
> 3400 < 3500ns 98.40 98.60
> 3500 < 3600ns 98.60 98.60
> 3600 < 3700ns 98.70 98.70
> 3700 < 3800ns 98.70 98.90
> 3800 < 3900ns 98.70 98.90
> 3900 < 4000ns 98.70 98.90
> 4000 < 4100ns 98.80 98.90
> 4100 < 4200ns 98.80 98.90
OP T0E0 T0E1 T1E0 T1E1 T2E0 T0E1-T0E0 T1E0-T0E1
27 16408 16446 13464 13506 14628 38 -2940
24 9370 9395 6621 6662 8061 25 -2733
20 8423 8449 6015 6041 6889 26 -2408
20 12217 12242 10274 10296 11092 25 -1946
25 8212 8253 6294 6328 7509 41 -1925
22 7093 7126 5249 5277 6188 33 -1849
21 7499 7526 5971 5999 6883 27 -1527
43 12814 12945 11372 11422 13503 131 -1523
21 11226 11253 9820 9842 10762 27 -1411
20 11591 11622 10281 10302 11137 31 -1320
17 5822 5849 4542 4560 5332 27 -1289
18 11620 11645 10364 10389 11091 25 -1256
21 11469 11495 10244 10265 11018 26 -1230
20 5911 5931 4713 4739 5505 20 -1192
19 11815 11843 10666 10692 11658 28 -1151
60 23146 23298 22419 22461 32074 152 -837
29 16809 16849 15994 16024 17875 40 -825
19 11809 11838 11046 11074 11869 29 -764
43 17490 17535 16814 16867 23218 45 -668
20 4688 4715 4155 4176 5628 27 -539
27 14300 14328 13806 13847 16378 28 -481
20 5068 5094 4613 4639 5798 26 -455
20 10352 10378 9943 9961 11138 26 -417
20 10271 10296 9861 9883 11006 25 -413
24 7164 7191 6790 6823 8923 27 -368
44 33923 33965 33526 33602 40109 42 -363
43 27643 27689 27336 27388 31655 46 -301
18 10211 10236 9937 9958 12965 25 -278
20 4982 5007 4719 4744 6700 25 -263
22 121220 121253 121111 121146 121581 33 -107
21 5280 5305 5172 5200 7616 25 -105
61 37741 37786 37622 37687 43875 45 -99
37 14324 14359 14256 14291 16857 35 -68
42 21869 21912 21804 21852 27509 43 -60
21 13078 13106 13032 13052 13448 28 -54
19 9495 9516 9447 9472 9543 21 -44
20 4089 4115 4141 4166 4866 26 51
17 9204 9231 9266 9293 10321 27 62
20 8901 8927 8987 9008 9810 26 81
25 4730 4755 4806 4838 7389 25 83
44 6608 6636 6722 6750 7376 28 114
21 7566 7588 7691 7713 8963 22 125
41 40045 40088 40157 40220 41312 43 132
20 9149 9169 9283 9308 10438 20 139
20 9998 10024 10140 10166 12262 26 142
20 8008 8027 8145 8171 9753 19 144
19 3934 3960 4079 4105 4586 26 145
19 4186 4213 4332 4358 4866 27 145
21 10118 10144 10265 10290 10928 26 146
21 4724 4751 4880 4907 5080 27 156
20 5003 5032 5166 5192 5465 29 160
21 9660 9686 9822 9849 10238 26 163
19 10042 10068 10205 10232 10645 26 164
34 17422 17471 17586 17635 18286 49 164
17 4179 4207 4346 4373 4659 28 166
20 3575 3596 3735 3762 4256 21 166
17 9826 9853 9994 10020 10674 27 167
21 11710 11736 11879 11905 12986 26 169
19 9109 9135 9280 9307 9820 26 172
20 10227 10253 10405 10429 11090 26 176
17 3750 3776 3929 3956 4828 26 180
34 6061 6096 6242 6277 8130 35 181
20 4344 4370 4527 4552 4959 26 182
19 7752 7773 7931 7957 8860 21 184
19 9575 9601 9762 9787 10280 26 186
21 4269 4296 4461 4493 5007 27 197
22 4513 4535 4697 4732 6670 22 197
33 12551 12603 12765 12800 14774 52 197
20 3590 3610 3782 3809 4721 20 199
21 10152 10180 10351 10379 11482 28 199
19 9079 9107 9280 9307 9583 28 200
53 4182 4203 4378 4405 5019 21 202
21 3598 3619 3796 3825 4889 21 206
21 3854 3881 4060 4087 4768 27 206
22 4098 4124 4305 4330 4938 26 206
22 3979 4007 4187 4215 5513 28 208
17 8449 8467 8651 8676 9463 18 209
24 10808 10837 11019 11047 11523 29 210
25 4649 4674 4851 4884 5687 25 210
30 8274 8295 8477 8505 8814 21 210
25 10444 10478 10661 10689 12004 34 211
18 5584 5611 5797 5823 8519 27 212
21 4533 4560 4744 4772 5335 27 212
19 3350 3370 3558 3583 5013 20 213
22 9758 9786 9970 9999 10408 28 213
17 3763 3790 3980 4006 4360 27 216
20 3216 3237 3427 3453 5117 21 216
21 9981 10007 10200 10223 10705 26 216
22 4270 4288 4479 4504 4774 18 216
21 4356 4384 4574 4601 5552 28 217
19 3741 3773 3972 3998 4530 32 225
22 3337 3364 3568 3589 4132 27 225
24 11379 11413 11610 11640 12968 34 227
22 4453 4476 4676 4704 5793 23 228
21 8116 8136 8340 8367 9154 20 231
19 3871 3891 4098 4123 4622 20 232
20 4472 4497 4704 4729 5375 25 232
21 3637 3656 3862 3888 5013 19 232
22 8227 8247 8452 8479 9524 20 232
19 3801 3832 4044 4065 4504 31 233
41 11578 11711 11901 11944 13536 133 233
19 11145 11171 11382 11407 11535 26 236
20 7859 7880 8089 8116 8960 21 236
20 8308 8329 8538 8565 9505 21 236
20 8985 9011 9221 9247 10808 26 236
19 7911 7934 8146 8171 9207 23 237
22 4637 4664 4875 4902 5069 27 238
19 4006 4038 4258 4277 4791 32 239
22 5305 5338 5545 5577 6705 33 239
18 4389 4417 4639 4666 4880 28 249
20 8886 8912 9136 9162 10273 26 250
21 9051 9077 9304 9327 9772 26 250
20 3612 3641 3851 3895 4686 29 254
35 6682 6731 6953 6986 7637 49 255
21 9570 9590 9826 9852 10931 20 262
20 3921 3946 4184 4210 4686 25 264
20 4216 4236 4473 4500 4824 20 264
21 10503 10531 10768 10796 11749 28 265
22 3625 3651 3891 3917 4920 26 266
21 9902 9930 10170 10198 11165 28 268
20 4092 4118 4363 4389 5622 26 271
49 48944 48985 49211 49257 90141 41 272
20 3375 3397 3644 3670 4089 22 273
21 4304 4332 4578 4605 5300 28 273
21 7584 7605 7859 7880 9624 21 275
25 5147 5187 5437 5462 5925 40 275
22 4006 4034 4288 4315 5752 28 281
27 5483 5523 5764 5804 6747 40 281
21 3891 3917 4174 4200 5153 26 283
30 6190 6229 6476 6512 7493 39 283
21 3748 3774 4032 4058 4651 26 284
22 9648 9674 9934 9959 10678 26 285
19 9039 9065 9326 9352 10589 26 287
21 9941 9966 10227 10253 11130 25 287
21 3391 3410 3672 3698 4644 19 288
21 4086 4108 4365 4399 6960 22 291
22 9515 9547 9819 9839 10371 32 292
18 7961 7982 8249 8275 10006 21 293
27 11208 11234 11495 11528 12574 26 294
18 3612 3634 3902 3929 4474 22 295
20 10461 10486 10755 10781 11707 25 295
20 9043 9069 9339 9364 9764 26 295
22 4747 4770 5038 5065 5459 23 295
20 8120 8141 8411 8437 9566 21 296
20 9433 9459 9732 9758 10509 26 299
20 7724 7745 8018 8045 8686 21 300
21 11365 11387 11661 11687 11849 22 300
22 6944 6966 7240 7267 7798 22 301
19 8650 8670 8947 8972 9617 20 302
22 8444 8466 8740 8768 9851 22 302
26 12350 12382 12656 12687 13405 32 305
21 7311 7330 7611 7637 8842 19 307
20 3745 3767 4050 4076 4200 22 309
21 10324 10351 10632 10660 11702 27 309
27 5340 5373 5656 5683 5968 33 310
21 3977 4003 4288 4314 4589 26 311
20 8947 8973 9259 9285 10093 26 312
25 12824 12865 13150 13177 13863 41 312
21 3747 3774 4066 4087 4761 27 313
20 8677 8702 8991 9017 10092 25 315
20 3488 3514 3810 3830 4392 26 316
21 3701 3728 4024 4044 4566 27 316
21 9540 9566 9857 9882 10016 26 316
21 10113 10141 10431 10458 10752 28 317
22 11281 11308 11598 11626 12058 27 318
34 17209 17257 17543 17577 18369 48 320
21 3785 3814 4113 4135 5140 29 321
21 3960 3982 4275 4303 4846 22 321
25 4852 4891 5188 5212 6617 39 321
21 3616 3643 3947 3968 5006 27 325
21 4167 4193 4492 4518 4975 26 325
25 10974 10999 11291 11324 11928 25 325
19 3761 3784 4084 4111 4954 23 327
19 8916 8942 9243 9269 10115 26 327
20 3737 3763 4063 4090 4634 26 327
24 4574 4599 4894 4927 6727 25 328
21 10581 10608 10910 10937 11733 27 329
23 9926 9954 10254 10283 11742 28 329
20 8069 8090 8395 8422 10279 21 332
23 3919 3946 4256 4278 5026 27 332
20 3512 3538 3851 3871 4372 26 333
18 9108 9133 9442 9467 10190 25 334
24 10930 10961 11263 11295 11579 31 334
27 5408 5449 5742 5783 6589 41 334
20 4091 4124 4429 4460 4982 33 336
20 4159 4185 4501 4523 4626 26 338
24 3832 3859 4170 4197 5368 27 338
33 5985 6036 6344 6374 6815 51 338
19 9058 9085 9397 9424 9901 27 339
21 8649 8670 8976 9009 9168 21 339
27 6675 6715 7013 7054 8073 40 339
21 3728 3754 4074 4094 4349 26 340
22 4143 4169 4483 4509 4707 26 340
22 9308 9333 9645 9673 10004 25 340
33 8032 8082 8371 8422 10177 50 340
22 9797 9824 10144 10165 12652 27 341
20 4126 4152 4470 4496 4690 26 344
21 3904 3932 4247 4276 5268 28 344
19 11497 11537 11855 11882 15262 40 345
27 5610 5635 5941 5980 7572 25 345
19 3250 3283 3612 3633 4743 33 350
21 8256 8277 8603 8628 9593 21 351
23 3955 3978 4289 4329 5081 23 351
22 8505 8528 8853 8882 10117 23 354
22 3329 3350 3678 3705 4042 21 355
20 3332 3354 3684 3710 5016 22 356
22 4113 4141 4468 4497 5101 28 356
21 9763 9783 10108 10140 10753 20 357
22 3369 3395 3732 3752 4692 26 357
22 3820 3847 4179 4205 4575 27 358
20 9309 9336 9670 9697 10202 27 361
21 4352 4378 4712 4739 4890 26 361
43 9272 9315 9629 9678 13682 43 363
27 4296 4323 4648 4688 6713 27 365
27 4387 4426 4757 4791 6661 39 365
25 10776 10801 11135 11168 12619 25 367
30 5978 6008 6335 6375 6912 30 367
20 3802 3830 4173 4200 4975 28 370
21 3804 3826 4168 4196 5154 22 370
27 12858 12890 13227 13260 13822 32 370
21 3999 4026 4373 4401 5139 27 375
21 3531 3558 3913 3934 4756 27 376
24 3981 4012 4367 4388 4689 31 376
19 7752 7773 8124 8150 9353 21 377
20 3582 3607 3965 3985 4283 25 378
20 9319 9344 9696 9722 9989 25 378
21 10931 10953 11302 11331 12205 22 378
34 20097 20146 20475 20524 26506 49 378
21 4278 4310 4658 4690 5277 32 380
21 7764 7784 8138 8165 9260 20 381
21 10760 10788 11142 11170 11577 28 382
21 3644 3666 4021 4048 4869 22 382
22 3724 3751 4115 4136 4743 27 385
19 4029 4053 4420 4439 4619 24 386
29 19262 19895 20243 20281 21544 633 386
22 4250 4278 4638 4666 5129 28 388
23 8383 8406 8762 8794 10323 23 388
20 4666 4693 5055 5082 5302 27 389
21 3141 3162 3530 3551 4739 21 389
21 9460 9486 9851 9875 10226 26 389
34 9503 9537 9878 9927 12908 34 390
21 3393 3420 3791 3811 4515 27 391
20 3521 3542 3909 3936 4135 21 394
20 9960 9985 10353 10379 10511 25 394
28 11805 11830 12194 12225 13000 25 395
22 10060 10087 10459 10487 10978 27 400
56 3859 3881 4253 4281 5651 22 400
24 4564 4603 4980 5004 5498 39 401
34 6347 6398 6766 6800 7346 51 402
51 10288 10339 10707 10741 12642 51 402
23 10413 10441 10817 10844 11841 28 403
22 9687 9714 10092 10119 10317 27 405
25 15184 15216 15590 15622 17441 32 406
19 8293 8325 8714 8736 9047 32 411
18 9613 9640 10024 10052 10217 27 412
19 3253 3273 3654 3685 4924 20 412
24 5032 5073 5445 5486 6256 41 413
27 4664 4695 5082 5108 5730 31 413
19 9668 9694 10082 10109 10604 26 415
43 22616 22666 23040 23082 24777 50 416
19 10189 10214 10611 10631 10934 25 417
26 127525 127554 127945 127971 128335 29 417
26 5324 5350 5726 5767 6104 26 417
33 7119 7154 7521 7571 8480 35 417
21 7527 7548 7954 7975 9128 21 427
21 9181 9207 9609 9635 10334 26 428
18 5308 5335 5740 5764 6503 27 429
22 9751 9894 10294 10323 11070 143 429
24 4618 4643 5031 5073 7065 25 430
25 10919 10944 11334 11376 12685 25 432
19 3895 3921 4329 4354 6650 26 433
19 3634 3660 4070 4094 4341 26 434
20 8779 8805 9216 9242 10027 26 437
29 12840 12879 13278 13316 14598 39 437
20 9868 9895 10307 10334 10486 27 439
21 3438 3459 3872 3899 4601 21 440
29 16839 16880 17281 17320 18251 41 440
20 9779 9799 10208 10240 10451 20 441
20 3990 4016 4432 4458 6963 26 442
35 15361 15394 15802 15836 19606 33 442
20 3510 3536 3960 3979 4321 26 443
20 8994 9020 9437 9463 10404 26 443
45 8288 8352 8746 8795 9710 64 443
20 11439 11464 11883 11908 13053 25 444
21 8584 8616 9037 9060 9418 32 444
29 5415 5446 5851 5891 6944 31 445
21 4155 4180 4601 4627 5713 25 447
19 3318 3338 3755 3788 5169 20 450
19 3252 3279 3710 3734 5015 27 455
34 7764 7813 8235 8270 11808 49 457
20 3578 3605 4040 4063 5335 27 458
18 8707 8728 9160 9187 9781 21 459
30 5252 5289 5710 5748 7436 37 459
32 17164 17213 17621 17672 18335 49 459
20 4312 4332 4760 4793 4958 20 461
21 3005 3025 3466 3488 4964 20 463
20 4001 4028 4472 4493 4716 27 465
21 9038 9063 9502 9528 11702 25 465
20 9328 9355 9797 9824 10280 27 469
20 9911 9937 10380 10406 10640 26 469
20 9323 9348 9796 9821 10185 25 473
28 4293 4326 4762 4799 5609 33 473
20 3576 3601 4049 4076 4532 25 475
28 11639 11676 12123 12151 13238 37 475
22 8455 8476 8925 8952 10774 21 476
25 13394 13434 13869 13910 14743 40 476
21 9584 9610 10057 10089 10317 26 479
24 13660 13700 14154 14179 14631 40 479
25 4087 4126 4583 4610 11202 39 484
20 9116 9142 9602 9628 10299 26 486
21 3182 3204 3666 3693 4146 22 489
23 4233 4260 4721 4749 5041 27 489
237 4950 4974 5432 5463 5682 24 489
29 15223 15265 15715 15755 16221 42 490
17 8771 8796 9262 9287 9581 25 491
22 12085 12106 12571 12599 15085 21 493
72 4935 4976 5435 5470 5633 41 494
22 9056 9084 9553 9580 10448 28 496
34 6809 6843 7293 7341 7966 34 498
21 3346 3367 3840 3866 4699 21 499
35 5986 6037 6503 6537 7091 51 500
34 17798 17849 18301 18351 18584 51 502
32 12580 12611 13074 13114 14581 31 503
20 8971 8997 9482 9502 10244 26 505
20 5512 5538 6021 6046 6729 26 508
22 7428 7449 7936 7958 8542 21 509
24 8236 8259 8735 8768 9844 23 509
19 7560 7581 8071 8091 8760 21 510
21 9177 9204 9687 9714 10037 27 510
19 8470 8495 8981 9007 9551 25 512
21 3605 3631 4117 4144 4651 26 513
21 9893 9919 10405 10432 11307 26 513
20 3901 3921 4410 4436 5711 20 515
25 10397 10426 10911 10942 11135 29 516
28 5592 5633 6120 6149 6542 41 516
18 3355 3381 3873 3898 5752 26 517
25 4826 4859 5345 5377 6483 33 518
20 4466 4492 4986 5011 5225 26 519
20 9043 9069 9563 9588 10135 26 519
20 9206 9233 9726 9752 10604 27 519
18 3407 3434 3928 3954 4277 27 520
27 10346 10378 10872 10898 11678 32 520
40 6589 6639 7123 7159 8484 50 520
35 6193 6229 6701 6750 8777 36 521
21 4126 4160 4661 4682 5064 34 522
20 3700 3720 4216 4243 4696 20 523
28 42176 42216 42700 42740 45658 40 524
43 14530 14573 15049 15097 18752 43 524
17 8801 8829 9327 9354 9774 28 525
21 9576 9602 10100 10127 10520 26 525
20 3582 3603 4103 4129 5006 21 526
19 3381 3404 3905 3931 4792 23 527
22 8340 8361 8876 8897 9227 21 536
28 4325 4357 4862 4894 5811 32 537
29 5671 5709 6208 6246 7437 38 537
34 6551 6583 7071 7120 8145 32 537
27 4456 4488 4995 5027 5682 32 539
21 3421 3447 3963 3988 4826 26 541
21 3462 3482 3997 4023 4364 20 541
21 3696 3721 4237 4264 4757 25 543
20 9729 9756 10274 10301 10502 27 545
22 4230 4252 4770 4797 7065 22 545
19 3396 3420 3944 3969 4849 24 549
20 8986 9013 9536 9563 10885 27 550
26 4545 4576 5101 5128 6631 31 552
20 2966 2986 3518 3539 4827 20 553
24 5265 5306 5834 5859 6601 41 553
17 3375 3400 3929 3954 5148 25 554
25 13744 13784 14301 14340 14564 40 556
34 4622 4654 5179 5210 6042 32 556
21 3515 3542 4073 4100 4607 27 558
25 12813 12853 13370 13411 14001 40 558
20 3707 3729 4262 4290 5231 22 561
21 9114 9140 9675 9701 10241 26 561
36 13302 13335 13846 13897 16062 33 562
43 28008 28051 28567 28615 34896 43 564
21 4054 4081 4617 4646 4952 27 565
21 9314 9341 9878 9906 10256 27 565
28 7289 7325 7855 7893 8440 36 568
33 5822 5871 6406 6440 7088 49 569
23 10445 10477 11015 11047 11538 32 570
35 7845 7895 8431 8465 12155 50 570
20 3469 3496 4042 4069 4344 27 573
33 14417 14451 14975 15024 18409 34 573
43 32849 32898 33428 33471 37428 49 573
25 3639 3666 4214 4243 5424 27 577
37 6904 6938 7463 7515 9451 34 577
25 10248 10274 10814 10854 12666 26 580
19 3547 3573 4134 4154 4291 26 581
19 4263 4289 4850 4875 7530 26 586
21 3669 3695 4257 4283 4731 26 588
27 4795 4834 5381 5422 7306 39 588
21 9719 9745 10309 10334 10773 26 589
23 4146 4175 4738 4767 5310 29 592
29 5243 5282 5842 5874 6175 39 592
25 12045 12078 12641 12672 13799 33 594
35 6220 6270 6814 6864 8214 50 594
23 3929 3950 4518 4545 5482 21 595
35 6657 6707 7266 7302 8471 50 595
22 10995 11022 11590 11618 12179 27 596
22 3593 3614 4184 4211 4480 21 597
26 10556 10588 11156 11185 11316 32 597
23 4264 4296 4872 4894 5207 32 598
43 7568 7614 8169 8212 8666 46 598
22 3707 3728 4300 4327 4573 21 599
22 4591 4620 5192 5220 6181 29 600
24 4151 4179 4751 4779 5024 28 600
26 5611 5636 6195 6236 7427 25 600
27 9893 9919 10487 10519 11128 26 600
33 5344 5378 5928 5978 7738 34 600
21 8832 8858 9434 9459 10707 26 601
26 4191 4222 4791 4823 5554 31 601
20 4340 4366 4945 4970 7674 26 604
35 14892 14942 15495 15546 15937 50 604
20 3547 3573 4154 4179 4544 26 606
18 3677 3703 4289 4315 4420 26 612
31 5366 5406 5979 6018 7203 40 612
26 14009 14180 14753 14793 17012 171 613
18 9527 9559 10142 10175 10651 32 616
20 10031 10057 10646 10673 10791 26 616
21 3586 3607 4196 4224 4414 21 617
20 3484 3505 4096 4123 4381 21 618
33 5571 5613 6199 6231 8076 42 618
24 14767 14806 15384 15426 15901 39 620
20 4785 4806 5401 5428 6275 21 622
21 3649 3723 4318 4345 4730 74 622
21 9419 9445 10042 10067 10533 26 622
26 13266 13306 13889 13928 14827 40 622
18 3347 3372 3972 3998 4699 25 626
32 17244 17278 17855 17905 18470 34 627
18 9213 9240 9842 9868 11407 27 628
22 10430 10459 11066 11088 11278 29 629
24 4493 4518 5109 5148 6335 25 630
22 10541 10567 11172 11198 11926 26 631
26 7414 7439 8030 8071 11786 25 632
38 6162 6196 6777 6829 7400 34 633
22 10589 10618 11228 11255 12577 29 637
21 3538 3564 4185 4205 4454 26 641
26 13083 13124 13725 13765 14447 41 641
31 13299 13348 13941 13991 14377 49 643
25 10284 10317 10932 10964 11366 33 647
27 9546 9572 10190 10222 11266 26 650
21 3415 3441 4073 4092 4229 26 651
21 4543 4569 5194 5220 5560 26 651
34 14488 14538 15153 15189 15459 50 651
20 3886 3918 4539 4572 4925 32 654
28 4288 4316 4937 4970 5957 28 654
23 3240 3261 3896 3916 5489 21 655
20 8627 8653 9283 9309 9844 26 656
38 6648 6698 7321 7357 7966 50 659
19 3785 3817 4449 4480 5282 32 663
19 7460 7482 8120 8145 8410 22 663
23 3790 3815 4456 4478 5504 25 663
27 12205 12243 12869 12906 13295 38 663
34 7283 7331 7944 7994 8236 48 663
24 4043 4076 4709 4740 5295 33 664
19 3300 3326 3966 3992 4599 26 666
25 13045 13070 13695 13736 13903 25 666
34 5822 5856 6471 6522 8158 34 666
20 3893 3926 4563 4594 5088 33 668
35 5903 5940 6562 6612 8256 37 672
35 12566 12597 13222 13271 15772 31 674
22 3412 3439 4088 4114 4847 27 675
23 4004 4036 4678 4711 5714 32 675
27 10099 10127 10780 10804 11094 28 677
20 9482 9507 10163 10189 10426 25 682
22 12675 12704 13358 13386 14036 29 682
18 10198 10226 10883 10910 11123 28 684
26 12762 12802 13460 13486 13903 40 684
20 3383 3409 4068 4094 6873 26 685
26 4492 4524 5178 5210 5865 32 686
24 5618 5658 6308 6347 6850 40 689
25 5462 5502 6149 6191 9460 40 689
35 6964 7015 7670 7705 8341 51 690
20 3351 3377 4044 4070 5860 26 693
22 4678 4700 5367 5394 5985 22 694
20 9018 9044 9709 9739 10145 26 695
17 3374 3401 4071 4097 4715 27 696
18 9207 9228 9897 9924 11491 21 696
20 9372 9398 10071 10098 10629 26 700
22 9239 9264 9940 9964 10309 25 700
20 3797 3824 4500 4526 4923 27 702
22 4217 4244 4925 4952 5256 27 708
26 12449 12481 13160 13192 13726 32 711
44 8926 8976 9647 9688 13053 50 712
33 5839 5889 6570 6602 7088 50 713
18 8299 8326 9018 9045 9225 27 719
21 4236 4263 4957 4984 5368 27 721
21 3313 3340 4035 4062 4418 27 722
35 13812 13847 14521 14570 15911 35 723
27 12843 12880 13566 13605 14024 37 725
19 2828 2847 3558 3577 3689 19 730
21 3247 3273 3978 4004 4750 26 731
26 9854 9880 10571 10611 11633 26 731
25 5279 5319 6008 6051 6679 40 732
20 3571 3597 4304 4330 4729 26 733
21 3747 3774 4486 4507 4744 27 733
21 8586 8606 9317 9339 9449 20 733
21 2851 2871 3592 3612 3737 20 741
21 3787 3808 4525 4553 5718 21 745
22 2965 2985 3708 3730 4493 20 745
20 8873 8892 9618 9638 9818 19 746
22 10281 10314 11028 11061 11578 33 747
20 2947 2969 3698 3717 3876 22 748
24 4302 4334 5049 5082 6260 32 748
24 4610 4644 5371 5393 5619 34 749
21 8935 8955 9686 9707 9875 20 752
19 8433 8454 9181 9207 9867 21 753
23 3872 3904 4633 4657 4968 32 753
18 8514 8534 9269 9290 9414 20 756
20 8632 8651 9387 9407 9950 19 756
25 13630 13668 14398 14426 15136 38 758
21 7770 7790 8529 8550 8694 20 760
22 8806 8827 9567 9587 11274 21 760
20 8356 8376 9116 9137 9256 20 761
21 8777 8797 9538 9558 9727 20 761
39 7034 7153 7864 7914 8433 119 761
20 2798 2819 3560 3581 4351 21 762
21 3034 3055 3796 3817 4410 21 762
28 11248 11276 12016 12041 12570 28 765
21 3032 3052 3800 3820 3971 20 768
18 3108 3132 3876 3901 4390 24 769
19 9877 9903 10646 10672 12104 26 769
20 7959 7977 8725 8746 10028 18 769
19 2769 2789 3540 3560 3691 20 771
19 8412 8431 9182 9202 9362 19 771
31 12044 12074 12807 12845 14552 30 771
24 4438 4470 5206 5242 5644 32 772
21 9119 9140 9892 9913 10546 21 773
20 3373 3398 4148 4173 4547 25 775
30 13377 13411 14149 14187 14825 34 776
25 4833 4874 5623 5651 7742 41 777
20 9404 9423 10182 10202 10355 19 779
22 8659 8679 9438 9459 9575 20 780
23 2848 2871 3628 3651 4241 23 780
22 9626 9658 10407 10439 11190 32 781
22 9099 9127 9889 9910 10551 28 783
21 9492 9514 10277 10299 10428 22 785
19 2777 2797 3564 3583 3696 20 786
21 4715 4736 5506 5526 5682 21 790
19 3343 3363 4134 4155 4311 20 792
20 8769 8795 9561 9587 9775 26 792
22 3529 3551 4314 4343 4535 22 792
20 3557 3578 4351 4372 4683 21 794
25 6506 6535 7290 7330 7676 29 795
21 3485 3512 4290 4311 4534 27 799
25 10147 10179 10949 10980 11040 32 801
27 5436 5468 6238 6270 6973 32 802
30 14631 14672 15442 15475 16285 41 803
21 59886 59906 60690 60710 60869 20 804
21 9058 9078 9855 9883 10255 20 805
20 3776 3807 4586 4617 5058 31 810
26 10108 10133 10901 10943 11388 25 810
32 5679 5729 6489 6539 7556 50 810
21 3095 3115 3903 3926 4328 20 811
19 9540 9566 10351 10378 10855 26 812
34 15839 15888 16650 16701 18098 49 813
21 10186 10213 11007 11027 11266 27 814
34 12778 12813 13578 13627 15641 35 814
20 30424 30445 31231 31260 32205 21 815
19 3632 3658 4449 4475 7056 26 817
37 12756 12793 13573 13610 15898 37 817
20 3486 3507 4304 4326 4730 21 819
27 4999 5025 5813 5845 8041 26 820
20 3892 3913 4714 4734 4902 21 821
32 5505 5538 6308 6359 7861 33 821
17 3748 3766 4563 4590 4668 18 824
18 4085 4112 4910 4936 5624 27 824
21 9095 9116 9920 9940 10088 21 824
27 11303 11335 12129 12160 12836 32 825
18 8752 8779 9577 9605 9881 27 826
18 9345 9371 10171 10197 10600 26 826
21 3019 3040 3847 3868 3996 21 828
18 3449 3477 4284 4306 4435 28 829
21 66235 66255 67065 67086 68586 20 831
35 14776 14813 15595 15644 16178 37 831
34 14597 14646 15447 15480 18676 49 834
35 6346 6381 7167 7216 8968 35 835
21 9830 9851 10665 10687 10826 21 836
33 17325 17351 18146 18187 18422 26 836
24 10571 10768 11572 11606 12187 197 838
22 2940 2961 3779 3800 4020 21 839
22 3956 3984 4795 4823 4912 28 839
20 10553 10579 11393 11419 12033 26 840
22 3832 3854 4674 4694 4838 22 840
21 3338 3364 4182 4208 4272 26 844
26 4186 4211 5015 5055 5748 25 844
20 3098 3119 3938 3964 4103 21 845
22 3421 3448 4270 4296 5255 27 848
26 5052 5078 5895 5927 6055 26 849
20 3031 3053 3883 3903 4453 22 850
20 9921 9948 10771 10798 11207 27 850
24 11696 11723 12541 12573 13087 27 850
34 17576 17608 18409 18458 20542 32 850
21 3453 3480 4311 4331 4702 27 851
21 3835 3875 4693 4726 5086 40 851
26 11330 11362 12180 12213 12500 32 851
35 5602 5639 6458 6492 7992 37 853
20 2970 2991 3824 3845 3963 21 854
21 3188 3209 4044 4064 4181 21 855
21 8760 8781 9610 9637 11012 21 856
42 19149 19192 19999 20048 22688 43 856
20 10487 10514 11344 11372 12916 27 858
21 8164 8185 9008 9043 9210 21 858
26 4311 4343 5171 5202 5494 32 859
34 5283 5317 6144 6177 8559 34 860
25 4774 4799 5619 5661 6876 25 862
20 7318 7338 8169 8201 8290 20 863
28 5507 5544 6372 6410 7518 37 866
21 3214 3236 4084 4106 4225 22 870
18 3376 3403 4247 4274 4404 27 871
24 5265 5306 6153 6179 6562 41 873
21 3240 3261 4114 4135 4862 21 874
38 12776 12810 13649 13685 14870 34 875
21 3829 3851 4702 4728 4811 22 877
21 2880 2901 3758 3779 3927 21 878
22 3789 3815 4670 4693 5284 26 878
37 6121 6169 7013 7048 8654 48 879
20 10457 10483 11338 11363 11853 26 880
21 4431 4457 5313 5338 5897 26 881
23 9214 9238 10097 10121 11674 24 883
21 3292 3318 4177 4203 4351 26 885
22 3012 3033 3900 3921 4040 21 888
16 5577 5606 6466 6495 6631 29 889
20 3847 3873 4736 4762 4870 26 889
28 15481 15518 16371 16409 16687 37 891
22 10847 10875 11739 11767 12352 28 892
22 3048 3070 3942 3963 4094 22 893
19 8831 8852 9726 9747 10395 21 895
28 4054 4086 4951 4983 5307 32 897
19 8490 8511 9393 9414 9562 21 903
33 15416 15467 16322 16371 17791 51 904
20 14117 14144 15025 15052 15195 27 908
20 7404 7424 8311 8336 8556 20 912
26 11090 11115 12004 12030 12216 25 915
19 3898 3924 4816 4843 5143 26 919
34 11712 11746 12633 12666 13687 34 920
18 3240 3266 4160 4187 4350 26 921
20 3466 3492 4387 4413 4686 26 921
21 3785 3813 4707 4735 5374 28 922
33 16319 16368 17247 17298 17815 49 930
21 3324 3350 4254 4281 4404 26 931
35 6427 6477 7376 7409 8036 50 932
23 10138 10166 11071 11099 11602 28 933
26 5539 5565 6462 6503 6789 26 938
20 3746 3773 4692 4713 4972 27 940
19 3839 3865 4782 4807 5283 26 942
22 4090 4119 5034 5062 5374 29 943
19 9937 9963 10881 10907 12601 26 944
26 4112 4143 5057 5089 6341 31 946
22 4029 4055 4974 5002 5702 26 947
29 4724 4752 5661 5699 7540 28 947
21 8787 8808 9734 9756 9885 21 948
25 11468 11494 12417 12442 12654 26 948
19 8512 8532 9456 9482 9625 20 950
20 8798 8819 9742 9769 9841 21 950
20 9385 9405 10339 10359 10471 20 954
20 2913 2933 3863 3889 4028 20 956
20 4498 4525 5457 5484 7676 27 959
21 8182 8204 9143 9163 9056 22 959
26 11687 11719 12645 12678 12961 32 959
19 9021 9048 9988 10008 9951 27 960
21 2927 2950 3892 3913 4469 23 963
34 5458 5491 6406 6455 7767 33 964
21 2722 2744 3689 3709 3576 22 965
21 3369 3396 4336 4361 4472 27 965
22 3748 3774 4714 4739 5203 26 965
25 11712 11739 12679 12706 13150 27 967
32 6083 6132 7049 7099 7502 49 967
20 3565 3591 4534 4562 4860 26 971
29 13937 13974 14908 14946 15307 37 972
23 4041 4069 5016 5045 5132 28 976
24 5974 6014 6952 6991 7402 40 977
20 3091 3111 4066 4091 4154 20 980
20 3585 3612 4571 4592 4914 27 980
21 7943 7963 8921 8943 9623 20 980
34 5694 5745 6674 6725 7314 51 980
26 5879 5920 6861 6902 7312 41 982
20 3266 3292 4252 4278 4826 26 986
21 4323 4349 5310 5336 5539 26 987
26 10831 10855 11816 11842 12004 24 987
20 3350 3376 4343 4368 4974 26 992
24 32614 32640 33607 33632 35466 26 992
21 8921 8941 9907 9934 10272 20 993
20 8835 8861 9829 9855 10112 26 994
21 8594 8614 9583 9608 9624 20 994
23 11844 11885 12848 12879 13372 41 994
27 12214 12241 13211 13235 13415 27 994
36 14136 14171 15134 15170 16418 35 999
17 8801 8828 9808 9828 10286 27 1000
21 3789 3815 4791 4815 7259 26 1000
25 3828 3854 4830 4857 5122 26 1003
20 9154 9181 10159 10187 11645 27 1006
26 12094 12118 13098 13124 13286 24 1006
28 4839 4877 5844 5883 9928 38 1006
24 3986 4010 4994 5021 5181 24 1011
21 4573 4601 5584 5613 5898 28 1012
37 6627 6676 7640 7691 9762 49 1015
34 7477 7510 8474 8526 9018 33 1016
36 5847 5897 6880 6914 7397 50 1017
19 10631 10656 11649 11674 12228 25 1018
24 3484 3508 4499 4526 5278 24 1018
26 3407 3431 4423 4450 4594 24 1019
37 13556 13591 14562 14612 15228 35 1021
20 9323 9349 10345 10371 10931 26 1022
22 9045 9071 10069 10095 10504 26 1024
16 4076 4103 5102 5129 5375 27 1026
19 2783 2804 3805 3831 3742 21 1027
21 7447 7469 8471 8496 8352 22 1027
36 5979 6030 7008 7058 8123 51 1028
21 8720 8741 9744 9770 9737 21 1029
31 7279 7312 8295 8344 9282 33 1032
21 9493 9515 10521 10549 10449 22 1034
21 5979 6012 7008 7047 7087 33 1035
38 14493 14528 15531 15566 16284 35 1038
20 9267 9294 10318 10345 10388 27 1051
21 4299 4326 5351 5378 5415 27 1052
22 8653 8674 9699 9726 9547 21 1052
25 3614 3641 4675 4701 4848 27 1060
26 4240 4267 5306 5331 5547 27 1064
31 5577 5610 6630 6679 7044 33 1069
21 9204 9225 10268 10295 10234 21 1070
22 3795 3821 4865 4891 5270 26 1070
34 13195 13227 14251 14300 15770 32 1073
18 2295 2322 3375 3403 3486 27 1081
20 9520 9540 10595 10621 10916 20 1081
25 6174 6213 7254 7294 7714 39 1081
20 10005 10031 11087 11113 11570 26 1082
20 9084 9109 10165 10191 10306 25 1082
20 10788 10815 11873 11899 12314 27 1084
26 4598 4630 5687 5718 5934 32 1088
20 9383 9410 10475 10502 10369 27 1092
22 4163 4189 5255 5281 5688 26 1092
33 12389 12421 13465 13514 15262 32 1093
20 3724 3750 4821 4847 5026 26 1097
35 6035 6084 7131 7182 7784 49 1098
17 8797 8825 9898 9925 9771 28 1100
20 8471 8492 9567 9594 9505 21 1102
26 4641 4673 5743 5775 6103 32 1102
26 9753 9779 10843 10882 10925 26 1103
20 3015 3037 4131 4152 5239 22 1115
20 4109 4135 5230 5251 5053 26 1116
26 12115 12140 13242 13267 13411 25 1127
22 3001 3021 4134 4155 5277 20 1134
20 4015 4040 5150 5176 5087 25 1136
32 7248 7281 8368 8417 9651 33 1136
20 9258 9283 10395 10420 10593 25 1137
21 2925 2945 4061 4082 4755 20 1137
28 5788 5829 6927 6966 7153 41 1137
43 6689 6732 7819 7869 12920 43 1137
26 12137 12163 13276 13302 13532 26 1139
27 4002 4029 5148 5176 5365 27 1147
26 11010 11041 12156 12189 13856 31 1148
32 12884 12917 14014 14065 14330 33 1148
26 11167 11199 12322 12349 12814 32 1150
32 4141 4172 5301 5331 5538 31 1159
40 15907 15955 17067 17115 17598 48 1160
22 2867 2908 4048 4074 4187 41 1166
35 5131 5167 6303 6339 7410 36 1172
47 22025 22069 23196 23245 27922 44 1176
19 9971 9997 11151 11177 10914 26 1180
20 2788 2808 3969 3989 4920 20 1181
25 4289 4321 5477 5503 5859 32 1182
29 11174 11206 12364 12389 12911 32 1183
24 6759 6791 7956 7978 8653 32 1187
22 9007 9028 10204 10231 10385 21 1203
32 11535 11568 12727 12776 13140 33 1208
24 10255 10287 11465 11496 12110 32 1209
28 12500 12528 13715 13743 13919 28 1215
33 4475 4507 5691 5724 6175 32 1217
36 15897 15946 17115 17164 17527 49 1218
27 3535 3560 4754 4779 4973 25 1219
19 4842 4867 6062 6087 6257 25 1220
32 7867 7919 9105 9139 10249 52 1220
33 13198 13231 14420 14453 15026 33 1222
20 4332 4359 5561 5583 5390 27 1224
21 10002 10035 11226 11260 11534 33 1225
33 14164 14197 15392 15426 16522 33 1229
25 11906 11933 13138 13164 13874 27 1231
20 4109 4135 5345 5369 5122 26 1234
21 9479 9502 10712 10740 11152 23 1238
25 11510 11535 12741 12773 12719 25 1238
25 6354 6393 7601 7642 8682 39 1249
28 12840 12868 14089 14117 15105 28 1249
44 8117 8163 9360 9412 10129 46 1249
26 3887 3913 5132 5166 5201 26 1253
20 4081 4107 5338 5364 5827 26 1257
21 3065 3085 4316 4343 3950 20 1258
44 8258 8308 9523 9566 13132 50 1258
22 9037 9057 10289 10316 11618 20 1259
35 5190 5224 6449 6483 7627 34 1259
33 13329 13359 14572 14619 14797 30 1260
40 4895 4931 6156 6191 6515 36 1260
33 4432 4465 5679 5726 6053 33 1261
20 8229 8249 9485 9511 9717 20 1262
21 8373 8392 9627 9654 10029 19 1262
20 9749 9775 11015 11041 10844 26 1266
21 7818 7845 9092 9113 9502 27 1268
32 14359 14392 15630 15664 15853 33 1272
28 6267 6305 7547 7586 7796 38 1281
20 8400 8420 9679 9704 9958 20 1284
21 9330 9350 10608 10635 10990 20 1285
42 7155 7198 8424 8485 11240 43 1287
22 8889 8910 10174 10201 10714 21 1291
45 23038 23089 24323 24380 26168 51 1291
43 8280 8329 9579 9623 12475 49 1294
21 10265 10285 11553 11580 11848 20 1295
22 3020 3041 4320 4340 5674 21 1299
24 4081 4111 5381 5414 5728 30 1303
18 8635 8661 9942 9969 10317 26 1308
28 3747 3772 5051 5083 5151 25 1311
43 9087 9131 10395 10445 12855 44 1314
33 5641 5673 6939 6988 8102 32 1315
24 12903 12943 14223 14263 14963 40 1320
22 8927 8949 10244 10272 10482 22 1323
21 8579 8600 9898 9926 10405 21 1326
26 4467 4493 5789 5823 5777 26 1330
43 9247 9291 10576 10626 12130 44 1335
37 4965 5001 6303 6339 6863 36 1338
26 4164 4190 5492 5531 5304 26 1341
20 8077 8096 9417 9443 10028 19 1347
20 2727 2747 4072 4097 4301 20 1350
24 10916 10941 12252 12294 12207 25 1353
43 7751 7793 9112 9153 10566 42 1360
44 9012 9056 10387 10436 12047 44 1380
36 6513 6564 7898 7947 31588 51 1383
45 33131 33174 34509 34557 33646 43 1383
57 6907 6955 8301 8339 9190 48 1384
21 3323 3345 4717 4744 4993 22 1399
33 4984 5018 6388 6421 6630 34 1403
21 9109 9135 10515 10542 10716 26 1407
34 14892 14942 16303 16351 16608 50 1409
34 6064 6100 7480 7516 8942 36 1416
31 5241 5280 6673 6711 8818 39 1431
27 5988 6028 7419 7460 9061 40 1432
20 3081 3101 4507 4535 5019 20 1434
48 40209 40245 41647 41681 41880 36 1436
43 49599 49642 51023 51082 54124 43 1440
34 5406 5439 6847 6881 7108 33 1442
33 13799 13833 15255 15289 15583 34 1456
33 5911 5959 7384 7418 7740 48 1459
33 4669 4702 6129 6163 6373 33 1461
25 3604 3630 5061 5093 6956 26 1463
30 5043 5082 6511 6550 6814 39 1468
35 4807 4842 6276 6310 7536 35 1468
20 4100 4133 5516 5603 5789 33 1470
36 7912 7947 9384 9420 9727 35 1473
21 4075 4096 5552 5578 6126 21 1482
33 18278 18312 19749 19801 20833 34 1489
194 10280 10308 11771 11799 12002 28 1491
21 8013 8040 9521 9541 9794 27 1501
34 5947 5996 7457 7507 7810 49 1511
27 11432 11460 12943 12972 13304 28 1512
23 4659 4686 6174 6203 5739 27 1517
20 151869 151892 153382 153416 154015 23 1524
26 10791 10816 12310 12343 12857 25 1527
43 6933 6982 8468 8510 9796 49 1528
43 19626 19669 21150 21199 22972 43 1530
19 4422 4448 5953 5979 6260 26 1531
36 16090 16139 17632 17682 18556 49 1543
26 11164 11193 12707 12747 12952 29 1554
35 4723 4757 6282 6317 7641 34 1560
36 15394 15432 16958 16992 17205 38 1560
27 6271 6312 7838 7878 8194 41 1566
34 16565 16614 18133 18184 19553 49 1570
35 15800 15836 17378 17414 17641 36 1578
28 5414 5452 7002 7040 6775 38 1588
31 4530 4565 6110 6156 7149 35 1591
33 13872 13906 15466 15499 16728 34 1593
31 4434 4468 6030 6062 6108 34 1594
48 6224 6258 7829 7864 9146 34 1606
43 19319 19364 20939 20988 21496 45 1624
19 2945 2967 4566 4592 4798 22 1625
25 10380 10406 12003 12035 12240 26 1629
28 4498 4530 6131 6164 6570 32 1634
34 7033 7083 8667 8717 9052 50 1634
44 19464 19505 21096 21147 22707 41 1642
33 21850 21884 23501 23533 24802 34 1649
21 4386 4415 6055 6082 6101 29 1667
28 15547 15585 17220 17258 16913 38 1673
35 16709 16743 18383 18417 18654 34 1674
26 3529 3554 5218 5250 5813 25 1696
33 14853 14892 16539 16590 17072 39 1698
20 3439 3466 5147 5173 5288 27 1707
21 2908 2928 4620 4646 5959 20 1718
28 4905 4945 6623 6664 7050 40 1719
27 5636 5674 7363 7401 7179 38 1727
27 10909 10936 12635 12667 13348 27 1731
43 8157 8199 9897 9939 10379 42 1740
34 18444 18493 20186 20234 20611 49 1741
46 9551 9592 11292 11336 12736 41 1744
33 13674 13708 15402 15453 15479 34 1745
43 8067 8112 9827 9871 10159 45 1759
42 7187 7231 8963 9008 9257 44 1777
34 16849 16884 18634 18668 18936 35 1784
23 3231 3255 5024 5045 5723 24 1790
27 11694 11724 13482 13522 13859 30 1798
20 1784 1810 3602 3628 6064 26 1818
32 5672 5703 7489 7521 7391 31 1818
43 19331 19374 21169 21214 21453 43 1840
43 10676 10721 12512 12563 13166 45 1842
60 17876 17910 19721 19772 20592 34 1862
44 8876 8929 10741 10792 11101 53 1863
43 7706 7748 9571 9614 9887 42 1866
43 18818 18861 20704 20753 22051 43 1892
35 15761 15798 17643 17691 18355 37 1893
25 13399 13439 15314 15354 15634 40 1915
26 5159 5200 7099 7124 6997 41 1924
33 14279 14311 16190 16240 16747 32 1929
60 28251 28294 30196 30240 30678 43 1946
43 10421 10509 12421 12463 12818 88 1954
42 55855 55916 57811 57871 58228 61 1955
43 19429 19477 21401 21450 22627 48 1973
21 8600 8620 10575 10601 10459 20 1981
44 7967 8030 9976 10028 10331 63 1998
45 9357 9405 11366 11417 12784 48 2012
28 4577 4604 6585 6623 7037 27 2019
31 5328 5376 7364 7412 7831 48 2036
44 19938 19986 21977 22026 22353 48 2040
32 17017 17067 19061 19110 18825 50 2043
17 2951 2978 5017 5039 4714 27 2061
20 9882 9908 11944 11969 11601 26 2061
33 15318 15367 17379 17429 18227 49 2062
34 16533 16565 18580 18628 19114 32 2063
26 14019 14052 16084 16116 15207 33 2064
20 8388 8408 10448 10474 11195 20 2066
46 19447 19496 21529 21578 25423 49 2082
21 8153 8173 10239 10265 10683 20 2092
33 4655 4688 6752 6784 7406 33 2096
33 17050 17083 19143 19191 19467 33 2108
39 5245 5279 7361 7396 9526 34 2117
63 13042 13102 15167 15222 15569 60 2120
25 12116 12156 14235 14277 14537 40 2121
21 8346 8366 10484 10509 10083 20 2143
49 10535 10580 12709 12751 15776 45 2171
43 8364 8413 10540 10604 11839 49 2191
18 3346 3373 5560 5580 5106 27 2207
28 14941 14978 17200 17237 17107 37 2259
18 8995 9020 11274 11293 10041 25 2273
43 6504 6549 8821 8870 11542 45 2321
37 15287 15322 17601 17651 19707 35 2329
22 3001 3021 5349 5371 4900 20 2350
30 16597 16621 18963 18995 19103 24 2374
20 3307 3332 5713 5734 4302 25 2402
35 6914 6963 9383 9419 10208 49 2456
37 6634 6666 9096 9146 9456 32 2480
44 6838 6882 9231 9365 9863 44 2483
37 6184 6219 8676 8725 9397 35 2506
36 15760 15796 18301 18351 18588 36 2555
36 5187 5221 7735 7784 8419 34 2563
43 32891 32940 35468 35518 35174 49 2578
21 2986 3007 5593 5620 5150 21 2613
43 40753 40802 43298 43431 43446 49 2629
25 10126 10152 12805 12837 12490 26 2685
34 6950 6999 9710 9759 10095 49 2760
21 11771 11799 14565 14593 13029 28 2794
20 3092 3117 5895 5920 5073 25 2803
25 11695 11721 14499 14532 14042 26 2811
33 7152 7202 9964 10014 10382 50 2812
35 27329 27378 30175 30211 30715 49 2833
43 45353 45397 48104 48237 48830 44 2840
22 1627 1654 4478 4499 4690 27 2845
21 9511 9538 12411 12439 11620 27 2901
22 10334 10362 13284 13305 12790 28 2943
46 5384 5410 8331 8357 7177 26 2947
43 8160 8203 11128 11177 12523 43 2974
22 9860 9888 12846 12873 12857 28 2985
27 4130 4156 7184 7216 6644 26 3060
44 7150 7193 10179 10255 9938 43 3062
18 9389 9410 12467 12491 11186 21 3081
18 8843 8870 12003 12028 10586 27 3158
20 3694 3720 6867 6901 7052 26 3181
18 3472 3498 6686 6706 5401 26 3208
22 1730 1751 4948 4975 5263 21 3224
43 18714 18764 22060 22103 22472 50 3339
26 4277 4303 7637 7670 7116 26 3367
44 9659 9702 13044 13094 13638 43 3392
34 15149 15183 18577 18609 18545 34 3426
45 17297 17347 20741 20786 21765 50 3439
22 10438 10465 13906 13934 12302 27 3469
44 14211 14256 17626 17761 18226 45 3505
18 3840 3867 7360 7387 5866 27 3520
32 14593 14626 18374 18426 17851 33 3800
77 25552 25691 29519 29561 29972 139 3870
31 18050 18084 21924 21974 19756 34 3890
34 13337 13373 17298 17334 17721 36 3961
39 15499 15533 19476 19525 18880 34 3992
20 3118 3140 7256 7290 7608 22 4150
22 3341 3431 7984 8005 8873 90 4574
32 4984 5025 9848 9889 7780 41 4864
34 17087 17128 22089 22139 19881 41 5011
34 15804 15837 21023 21072 18957 33 5235
20 3550 3577 8878 8904 5905 27 5327
57 17850 17884 23292 23327 23772 34 5443
20 3725 3746 9248 9339 8655 21 5593
43 7463 7506 13697 13740 9718 43 6234
43 3103 3154 9437 9487 9764 51 6333
43 6877 6923 13648 13698 10798 46 6775
19 1262 1289 9447 9474 9180 27 8185
35 23421 23455 38035 38071 24839 34 14616
44 40136 40187 66928 67270 42505 51 27083
#!/usr/bin/env nix-shell
#! nix-shell -i python3 -p python3Packages.pandas
import sys, os
import pandas as pd
import re
print(
re.sub(r"\.\d+", "",
pd.read_table(sys.argv[1]).describe().transpose().to_csv()))
# convert content of those outputs in a csv like
# count mean std min 25% 50% 75% max
#Setup 1000 101 63 41 56 87 120 831
#Serial 1000 41 17 20 26 37 56 173
#Spwn0000E0 - Epoch 1000 9857 10677 2334 4597 7322 12166 161291
#Spwn0001E0 - Epoch 1000 10807 11054 2534 5502 8237 13308 173191
#Spwn0002E0 - Epoch 1000 11466 11230 2686 6068 8860 13825 174548
#Spwn0001E0 - Spwn0000E1 1000 908 1360 -3608 267 498 1140 22993
#Spwn0002E0 - Spwn0001E1 1000 618 1255 -3788 163 359 798 15018
## The objective is mesure the actual cost os thread releated operations
##
## I studies 4 scenarios
##
## 1º news lock/cond operated N times
## 2º news lock/cond reused N times with sleep to clear cpu cache
## 3º news lock/cond reused N/10 times
## 4º news lock/cond reused N/10 times with sleep to clear cpu cache
##
## case: scneario
## op: max precision in nanoseconds
## lock: time required to acquire lock
## rele: time required to release the lock
## tryT: time required to tryAcquire lock with success
## tryF: time required to tryAcquire lock with failure
## sign: time required to send signal
##
## I'm not sure if I could mesure wait or locked acquire
import std/monotimes
import std / [locks, times, os]
import strutils
proc header(sep: string) =
echo [
"case",
"op",
"lock",
"rele",
"tryT",
"tryF",
"sign",
].join(sep)
proc getThreadOpTime(scenario: int;runs: int = 100; sep: string = "\t"; clear: bool = false) =
var epoch = getMonoTime()
var precT = 0.int64
var lockT = 0.int64
var releT = 0.int64
var tryTT = 0.int64
var tryFT = 0.int64
var signT = 0.int64
for i in 0..(runs - 1):
var lock: Lock
var cond: Cond
initLock(lock)
initCond(cond)
epoch = getMonoTime()
precT = inNanoseconds(getMonoTime() - epoch)
if clear: sleep(10)
epoch = getMonoTime()
acquire(lock)
lockT = inNanoseconds(getMonoTime() - epoch)
if clear: sleep(10)
epoch = getMonoTime()
release(lock)
releT = inNanoseconds(getMonoTime() - epoch)
if clear: sleep(10)
epoch = getMonoTime()
discard tryAcquire(lock)
tryTT = inNanoseconds(getMonoTime() - epoch)
if clear: sleep(10)
epoch = getMonoTime()
discard tryAcquire(lock)
tryFT = inNanoseconds(getMonoTime() - epoch)
release(lock)
if clear: sleep(10)
epoch = getMonoTime()
signal(cond)
signT = inNanoseconds(getMonoTime() - epoch)
echo [
scenario,
precT,
lockT,
releT,
tryTT,
tryFT,
signT
].join(sep)
let sep = "\t"
let runs = 50
header(sep)
getThreadOpTime(1, runs, sep)
sleep(10)
getThreadOpTime(2, runs, sep, true)
sleep(10)
for i in 1..(runs div 10):
getThreadOpTime(3, 10, sep)
for i in 1..(runs div 10):
sleep(10)
getThreadOpTime(4, 10, sep)
header(sep)
case op lock rele tryT tryF sign
1 37 452 529 28101 203 363
1 14 45 47 32 21 25
1 14 16 15 19 19 16
1 14 16 15 21 19 15
1 15 15 15 18 20 15
1 15 15 15 20 20 13
1 14 16 15 19 19 14
1 14 15 15 19 19 14
1 15 16 17 20 18 14
1 14 15 15 21 17 14
1 15 16 16 20 19 14
1 15 16 24 21 18 13
1 12 15 15 20 18 13
1 15 16 15 19 18 14
1 14 15 17 22 19 15
1 15 15 16 20 19 14
1 13 16 14 20 19 14
1 15 16 15 21 20 15
1 15 15 15 20 21 16
1 15 15 15 20 18 14
1 14 15 16 19 20 16
1 15 15 15 20 20 14
1 15 16 18 21 20 15
1 16 17 15 20 19 14
1 14 15 16 19 20 14
1 14 14 16 19 21 16
1 14 15 14 21 18 15
1 14 16 14 20 18 14
1 14 16 16 19 18 14
1 24 21 33 26 22 21
1 19 23 32 22 22 26
1 25 27 39 28 24 18
1 14 16 17 20 20 14
1 14 16 14 19 20 14
1 15 16 15 20 18 14
1 15 15 18 20 19 14
1 15 15 16 20 18 14
1 14 15 15 21 20 15
1 14 15 15 20 19 14
1 15 14 18 19 20 14
1 14 17 15 22 19 13
1 14 16 14 21 18 14
1 15 15 16 20 19 14
1 15 16 18 21 18 15
1 15 16 15 20 19 14
1 14 17 15 20 21 14
1 14 16 15 19 20 16
1 28 31 23 32 21 34
1 17 20 19 25 23 20
1 18 20 20 25 25 18
2 22 448 694 940 1604 1071
2 160 1285 1602 2005 1383 633
2 108 1722 1318 1503 1468 662
2 75 728 1038 1537 2016 640
2 69 902 1050 1088 1834 622
2 72 980 1266 1694 1235 589
2 61 940 901 762 2414 683
2 86 1238 1492 707 1039 567
2 55 706 602 881 2031 610
2 48 1420 1718 1381 872 789
2 110 1728 2206 1838 1024 835
2 86 1116 893 1111 1263 1098
2 113 1156 1945 1272 1363 952
2 106 1203 988 1848 1380 1239
2 105 1808 1149 1570 1369 1050
2 108 736 931 1328 1364 938
2 108 869 1356 1150 1498 729
2 87 665 1210 873 2388 1002
2 87 913 1164 737 1515 782
2 86 797 1324 929 2266 526
2 83 817 1234 1135 1221 574
2 92 923 635 1033 1308 756
2 112 722 1347 980 986 501
2 56 770 661 898 2085 1064
2 202 1179 1436 981 1878 584
2 70 1033 1502 995 1608 633
2 90 923 1400 1973 1261 644
2 95 697 883 1287 1243 905
2 150 956 1097 1580 1100 1216
2 156 917 861 1880 2161 1063
2 157 938 1005 1402 1301 942
2 104 983 1043 1146 1595 997
2 112 857 1157 1137 1316 859
2 99 728 1088 1089 987 690
2 90 959 921 1380 1062 1246
2 54 764 1318 974 1317 618
2 100 888 1033 917 1113 570
2 103 841 1844 1415 1247 800
2 121 1546 1842 853 1533 619
2 82 789 1605 883 1749 579
2 68 916 1177 1189 1923 627
2 74 1040 1270 1268 2039 546
2 65 855 1227 1734 1207 638
2 95 779 908 1678 1310 896
2 108 893 1036 1138 1325 901
2 135 833 1177 1133 1461 1063
2 199 946 1087 1028 1767 922
2 125 785 1003 905 1340 625
2 109 822 1004 848 1205 673
2 75 963 1342 1174 1165 640
3 61 569 531 688 362 141
3 73 63 59 108 76 63
3 56 89 60 77 80 61
3 58 88 90 76 78 60
3 56 62 60 76 78 61
3 56 61 60 76 78 61
3 57 60 60 77 78 61
3 57 60 60 78 78 61
3 63 68 78 84 78 68
3 57 61 58 76 78 62
3 57 62 60 79 78 60
3 59 64 61 79 78 59
3 57 61 64 79 76 60
3 57 61 60 76 79 59
3 58 64 64 78 77 60
3 56 61 61 77 78 59
3 57 60 61 77 78 60
3 56 62 61 77 76 59
3 57 63 60 76 78 58
3 57 61 60 77 78 61
3 55 59 59 78 78 60
3 57 61 60 78 78 58
3 57 62 59 78 77 61
3 58 62 61 78 78 61
3 58 63 62 77 80 58
3 59 64 63 78 75 58
3 57 59 60 76 78 59
3 57 63 59 80 77 60
3 57 61 59 76 79 61
3 57 63 63 80 78 59
3 57 62 60 76 78 59
3 58 62 62 78 78 61
3 57 63 62 78 78 59
3 58 82 63 80 77 61
3 57 63 63 79 78 61
3 56 63 61 77 78 61
3 57 61 61 78 78 59
3 58 63 62 79 78 60
3 57 61 59 78 78 58
3 57 62 60 78 77 60
3 57 60 61 78 79 59
3 58 64 62 78 78 60
3 55 62 61 79 78 59
3 56 63 64 80 77 58
3 57 63 61 79 76 58
3 56 62 61 80 77 59
3 58 61 59 77 77 59
3 57 61 59 78 79 60
3 57 61 61 78 76 59
3 57 61 61 78 76 60
4 50 546 610 741 348 112
4 58 53 54 93 64 77
4 46 48 48 62 62 50
4 46 48 50 64 63 49
4 46 49 49 60 61 48
4 44 50 50 63 60 47
4 46 48 49 61 62 47
4 48 51 50 65 61 48
4 46 49 48 63 63 47
4 47 49 49 64 62 48
4 56 478 517 733 347 131
4 80 67 61 109 77 86
4 53 84 57 70 73 55
4 55 59 58 72 70 57
4 54 58 55 73 73 57
4 52 57 55 70 72 55
4 52 57 56 73 72 54
4 103 374 357 456 214 133
4 47 98 89 66 71 63
4 54 63 104 969 65 58
4 29 457 329 458 36 27
4 40 34 42 51 39 32
4 42 32 50 41 40 43
4 38 56 67 51 50 52
4 41 50 77 51 50 52
4 32 48 73 54 52 46
4 48 73 100 71 78 57
4 56 57 100 67 66 66
4 54 69 126 76 73 62
4 50 71 99 78 65 67
4 156 193 197 241 201 152
4 52 55 60 71 74 54
4 53 57 54 72 70 51
4 52 55 54 75 68 52
4 50 54 55 75 71 52
4 53 58 57 74 71 50
4 58 58 56 72 68 52
4 39 47 49 59 57 40
4 41 46 54 57 53 42
4 42 47 44 59 57 40
4 73 626 608 728 173 169
4 52 56 103 73 74 56
4 54 58 57 71 69 48
4 57 59 53 71 67 52
4 55 57 57 74 67 54
4 52 55 53 76 68 53
4 52 57 55 73 65 52
4 57 56 55 72 71 50
4 52 57 55 76 68 51
4 56 56 59 71 73 57
case op lock rele tryT tryF sign
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment