Benchmarks vs 0.6 in prep for 0.7 release [do not merge]

[KristofferC]

Issue to see how our benchmarks are compared to the previous release.

Updated table:

Intersect:

E.g:

["collection", "set operations", "(\"Vector\", \"Int\", \"intersect\", \"Vector\", \"Vector\")"] | 2.69 (25%) ❌ | 42.68
...

Problem

I regard these as extra important because they tend to be non toy-examples.

["problem", "json", "parse_json"] | 1.71 (15%) ❌ | 2.63 (1%) ❌
["problem", "laplacian", "laplace_iter_sub"] | 2.74 (15%) ❌ | 3829.61 (1%) ❌
["problem", "laplacian", "laplace_sparse_matvec"] | 1.39 (15%) ❌ | 1.00 (1%)
["problem", "monte carlo", "euro_option_devec"] | 1.53 (15%) ❌ | 1.00 (1%)
["problem", "grigoriadis khachiyan", "grigoriadis_khachiyan"] | 1.55 (15%) ❌ | 1.76 (1%) ❌
["problem", "spellcheck", "spellcheck"] | 1.72 (15%) ❌ | 2.20 (1%) ❌
["problem", "ziggurat", "ziggurat"] | 1.94 (15%) ❌ | 2.83 (1%) ❌

Spellcheck analysis: #27030 (comment)

Shootout

["shootout", "regex_dna"] | 60.65 (15%) ❌ | 1.00 (1%)

Regex DNA is: #26731

Simd

["simd", "(\"conditional_loop!\", \"Float32\", 4096)"] | 5.98 (20%) ❌ | 1.00 (1%)
["simd", "(\"conditional_loop!\", \"Float64\", 4095)"] | 6.31 (20%) ❌ | 1.00 (1%)
["simd", "(\"conditional_loop!\", \"Float64\", 4096)"] | 6.33 (20%) ❌ | 1.00 (1%)
["simd", "(\"conditional_loop!\", \"Int32\", 4095)"] | 3.41 (20%) ❌ | 1.00 (1%)
["simd", "(\"conditional_loop!\", \"Int32\", 4096)"] | 6.55 (20%) ❌ | 1.00 (1%)
["simd", "(\"conditional_loop!\", \"Int64\", 4095)"] | 1.77 (20%) ❌ | 1.00 (1%)
["simd", "(\"conditional_loop!\", \"Int64\", 4096)"] | 1.79 (20%)

["simd", "(\"loop_fields!\", \"Float32\", \"BaseBenchmarks.SIMDBenchmarks.ImmutableFields\", 4096)"] | 4.91 (20%) ❌ | 1.00 (1%)
["simd", "(\"loop_fields!\", \"Float32\", \"BaseBenchmarks.SIMDBenchmarks.MutableFields\", 4095)"] | 5.03 (20%) ❌ | 1.00 (1%)
["simd", "(\"loop_fields!\", \"Float32\", \"BaseBenchmarks.SIMDBenchmarks.MutableFields\", 4096)"] | 5.09 (20%) ❌ | 1.00 (1%)
["simd", "(\"loop_fields!\", \"Float64\", \"BaseBenchmarks.SIMDBenchmarks.ImmutableFields\", 4095)"] | 2.30 (20%) ❌ | 1.00 (1%)
["simd", "(\"loop_fields!\", \"Float64\", \"BaseBenchmarks.SIMDBenchmarks.ImmutableFields\", 4096)"] | 2.28 (20%) ❌ | 1.00 (1%)
["simd", "(\"loop_fields!\", \"Float64\", \"BaseBenchmarks.SIMDBenchmarks.MutableFields\", 4095)"] | 2.38 (20%) ❌ | 1.00 (1%)
["simd", "(\"loop_fields!\", \"Float64\", \"BaseBenchmarks.SIMDBenchmarks.MutableFields\", 4096)"] | 2.38 (20%) ❌ | 1.00 (1%)
["simd", "(\"loop_fields!\", \"Int32\", \"BaseBenchmarks.SIMDBenchmarks.ImmutableFields\", 4095)"] | 4.78 (20%) ❌ | 1.00 (1%)
["simd", "(\"loop_fields!\", \"Int32\", \"BaseBenchmarks.SIMDBenchmarks.ImmutableFields\", 4096)"] | 4.93 (20%) ❌ | 1.00 (1%)
["simd", "(\"loop_fields!\", \"Int32\", \"BaseBenchmarks.SIMDBenchmarks.MutableFields\", 4095)"] | 4.62 (20%) ❌ | 1.00 (1%)
["simd", "(\"loop_fields!\", \"Int32\", \"BaseBenchmarks.SIMDBenchmarks.MutableFields\", 4096)"] | 4.74 (20%) ❌ | 1.00 (1%)

["simd", "(\"manual_example!\", \"Float32\", 4095)"] | 14.62 (20%) ❌ | 1.00 (1%)
["simd", "(\"manual_example!\", \"Float32\", 4096)"] | 15.00 (20%) ❌ | 1.00 (1%)
["simd", "(\"manual_example!\", \"Float64\", 4095)"] | 6.45 (20%) ❌ | 1.00 (1%)
["simd", "(\"manual_example!\", \"Float64\", 4096)"] | 6.49 (20%) ❌ | 1.00 (1%)
["simd", "(\"manual_example!\", \"Int32\", 4095)"] | 9.91 (20%) ❌ | 1.00 (1%)
["simd", "(\"manual_example!\", \"Int32\", 4096)"] | 10.46 (20%) ❌

Parsing

["misc", "parse", "DateTime"] | 1.51 (15%) ❌ | 0.00 (1%
["dates", "parse", "DateTime"] | 1.47 (15%) ❌ | 0.00 (1%) ✅
["micro", "parseint"] | 1.70 (15%) ❌ | 1.17 (1%) ❌

Latest benchmark at: https://github.com/JuliaCI/BaseBenchmarkReports/blob/a6f383f75c9cb427e1e93b97e559cd31189c1bdf/e1dfb30_vs_df1c1c9/report.md

[KristofferC]

@nanosoldier runbenchmarks(ALL, vs = ":aa/backports-0.6.3")

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @ararslan

[ararslan]

Wow, overall that actually looks really good.

[ararslan]

@nanosoldier runbenchmarks("problem", vs=":aa/backports-0.6.3")

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @ararslan

[KristofferC]

@nanosoldier runbenchmarks(ALL, vs = "@df1c1c939bf1c582cf040b5ad235933242aeacfb")

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @ararslan

[KristofferC]

I updated the OP with the results of the latest benchmark. I scrolled through kinda quickly, so feel free to add things to it. While we are doing great on a bunch of micro benchmarks it is a bit worrying that many of the larger examples (problem) are quite heavily regressed.

[KristofferC]

What happened with replace? Responsible for the regex_dna benchmark slowdown.

julia> const seq = "atgatctaattctttttgggctt\ntgttcgatggtgattcagaatctttatccggtcgcttccctgtagctactttgtggggat\nattgcccggggattatagggttgagatcgtttcctaaaagtatttaaaccaagtagactt\ncaactaaactacatcagaacatcgtgaagacaccatacgcggtacctttatttaccgata\nacatttcttcaagaaataccggtaagcagcataatgaccctaaacagctcggggtatcgt\ncgtagttttaaattttatttaggttactgctcaaggaataaaaactaactatttaattta\ntaataatattacaaggctcacactgattagatttgtctataagacttcgcgatcccccat\ntaccggattgtcttaagaataaactagataaaccatgcattttctagataaggcctttag\ntctaattagatacaaaaaacacgatagttgcatccttaatttattgtgtcaaacctggaa\nccttttaattacccgcaaatcactttatgtcgagactacctctgaaatttattatctacc\ntaccgcatgaggacttgaaccatcttgtaggagttatgtttattagctaagattcgttta\ntcctgtagcggtccatgtatattcaacaagcaaaaagcactcagaattgtttttagttga\ngtcaagactgatatataaataagtttccctagttttttcgtggtgggacgatattgaatt\ngaatcttaaccgaagagtttcccactctgtcgcacaataatacacgccaatatttccagc\ncctgcttatgccttaatcggttactcaatctcccattgaagttcattttgatctgcatag\naagtttcgggcccagccttttttctgccaccttcctccaagctctgtagacgcactctaa\ngattgatgctcacatgtattaattctacattaacataaatatataagtcatgcatcttcg\nagtaaaatatctggttctccaacatgtcctggcacgtatcgttataatgcccatacatgt\nagtattaaaatgattgggttaactggatattaagatcatcgaaattgtaaagtcaaatta\nacaatactgtctcaagaccgtgtattcctcgtgctcggaagggctattacgcttacttcc\ngttttggtatcttaatatgactttcaaaaattaagttgcagtgagtcctacctgcgtgca\ntcggttagcaagagtataaaagttgtttaaacgaactacttgctttacaataccggtcgt\natatatcgccgtgaatccagaagattgtcttctttggattatcaaccgagatcctgtgga\nccgatgttttgggaccttcacagaggactccaggtagagctcgcttttgcattaatctaa\ngaattgtacctctctaaaagatctaaaacagtgaatgtgtatttcatggaaaaacacaga\ngaaacgtaaattactttaggccgaaaggcacatgagttattatacatatacgagatggtg\ngtatacatcgaattcggggcatacactatagttgcattgtatttagctgctttaaataat\natgatattaccttccttacataagacattaccggcataccctggttttcaacttgtgggg\nctttttgacgatcgcactctcatttgatccgagtagggcggtgacccctgcttttcaaat\nacaaaaatttcgctatgaaggtaatagattacttttcgctgttatgatagaaacggtaaa\ntttaaaattgaaacttctagaaaagtaaagtaacgagaaatgattttgtgaataatgcgg\ntcatgattgcgcaagtaagaaaaaaaggcaaaaggatgcgcggaatagaaacttatcagt\ncacgggtatcttgatttcattcttcttgtcaattgccgacataggatgaaatcagattcc\naatgcaatacacagtaacccccacccttgattgtaatgtcgatttgaagttgtacgcgtc\ngacgaagtggatagtatacgggccttttgtacggtgcgatcaactatgaatctcggcgag\nttagatggtcgtacaatctcacacatagaggtcacttgcctgtaatgacgaattttcggc\ntaggtactcgaactttattagaagtaaaaatgtgggcaaaagaaggattccattttacaa\ngacgattacaatgagttacatgtctctcaacgtagtctttccctagtagtctttgaacta\ntttaggtactccagaaaattttagcaaagggtttctgtgtgaatccgccattcatgttta\ntgatggaacaataagaataacgccctcgtatgttatcgacagtgaagtcagcagttcggc\ncaaaaacatattcaatttagtacagatccccagaagttaagctaagtgctctaaaatggc\nctaaacggttatcaaagtaggtctaattactatactaacgggtgcatcgtaataactgct\ngtcgatgcaacactatatgatagtgtcgttttgctatatatgtacaatgtgacaaagaag\nccttagcgattcttgcaaacttaggacttcggattctcaatcttaaatgtccgaaaacgc\naaagattcaaaaatttaatctatgagcagatatgcctgatggtgactacgcgtatgttaa\nggctaaatgttgacaaccgcacacataatcgaactattgatagtcgggagcataaccagg\ntgaacgtactttgttcacgacatttattgacatgttctaaatacgtctcaaaatcacggc\ngcactagaaaacgcaatcaaatcattgtcctggtttaagggccgtaatgccggtagtgtc\naaacttcatgagaactttagctggcttttggccagtatttagggaccaagagcactagcc\nttaagctgaatattttgccatttatctactgttataactttaaaacttggtggcaccaga\ncttgtcgatacacacgcatcaatctgtaacgtaaaaggtttactaagaacaagcgtagga\nattgagtttatattatatttaaactaaaagatgatattagcttctgagggcgatagggct\nccaaatcataaagaggaatatattattacacgattagaaacccacaacatacctcgaatc\ngcccaaaagtttgacgaaacttggcagtactccacatctcagtaatacagttgggagagt\nctcaaatgttgttttattactcaatgaaccaccctcataatttcactgctgttccattaa\natttgcaaacgatcatttgctttgaagaaacgtaaaatcgacaaaattacagataagtag\natgcataataaaaaaaactgctcgctataacacgatcatcgtgcattcttacttaggagc\natcacccgcacaataacgtaccttaaactacaacactattagaccgagtactgtaattca\ncgaaagctcaagctcgcattgtaaagaacttgctctctcgtaaaatgtgataatagtttg\ncggagaggattcaattattttccattgcacctactccactagattcgataaaagaaggtg\ngtcctcccttaaaaagaaatgttaagtaacatcggaaccataagcaaagcatgtaagtga\naccgtcatccttccctaagaaacataaaggtttttaataatgtcgactgtgaactataac\ntgcatcctttcctgacctactccggttccttgttgttatttctgaacgagaccagtagat\naaacaatgtaaaccacagtgggtaccaatggtgcatgtgacgctaccgttgttttaagtg\ncccgtacaaacataagaagtcataatcttacttgaaattaattttgccttttattttttt\ntcaggctcgaaattaatgatttgttttttttgaccttctagttacgctaatatgcggtcg\ncctgtggtttctattgagtcctataacgggatgggatctaatacgtttggttactagtaa\nacaaggtataaatttgataccggagtatcaactgtataacatcaagctttatgactcata\ncgcgaagtaatgacacaaggctttcaggagatcgcgagtacagagccactaaggggtgta\nttacgatagtgacaccaccgagcgcactcactccccaagtagatttatgatcctacgcta\nagtattagatatataaccaaagaggttctagtcagtgcaactcttagaataataattagc\ncggttttgcctttttaggcctaatgcaatattcagctagcccttatgtatctcgcgttcc\nacagcaccactcatggcacgcgtttaaactaatcaaatataatctatgaatgttatgcca\ngtacttgaataaatcaggttttttataagtccttgcatactctcgttatatactgttaga\ngtcttaccccatagaaattctttcatctgcaaacttagaagaattctcagctacggggag\ncataaagtccccaggatgttgacaaatacaacaaatgtggcttatacaaacactccatat\ngaaaatcgaaccctcgtggtagttttagccgaaccttgtacggataaatccctccatttt\nccaatagcagatacctatcctactacctcgtggtattaaattaaagcttgaaatatagag\nctgcatagcttatccaattcccaagcacgagtctaccgtcgtaaccacgatttgatttac\nagacgctagagcaaacccatctttaaacatataagtaaaaattaaagggtgagtgcgtac\ngtgtttactagcaacttcgcttattaagacaattgtttataagccataattaaaaacata\ntgttcaacaggttcattgatatttgtaattgcacaggtttttaataaggatctacgtaag\ntataatgaacaaactttttaccagagttatattctgtactttgaaaatgctcctctaccg\nccttagagactttcaattagattttttgcagttaatctatgcgtaagtgaaccatgcaag\nggatgcgattcaaccgcctcgtgctaaccctatcgtctgtctcataactgtaggtctaat\nataattttcagttttcgaacacataaccctttgaaaatctgctatttaatgtctcacctg\ncatgcactatcttctatactgctcagaacggctatacgtcactatgctccaagtgacgat\nttaaacgaagcaaggaataataggtttattttagtgcaaaacaattaagtgcggactacg\ntgctctttacaataagccttgtgattgggctataggttaagtcccatattaacgatctcc\naatgtacaaaatcgacaatcgctttgcattacccggttactagtcgaattacagatagct\ngttagatactcactctaattttggacaacaatcccaatcttggggtcgtctatcgcctga\nagctcgtaaatccttccatcttaaacgattacatattatagacttgttcggggtagagat\natcacagttgtgcaaacattgtaaatcgatactagtttatgttggtagtctagttgcttt\ntaccattccccgaaaaacttgatctactatttcgacaacagtaaacttgaactaggtaag\ntgaaaacagagaatgcctcatagtgccactatttgtccactatatgtaagtgtagcttta\ncataatccactatgactgagatcattacggcctaggaaagcagcgtagaaaaaaagggcc\ncggatattacgactgtaactataaaactagttactggtagcgcgccatgtatagatttgt\ntttaccggttgtggttgcgttaacgaatttcagccgcgaaaattgatccgttaaccagtc\ncatctcgacttctataaaacgataaagtaaagttgatgttcagcctccttcttatggttg\ncatcgagagtacactactcagtgggaaatagatcggggttcctacttcagattgtattat\nctaggcaattgccgattgtgccatacctggataaaataagctacctacatgtgatgctta\ntctattatcgtcatactaccttagggtgtcctgttgaacgctacattaatctttagccgt\nttgagatgttccaatggataggagtctaacgcatgatgaagtttaggaaggcagagcatc\nccactaagtatgtgacagtgtatttcgaaacgagacgttataaatagaaaaaaggtcctt\nctggttctattctgctgaactattgaatggaaagattggttgacctacgtactatttgct\ntgaagtcatcaatttgacggggtgagagacatatggtgcatactttacggactctatatt\nttagatcagaagcttagcagtcttctctacaccccctcacgacataattgcttttaagaa\ntctatgtttgattcctctacgggaattcggatccgttcgcatgtgcggtttatctaaacc\naggggacatatgttcagctaaagcatacgaacactttgctaactagacgtatgtatagta\ngctataaatcccgacgatatttacaaaaagaaatgagactcaaatatatacatagcgacc\nctacacttattcgcaccctgatctaggcgatcctagcacccacacccgaaagtgagcact\nagtgtcttccgtattaaatttactgcagttgagattttagttgtctactaaggattactc\ntaacccgtaataaggatcaagactcggtactagctttactatcattccctatgtgttttc\nctaactcacaagggtacgtaccagcctatgtaattacaataatgataaagacacaaagga\nagtaactttacaaatgagtctccagttacactagcttagtccctcccatcttgctttgaa\ngtctaaatacgcaatctctgaggatatacagcagaagaacactcataacgttggagtcca\nagaattagactcatagggcccccaacatttaatatgtactgtgagtttgaaggtgttcta\nttgttaattcctgctcttgatacatgacacgtactccgtgtttaaggcttcggactgact\nttctttcataagttgagcaacgaaaatttcagaatcgataagttggattcactaactaat\nacggctgattgaaaactccactccggacctatatggtcgacctttatacgtaaccgatat\naaaacttataggctggtatatcgagccttcctagcgcaatttcggatggggtttcttcta\nctactcaacaacggaatagtctttgtttagtaaaccagagctcaggacgcccaatacgta\nggagagcgctgtggagcatgtgtcattatggactggagcactcttaaatcactctgcgtg\ntgctaaacgatagatcataacatgtcctgagtaaattttcttgatacgtcgcaatatacc\ngttattagttaaacgttctcatccgtcatgcgtgaaatacggctgtcgtgctcagatata\nctattagcgactcatctcgcctaacacgcacacgtataaactcggaatgactgccgctct\ntacatattagaaatacagactacaccacggaagcattgggtcattctcaaccgctgtata\naaagatgattagtcttataataagattaccaaagaggcagaatcatgggtagtaaatcta\nttattcaagtgattaccgtcgtgtaggcagggagtgaggacgagatggtactcaggacaa\natattaaccggacgaagtggtttacgtcgtactttcactattagtagtaaatacaaggta\nacaccggggaatagtactaaatataatgatatctatcttcgggagaacgagtcgtctatt\ngctttgaacattctcaaggcgtaaaatgtgctgacttatagcatgatacaaccgattgtt\nacttttgtctattcaaaagattgaatagttttttatacaaaagccgcatacttatgacgg\nctagtatacagtttcatcccctagcatcaatgctatggacagtattgaacttataggaaa\nttcttctaatagggcaaatccgtcgtgatgcctattttttttcagtcacatcctcaaatg\ngcactagtattgtcgggatcccattaacaggctcaaccacgagctcacgcgaggacatgt\nagtccgtatctttaacgaagcgacagcgacagaactcccatggataaccaattataaggc\nccgtaatcctctagacatcgtttaccaataaatccgctttctccgtaatcatgttgaata\nccccagagtagtccagatgataaccgatgaaacacaagtctttctcaatgcacttacggt\ngaacttattaccgccaacgtagctcatcaaggttgcgacatctagttgtgtgtttgcgac\ngagcccagcgaacttcatcaactttcgtatattcaacgccttgtaattttactttaagac\ngcctggtgatgtagattcttagataatcagtttgttatcggctgtactttaccataattt\ncacaggtttcaggtcaagaagattatagctgtatatacagttccatgctcggtgcacaga\naacgtgatcggataataatcaatcgcttatgtcgtctttaggcgtatccaatacatgccc\ncgataccgcagtgtatttcgacatgtaggtataccgtcgcatttgagctcgagtcaggac\ngtcagctagattagattccttaatagaatataccgacctctagtccgaactaaactatag\nataacgccaacttcaggttaattgtctagtcgtctgtttgcagatgggattcttagatga\ngtgagtatcggccatattggttcgagcactttagtttttgatgcataggatatgcaatgt\natagctgaaagtactttatctgtttcaaactcacattgattaaaccggtaaacctttaaa\ngactacaagaaaatattcagtgagggcaattttgtcaatcacaatcttccagctagagat\nacttcacaatttgtcttgaggctacgcaacattagacggattttcgcgttttattgaaat\naatcgaggggcccaagagtatccatagttcattttgtaagatttctttacaggcttatta\ncagcttcttcagactcctacatgcttacgagttatatgctagcatgtgaacaatagatta\natatacaggaaaacgtacattgagagagatgaccctacacagcgcaaccgttgagtactt\ntcattaaagggtaacgctctcgagacagcatccttaagatggccttattgtcaaatcatt\ntgcagaagtacgcaagatccctaaccaacgtagaagaatccctacaaacacatgagacgc\nggtgaaaatagacagggtgttagtattcaatcttcggagtatcaatttcgccaatcttgg\ntgagaaagcataccctttcttcagagaaagaagatcaatcataacactatctttaacgag\ngtacgcacgcgcatcattacctgcctccatggatctttaggatagcggaaagtattggca\ngcgtattgtgatttcgttcctactttatcaatttcacattcatatacatgtcttttatca\naaatcgccaataagataggatgagctatattagatgctagtagagttcgcgccaacatca\ntcgataggaatactcaggacagcgtgataggacttttcaatccctaatactctctataat\ntataactctctcttaagtttggaggcagtaacgcgctctatataatcagtttgctgcacc\nattcttcagcctctgatacatacaaataaattccacagcagtaagagggtttaattgaga\ncatcttgggaacttaggattttactctaacatcaccgaaacgattattggataccgtacc\ntaaacgaactttctcaaggcagtaatataggacatccgcaataacacaaatgctgcctcc\nccaggagttatgtcttcctggaggctatatcttacacccactcactataggcaaactaaa\ngtttaaatgttgattgtctaaaaaaaagatagataagagttggccggcgtagcacatgcg\naaagtgaatcgtaagctataattctctggacttgaagttctgtcctgttcctctgcaaga\naacaaacttcctttaaagctatttacgacgcacatctcagcaagttataaacatgttgga\nagtttctagtcggaattcccaaagaacggatctatctaatgcattcctacatttttcctg\ntctgccgatggtgccatcctattcaaagaatttcttaaaagtagattaaatgggactttt\naacaatgagtaaccttacgcctctaagggttcctcgagtgccatacaccagtcaggtccg\nagccacatacacggagaacattctaacatagcattctcaactcgatcatttgcaggttac\nttctttcctatcctagtgctaaaaatcatacttgcaatcccatagcacggattaagaacc\ntaagaaacaattcagtaaaacatgttcgaattcttggtatgggaacatcattgcagctat\nggtctaacgcattaatgtttgggtacatcttccatcatataaacaggaagagtctgacga\ncagggagtgcttgcgatcatgtctatcattgtgaaatcaaattgtagctcacatgtcgtc\ntatgagagcgtgtatccgataagatttagaaaaatagaagtcgtataagatctcactgaa\ncttttgaatgaatgtgaagcatatatgatctgctttaataaaactttatccataggatac\ngtttccaaatcaattcaataattattagtcaaaatagataaggatgaacaacctgaaggc\ncgatcggacgtagaaagtggtcccatcactttgagttgatattgttgaaccacacgttat\ntatggttttcaaacagtctcaggatattgtatatacagataatccgataccagttgtctg\nacgcccctcttacgtaccccaccctttgtgacgtttaaagcagttgttcagtattttaaa\nctaggcggcaactaatttggaaagaagcacagtggatatgtctaaattcttgttattcag\ngcctgaatttaatacaccgcatagttaacttcgcggtagagttgttcatcatgcctcctc\ntaagctaccacttctatgatacaccaatagttgttctacggaatctgataattggccaag\ntcataaacttccgctgcgttcaacccccttgctcgaatatccaactcgaaaagacagcct\ntttggtgtccggaacaaatcagttacttcttttctgatgttaattctctgtggtcagata\ncagaccaaaaactccgcggatttaccatcctccaagaacaaatttgcatcaacatagcat\ntttggctacatattctaagtctcaatagtttaggttttcaactacattatcccaacatta\nggattggaggaataatagctgggtaagtccccttgcgtctacaatcgactattttttatg\naatatgcttctgccgcacctatggttattaaaaaagtcatgactttgaagaaccctgaaa\nagatagatgaatcaggtgtaatggcagcagccaaagagcatataattagcaacactctaa\ngaacattatagatatgatgatagcgatcgtcatgatgttatccggtcacaatagtagctt\ncatcagctaattcgttttgccagtggtgacttgcgctggaagaatcgttatacggtccct\ntccctcttgatacggtgggggcttattcaaccgcgtggattgggttgtcatacttgcatt\naaacgatgtaaaccatctagtagtcaactatactaaatcacaaaatagtgatcaatacat\nacccgcttcatggttttaaccatttaattgattaaagatattccgctaagaaccattatc\ntacctaaactgatcgccgtatcctagtagtttgaaatttgatgtaccgtaatgatcaacg\naagtaaaacgttatattgtatgtagaataataggtcttggagctaaatgatgtgattggt\nagtgaagacttacccttacaactttaccggtttctcggaagaatatactagagaatcaat\ngcatgggctacataagcactttagtctaatgagataaaaaatacacgagtcttccatcat\ngaattttttgtcgaaaaactcgaacctggtaatttaaaccatatatctttatgtcgtcaa\ntaactctcatatgttttatataacttcccaatcacgacttgtaactgcttgttcgactga\ngctgtttgagctatgaggccgggatccggttgagctacatctatttgctacaagaaaaat\ngaaagcacatttgttgggagttctggctacactcatagagaaataagtggcccgagtggg\ntgcggcctgcctccatattcaagtgtatcttaaaccaagtggttccaacgctcgcgctaa\nagaattaaagcctttatttcctccacggagtagcccgtaatccggttcgaaagagaccat\ntgaagttaattttcatatccagtgaagtttaggcacaagcatgtgttctgccacatgcct\ncaaagcgctcttcaaccaagatatgattcatcctaacttcgatgaatgcgtctgtaacat\naaatatagaaggaatgattcggcgagttaattttcgccttctccaacatggcatccctac\ngttcgttataaggaccatacatgtaggttttaaaggtttgcggttaatcgatatttacat\ncatagaaattctatagtcaaatttacaagactctagatactcactcgttgcagccggcta\nggaagcgctttgtaccttacttcccttttcgttgcgtaatatgaatttcatatagtaagt\ntcaaggcactcatacctccgtgaagagggtagatagactattaaagttgtttaatagtac\ngtattgatggaaatgacccgtaggagatttaccactcaatccacaagattcgctgctgtg\ncattatcaaaacagtgcatgtcgaaacatgggttgggtccttcaaacacgaatccaggta\ngagatacctttgcaattttt\n"

# 0.7
julia> @time replace(seq, r">.*\n|\n" => ""); 
  0.057133 seconds (12 allocations: 119.766 KiB)

# 0.6
julia> @time replace(seq, r">.*\n|\n" => "");
  0.000565 seconds (11 allocations: 119.750 KiB)

Reduced to:

0.6:

julia> f(seq) = search(seq, r">.*\n|\n", 1)
f (generic function with 1 method)

# 0.6
julia> @btime f(seq)
  100.660 ns (1 allocation: 32 bytes)

0.7

julia> f(seq) = findnext(r">.*\n|\n",seq,1)
f (generic function with 1 method)

julia> @btime f(seq)
  60.167 μs (1 allocation: 32 bytes)

Ok, it is the new match options

julia> @btime findnext(r,seq,1)
  60.168 μs (1 allocation: 32 bytes)

julia> r.match_options = Base.PCRE.NO_UTF_CHECK
0x40000000

julia> @btime findnext(r,seq,1)
  108.639 ns (1 allocation: 32 bytes)
1:22

Ref #26731

[KristofferC]

Regression for shooutout k_nucleotide seems to be due to join being slower

julia> chars = [rand(Char) for i in 1:10^4];

0.7
julia> @btime join(chars, "");
  4.729 ms (30019 allocations: 1004.14 KiB)

0.6
julia> @btime join(chars, "");
  771.538 μs (20 allocations: 66.67 KiB)

Relevant part of the profile:

            2228 ./iterators.jl:1085; iterate
             2225 ./iterators.jl:1078; popfirst!
              22   ./array.jl:682; iterate
               17 ./array.jl:707; getindex
               4  ./array.jl:174; length
               1  ./int.jl:53; +
              2109 ./sysimg.jl:19; setproperty!(::Base.Iterators.Stateful{Array{Char,1},Union{Nothing, Tuple{Char,Int64}}}, ::Symbol, ::Tuple{Char,Int64})
               2 ./essentials.jl:123; convert(::Type{Union{Nothing, Tuple{Char,Int64}}}, ::Tuple{Char,Int64})

Seems assigning to the nextvalstate::Union{VS, Nothing} in Stateful is slow, @Keno.

Probably fixed by #27685.

[KristofferC]

Regression for perf_revcomp seems to be that julia 0.7 is a bit worse on dealing with the Core.Box that comes from the comprehension in this function https://github.com/JuliaCI/BaseBenchmarks.jl/blob/master/src/shootout/revcomp.jl#L40:

Without changes:

0.6:
julia> @btime perf_revcomp()
  1.547 ms (1051 allocations: 92.67 KiB)

0.7:
julia> @btime perf_revcomp()
  2.443 ms (1223 allocations: 87.31 KiB)

Adding let line = line around the comprehension so no Boxes anymore:

0.6:
julia> @btime perf_revcomp()
  203.506 μs (882 allocations: 87.41 KiB)

0.7:
julia> @btime perf_revcomp()
  164.448 μs (883 allocations: 79.38 KiB)

Damn Boxes.

[KristofferC]

Regression for spellcheck seems to be reduced to:

const ALPHABET = "abcdefghijklmnopqrstuvwxyz"

function edits(splits)
    s = splits[1:end-1]
    m = Matrix{String}(undef, length(ALPHABET), length(s))
    i = 1
    for (a,b) in s
         j = 1
        for c in ALPHABET
            m[j, i] = string(a, c, b[2:end])
            j += 1
        end
        i += 1
    end
    return m
end

word = "Foobar"
splits = [(word[1:i], word[i+1:end]) for i=0:length(word) ]

# 0.6
julia> @btime edits(splits)
  9.300 μs (288 allocations: 10.39 KiB)

julia> @btime edits(splits)
  75.803 μs (1380 allocations: 49.39 KiB)

Further reduced to:

# 0.6
julia> @btime string("foo", 'b', "ar")
  36.804 ns (1 allocation: 32 bytes)

# 0.7
julia> @btime string("foo", 'b', "ar")
  443.838 ns (8 allocations: 288 bytes)

Contrast with:

0.6
julia> @btime string("foo", "b", "ar")
  35.629 ns (1 allocation: 32 bytes)

0.7
julia> @btime string("foo", "b", "ar")
  32.122 ns (1 allocation: 32 bytes)

@StefanKarpinski it seems the replacement of string(::Union{String,AbstractChar}...) in #24999 lead to this regression. Doing something like

function string2(a::Union{String,AbstractChar}...)
    s = IOBuffer()
    for x in a
        print(s, x)
    end
    return String(resize!(s.data, s.size))
end

helps a bit:

julia> @btime string2("Foo", 'b', "ar")
  110.308 ns (3 allocations: 192 bytes)

but still 3x slower than before.

[ararslan]

I think BaseBenchmarks should be working now?

@nanosoldier runbenchmarks(ALL, vs="@df1c1c939bf1c582cf040b5ad235933242aeacfb")

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @ararslan

[iamed2]

Has anyone looked at the a::Vector{Int} == b::UnitRange{Int} benchmarks yet?

julia> a = 1:10000
1:10000

julia> b = collect(a);

julia> @benchmark a == b
BenchmarkTools.Trial:
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     8.973 μs (0.00% GC)
  median time:      9.374 μs (0.00% GC)
  mean time:        10.479 μs (0.00% GC)
  maximum time:     193.336 μs (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     3

This should be nanoseconds

[KristofferC]

The reason for the regression is quite simple. In 0.6, arrays and ranges were not considered == so it returned false straight away. They now are so it actually does the loop.

[KristofferC]

Updated OP with the latest result. Now with deprecation warnings fixed and some performance fixes in, it is starting to look quite nice!

[vtjnash]

From ae8e95f#commitcomment-29628207 (update of #27539), there's a couple more potential regressions to investigate:

mapr_access (array) - we appear to be faster than v0.6 by 2-3x, but slower than we were 2 months ago by up to 40x
BaseBenchmarks.ArrayBenchmarks.perf_mapreduce - this isn't mentioned in the v0.6 report, but it's reported as being 50x slower than two months ago
problems: euro_option_devec, seismic

[vtjnash]

Perhaps #26198 relevant? Interesting that a performance PR got merged without running Nanosoldier.

I don't think there's any nanosoldier tests for this. This change is due to #23528, which changed the definition from symdiff(x) = x to actually computing the actual symmetric difference. The expected nanosoldier results from this are #23528 (comment)

[KristofferC]

Let's run an update after #27945 (comment).

@nanosoldier runbenchmarks(ALL, vs = ":master")

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @ararslan

[KristofferC]

Sigh, last one was against the wrong target...

@nanosoldier runbenchmarks(ALL, vs="@df1c1c939bf1c582cf040b5ad235933242aeacfb")

but it is interesting that this PR that does nothing gives so many memory improvements

[vtjnash]

Er, it's nice to know that the version number bump will make several benchmarks much faster relative to master 😜

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @ararslan

[jebej]

what's wrong with the Laplacian benchmark? Deprecation?

[KristofferC]

I think the iter_sub in Laplacian comes from #25890.

[KristofferC]

Some more fixes in (mostly to SIMD). Let's check again.

@nanosoldier runbenchmarks(ALL, vs="@df1c1c939bf1c582cf040b5ad235933242aeacfb")

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @ararslan

[KristofferC]

Just to make sure nothing crept in:

@nanosoldier runbenchmarks(ALL, vs="@df1c1c939bf1c582cf040b5ad235933242aeacfb")

[nanosoldier]

Something went wrong when running your job:

NanosoldierError: failed to run benchmarks against primary commit: failed process: Process(`sudo cset shield -e su nanosoldier -- -c ./benchscript.sh`, ProcessExited(1)) [1]

Logs and partial data can be found here
cc @ararslan