Added :iterate-max to control number of function calls

[jeroenvandijk]

This is an example implementation for issue #348

It allows to control the number of functions call via the iterate-max.

In local script/perf checks this doesn't seem to affect performance (before and after).

things to be discussed

• should this code be moved to analyzer.clj as suggested by @borkdude
• naming of the option
• adding an elapsed time limit

[borkdude]

@jeroenvandijk The case I've been optimizing is this:

(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))

I do that using multitime:

$ multitime -n10 -s0 tmp/sci-without-checks "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
1000000
1000000
1000000
1000000
1000000
1000000
1000000
1000000
1000000
1000000
===> multitime results
1: tmp/sci-without-checks "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
            Mean        Std.Dev.    Min         Median      Max
real        1.745       0.033       1.712       1.726       1.812
user        1.660       0.031       1.627       1.643       1.723
sys         0.078       0.002       0.074       0.078       0.083

$ multitime -n10 -s0 tmp/sci-with-checks "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
1000000
1000000
1000000
1000000
1000000
1000000
1000000
1000000
1000000
1000000
===> multitime results
1: tmp/sci-with-checks "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
            Mean        Std.Dev.    Min         Median      Max
real        1.636       0.016       1.613       1.635       1.666
user        1.553       0.016       1.532       1.552       1.586
sys         0.076       0.003       0.071       0.076       0.081

What's weird is that your branch is even a tad faster than the original. It would be good to find out what causes the speedup :).

[jeroenvandijk]

I can imagine the speedup comes from extracting the interpretation of the body 25ace7c

This is one extra function call, but two function calls less (first and count)

[borkdude]

If the speedup is reproducible then a separate PR with only the speedup fix would be a no-brainer.

[jeroenvandijk]

I'm getting different results. Not exactly sure how to read or explain the results:

git checkout 6c8852d4b78f037e4594727c17b10c018a7ac25b
COMMIT=$(git log --pretty=format:'%h' HEAD^..HEAD | cat)
script/compile
mv ./sci tmp/sci-${COMMIT}
~/bin/multitime -n10 -s0 tmp/sci-${COMMIT} "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"

git checkout 25ace7c
COMMIT=$(git log --pretty=format:'%h' HEAD^..HEAD | cat)
script/compile
mv ./sci tmp/sci-${COMMIT}
~/bin/multitime -n10 -s0 tmp/sci-${COMMIT} "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))" 

git checkout 3c01531
COMMIT=$(git log --pretty=format:'%h' HEAD^..HEAD | cat)
script/compile
mv ./sci tmp/sci-${COMMIT}
~/bin/multitime -n10 -s0 tmp/sci-${COMMIT} "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))" 


===> multitime results (master)
1: tmp/sci-6c8852d "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
            Mean                Std.Dev.    Min         Median      Max
real        1.728+/-0.1915      0.191       1.591       1.665       2.271
user        1.592+/-0.0473      0.047       1.517       1.590       1.688
sys         0.067+/-0.0049      0.005       0.060       0.068       0.075

===> multitime results (Extract body interpretation as an inline function )
1: tmp/sci-25ace7c "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
            Mean                Std.Dev.    Min         Median      Max
real        1.964+/-0.3633      0.363       1.625       1.786       2.768
user        1.706+/-0.1910      0.191       1.537       1.653       2.197
sys         0.089+/-0.0158      0.016       0.074       0.083       0.125


===> multitime results (Implement safety check)
1: tmp/sci-3c01531 "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
            Mean                Std.Dev.    Min         Median      Max
real        1.702+/-0.2160      0.215       1.546       1.620       2.316
user        1.551+/-0.0616      0.062       1.466       1.533       1.656
sys         0.073+/-0.0035      0.004       0.068       0.073       0.079

[jeroenvandijk]

With n=100. I don't really see anything that is significant or makes sense (3c01531 should be slower than 25ace7c)

===> multitime results (Implement safety check)
1: tmp/sci-3c01531 "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
            Mean                Std.Dev.    Min         Median      Max
real        1.565+/-0.0112      0.044       1.499       1.561       1.687
user        1.479+/-0.0096      0.037       1.419       1.475       1.596
sys         0.077+/-0.0014      0.005       0.068       0.076       0.099

===> multitime results (master)
1: tmp/sci-6c8852d "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
            Mean                Std.Dev.    Min         Median      Max
real        1.599+/-0.0137      0.053       1.523       1.592       1.825
user        1.510+/-0.0114      0.044       1.443       1.507       1.702
sys         0.077+/-0.0020      0.008       0.063       0.075       0.109

===> multitime results (Extract body interpretation as an inline function )
1: tmp/sci-25ace7c "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
            Mean                Std.Dev.    Min         Median      Max
real        1.606+/-0.0150      0.058       1.500       1.605       1.856
user        1.517+/-0.0126      0.049       1.420       1.520       1.702
sys         0.077+/-0.0017      0.006       0.062       0.076       0.097

[borkdude]

OK, thanks. At least we know how to test a performance regression now :).

[jeroenvandijk]

Install instructions to be complete :)

git clone git@github.com:ltratt/multitime.git 
cd multitime
make -f Makefile.bootstrap
./configure
make install

I used multitime version 2eaf07774a

[borkdude]

I just did brew install multitime :)

[jeroenvandijk]

Hmmm odd

brew install multitime
Error: No available formula with the name "multitime"

Guess I was unlucky 😐

[borkdude]

Maybe no Catalina support?

[jeroenvandijk]

Btw, not sure what you think of this:

===> multitime results
1: ./sci "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
            Mean                Std.Dev.    Min         Median      Max
real        1.616+/-0.0569      0.221       1.475       1.583       3.677
user        1.510+/-0.0183      0.071       1.400       1.498       1.814
sys         0.072+/-0.0013      0.005       0.062       0.072       0.089

Changed this line from bindings (:bindings ctx) to bindings (.get ^java.util.Map ctx :bindings)

Now I wonder how to do a proper performance test. Maybe these differences are just too small

[borkdude]

Yes, these types of changes are good for performance and PRs for them are welcome.

[borkdude]

Posted a comment in #348.