-
-
Notifications
You must be signed in to change notification settings - Fork 642
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Slower than Catch in realistic test cases #14
Comments
It should be because of the macros and code bloat - currently the CHECK/REQUIRE macros generate more code than they should - this is a planned optimisation - and when I fix this - it should be on-par with Catch for heavy usage as well - but making it any faster is maybe impossible. see the comparison in code generated by the macros - doctest versus catch many doctest internals will be reimplemented - I just rushed this release and atleast the interface should be stable. A step further for even faster compile times would be to introduce macros like |
You're probably right, a bit of encapsulation would save some compilation time if this block of code is not compiled for every assertion.
This is me indeed :) I was very surprised to find such a large improvement. Thanks for the time invested so far and congratulations for the very fast include times :) |
If you are interested, I have been able to reduce the compilation time even further: http://baptiste-wicht.com/posts/2016/06/reduce-compilation-time-by-another-16-with-catch.html I have reduced the REQUIRE_EQUALS macro even further for another 16% reduction in compilation, this is a total of 27% reduction from the original compilation time of 764 seconds to 554 seconds. |
I will probably supply such macros in a separate header as an extension - but before that I will optimize the current macros |
…r compile times and less code bloat (not benchmarked yet) - relates #14
100 files variables and assertions like this:
MSVC Debug - 13 sec (old was 19 sec) MinGW-w64 GCC 5.3.1 Debug - 47 sec (old was 100 sec) Catch: MSVC Debug - 62 sec So the dev version of doctest should now be quite faster than it's previous version (by around 40% for MSVC and 53% for MinGW-w64 gcc 5.3.1) - even though Catch is slower than both the old and the new version of doctest according to my benchmarks - but I suspect they still ain't real enough. The single header is now assembled from it's 2 parts - I'm going to make a header - But I'm not sure if I should circumvent the Also you don't handle the case if the expression throws - so in your version of the macros you have cut away more than just the expression decomposition - I think I will save the try/catch blocks though. All this is in the dev branch - I will make a 1.1 release soon (in the upcoming weeks) and this will go into it. Thanks for pushing me to optimize this - it was on the roadmap but I only suspected it was slower - didn't have a clue how much. What do you think? Test it in your codebase and report :) |
removed WARN assertions from the assertion count
Sooo I made the first set of macros in the and here are some compile time benchmarks with MSVC 2015 in Debug (no other compilers/modes tested): 500 files 93 sec (expression decomposition) vs 35 sec (fast macros) - but the majority of the time in the slow case is spent in linking 1 file 48 sec (expression decomposition) vs 4.6 sec (fast macros) - majority of time spent NOT in linking - but in compiling!!! And I even retained all the functionality of the macros - they break into the debugger and the REQUIRE_EQ macro ends the test case on failure! I could have made it even more optimal but I don't want to cut away these features. Give it a try and report what happens in the real world :) I will make macros also for less than/greater than/etc. soon. |
… header - implemented the rest of the comparison fast macros (and also the unary ones) fixed #14
I ended up removing the |
This sounds great! I'm in vacation now, but I'll test this on my codebase next week. |
I've been able to test the last version directly on my codebase. I made it so I just have to define a macro to switch between Catch and doctest. Here are the timings:
(The timings are not the same from my blog post because I did this experiment with less features enabled) So, it's faster than Catch in all cases! Even in the case when using my very minimal Catch macros, it's faster and has more features (since you retained much more features than I did). The stripped macros are stripped so that they contain only the call to fast_assert and fast_assert_unary. Not sure if it's worth doing for me and certainly not worth doing in the library itself. I had to do one change though. In fast_assert_unary, you are taking the expression by reference and then converting bool inside. The problem with that is that the expression is odr-used and in my case this leads to undefined references at link time with fields that should not be odr-used. The same problem could occur in fast_assert, but I don't compare binary things that cannot be odr-used in my case. I simply did a new version of fast_assert with a bool value as parameter. Is it really important to do the conversion to bool inside fast_assert than simply taking a bool parameter ? Thanks for this nice update! |
well if about the odr-used stuff - can you give an example? I'm a bit brain-dead at the moment |
But only the conversion to bool will be evaluated at this place, not the evaluation of the expression itself, isn't it ? If you have a function f(x) that returns a bool, the evaluation of f(x) inside REQUIRE_UNARY(f(22)) will not be evaluated inside the try/catch block, but only inside the DOCTEST_FAST_ASSERTION_UNARY. Only the bool will then be passed to fast_assert_unary and therefore the try catch should not catch anything. The only thing that would be caught by your try/catch is an exception thrown by the conversion of the passed value to bool and in most cases, this will simply be a bool. Unless I'm missing something big.
Yes. Imagine you are testing type_traits:
somewhere:
In this case, there is an undefined reference because is_bounded is "used" (because of the reference to it), its value cannot be directly replaced by the compiler. And therefore a definition should be present. This is not a huge issue, since in most cases, the definition will be available. This is mostly a corner case and I can deal with it myself I guess. |
damn - I completely missed both of these things... I think I'll end up making 2 sets of assertions - those who can catch exceptions - in the macro, and those that cant - the fast ones. And maybe the user will switch between them with a define, or they will be separate macros - perhaps and the fast versions of the unary macros will take just a bool and not a Thanks again for pointing out these issues to me - and also that the original macros were so slow! I do care about this. |
Sorry to give you so much work ;) As for the reference problem, it's probably secondary. The first problem is more important for people who care about exceptions. |
after 3+ months of radio silence - I've fixed all the new assertions! checkout the benchmarks! roughly said: doctest 1.1 (which will be released today probably) is around 3 times faster than version 1.0 (released on 2016.05.22) when expression decomposing macros like Additionally I've added normal binary macros that don't do any expression decomposition - Additionally I've added faster normal binary macros that don't do any expression decomposition - Additionally I've added the I've also added the All macros evaluate the expressions only once. All macros stringify properly the values when the assertion fails. There are also unary assertions - like I've also added benchmarks with gcc/clang under linux. I assume working with Let me know what you think! :) I think I've reached the limits of compile time for assertion macros. |
That sounds so awesome! As soon as the version 1.1 is released, I'll update my code and do the measurements again and make a post on my blog. I'll keep you up-to-date with the results. |
…r compile times and less code bloat (not benchmarked yet) - relates #14
removed WARN assertions from the assertion count
A blog post would be much appreciated - any kind of publicity for this framework is much needed! damn rebasing... I should really learn git. |
I made all the tests. The results are available on my blog: http://baptiste-wicht.com/posts/2016/09/blazing-fast-unit-test-compilation-with-doctest-11.html For reference, here are the final results I got (on my expression templates framework): Catch 724.22 That's pretty good results, I think. |
damn - nothing compared to my synthetic benchmarks :D but still - my fastest macros are just a single function call - just like the ones u've made for Catch. I should make a note in the documentation that for real use cases the results are not that spectacular :D |
I think it's still great. Here, we are still compiling more than 1000 test case with several thousand assertions. Moreover, this a template-heavy library so we can expect some lengthy compilation times. If we could compute the real compilation of the test cases without the test (:P), then we could compare the real difference between catch and doctest. |
Hi,
I did some benchmarking of doctest on my machine (Linux, gcc-4.9.3). I adapted a bit your Python script to work on Linux (with_gcc = 1 and removed the forced CMake generator).
First, I can confirm that in the simple benchmark case, it is indeed faster to compile than Catch by a large margin. However, this is not a very realistic test case.
I have modified the benchmark to generate 50 files, each with 25 test cases and each with 10 CHECK (using a variable and not two constants). Here are the results:
Catch: 1 minute 56 seconds
Doctest: 2 minutes 32 seconds
In a realistic test case, doctest is significantly slower than Catch. I can see the same thing when I replace Catch by doctest in one of my large test case of my ETL library.
Do you have any idea why this happen ?
I can only guess that the overhead comes from the templates used to extract lhs and rhs or by the macros generating the test cases.
The text was updated successfully, but these errors were encountered: