[P4Testgen] Add a rudimentary P4Testgen benchmark with via GTest.

[fruffy]

Generate 1000 P4Testgen tests for the fabric P4 program. This should take around 30 seconds.
GTest is not really intended for this but it seems to work well enough.

[fruffy]

@asl This can be another benchmarking data point. P4Testgen makes heavy use of visitors of all forms and should be affected by many of these optimizations.

[asl]

@asl This can be another benchmarking data point. P4Testgen makes heavy use of visitors of all forms and should be affected by many of these optimizations.

Yes, saw this PR. Will certainly compare

[fruffy]

Yes, saw this PR. Will certainly compare

If you can get the framework to build on MacOS, never tried.

[asl]

Yes, saw this PR. Will certainly compare

If you can get the framework to build on MacOS, never tried.

I can build p4test now. Let me check if I can run this :)

[asl]

Maybe there is a way to grab absolute filepath? Otherwise running from "wrong" directory by hands yields:

Note: Google Test filter = P4TestgenBenchmark.SuccessfullyGenerate1000Tests
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from P4TestgenBenchmark
[ RUN      ] P4TestgenBenchmark.SuccessfullyGenerate1000Tests
/Users/asl/Projects//p4c/backends/p4tools/modules/testgen/targets/bmv2/test/testgen_api/../../../../../../../../testdata/p4_16_samples/fabric_20190420/fabric.p4:37:10: fatal error: 'core.p4' file not found
#include <core.p4>
         ^~~~~~~~~
1 error generated.
/Users/asl/Projects//p4c/backends/p4tools/modules/testgen/targets/bmv2/test/testgen_api/../../../../../../../../testdata/p4_16_samples/fabric_20190420/include/control/filtering.p4(24):syntax error, unexpected IDENTIFIER "standard_metadata_t"
                   inout standard_metadata_t
                         ^^^^^^^^^^^^^^^^^^^
[--Werror=I/O error] error: Preprocessor returned exit code 256; aborting compilation
[--Werror=overlimit] error: 2 errors encountered, aborting compilation
Internal error: bad_optional_access
============ Timers ============
Total: 0 ms
[       OK ] P4TestgenBenchmark.SuccessfullyGenerate1000Tests (29 ms)
[----------] 1 test from P4TestgenBenchmark (29 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (29 ms total)
[  PASSED  ] 1 test.

Note that test passed even when it failed... :)

[asl]

Oh, I was mistaken, it's the include path issue, not the input. Looks like these are not provided. I had to run it like this:

env P4C_16_INCLUDE_PATH=p4include ./backends/p4tools/modules/testgen/testgen-gtest

[asl]

Here we go:

Note: Google Test filter = P4TestgenBenchmark.SuccessfullyGenerate1000Tests
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from P4TestgenBenchmark
[ RUN      ] P4TestgenBenchmark.SuccessfullyGenerate1000Tests
/Users/asl/Projects//p4c/backends/p4tools/modules/testgen/targets/bmv2/test/testgen_api/../../../../../../../../testdata/p4_16_samples/fabric_20190420/fabric.p4(84): [--Wwarn=invalid_header] warning: accessing a field of a potentially invalid header hdr.gtpu_ipv4
        spgw_ingress.apply(hdr.gtpu_ipv4, hdr.gtpu_udp, hdr.gtpu,
                           ^^^^^^^^^^^^^
warning: Ingress parser exception handler not fully implemented
============ Timers ============
Total: 31454 ms
branch_selection: 0 ms (0.00 % of parent)
backend: 1602 ms (5.09 % of parent)
z3: 8069 ms (25.65 % of parent)
	z3.getModel: 2045 ms (25.34 % of parent)
	z3.checkSat: 4507 ms (55.86 % of parent)
step: 17234 ms (54.79 % of parent)
[       OK ] P4TestgenBenchmark.SuccessfullyGenerate1000Tests (32215 ms)
[----------] 1 test from P4TestgenBenchmark (32215 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (32215 ms total)
[  PASSED  ] 1 test.

[fruffy]

I think I also need to add the include paths of the compiler here. The path is definitely awkward, maybe there is a way to add a compiler base path to the gtest utilities.

[asl]

Definitely looks better now!