verify-input target derrived from #1909 #1921
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ult of testing Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Conceptualy derived from a pull request, universal-ctags#1909 sent by @ahakanbaba. A example session: $ bash misc/units verify-input Units misc/verifiers .. Category: ROOT ------------------------------------------------------------ simple-json.d/input.json with jq valid zephir-simple.d/input.zep with zoop unavailable Summary ------------------------------------------------------------ #valid: 44 #invalid: 19 #skipped (known invalidation) 1 #skipped (verifier unavailable) 1 Unavailable verifiers ------------------------------------------------------------ zoop You can specify verfiiers to use with --verifiers= option. $ dash misc/units verify-input --verifiers=jq Units misc/verifiers .. Usage: A test case developer puts "verfier" file to one's test case directory or one's category directory. Put a name of verifier to the file. If the file is at a category directory (Units/*.r), the verifier specified in a file is used for verifying all input files under the category. Put "verify" file in a test case directory to override the verifier specified in the category directory. "KNOWN-INVALIDATION" is a special verifier. If the input is kept invalid intentionally, specify the verifier. A verifier developer writes verifier-foo file and puts it to misc/verifiers. Shell is suitable for writing the file. The execution bit of the file must be set. Here, "foo" is the name of verifier which may be specified in "verfier" files under Units. The file name is made a bit redundant. misc/units runs a verifier command in two different modes. If "is_runnable" subcommand is given as the first argument, return 0 if the verifier is ready to run. A verifier may depend on external tools. If the verifier cannot find the external tools in the runtime environment, the verifier exits with non-zero. misc/units never runs the verifier if the verifier exits with non-zero for the subcommand. If "verify" subcommand is given as the frist argument, the script do verify the input file passed as the second argument. It returns 0 if the input is valid, or non-zero if it is invalid. TODO: The above memo should be written in docs/testing.rst. Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Signed-off-by: Masatake YAMATO <yamato@redhat.com>
Derrived from universal-ctags@234540f , chainge in a pull request submitted by @ahakanbaba.
… Fedora Signed-off-by: Masatake YAMATO <yamato@redhat.com>
razum2um
pushed a commit
to razum2um/ctags
that referenced
this pull request
Oct 20, 2018
Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 142.32s user 21.10s system 99% cpu 2:43.67 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 73.08s user 17.89s system 99% cpu 1:31.16 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
razum2um
pushed a commit
to razum2um/ctags
that referenced
this pull request
Oct 20, 2018
Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 73.08s user 17.89s system 99% cpu 1:31.16 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 49.81s user 15.95s system 99% cpu 1:05.91 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
razum2um
pushed a commit
to razum2um/ctags
that referenced
this pull request
Oct 20, 2018
Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 49.81s user 15.95s system 99% cpu 1:05.91 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 46.45s user 15.58s system 99% cpu 1:02.12 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
razum2um
pushed a commit
to razum2um/ctags
that referenced
this pull request
Oct 20, 2018
…erals Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 46.45s user 15.58s system 99% cpu 1:02.12 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 4.48s user 1.01s system 99% cpu 5.498 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
razum2um
pushed a commit
to razum2um/ctags
that referenced
this pull request
Oct 20, 2018
Releated to universal-ctags#1912. example.pp.zip submitted to universal-ctags#1921 is used to take a benchmark. Before: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 4.48s user 1.01s system 99% cpu 5.498 total After: [yamato@master]~/var/ctags-github% time ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 ./ctags --verbose /tmp/example.pp > /tmp/LOG 2>&1 3.46s user 0.84s system 99% cpu 4.312 total Signed-off-by: Masatake YAMATO <yamato@redhat.com>
|
Thanks @masatake |
|
FYI, if you need help, I can work on the puppetManifest verifier. |
|
@ahakanbaba, thank you for trying the branch. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This doesn't use "make" feature much. Therefore, this cannot run verifiers in parallel. However, you can specify a verifier for an input file in a test case directory. A test case developer doesn't have to touch the test harness side and Makefile much.
The original pull requests target puppetManifest inputs and use puppet command for verification.
This pull request target json inputs and use jq to make this pull request small.
After merging I will introduce verifiers for java and puppetManifest inputs.
Conceptually, the changes are completely based on #1909, but the changes for misc/units is written from scratch.
About CI, this utilizes only circleci on Fedora. The most of all inputs, verifying just on Fedora may be enough. However, in future, we may have to think about windows specific tools for verifying a kind of input files.
Writing a section about input verification to docs/testing.rst is a todo item.