Use Python's unittest for validating bundles and configurations?

[wking]

Currently we're using Go, which means it's hard to build a single validator that can process multiple versions of the spec, because you'd have to use tedious interface{} manipulation or load separate packages with structures for each supported version.

We're also not using a test suite, which means it's hard to test multiple aspects of the configuration and report on several errors (the current code just dies on the first error).

By using Python's unittest, we get more convenient handling of generic JSON and an established test framework that doesn't need a lot of boilerplate. I've started working up this approach here if folks want to kick the tires. Examples of potentially tricky things and how this approach lets us handle them easily:

• Skipping particular tests if the configuration has an unrecognized version (wking/oci-runtime-config-validator@c607380f).
• Applying the Windows-specific mount-nesting restriction (here).
• Applying the 0.5.0-specific (and previous, but I haven't bothered supporting them) relative root.path requirement (here).

Example compact output with the current tip (wking/oci-runtime-config-validator@cd0facf0a. I still haven't finished process, later entries in config.md or anything from the platform-specific files):

$ BUNDLE=~/src/opencontainers/my-app python3 -m unittest
...s.............
----------------------------------------------------------------------
Ran 17 tests in 0.006s

OK (skipped=1)

and verbose output:

$ BUNDLE=~/src/opencontainers/my-app python3 -m unittest -v
test_configuration (test.test_bundle.TestBundle)
config.json MUST reside in the root of the bundle directory. ... ok
test_root (test.test_bundle.TestBundle)
The bundle directory MUST contain the root filesystem. ... ok
test_destination (test.test_mounts.TestMounts)
destination (string, required). ... ok
test_destination_nesting (test.test_mounts.TestMounts)
Mount destinations MUST not be nested within another mount. ... skipped 'the destination-nesting restriction only applies to Windows for specification version 1.0.0-rc1.'
test_options (test.test_mounts.TestMounts)
options (list of strings, optional). ... ok
test_source (test.test_mounts.TestMounts)
source (string, required). ... ok
test_type (test.test_mounts.TestMounts)
type (string, required). ... ok
test_args (test.test_process.TestProcess)
args (array of strings, required). ... ok
test_cwd (test.test_process.TestProcess)
cwd (string, required). ... ok
test_env (test.test_process.TestProcess)
env (array of strings, optional). ... ok
test_process (test.test_process.TestProcess)
process (object, required). ... ok
test_terminal (test.test_process.TestProcess)
terminal (bool, optional). ... ok
test_path (test.test_root.TestRoot)
path (string, required). ... ok
test_readonly (test.test_root.TestRoot)
readonly (bool, optional). ... ok
test_syntax (test.test_syntax.TestSyntax)
All configuration JSON MUST be encoded in UTF-8. ... ok
test_recognized_version (test.test_version.TestVersion)
Check for a recognized configuration version. ... ok
test_semantic_version (test.test_version.TestVersion)
ociVersion (string, required) MUST be in SemVer v2.0.0 format. ... ok

----------------------------------------------------------------------
Ran 17 tests in 0.007s

OK (skipped=1)

And with an unrecognized version:

$ BUNDLE=~/src/opencontainers/my-app python3 -m unittest
.sssssssssssss.F.
======================================================================
FAIL: test_recognized_version (test.test_version.TestVersion)
Check for a recognized configuration version.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/wking/src/opencontainers/oci-runtime-config-validator/test/test_version.py", line 39, in test_recognized_version
    .format(util.VERSION))
AssertionError: '1.0.0-rc2' not found in ['1.0.0-rc1', '0.5.0'] : Unrecognized configuration version.  Either your configuration does not match an OCI specification or the test suite has not been taught to process the version you are using.

----------------------------------------------------------------------
Ran 17 tests in 0.002s

FAILED (failures=1, skipped=13)
$ BUNDLE=~/src/opencontainers/my-app python3 -m unittest -v
test_configuration (test.test_bundle.TestBundle)
config.json MUST reside in the root of the bundle directory. ... ok
test_root (test.test_bundle.TestBundle)
The bundle directory MUST contain the root filesystem. ... skipped 'cannot validate an unrecognized version'
test_destination (test.test_mounts.TestMounts)
destination (string, required). ... skipped 'cannot validate an unrecognized version'
test_destination_nesting (test.test_mounts.TestMounts)
Mount destinations MUST not be nested within another mount. ... skipped 'cannot validate an unrecognized version'
test_options (test.test_mounts.TestMounts)
options (list of strings, optional). ... skipped 'cannot validate an unrecognized version'
test_source (test.test_mounts.TestMounts)
source (string, required). ... skipped 'cannot validate an unrecognized version'
test_type (test.test_mounts.TestMounts)
type (string, required). ... skipped 'cannot validate an unrecognized version'
test_args (test.test_process.TestProcess)
args (array of strings, required). ... skipped 'cannot validate an unrecognized version'
test_cwd (test.test_process.TestProcess)
cwd (string, required). ... skipped 'cannot validate an unrecognized version'
test_env (test.test_process.TestProcess)
env (array of strings, optional). ... skipped 'cannot validate an unrecognized version'
test_process (test.test_process.TestProcess)
process (object, required). ... skipped 'cannot validate an unrecognized version'
test_terminal (test.test_process.TestProcess)
terminal (bool, optional). ... skipped 'cannot validate an unrecognized version'
test_path (test.test_root.TestRoot)
path (string, required). ... skipped 'cannot validate an unrecognized version'
test_readonly (test.test_root.TestRoot)
readonly (bool, optional). ... skipped 'cannot validate an unrecognized version'
test_syntax (test.test_syntax.TestSyntax)
All configuration JSON MUST be encoded in UTF-8. ... ok
test_recognized_version (test.test_version.TestVersion)
Check for a recognized configuration version. ... FAIL
test_semantic_version (test.test_version.TestVersion)
ociVersion (string, required) MUST be in SemVer v2.0.0 format. ... ok

======================================================================
FAIL: test_recognized_version (test.test_version.TestVersion)
Check for a recognized configuration version.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/wking/src/opencontainers/oci-runtime-config-validator/test/test_version.py", line 39, in test_recognized_version
    .format(util.VERSION))
AssertionError: '1.0.0-rc2' not found in ['1.0.0-rc1', '0.5.0'] : Unrecognized configuration version.  Either your configuration does not match an OCI specification or the test suite has not been taught to process the version you are using.

----------------------------------------------------------------------
Ran 17 tests in 0.003s

FAILED (failures=1, skipped=13)

This approach would also likely work fine in other languages and test frameworks that make it easy to handle generic JSON. I'm familiar with Python, but if the OCI community prefers a different language, I'm game to try. So:

1. Does this sound like a useful direction? If so,
2. Is it worth pulling into ocitools? Go is a reasonable language for compiled binaries like runtimetest (certainly a better choice for that than Python), and multi-language repositories can get awkward. The validation tasks (configuration validation and runtime validation) seem to decouple well, so I'd rather not stuff both into a single repository. But I can live with a single repo if it's the maintainer preference ;).

[liangchenye]

I think it is the direction. There are three main tasks of ocitools in my mind:

1. validate all the MUST restriction in spec runtimespec VS ocitools #66

2. easier for user or runtime developer Suggestion: Add a Getting Started  #69

3. this one - test cases and test framework
   OCT does the similar thing. It tests different cases for example readonly, terminal just like you do. It outputs successful details and statistics for example how many cases pass the test.
   A shortage is it does not support different versions. @zenlinTechnofreak @wangkirin and I had discussed about the version issue. We tried to support different versions at the beginning, but found it is hard to maintain. Then we read that oci does not guarantee to be backwards-compatible before 1.0, so we choose a lazy way: rolling update.
   [WIP] update initruntimetest by linzhinan #61 is a first step to merge OCT with ocitools. But your comment makes sense that is why I don't know how to convince you golang is a better solution :)

   Does python unittest depend on a working python environment? I hope we can ship ocitools as a single binary. It is also a reason why we wrote OCT but did not use the native 'go test' or other golang testing framework like 'ginkgo'.

[wking]

On Sun, Jun 05, 2016 at 09:04:58PM -0700, 梁辰晔 (Liang Chenye) wrote:

OCT did the similar thing. It tests different cases for example
readonly, terminal just like you do.It outputs successful details
and statistics just like how many cases pass the test. A shortage
is it does not support different versions… We tried to support
different versions at the beginning, but found it is hard to
maintain. Then we read that oci does not guarantee to be
back-comparability before 1.0, so we choose a lazy way: rolling
update.

Would you attribute the multi-version issues more to the spec shifting
around in the early versions (v0.5.0 to v1.0.0-rc1 wasn't all that big
a shift), or to the difficulty in importing the spec's Go structures
multiple times (once for each version)?

And I can't find configuration-validation logic in OCT for ‘readonly’.
The only hit seems to be for runtime validation 1. Can you post a
more specific link to the configuration-validation code?

#61 is a first step to merge OCT with ocitools.

That's about testing the runtime (e.g. is runC conformant?), and I
think Go is a better choice than Python there (see the wording around
“Go is a reasonable language for compiled binaries like runtimetest”
in my topic post 2). This Python work is about testing the
configuration (e.g. is the Redis configuration conformant?), where the
compiled-ness of the test suite is not important.

Does python unittest depend on a working python environment?

Yes, and the stuff I have there now uses features specific to Python
3. But Software Carpentry expects green students to be able to
install Python 3 without much assistance [3](and the students seem to
be fairly successful in accomplishing that), so I'm not terribly
concerned about the dependency. And for systems with a package
manager, it's probably just one emerge/apt-get/yum/brew/… away ;).

I hope we can ship ocitools as a single binary.

I agree that this would be nice to have, but I'm becoming increasingly
sure it's not worth the hoops we'd have to jump through to get there.
The different tasks:

a. Generating configurations based on command-line options (‘ocitools
generate’).
b. Validating configurations (‘ocitools validate’).
c. Validating runtimes (‘test_runtime.sh’, OCT, #61).

seem distinct enough that the benefit from a single binary does not
outweigh the benefit of choosing appropriate tools for each job.

[liangchenye]

Would you attribute the multi-version issues more to the spec shifting

Both. The spec changes so fast between 0.1 ~0.3 that our update had to be very fast accordingly.
I don't know how to import different versions in one program, so we added a commit id in every OCT implementation version, checked spec out before run/compile it. (Looks like a pretest hook)
It is less painful if we only support 1.0/2.0. But we need to re-organize the spec directory structure once we start to work on 2.0 version. Maybe like this:

runtime-spec/specs-go/v1/
runtime-spec/specs-go/v2/

And I can't find configuration-validation logic in OCT for ‘readonly’.
The only hit seems to be for runtime validation [1]. Can you post a
more specific link to the configuration-validation code?

Yes, it is just runtime validation. Sorry I did not explain clearly, I just want to make an analogy:
OCT is also test runtime options by one by one and output individual result,
not just exit at the first failure.

[wking]

On Sun, Jun 05, 2016 at 11:34:27PM -0700, 梁辰晔 (Liang Chenye) wrote:

It is less painful if we only support 1.0/2.0. But we need to
re-organize the spec directory structure once we start to work on
2.0 version. Maybe like this:

runtime-spec/specs-go/v1/
runtime-spec/specs-go/v2/

Ugh. I'm not excited about maintaining old Go structures in the
repository. I'd rather have old versions get posted somewhere else,
and have folks access them through Git tags. I'm not sure how easy
that is to do for Go structures, but it's pretty convenient for JSON
Schema (see opencontainers/runtime-spec#490).

Yes, [OCT] is just runtime validation. Sorry I did not explain
clearly, I just want to make an as analogy: OCT is also test runtime
options by one by one and output individual result, not just exit at
the first failure.

Got it. And I agree that this is a useful goal for runtime testing as
well, but think that issue is largely orthogonal to whether we go down
this route for configurations testing. I recommend an off-the-shelf
test framework for runtime testing too, but I don't think Python's
unittest is a good fit there. I do think Python's unittest is a good
fit for configuration validation.