115 Improve integration tests to start a C* version then run all tests against that version #152

absurdfarce · 2021-06-10T05:31:04Z

Change in the title is the core change here but there are a number of other improvements as well

…e test we don't really need export-specific packages

…ma creation a class-level op

absurdfarce · 2021-06-10T05:33:39Z

python/adelphi/bin/run-adelphi-tests.py

+# collected tests for each of them.  Rather than fight with the frameworks we
+# manage the fixture iteration manually in this script.  This also has the
+# nice side effect of moving a lot of C* checking/session code out of the test
+# suite, which in turn should allow us to write simpler tests.


I spent a considerable amount of time trying to find a clear way to manage Cassandra-versions-as-test-fixtures within pytest directly. There wasn't any obvious way to iterate over a set of fixtures (executing any discovered tests for each) so rather than continue to work against the tooling I moved up a level and implemented the fixture iteration in a front-end script.

absurdfarce · 2021-06-10T05:36:10Z

python/adelphi/bin/run-adelphi-tests.py

+        "commands": "pytest {posargs}", \
+        "setenv": "CASSANDRA_VERSION = {}".format(version)}
+    with open(TOX_CONFIG, 'w') as configfile:
+        config.write(configfile)


We now generate the tox.ini file rather than managing it explicitly. Underlying motivation here was to pass Cassandra version into the tests themselves; individual test cases need to know what version they're testing against in order to validate observed output.

absurdfarce · 2021-06-10T05:40:03Z

python/adelphi/bin/run-adelphi-tests.py

+@click.command()
+@click.option('--cassandra', '-c', multiple=True, type=str)
+@click.option('--python', '-p', multiple=True, type=click.Choice(["py2","py3"], case_sensitive = False))
+@click.option("--pytest", "-t", type=str, help="Arguments to be passed to pytest")


Click represents a significant improvement to usability here. Using these args users can select (at run-time) any combination of the following:

One or more Cassandra versions ("python bin/run-adelphi-tests.py -c 2.1.22 -c 2.2.19")

Which Python version to use (if a specific one is required) ("python bin/run-adelphi-tests.py -p py2")

Any args which should be passed directly to pytest (as executed by tox). This allows users to leverage the full set of pytest functionality which is really pretty extensive. (e.g. 'python bin/run-adelphi-tests.py -t "-k TestNb"' to find a specific test by keyword)

As expected these args can be mixed and matched as desired.

In some cases these params were passed via env vars, in some cases via tox args, and prolly a few other mechanisms I'm forgetting now. Consolidating it all in a single place (with good --help support) should help quite a bit.

absurdfarce · 2021-06-10T05:41:35Z

python/adelphi/test-requirements.txt

@@ -0,0 +1,3 @@
+docker ~= 4.4
+tenacity ~= 7.0
+click ~= 7.1


A set of dependencies which must be in place when trying to run run-adelphi-tests.py. These were previously declared in tox.ini (minus click which is new) but now that some of that functionality has moved out to the script we have to have them available earlier.

absurdfarce · 2021-06-10T05:47:08Z

python/adelphi/tests/integration/test_cql.py

@@ -0,0 +1,146 @@
+import glob


Another major change. We used to have individual test classes (each contained in their own file) for a specific test case grouped within a module representing the export type we wanted to test. The connective tissue between those tests and unittest wasn't completely intuitive; it wasn't clear which test function was actually invoked in a given case.

This PR changes all that by moving all tests for a specific export type into a single file. Each test case now has a descriptive method name (by itself a massive improvement) and the module hierarchy was simplified. The odd inheritance + double mixin strategy we were using has also been removed in favor of a much simpler inheritance model.

Even a cursory walk-through should make it clear how this approach vastly simplifies #110.

absurdfarce · 2021-06-10T05:48:09Z

python/adelphi/tests/integration/test_cql.py

+        log.warning("Dropping keyspace testkeyspace my_ks_0")
+        dropKeyspace("my_ks_0")
+        log.warning("Dropping keyspace testkeyspace my_ks_1")
+        dropKeyspace("my_ks_1")


In this case test schemas are managed via instance methods but it's entirely up to the test author how they want to do this.

absurdfarce · 2021-06-10T05:49:10Z

python/adelphi/tests/integration/test_nb.py

+    @classmethod
+    def tearDownClass(cls):
+        log.warning("Dropping keyspace testkeyspace")
+        dropKeyspace("testkeyspace")


Re: the conversation above... this test has no dependency upon particular intstances to setup it's schema so it's safe to do so via class methods. There's no need to do it one way or the other... the decision is entirely about what's good for a given test suite.

absurdfarce · 2021-06-17T17:57:55Z

A few notes on running the tests now.

You'll need Docker installed and dockerd up and running. You'll probably also want a Python virtualenv for these tests, although this isn't strictly required.

In whatever environment you're in you'll want to install an "adelphi" package built from this source. You can do this with the following:

cd python
pip install ./adelphi

If you already have a previous version of the "adelphi" package installed in your environment you'll want to uninstall it first. I use the following as my default command for installing a new version of the package in my virtualenv:

pip uninstall adelphi; pip install ./adelphi

Once you have the package installed you'll also want to install the test dependencies. These are additional packages required by the testing infrastructure but not by the package itself. These are managed in "test-requirements.txt" so the following should work for you:

pip install -r test-requirements.txt

Note that the setup for the GH action support in #154 automates all of this for CI testing.

Now that your environment is setup you'll then want to navigate to python/adelphi in order to run the tests. "test-adelphi" effectively does the same thing as running "tox" in the current working directory and tox relies on setup.py to do it's work. You'll also need to be able to find tests.unit, tests.integration and tests.util modules so make sure python/adelphi is in your PYTHONPATH. Since you should be running everything from python/adelphi anyway it's enough to just set PYTHONPATH to "." instead.

From there you can run the app with "python bin/test-adelphi". This changes with some of the later PRs; #154 specifically makes "test-adelphi" part of the regular Python package, but as of the creation of this PR I was still trying to avoid that (for reasons that were misguided). "test-adelphi" leverages click for CLI processing so help is available:

(python3-venv) [mersault@localhost adelphi]$ python bin/test-adelphi --help
Usage: test-adelphi [OPTIONS]

Options:
  -c, --cassandra TEXT
  -p, --python [py2|py3]
  -t, --pytest TEXT       Arguments to be passed to pytest
  --help                  Show this message and exit.

-p allows you to specify which Python version you'd like to test. You can specify "-p py2", "-p py3" or both to explicitly request both be tested (this is also the default behaviour). -c allows you to specify the Cassandra version you'd like to test. You can have any number of -c args on the command line, although at the moment you have to specify the full Cassandra version (major.minor.path or 4.0-rc1) rather than use any kind of shorthand. #154 relaxes this considerably, allowing you to say "-c 2.1" (to get the most recent 2.1.x version from the default list) or "-c 2" (to do the same with the latest 2.x.y version) but that functionality isn't in place for this PR yet.

-t allows you to pass additional args to pytest via tox. This can be useful for, say, selecting certain tests. 'python bin/test-adelphi "-k TestCql"' for instance will use pytest's keyword matching functionality to match only tests that start with "TestCql". See https://docs.pytest.org/en/6.2.x/usage.html for some of the additional things you can do here.

Once you invoke it test-adelphi will iterate through your requested Cassandra versions. For each it will start a Docker container of that version, wait until it's up and running and then (effectively) invoke "tox" on the test suite with any additional args you supplied. We don't shell out to invoke "tox" as another process; we invoke it directly via it's Python API, although the output should look pretty much the same.

jdonenine · 2021-06-17T19:22:22Z

A few notes on running the tests now.

@absurdfarce the content there in that comment was super helpful, was able to run all of the tests as discussed. I think it would be super worth it to get this content into some docs in tree somewhere. Perhaps as the start of a developer/contributor guide we could start to flesh out.

absurdfarce · 2021-06-17T19:29:33Z

Thx @jdonenine ! I don't disagree, although I might want to wait until #154 lands. It makes some pretty significant changes to the process in place here.

I might look at adapting those comments into a README in the "tests" directory on #154... I could definitely see that being useful.

jdonenine

Testing things out and all of the tests functioned as expected, looks good. Definitely simplified reading through the test definitions. 👍

jdonenine · 2021-06-17T19:51:32Z

Thx @jdonenine ! I don't disagree, although I might want to wait until #154 lands. It makes some pretty significant changes to the process in place here.

Yup, no reason to hold this up, I'll capture the idea in a new issue to follow up with on the docs side. Would definitely be a good add to the project.

jdonenine · 2021-06-17T19:57:15Z

python/adelphi/bin/test-adelphi

+
+
+# Default C* versions to include in all integration tests
+DEFAULT_CASSANDRA_VERSIONS = ["2.1.22", "2.2.19", "3.0.23", "3.11.9", "4.0-rc1"]


One thing I noticed is that we don't have 3.11.10 in the list, probably makes sense to include that since it's the latest stable 3.11 release.

I'll create a separate issue to add that to the mix.

@jdonenine This was easy enough to address now so I just did it... 98fa005

absurdfarce added 11 commits June 8, 2021 23:19

First sketch of an idea, still lots to flesh out yet

d5d51fe

Add automated generation of tox.ini

c99f0c0

Single file nb tests now working

2419d8c

Forgot to clean out tests.integration.nb package init

98345e3

First cut at a consolidated test_cql.py

47869f9

Tests appear to be passing on py2 and py3 again

5dfe8b1

Now that export-specific functionality is consolidated within a singl…

ed51b67

…e test we don't really need export-specific packages

Would like to keep unittests runnable on their own as well

4002710

Fix tox invocation to avoid exiting after first run completes

2adabce

test_nb doesn't need method-specific data for init so let's make sche…

fbc8476

…ma creation a class-level op

Add click for handling CLI args

21ef1c2

absurdfarce commented Jun 10, 2021

View reviewed changes

Comment updates

8a38d9a

absurdfarce linked an issue Jun 10, 2021 that may be closed by this pull request

Make integration test infrastructure smarter w.r.t. Cassandra versions #115

Closed

absurdfarce requested review from grighetto and jdonenine June 10, 2021 05:53

absurdfarce mentioned this pull request Jun 10, 2021

129 Make cycles, scenarioname NB template variables by default #148

Open

absurdfarce added 2 commits June 10, 2021 10:45

Rename + making test app executable

e707cf8

Update keyspace drop logic to be more automatic

4298a8e

absurdfarce self-assigned this Jun 11, 2021

absurdfarce mentioned this pull request Jun 11, 2021

CI/CD integration of anonymizer script testing #58

Open

Forgot to add tox to the test dependencies

69f3287

Update .gitignore to exclude generated tox file (#156)

7ed1cc0

jdonenine approved these changes Jun 17, 2021

View reviewed changes

jdonenine reviewed Jun 17, 2021

View reviewed changes

Bump 3.11.x C* version to 3.11.10

98fa005

absurdfarce merged commit 3d42c1e into master Jun 21, 2021

absurdfarce deleted the 115-breadth-first-testing branch June 21, 2021 19:23

grighetto mentioned this pull request Jul 12, 2021

113 Improve reporting of test failures for CQL tests, take two #159

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

115 Improve integration tests to start a C* version then run all tests against that version #152

115 Improve integration tests to start a C* version then run all tests against that version #152

absurdfarce commented Jun 10, 2021

absurdfarce Jun 10, 2021

absurdfarce Jun 10, 2021

absurdfarce Jun 10, 2021

absurdfarce Jun 10, 2021

absurdfarce Jun 10, 2021

absurdfarce Jun 10, 2021

absurdfarce Jun 10, 2021

absurdfarce commented Jun 17, 2021 •

edited

Loading

jdonenine commented Jun 17, 2021

absurdfarce commented Jun 17, 2021

jdonenine left a comment

jdonenine commented Jun 17, 2021

jdonenine Jun 17, 2021

jdonenine Jun 17, 2021

absurdfarce Jun 21, 2021



		# Default C* versions to include in all integration tests
		DEFAULT_CASSANDRA_VERSIONS = ["2.1.22", "2.2.19", "3.0.23", "3.11.9", "4.0-rc1"]

115 Improve integration tests to start a C* version then run all tests against that version #152

115 Improve integration tests to start a C* version then run all tests against that version #152

Conversation

absurdfarce commented Jun 10, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

absurdfarce commented Jun 17, 2021 • edited Loading

jdonenine commented Jun 17, 2021

absurdfarce commented Jun 17, 2021

jdonenine left a comment

Choose a reason for hiding this comment

jdonenine commented Jun 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

absurdfarce commented Jun 17, 2021 •

edited

Loading