reorganize tests and implement paver

[pwnage101]

I realize this is two different things; if necessary I can split into two different PRs.

Reorganize tests: all the tests are now located beneath the loadtests directory, and the canonical locust invocation changed to:

locust --host=<hostname> -f loadtests/<test-name>

So this would be a breaking change since -f lms wouldn't work anymore.

Implement paver: refer to the updated README.rst for detailed usage, but here's some output:

% paver settings --loadtest-name=lms
---> pavement.settings
cp settings_files/lms.yml.example settings_files/lms.yml
NOTICE: settings for the lms load test can be changed by modifying
the following file:

  settings_files/lms.yml

The primary motivation for doing this is so we can better insulate the locust ansible role from repository semantics.

[pwnage101]

@rlucioni @tobz

FYI @benpatterson

[rlucioni]

Can you show me the relevant sys.path manipulation? Can we make a change to the project structure to avoid the need for this in every single locustfile?

[pwnage101]

the offending lines are around https://github.com/locustio/locust/blob/master/locust/main.py#L293

I have to run so I can't elaborate further, but I remember stepping through and noticing that the project root was not in sys.path at the time of loading the locustfile module.

[rlucioni]

@pwnage101 when you get back: how were you running locust? From the project root?

[pwnage101]

yes, the root. I'm beginning to remember what the exact sequence of events are:

1. user provides -f loadtests/lms on the command line
2. locust splits the locustfile path into "loadtests" and "lms"
3. locust adds "loadtests" to the sys.path
4. locust calls __import__("lms"), triggering our helper imports

That method of importing the locustfile precludes the possibility of referencing anything above the loadtests directory without adding it manually.

[rlucioni]

I see. So it was working before because edx-load-tests (the project root) was being added to sys.path. Putting the tests in a loadtests directory causes edx-load-tests/loadtests to be added to sys.path instead, so you need to manually add edx-load-tests back to be able to use the helper imports.

I wonder why Locust imports the locustfile that way. If this line were to instead call __import__('loadtests.course_discovery'), this sys.path hacking wouldn't be necessary.

[rlucioni]

@pwnage101 done with my pass. I've left comments inline. Not to belabor the point, but note how much cleaner the Invoke implementation is.

[rlucioni]

@pwnage101 your original Makefile PR included a target that could be used to run tests. I don't see a corresponding task here. Is that intentional?

[pwnage101]

@pwnage101 your original Makefile PR included a target that could be used to run tests. I don't see a corresponding task here. Is that intentional?

Ah, I meant to write a comment about that. It didn't make sense to me to abstract locust invokation because we already use a large subset of its features in the way described by locust --help.

However, it does occur to me that knowing to use -f loadtests/lms vs. -f lms is somewhat tribal knowledge (and subject to change). For the rest of the args, it should be straightforward to passthrough args from paver to subprocesses. Would something like this seem reasonable to you:

$ paver loadtest --loadtest-name=lms [passthrough args]

[rlucioni]

@pwnage101 yes, seems reasonable.

[pwnage101]

This is still WIP, but pushing up my progress so @cpennington and @jzoldak can have some visibility.

[jzoldak]

@pwnage101 I do think it would be easier to review if as you suggested you split off "reorg the repo" and "implement paver" into 2 separate PRs.

[pwnage101]

splitting.