-
Notifications
You must be signed in to change notification settings - Fork 317
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New Feature: Auto-generation of test code #1817
Merged
Merged
Changes from 31 commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
a7bed7e
initial conversion of autotest code from 0.7.x to 0.9.x
trevorcampbell 19b089e
added docs html files for new problems to .gitignore
trevorcampbell 6705227
minor bugfix: shortened --source_with_tests flag in tests
trevorcampbell d823f3e
output for source_with_tests set to the right directory
trevorcampbell 48837d9
simplified kernel execution code
trevorcampbell f0140fc
store kernel name in resources
trevorcampbell 924a26d
minor ed
trevorcampbell 9ea6577
remove problem3.ipynb from the ui-tests
trevorcampbell 5a521ac
removed commented out timestamp code from earlier snippet execution
trevorcampbell a8ab39f
tests.yml -> autotests.yml
trevorcampbell 50e40c6
logging string interpolation
trevorcampbell be7f998
optional success code (and fix execute success code bug)
trevorcampbell ec63545
reraise filenotfound error when find autotest directive but no autote…
trevorcampbell e8c3d7e
hash optional; raise error if not set
trevorcampbell 2ceab61
minor comment rephrasing
trevorcampbell c53a319
better if statement formatting
trevorcampbell 52ae394
remove unnecessary spaces in start new kernel arg
trevorcampbell 577c733
clear outputs from test notebook
trevorcampbell 702d10d
added jinja to pyproject
trevorcampbell da8c972
initial commit of advanced topics docs, minor polish on creating assi…
trevorcampbell db9b410
done advanced docs section on autotest
trevorcampbell a881d2b
moved problem3 to it's own problem set ps1_autotest
trevorcampbell 1af6da2
moved problem3 to its own ps1_autotest assignment
trevorcampbell 0ca7287
remove problem3 from ps1
trevorcampbell cf1d605
ignore autotest assignment files; make quickstart rename ps1_autotest…
trevorcampbell 4b76b89
remove ps2 R assignment; check for autotest directive in quickstart t…
trevorcampbell 2702e2c
minor code tags syntax fix in docs
trevorcampbell 6964065
set kernel working dir to notebook resources path
trevorcampbell cb09c37
Merge pull request #19 from UBC-DSCI/set-kernel-resource-directory
trevorcampbell d271f9c
Convert to deterministic salt computed using cell source/index
trevorcampbell c58646d
minor ed
trevorcampbell cc5acbf
remove extra jinja from pyproject
trevorcampbell 05570a0
add tests for kernel working dir and release nb consistency
trevorcampbell 865f418
minor bugfix in workingdir tests
trevorcampbell File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
import os | ||
import re | ||
|
||
from traitlets import List, default | ||
|
||
from .base import BaseConverter | ||
from ..preprocessors import ( | ||
InstantiateTests, | ||
ClearOutput, | ||
CheckCellMetadata | ||
) | ||
from traitlets.config.loader import Config | ||
from typing import Any | ||
from ..coursedir import CourseDirectory | ||
|
||
|
||
class GenerateSourceWithTests(BaseConverter): | ||
|
||
@default("permissions") | ||
def _permissions_default(self) -> int: | ||
return 664 if self.coursedir.groupshared else 644 | ||
|
||
@property | ||
def _input_directory(self) -> str: | ||
return self.coursedir.source_directory | ||
|
||
@property | ||
def _output_directory(self) -> str: | ||
return self.coursedir.source_with_tests_directory | ||
|
||
preprocessors = List([ | ||
InstantiateTests, | ||
ClearOutput, | ||
CheckCellMetadata | ||
]).tag(config=True) | ||
|
||
def _load_config(self, cfg: Config, **kwargs: Any) -> None: | ||
super(GenerateSourceWithTests, self)._load_config(cfg, **kwargs) | ||
|
||
def __init__(self, coursedir: CourseDirectory = None, **kwargs: Any) -> None: | ||
super(GenerateSourceWithTests, self).__init__(coursedir=coursedir, **kwargs) | ||
|
||
def start(self) -> None: | ||
old_student_id = self.coursedir.student_id | ||
self.coursedir.student_id = '.' | ||
try: | ||
super(GenerateSourceWithTests, self).start() | ||
finally: | ||
self.coursedir.student_id = old_student_id |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be possible to use the autotests only if a flag is provided (e.g.
nbgrader quickstart --autotests
) ?Currently the quickstart generate ps1/Problem3.ipynb, which is a copy of the ps1/Problem1.ipynb with autotests. It can be confusing. We could generate only one of these example, depending on the flag above.
Same for the
tests.yml
file.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree. But the point of this example was to show something identical to
problem1.ipynb
except implemented using Autotests (so instructors can compare the manual one to the automatic one). But having it namedproblem3.ipynb
is indeed confusing.Maybe I can create
ps1_autotests/
as a new assignment entirely, and make it clear in the documentation that it's a duplicated assignment except implemented using automatic tests?I suspect if we make it clearer that it's meant to duplicate
ps1
but show off autotest functionality, we don't need a separate flag for it.Thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the idea of the
quickstart
generating an assignment as little as possible.My opinion is that autotests is an advanced feature, and bringing a new configuration file (autotest.yml) in the quickstart can be confusing for people who just try to understand the basics of
nbgrader
.cc. @jhamrick @BertR @perllaghu if you have an opinion on it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@brichet I'm happy to do as you suggest -- you're right, the quickstart should produce a very simple example to start from, and the flags can be used for advanced users.