[red-knot] Add fuzzer to catch panics for invalid syntax #14678
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dhruvmanila
force-pushed
the
dhruv/red-knot-fuzzer
branch
from
aa40db7 to
94698f8
Compare
dhruvmanila
force-pushed
the
dhruv/red-knot-fuzzer
branch
from
94698f8 to
4df8284
Compare
|
sharkdp
reviewed
Dec 3, 2024
| cp -r "../../../crates/ruff_python_parser/resources" "ruff_python_parser" | ||
|
|
||
| # Delete all non-Python files | ||
| find . -type f -not -name "*.py" -delete |
There was a problem hiding this comment.
Should we also keep stub files?
Using
Suggested change
| find . -type f -not -name "*.py" -delete | |
| find . -type f -not \( -name "*.py" -or -name "*.pyi" \) -delete |
or
Suggested change
| find . -type f -not -name "*.py" -delete | |
| find . -type f -not -regex '.*\.pyi?' -delete |
There was a problem hiding this comment.
We could but at the end we would still use a .py file even if the content is coming from a stub file. The reason is that the fuzzer gives us the raw bytes of the source code that it generates.
Hmm, maybe we should run red knot once on a .py file and then on a .pyi file with the same content as we do on the corpus tests.
MichaReiser
approved these changes
Dec 3, 2024
dcreager
added a commit
that referenced
this pull request
Dec 4, 2024
* main: [red-knot] Test: Hashable/Sized => A/B (#14769) [`flake8-type-checking`] Expands TC006 docs to better explain itself (#14749) [`pycodestyle`] Handle f-strings properly for `invalid-escape-sequence (W605)` (#14748) [red-knot] Add fuzzer to catch panics for invalid syntax (#14678) Check `AIR001` from builtin or providers `operators` module (#14631) [airflow]: extend removed names (AIR302) (#14734)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a fuzzer harness for red knot that runs the type checker on source code that contains invalid syntax.
Additionally, this PR also updates the
init-fuzzer.shscript to increase the corpus size to:And, remove any non-Python files from the final corpus so that when the fuzzer tries to minify the corpus, it doesn't produce files that only contains documentation content as that's just noise.
Test Plan
Run
./fuzz/init-fuzzer.sh, say no to the large dataset.Run the fuzzer with
cargo +night fuzz run red_knot_check_invalid_syntax -- -timeout=5