Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Outcomes not correct in all cases #1

Open
jeffwright13 opened this issue Mar 6, 2023 · 1 comment
Open

Outcomes not correct in all cases #1

jeffwright13 opened this issue Mar 6, 2023 · 1 comment
Assignees

Comments

@jeffwright13
Copy link
Owner

Need to fix internal outcome classification so they match what Pytest shows in the == short test summary == section.

@jeffwright13 jeffwright13 self-assigned this Mar 6, 2023
@jeffwright13
Copy link
Owner Author

jeffwright13 commented Mar 6, 2023

Improved with upcoming relrease, but still needs tweaking, because not all outcomes match.

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┓
┃ Test NodeId                                                                                         ┃ Duration  ┃ Outcome ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━┩
│ demo_tests/test_0.py::test0_pass_1                                                                  │ 50.4 ms   │ failed  │
│ demo_tests/test_0.py::test0_pass_2_logs                                                             │ 2.0039 ms │ passed  │
│ demo_tests/test_0.py::test0_pass_3_error_in_fixture                                                 │ 2.0983 ms │ passed  │
│ demo_tests/test_0.py::test0_fail_1                                                                  │ 6.227 ms  │ failed  │
│ demo_tests/test_0.py::test0_skip                                                                    │ 18.271 ms │ passed  │
│ demo_tests/test_0.py::test0_xfail                                                                   │ 6.0532 ms │ XFailed │
│ demo_tests/test_0.py::test0_xpass                                                                   │ 2.8989 ms │ XPassed │
│ demo_tests/test_0.py::test0_warning                                                                 │ 2.5411 ms │ passed  │
│ demo_tests/test_0.py::test_flaky_3                                                                  │ 3.1979 ms │ passed  │
│ demo_tests/test_1.py::test_a_ok                                                                     │ 992.4 ms  │ passed  │
│ demo_tests/test_1.py::test_b_fail                                                                   │ 1.051 s   │ failed  │
│ demo_tests/test_1.py::test_c_error                                                                  │ 1.641 ms  │ failed  │
│ demo_tests/test_1.py::test_d1_skip_inline                                                           │ 1.0406 s  │ skipped │
│ demo_tests/test_1.py::test_d2_skip                                                                  │ 1.0613 s  │ passed  │
│ demo_tests/test_1.py::test_d3_skip_decorator                                                        │ 1.0233 s  │ skipped │
│ demo_tests/test_1.py::test_e1_xfail_by_inline_and_has_reason                                        │ 1.0664 s  │ XFailed │
│ demo_tests/test_1.py::test_e2_xfail_by_decorator_and_has_reason                                     │ 1.0059 s  │ XFailed │
│ demo_tests/test_1.py::test_f1_xfails_by_inline_even_though_assertTrue_happens_before_pytestDotXfail │ 1.0997 s  │ XFailed │
│ demo_tests/test_1.py::test_f2_xpass_by_xfail_decorator_and_has_reason                               │ 1.0323 s  │ XPassed │
│ demo_tests/test_1.py::test_g_eval_parameterized[3+5-8]                                              │ 988.06 ms │ passed  │
│ demo_tests/test_1.py::test_g_eval_parameterized[2+4-6]                                              │ 999.88 ms │ passed  │
│ demo_tests/test_1.py::test_g_eval_parameterized[6*9-42]                                             │ 979.67 ms │ failed  │
│ demo_tests/test_1.py::test_1_passes_and_has_logging_output                                          │ 1.0405 s  │ passed  │
│ demo_tests/test_1.py::test_2_fails_and_has_logging_output                                           │ 1.0608 s  │ failed  │
│ demo_tests/test_1.py::test_3_fails                                                                  │ 1.0174 s  │ failed  │
│ demo_tests/test_1.py::test_4_passes                                                                 │ 1.0101 s  │ passed  │
│ demo_tests/test_1.py::test_5_marked_SKIP                                                            │ 17.155 ms │ passed  │
│ demo_tests/test_1.py::test_6_marked_xfail_by_decorator_but_passes_and_has_no_reason                 │ 1.0191 s  │ XPassed │
│ demo_tests/test_1.py::test_7_marked_xfail_by_decorator_and_fails_and_has_no_reason                  │ 994.77 ms │ XFailed │
│ demo_tests/test_1.py::test_8_causes_a_warning                                                       │ 1.0593 s  │ failed  │
│ demo_tests/test_1.py::test_9_lorem_fails                                                            │ 10.888 ms │ failed  │
│ demo_tests/test_1.py::test_10_fail_capturing                                                        │ 995.12 ms │ failed  │
│ demo_tests/test_1.py::test_10b_failed_capturing                                                     │ 1.0882 s  │ failed  │
│ demo_tests/test_1.py::test_11_pass_capturing                                                        │ 44.739 ms │ failed  │
│ demo_tests/test_1.py::test_12_fails_and_has_stdout                                                  │ 1.0144 s  │ failed  │
│ demo_tests/test_1.py::test_13_passes_and_has_stdout                                                 │ 946.47 ms │ passed  │
│ demo_tests/test_1.py::test_14_causes_error_pass_stderr_stdout_stdlog                                │ 1.637 ms  │ passed  │
│ demo_tests/test_1.py::test_15_causes_error_fail_stderr_stdout_stdlog                                │ 2.3451 ms │ passed  │
│ demo_tests/test_1.py::test_16_fail_compare_dicts_for_pytest_icdiff                                  │ 12.027 ms │ failed  │
│ demo_tests/test_random_results.py::test_1                                                           │ 12.407 ms │ failed  │
│ demo_tests/test_random_results.py::test_0                                                           │ 18.41 ms  │ failed  │
│ demo_tests/test_random_results.py::test_10                                                          │ 4.5941 ms │ passed  │
│ demo_tests/test_warnings.py::test_1_fails_with_warnings                                             │ 2.4029 s  │ failed  │
└─────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────┴─────────┘

versus

======================================= test session starts ========================================
platform darwin -- Python 3.10.5, pytest-7.2.2, pluggy-1.0.0 -- /Users/jwr003/coding/pytest-tally/venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/jwr003/coding/pytest-tally
plugins: Faker-17.6.0, tally-0.1.0
collected 82 items / 38 deselected / 44 selected

demo_tests/test_0.py::test0_pass_1 FAILED                                                    [  2%]
demo_tests/test_0.py::test0_pass_2_logs PASSED                                               [  4%]
demo_tests/test_0.py::test0_pass_3_error_in_fixture ERROR                                    [  6%]
demo_tests/test_0.py::test0_fail_1 FAILED                                                    [  9%]
demo_tests/test_0.py::test0_skip PASSED                                                      [ 11%]
demo_tests/test_0.py::test0_xfail XFAIL                                                      [ 13%]
demo_tests/test_0.py::test0_xpass XPASS                                                      [ 15%]
demo_tests/test_0.py::test0_warning PASSED                                                   [ 18%]
demo_tests/test_0.py::test_flaky_3 PASSED                                                    [ 20%]
demo_tests/test_1.py::test_a_ok PASSED                                                       [ 22%]
demo_tests/test_1.py::test_b_fail FAILED                                                     [ 25%]
demo_tests/test_1.py::test_c_error ERROR                                                     [ 27%]
demo_tests/test_1.py::test_d1_skip_inline SKIPPED (Skipping this test with inline call t...) [ 29%]
demo_tests/test_1.py::test_d2_skip PASSED                                                    [ 31%]
demo_tests/test_1.py::test_d3_skip_decorator SKIPPED (Skipping this test with inline cal...) [ 34%]
demo_tests/test_1.py::test_e1_xfail_by_inline_and_has_reason XFAIL (Marked as Xfail with...) [ 36%]
demo_tests/test_1.py::test_e2_xfail_by_decorator_and_has_reason XFAIL (Marked as Xfail w...) [ 38%]
demo_tests/test_1.py::test_f1_xfails_by_inline_even_though_assertTrue_happens_before_pytestDotXfail XFAIL [ 40%]
demo_tests/test_1.py::test_f2_xpass_by_xfail_decorator_and_has_reason XPASS (Marked as X...) [ 43%]
demo_tests/test_1.py::test_g_eval_parameterized[3+5-8] PASSED                                [ 45%]
demo_tests/test_1.py::test_g_eval_parameterized[2+4-6] PASSED                                [ 47%]
demo_tests/test_1.py::test_g_eval_parameterized[6*9-42] FAILED                               [ 50%]
demo_tests/test_1.py::test_1_passes_and_has_logging_output PASSED                            [ 52%]
demo_tests/test_1.py::test_2_fails_and_has_logging_output FAILED                             [ 54%]
demo_tests/test_1.py::test_3_fails FAILED                                                    [ 56%]
demo_tests/test_1.py::test_4_passes PASSED                                                   [ 59%]
demo_tests/test_1.py::test_5_marked_SKIP SKIPPED (unconditional skip)                        [ 61%]
demo_tests/test_1.py::test_6_marked_xfail_by_decorator_but_passes_and_has_no_reason XPASS    [ 63%]
demo_tests/test_1.py::test_7_marked_xfail_by_decorator_and_fails_and_has_no_reason XFAIL     [ 65%]
demo_tests/test_1.py::test_8_causes_a_warning FAILED                                         [ 68%]
demo_tests/test_1.py::test_9_lorem_fails FAILED                                              [ 70%]
demo_tests/test_1.py::test_10_fail_capturing FAIL stdout not captured, going directly to sys.stdout
FAIL stderr not captured, going directly to sys.stderr
FAILED                                          [ 72%]
demo_tests/test_1.py::test_10b_failed_capturing FAILED stdout not captured, going directly to sys.stdout
FAILED stderr not captured, going directly to sys.stderr
FAILED                                       [ 75%]
demo_tests/test_1.py::test_11_pass_capturing FAILED                                          [ 77%]
demo_tests/test_1.py::test_12_fails_and_has_stdout FAILED                                    [ 79%]
demo_tests/test_1.py::test_13_passes_and_has_stdout PASSED                                   [ 81%]
demo_tests/test_1.py::test_14_causes_error_pass_stderr_stdout_stdlog ERROR                   [ 84%]
demo_tests/test_1.py::test_15_causes_error_fail_stderr_stdout_stdlog ERROR                   [ 86%]
demo_tests/test_1.py::test_16_fail_compare_dicts_for_pytest_icdiff FAILED                    [ 88%]
demo_tests/test_hoefling.py::test_1 FAILED                                                   [ 90%]
demo_tests/test_random_results.py::test_0 PASSED                                             [ 93%]
demo_tests/test_random_results.py::test_1 SKIPPED (Skipped: Serious inside else memory i...) [ 95%]
demo_tests/test_random_results.py::test_10 FAILED                                            [ 97%]
demo_tests/test_warnings.py::test_1_fails_with_warnings FAILED                               [100%]

@jeffwright13 jeffwright13 changed the title Outcomes always show 'Passed' Outcomes not correct in all cases Mar 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant