Tests fail on all recent PRs

[smile13241324]

Description

I have changed some details on the "org" mode documentation and noticed that some tests
do fail consistently for all recent PRs.

Namely these are:
FAILED test-configure-packages-2--excluded-package-is-not-configured
FAILED test-configure-packages-2--lazy-install-package-is-not-configured
FAILED test-configure-packages-2--local-package-w/-bad-string-location-gives-warning
FAILED test-configure-packages-2--local-package-w/-dotfile-owner-update-load-path
FAILED test-configure-packages-2--local-package-w/-layer-owner-update-load-path
FAILED test-configure-packages-2--local-package-w/-string-location-update-load-path
FAILED test-configure-packages-2--local-package-w/o-owner-doesnt-update-load-path
FAILED test-configure-packages-2--package-owned-by-dotfile-is-not-configured
FAILED test-configure-packages-2--package-w/-layer-owner-is-configured
FAILED test-configure-packages-2--package-w/o-owner-is-not-configured
FAILED test-configure-packages-2--protected-excluded-package-is-configured
FAILED test-configure-packages-2--protected-package-is-configured
FAILED test-configure-packages-2--site-package-is-configured
FAILED test-configure-packages-2--toggle-nil-is-not-configured
FAILED test-configure-packages-2--toggle-t-is-configured

Reproduction guide 🪲

• Start Emacs
• Run the tests or open a PR
• Notice that 15 tests fail on all Emacs versions

Observed behaviour: 👀 💔
Tests fail on all recent branches

Expected behaviour: ❤️ 😄
Tests only fail if there is an issue with the contribution

System Info 💻

• OS: gnu/linux
• Emacs: 26.1
• Spacemacs: 0.300.0
• Spacemacs branch: develop (rev. b54ea8e)
• Graphic display: t
• Distribution: spacemacs
• Editing style: vim
• Completion: helm
• Layers:

((helm :variables helm-enable-auto-resize t)
 nim ocaml scheme
 (auto-completion :variables auto-completion-return-key-behavior 'complete auto-completion-tab-key-behavior 'cycle auto-completion-complete-with-key-sequence nil auto-completion-complete-with-key-sequence-delay 0.0 auto-completion-private-snippets-directory nil auto-completion-enable-snippets-in-popup nil auto-completion-enable-help-tooltip t auto-completion-enable-sort-by-usage t)
 better-defaults
 (clojure :variables clojure-enable-fancify-symbols t)
 gpu graphviz
 (plantuml :variables plantuml-jar-path "~/.plantuml/plantUml.jar" org-plantuml-jar-path "~/.plantuml/plantUml.jar")
 salt ibuffer emacs-lisp pdf cscope gtags
 (org :variables org-want-todo-bindings t org-enable-github-support t org-enable-bootstrap-support t org-enable-reveal-js-support t org-enable-hugo-support t)
 (shell :variables shell-enable-smart-eshell t shell-default-shell 'ansi-term shell-default-height 30 shell-default-position 'bottom shell-default-full-span nil shell-default-term-shell "/usr/bin/fish" syntax-checking)
 version-control semantic templates themes-megapack
 (c-c++ :variables c-c++-default-mode-for-headers 'c++-mode c-c++-enable-clang-support t c-c++-enable-clang-format-on-save t)
 (cmake :variables cmake-enable-cmake-ide-support t)
 git github sml pass common-lisp
 (python :variables python-backend 'lsp python-test-runner 'pytest python-enable-yapf-format-on-save t python-sort-imports-on-save t)
 windows-scripts agda jr shell-scripts prodigy emoji
 (markdown :variables markdown-live-preview-engine 'vmd)
 bibtex json debug
 (latex :variables latex-enable-auto-fill t latex-enable-magic nil latex-enable-folding t)
 lua html
 (javascript :variables javascript-disable-tern-port-files nil)
 react yaml web-beautify nginx docker csv search-engine
 (restclient :variables restclient-use-org t)
 restructuredtext selectric octave purescript speed-reading systemd imenu-list
 (java :variables java-backend 'meghanada)
 groovy
 (go :variables go-use-gometalinter t gofmt-command "goimports" go-tab-width 4 go-use-gocheck-for-testing t go-use-test-args "-race -timeout 10s" godoc-at-point-function 'godoc-gogetdoc)
 coq django
 (elm :variables elm-sort-imports-on-save t elm-format-on-save t)
 idris
 (haskell :variables haskell-enable-hindent-style "fundamental" haskell-completion-backend 'dante haskell-enable-hindent t)
 pandoc sphinx lsp parinfer asciidoc neotree ansible puppet rebox rust xkcd typography vimscript
 (terraform :variables terraform-auto-format-on-save t)
 (ruby :variables ruby-enable-enh-ruby-mode t ruby-version-manager 'rvm)
 ruby-on-rails asm
 (sql :variables sql-capitalize-keywords t)
 perl6 autohotkey
 (elixir :variables flycheck-elixir-credo-strict t)
 faust vagrant erlang
 (dash :variables helm-dash-docset-newpath "~/.local/share/Zeal/Zeal/docsets")
 games php nginx racket
 (colors :variables colors-colorize-identifiers 'all)
 d
 (typescript :variables typescript-fmt-tool 'typescript-formatter typescript-fmt-on-save t))

• System configuration features: XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 MODULES THREADS LIBSYSTEMD LCMS2

[sdwolfz]

I'll take a look and see if I can figure it out.

[sdwolfz]

Pushed 2582726 to disable the broken assertions but I don't know how to fix it properly, waiting for @syl20bnr to continue his work in that area.

[dalanicolai]

It appears to me that some tests still fail consistently, i.e. the elisp tests / test (windows-latest, 26.3, core) (pull_request) fails in the latest eight pull requests (starting from #13907).

Now I wonder if these PR's are still candidates/evaluated for merging (???). I have created #13927, and I see no reason why it should have broken anything.

[smile13241324]

@dalanicolai I don't think that your PR is causing these problems in CI and we are seeing the two mentioned PRs as candidates for merging. However #13907 is a draft release, therefore I have not checked it. I have read through #13927 already though I am not yet totally clear about what behaviour you have changed.

I don't know why the mentioned test fails, but I think @JAremko knows about these already.

[JAremko]

Yeah we added more tests and found more bugs... I'm thinking about disabling some tests in PR testing and such, for now. But it kinda stinks. 🤔 windows-latest, 26.3, core seems to be busted in spacelpa department and @syl20bnr planned to work on it.

[dalanicolai]

Thanks for the replies!

@smile13241324 I will elaborate on the (small but very useful) changes in #13927 in the conversation of the PR itself.