Refactor test execution #1756

bjaglin · 2023-05-01T12:20:29Z

Closes #1593
Unlocks #1728 (see comment)

On the projectsGraphPng below

the relationships between expectA_BTargetX_Y_Z and inputX_Y_Z/outputX_Y are not visible as they are task dependencies, not project ones
the runtime dependency (via a coursier fetch) from interface to cliA_B is not visible

bjaglin · 2023-05-14T23:23:56Z

scalafix-tests/integration/src/test/scala/scalafix/tests/reflect/RuleDecoderSuite.scala

@@ -34,7 +34,7 @@ class RuleDecoderSuite extends AnyFunSuite {
    assert(expectedName == rules.name.value)
  }

-  test("relative resolves from custom working directory") {
+  test("relative resolves from custom working directory", SkipWindows) {


integration3/test fails on windows because of this

[info] - relative resolves from custom working directory *** FAILED *** (34 milliseconds) [info] java.nio.file.InvalidPathException: Illegal char <:> at index 2: /D:/a/scalafix/scalafix/scalafix-tests/integration/target/jvm-3/test-classes/ [info] at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182) [info] at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) [info] at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) [info] at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92) [info] at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:232) [info] at java.base/java.nio.file.Path.of(Path.java:147) [info] at java.base/java.nio.file.Paths.get(Paths.java:69) [info] at dotty.tools.dotc.core.MacroClassLoader$.$anonfun$1(MacroClassLoader.scala:25) [info] at scala.collection.immutable.List.map(List.scala:246) [info] at dotty.tools.dotc.core.MacroClassLoader$.makeMacroClassLoader(MacroClassLoader.scala:25)

bjaglin · 2023-05-14T23:54:49Z

scalafix-tests/integration/src/test/scala-2/scalafix/tests/cli/BaseCliSuite.scala

-  // TODO: switch defaults to rely on much more simple test input/rules so
-  // that we don't need to hardcode a scala version here.
-  val sourceDir = "scalafix-tests/input/src/main/scala-2"


removing this unlocks #1728

bjaglin · 2023-05-14T23:56:46Z

project/ScalafixBuild.scala

-    // There is flakyness in CliGitDiffTests and CliSemanticTests
-    Test / parallelExecution := false,


I only observed flakiness in integration (expected because we have side effects), so I disabled parallel execution there specifically

bjaglin · 2023-05-14T23:58:15Z

build.sbt


 lazy val docs = projectMatrix
  .in(file("scalafix-docs"))
  .settings(
    noPublishAndNoMima,
+    fork := true,


prevents

[error] scala.reflect.internal.MissingRequirementError: object scala.runtime in compiler mirror not found. [error] at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:24) [error] at scala.reflect.internal.Mirrors$RootsBase.$anonfun$getModuleOrClass$6(Mirrors.scala:63) [error] at scala.reflect.internal.Mirrors$RootsBase.getPackage(Mirrors.scala:63) [error] at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage$lzycompute(Definitions.scala:200) [error] at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage(Definitions.scala:200) [error] at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass$lzycompute(Definitions.scala:201) [error] at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass(Definitions.scala:201) [error] at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr$lzycompute(Definitions.scala:1389) [error] at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr(Definitions.scala:1388) [error] at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses$lzycompute(Definitions.scala:1553) [error] at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses(Definitions.scala:1552) [error] at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode$lzycompute(Definitions.scala:1596) [error] at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode(Definitions.scala:1596) [error] at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1652) [error] at scala.tools.nsc.Global$Run.<init>(Global.scala:1236) [error] at scala.tools.nsc.interactive.Global$TyperRun.<init>(Global.scala:1351) [error] at scala.tools.nsc.interactive.Global.newTyperRun(Global.scala:1374) [error] at scala.tools.nsc.interactive.Global.<init>(Global.scala:294) [error] at scala.meta.interactive.InteractiveSemanticdb$.newCompiler(InteractiveSemanticdb.scala:34) [error] at scalafix.docs.PatchDocs$.compiler$lzycompute(PatchDocs.scala:118) [error] at scalafix.docs.PatchDocs$.compiler(PatchDocs.scala:117) [error] at docs.Main$.main(Main.scala:59) [error] at docs.Main.main(Main.scala) [error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [error] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [error] at java.base/java.lang.reflect.Method.invoke(Method.java:566) [error] stack trace is suppressed; run last docs2_13 / Compile / run for the full output

https://github.com/olafurpg/setup-scala/releases/tag/v14

- integration: seggregate slow, I/O tests (`shared` folded into it) - expect: stop building & running target-independant suites twice - BaseCliSuite no longer relies on teskit (except for basic diff features), to focus on actual CLI interactions rather than input/output checks handled in `expect`

- make all expect tests explicit with sbt-projectmatrix - keep only the 3 latest patch versions as targets (instead of 7) - run tests for all scala versions on windows - move away from one command per scala version - run docs checks in a separate CI job - speed up test execution by removing forking and limiting sequential run to `integration` suites - speed up windows test execution by avoiding compiling upfront scala 3 projects as part of publishLocalTransitive (`publishLocal / skip`) and by expressing the dependency to `publishLocalTransitive` only where needed by hooking into the task graph (`testWindows`) instead of using a sequential command

bjaglin · 2023-06-07T21:04:13Z

project/ScalafixBuild.scala

+    publishLocal / skip := (if ((publishLocal / skip).value) true
+                            else scalaBinaryVersion.value == "3"),


this turned out to have no effect as its intent is already achieved by the line just above, see #1776

bjaglin force-pushed the setup-scala branch 9 times, most recently from 5542a7b to 29af940 Compare May 7, 2023 18:20

bjaglin changed the title ~~GitHub actions tweaks~~ Refactoring testing May 14, 2023

bjaglin changed the title ~~Refactoring testing~~ Refactor testing May 14, 2023

bjaglin force-pushed the setup-scala branch 19 times, most recently from d54a069 to 5b712f4 Compare May 14, 2023 23:23

bjaglin commented May 14, 2023

View reviewed changes

bjaglin force-pushed the setup-scala branch 2 times, most recently from 18f178d to d821d0f Compare May 14, 2023 23:52

bjaglin changed the title ~~Refactor testing~~ Refactor testing setup May 14, 2023

bjaglin commented May 14, 2023

View reviewed changes

bjaglin force-pushed the setup-scala branch 2 times, most recently from 4049325 to 2494b7f Compare May 15, 2023 00:06

bjaglin marked this pull request as ready for review May 15, 2023 00:18

bjaglin mentioned this pull request May 15, 2023

add support for -Xsource:3 testkit input & output #1666

Merged

bjaglin added 2 commits May 16, 2023 20:16

unify way to deep clone

f97118d

move away from deprecated olafurpg/setup-scala

34bf3d0

https://github.com/olafurpg/setup-scala/releases/tag/v14

bjaglin force-pushed the setup-scala branch 2 times, most recently from ec8c956 to 85b867e Compare May 16, 2023 18:38

bjaglin added 2 commits May 16, 2023 20:52

bjaglin force-pushed the setup-scala branch from 85b867e to 86a0a9b Compare May 16, 2023 18:59

bjaglin changed the title ~~Refactor testing setup~~ Refactor test execution May 16, 2023

bjaglin merged commit afb12ac into scalacenter:main May 16, 2023

bjaglin commented Jun 7, 2023

View reviewed changes

bjaglin mentioned this pull request Aug 11, 2024

Publish cli_3 & rules_3, built against core_2.13 #1680

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor test execution #1756

Refactor test execution #1756

bjaglin commented May 1, 2023 •

edited

Loading

bjaglin May 14, 2023 •

edited

Loading

bjaglin May 14, 2023 •

edited

Loading

bjaglin May 14, 2023

bjaglin May 14, 2023

bjaglin Jun 7, 2023

		// There is flakyness in CliGitDiffTests and CliSemanticTests
		Test / parallelExecution := false,

		publishLocal / skip := (if ((publishLocal / skip).value) true
		else scalaBinaryVersion.value == "3"),

Refactor test execution #1756

Refactor test execution #1756

Conversation

bjaglin commented May 1, 2023 • edited Loading

bjaglin May 14, 2023 • edited Loading

Choose a reason for hiding this comment

bjaglin May 14, 2023 • edited Loading

Choose a reason for hiding this comment

bjaglin May 14, 2023

Choose a reason for hiding this comment

bjaglin May 14, 2023

Choose a reason for hiding this comment

bjaglin Jun 7, 2023

Choose a reason for hiding this comment

bjaglin commented May 1, 2023 •

edited

Loading

bjaglin May 14, 2023 •

edited

Loading

bjaglin May 14, 2023 •

edited

Loading