NoTestRun Not Accepted When Validating in a Sandbox Org

[mcarvin8]

Summary

Setting the Test Level to NoTestRun is currently accepted when running a deployment (sf project deploy start) to a sandbox org.

NoTestRun is not accepted when attempting to run a validation (sf project deploy validate -x manifest/package.xml -l NoTestRun --verbose).

Additionally, when I omit the Test Level from the validation command above (sf project deploy validate -x manifest/package.xml --verbose), all local tests in the org run during the validation.
Omitting the Test Level during a Deployment of Non-Apex to a sandbox will set tests to Not Required after Salesforce scans the package.xml.

Steps To Reproduce

1. Declare a package.xml with non-apex components (custom field)
2. Run this command to start a validation with NoTestRun: sf project deploy validate -x manifest/package.xml -l NoTestRun --verbose
3. Run this command to start a validation without the Test Level: sf project deploy validate -x manifest/package.xml --verbose

Expected result

Validation to the sandbox org starts with No Tests Required when setting Test Level to NoTestRun or omitting the Test Level to default to Salesforce scanning the package.xml.

Actual result

Validation with NoTestRun does not start with this error statement.

Error (1): Expected --test-level=NoTestRun to be one of: RunAllTestsInOrg, RunLocalTests, RunSpecifiedTests
See more help with --help

Validation which Omits Test Level runs all local tests in the org.

System Information

PowerShell on Windows 10

{
  "cliVersion": "@salesforce/cli/2.6.7",
  "architecture": "win32-x64",
  "nodeVersion": "node-v18.15.0",
  "osVersion": "Windows_NT 10.0.19045",
  "shell": "cmd.exe",
  "rootPath": "C:\\Program Files\\sf\\client",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 2.3.6 (core)",
    "@oclif/plugin-commands 2.2.23 (core)",
    "@oclif/plugin-help 5.2.17 (core)",
    "@oclif/plugin-not-found 2.3.37 (core)",
    "@oclif/plugin-plugins 3.2.7 (core)",
    "@oclif/plugin-search 0.0.22 (core)",
    "@oclif/plugin-update 3.1.32 (core)",
    "@oclif/plugin-version 1.3.8 (core)",
    "@oclif/plugin-warn-if-update-available 2.0.48 (core)",
    "@oclif/plugin-which 2.2.31 (core)",
    "@salesforce/cli 2.6.7 (core)",
    "apex 2.3.11 (core)",
    "auth 2.8.13 (core)",
    "data 2.5.7 (core)",
    "deploy-retrieve 1.17.5 (core)",
    "info 2.6.39 (core)",
    "limits 2.3.31 (core)",
    "login 1.2.28 (core)",
    "org 2.10.3 (core)",
    "schema 2.3.23 (core)",
    "settings 1.4.26 (core)",
    "sobject 0.2.6 (core)",
    "source 2.10.32 (core)",
    "telemetry 2.3.1 (core)",
    "templates 55.5.10 (core)",
    "trust 2.6.3 (core)",
    "user 2.3.29 (core)",
    "sfdx-git-delta 5.25.2 (user)"
  ]
}

Additional information

Setting Test Level to NoTestRun for a Validation of Non-Apex Package

Omit Test Level for a Validation of Non-Apex Package

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[github-actions]

Hello @mcarvin8 👋 None of the versions of sf you shared match the latest release.

Shared: 2.6.7
Latest: 2.7.11

Update to the latest version of Salesforce CLI (docs) and confirm that you're still seeing your issue.
You can also try the rc and nightly releases! (docs)

After updating, share the full output of sf version --verbose --json

[mcarvin8]

Same behavior seen when updating to the latest CLI

{
  "cliVersion": "@salesforce/cli/2.7.11",
  "architecture": "win32-x64",
  "nodeVersion": "node-v18.15.0",
  "osVersion": "Windows_NT 10.0.19045",
  "shell": "cmd.exe",
  "rootPath": "C:\\Users\\matthew.carvin\\AppData\\Local\\sf\\client\\2.7.11-5a5b1de",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 2.3.6 (core)",
    "@oclif/plugin-commands 2.2.24 (core)",
    "@oclif/plugin-help 5.2.18 (core)",
    "@oclif/plugin-not-found 2.4.0 (core)",
    "@oclif/plugin-plugins 3.3.2 (core)",
    "@oclif/plugin-search 0.0.22 (core)",
    "@oclif/plugin-update 3.2.0 (core)",
    "@oclif/plugin-version 1.3.9 (core)",
    "@oclif/plugin-warn-if-update-available 2.0.49 (core)",
    "@oclif/plugin-which 2.2.32 (core)",
    "@salesforce/cli 2.7.11 (core)",
    "apex 2.3.12 (core)",
    "auth 2.8.14 (core)",
    "data 2.5.7 (core)",
    "deploy-retrieve 1.17.6 (core)",
    "info 2.6.40 (core)",
    "limits 2.3.32 (core)",
    "login 1.2.28 (core)",
    "marketplace 0.1.1 (core)",
    "org 2.10.5 (core)",
    "schema 2.3.24 (core)",
    "settings 1.4.27 (core)",
    "sobject 0.2.6 (core)",
    "source 2.10.32 (core)",
    "telemetry 2.3.1 (core)",
    "templates 55.5.10 (core)",
    "trust 2.6.7 (core)",
    "user 2.3.31 (core)",
    "sfdx-git-delta 5.25.2 (user)"
  ]
}

[mshanemc]

@mcarvin8 we designed 2 distinct paths for validation in sf.

project deploy validate is meant to be run against a production org as a precursor to project deploy quick (and that's why it requires the tests in the validation).

project deploy start has a --dry-run option for any other sort of validation you might want to do (ex: deploy but not run any tests, deploy to a non-prod org with or without tests, etc. It's possible to run this in a way that won't be valid for quick deploy, but it's also way more flexible.

[mcarvin8]

Thanks @mshanemc for the update. The --dry-run option seems to be what I'm looking for in terms of validating separately ahead of a deployment. Tested that out with NoTestRun and that does the trick.