Example using config file for WebDriver settings

[maniax89]

There doesn't seem to be any way to figure out how to configure the WebDriver settings.

When I run look at the documentation for sfdx force:lightning:test:run (https://developer.salesforce.com/docs/atlas.en-us.sfdx_cli_reference.meta/sfdx_cli_reference/cli_reference_force_lightning.htm#cli_reference_test_run) - there is a --configfile option that says the following:

Path to a test configuration file to configure WebDriver and other settings. For details, see the LTS documentation.

Where is this documentation? I am attempting to run this on a CI tool (like Jenkins) for automated testing and seeing the following error message:

ERROR: not found: java.

I assume it is because I need to install a WebDriver, but there doesn't seem to be any documentation around this. Could you please point me in the right direction?

[esalman-sfdc]

Regarding how to specify web-driver options, see here and here

@alderete-sfdc plans on moving some of these details from issues to an FAQ section.

May be first try ensuring that java is installed and on the path? Basically The command uses webdriverio which has jre as a pre-requisite.

[innovate42]

I had quite a few problems getting the force:lightning:test:run process to work in a docker container under gitlab ci. Just adding some notes to help others.

Under the hood the sfdx client uses the npm package selenium-standalone. (Version 5.7.1 for sfdx 6.0.16.) The --configfile file on the sfdx command line is passed to the selenium-standalone module as the options. So see this module for what you can define in the config file. additionally the following sections of the config file are used :

• webdriverio is also passed to the npm webdriverio module as the options to the remote command.

• outputDivId is used to wait for the test results. It defines what the test harness is expecting to see on the html page at the end of the testing.

You need to make sure java is avaliable in the path, as well as google-chrome in the docker container. The selenium-standalone npm package automatically downloads selenium and the chromedriver files, you don't need those.

I had a lot of problem trying to debug issues where things seemed to work but nothing happened.
The problem was that I couldn't see what the sfdx client was doing. To help i added a shell script in the path before the real java command. This ran java and added some logging to chromedriver as well as re-directing the output of the java command to a file (so i could see it). There doesn't seem to be any other way to grab this output of the command executed by the sfdx client, the stderr and stdout of the subprocess are not captured by the logger.

For example

/usr/local/bin/java:

echo RUNNING java -Djava.util.logging.config.file=config/logging.properties -Dwebdriver.chrome.verboseLogging=verbose -Dwebdriver.chrome.logfile=/tmp/cd.log $@ 
/usr/bin/java -Djava.util.logging.config.file=config/logging.properties -Dwebdriver.chrome.verboseLogging=verbose -Dwebdriver.chrome.logfile=/tmp/cd.log $@ > /tmp/java.log 2>&1

I got force:lightning:test:run working with headless chrome using the following config file:-

{
  "drivers": {
    "chrome": {
      "version": 2.33,
      "arch": "x64",
      "baseURL": "https://chromedriver.storage.googleapis.com"
    }
  },
  "requestOpts": {
    "timeout": 100000
  },
  "webdriverio": {
    "desiredCapabilities": {
      "browserName": "chrome",
      "chromeOptions": {
        "args": [
          "--headless",
          "--disable-gpu",
          "--no-sandbox"
        ]
      }
    }
  }
}

Some notes:

• The driver section was necessary as the default driver version 2.23 didn't support chrome headless.

• Be careful with the chrome options. I added --snapshot but this caused the broswer load to fail as the option was not compatible with the google --remote-debugging-port flag (which is added automatically by selenium). Some other combination of options failed to start chrome.

• In my case the tests that i implemented took a long time, if there is a timeout with testing then you don't get a timeout message. It just fails. So if things appear to be working but you don't get the expected output then it might be a timeout issue.

• I had to add the --no-sandbox option above, this isn't a good idea, however i was having a problem with chrome running on a docker debian image. Chrome would fail to run and I would get the following error:-

Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted

There seems to be a relevant discussion here jessfraz/dockerfiles#65 . Although i can't quite follow the discussion it seems there are some permissions that are required on the docker and/or the kernel images.

[maniax89]

After using the setup you described @innovate42 (which is extremely helpful!) - I still seem to get the least helpful error message:

Invoking Lightning tests using ghenkins-ci...
ERROR: Cannot read property 'endsWith' of undefined.

I'm pretty sure I will have to figure out 2 things

1. how to install google-chrome onto my docker image (seems like it isn't part of the apt-get repositories)
2. how to make the logging more verbose, potentially using your suggestion for the the java override

It would be a lot more helpful if the CLI tool itself helped users debug a bit better

[innovate42]

Hi @maniax89 I've found that the endsWith error seems to mean that the command that sfdx-clie was trying to run has failed. The best bet would be to try and see the output - and the only way i've found to do that is to create a wrapper around java as mentioned above.

I've added the Dockerfile that we are using for the test builds here https://github.com/innovate42/dockerfiles/tree/master/sfdx-testenv

[esalman-sfdc]

endsWith error is a being thrown by sfdx cli while trying to log the actually error see here. Cli team knows about it and its being tracked under bug W-4442231.

When people have encountered it in context of lightning:test:run. It normally has been resolved by setting DISPLAY/Xvfb.

[maniax89]

I was able to successfully get things running with #46 (comment) after installing google-chrome-stable

It would be good to have this issue archived/turned into documentation so that others may benefit (as it was extremely helpful)

[esalman-sfdc]

@alderete-sfdc FYI for doc

[cnaccio]

Finally got everything working; hope this helps others that come behind me. While trying to get aura component testing working with Gitlab CI/CD using a Ubuntu 14.04 Docker image, I had to do the following three key things to get it working:

1. Install Java 8 - Instructions: https://askubuntu.com/questions/464755/how-to-install-openjdk-8-on-14-04-lts
2. Install google-chrome-stable - Instructions: npm install selenium-standalone@latest -g and then selenium-standalone install
3. Add selenium/standalone-chrome:latest to services section in gitlab-ci.yml config file.
4. Used the following LTS command and config file. I specifically had to to add remote host and port settings in both seleniumArgs and webdriverio; seleniumArgs is required for a check that verifies that Selenium is running and webdriverio ensures the remote Selenium service is used instead of the localhost instance as that one fails to run on Gitlab CI runner (at least for me).

Test Command: sfdx force:lightning:test:run -a auraTests.app --configfile config/lts-config.json

lts-config.json

{
    "requestOpts": {
      "timeout": 100000
    },
    "seleniumArgs": [
      "-host", "selenium__standalone-chrome",
      "-port", "4444"
    ],
    "webdriverio": {
      "desiredCapabilities": {
        "browserName": "chrome",
        "chromeOptions": {
          "args": [
            "--headless",
            "--disable-gpu",
            "--no-sandbox"
          ]
        }
      },
      "host": "selenium__standalone-chrome",
      "port": 4444
    }
  }

gitlab-ci.yml

image: billable/ci-sfdx:latest

services:
  - selenium/standalone-chrome:latest

cache:
  key: "${CI_COMMIT_REF_SLUG}"
  paths:
    - node_modules/

variables:
  SELENIUM_REMOTE_URL: "http://selenium__standalone-chrome:4444/wd/hub/"
  CI_TEMP_DIR: /tmp
  SCRATCH_ORG_DEF: config/ci-scratch-def.json
  FORCE_APP: force-app/main/default

stages:
  - build
  - test
  - analysis
  - packaging

# Environment Setup
before_script:
  - sfdx --version
  - sfdx plugins --core
  - mkdir ${CI_TEMP_DIR}/sfdx-keys
  - openssl enc -d -aes-256-cbc -in ./build/assets/server.key.enc -out ${CI_TEMP_DIR}/sfdx-keys/server.key -pass pass:${DEVHUB_SERVER_PASS}
  - sfdx force:auth:jwt:grant --clientid ${DEVHUB_CONSUMER_KEY} --jwtkeyfile ${CI_TEMP_DIR}/sfdx-keys/server.key --username ${DEVHUB_USERNAME} --setdefaultdevhubusername -a DevHub

# Build Application
build:
  stage: build
  script:
    - npm install
    - npm run build
  artifacts:
    paths:
      - ${FORCE_APP}/aura/baseStyle/baseStyle.css
      - ${FORCE_APP}/labels/*.xml
  only:
    - develop

# Test Automation
test:
  stage: test
  script:
    
    # Testing Environment Setup
    - sfdx force:org:create -s -f ${SCRATCH_ORG_DEF} -a gitlab_build_${CI_JOB_ID}
    - sfdx force:lightning:test:install -t jasmine
    - sfdx force:source:push -f -u gitlab_build_${CI_JOB_ID}
    - sfdx force:user:permset:assign -n Billable_Admin
    - mkdir -p "./build/test-results"

    # Code Quality

    # UI Testing
    - source ./build/enable-lts-debug.sh
    - npm run test:aura:remote
    #- npm run test:lwc

    # Apex Unit Testing
    - sfdx force:apex:test:run  -d ./build/test-results -u gitlab_build_${CI_JOB_ID} -l RunLocalTests -r human -w 10 -c
    - node scripts/coverage.js
    
    #Testing Environment Teardown
    - sfdx force:org:delete -u gitlab_build_${CI_JOB_ID} -p
  coverage: '/^Total\sCoverage:\s+(\d+.\d+\%)$/'
  artifacts:
    reports:
      junit: ./build/test-results/test-*.xml
  only:
    - develop

[maniax89]

for anyone coming back to this comment by @ghost - make sure to use a string (not a decimal) for the version value under the driver selected - i.e.

{
  "drivers": {
    "chrome": {
-     "version": 2.33,
+     "version": "2.33",
      "arch": "x64",
      "baseURL": "https://chromedriver.storage.googleapis.com"
    }
  },
  "requestOpts": {
    "timeout": 100000
  },
  "webdriverio": {
    "desiredCapabilities": {
      "browserName": "chrome",
      "chromeOptions": {
        "args": [
          "--headless",
          "--disable-gpu",
          "--no-sandbox"
        ]
      }
    }
  }
}