[FEAT] enable external partner testing #5567
Conversation
| const projectRoot = path.resolve(__dirname, '../../'); | ||
| const externalProjectName = process.argv[2]; | ||
| const gitUrl = process.argv[3]; | ||
| const tempDir = path.join(projectRoot, '../__external-test-cache'); |
There was a problem hiding this comment.
I intend to set this up such that we can cache the repositories we test on travis and use git pull to get latest most of the time vs needing to blow away and start over all the time.
There was a problem hiding this comment.
Alternatively, you could shallow clone for some speed improvement and not have to worry about caching mechanisms.
There was a problem hiding this comment.
@kategengler how does that work?
There was a problem hiding this comment.
git clone --depth=1 <repo> will get you just the HEAD of the repo, so you don't get all history.
lib/scripts/test-external.js
Outdated
|
|
||
| // run project tests | ||
| console.log(`Running tests for ${externalProjectName}`); | ||
| execWithLog(`cd ../__external-test-cache/${externalProjectName} && ember test`, true); |
There was a problem hiding this comment.
I'm unsure if an unsuccessful test run would cause us to throw an error and fail, likely need to do more here.
|
cc @backspace @kategengler @jakesjews :) |
|
Ah, this may be painful/brittle if the external apps fail for reasons unrelated to ember-data. My approach was going to be to add ember-try config to the various apps to test against canary ember-data + create a dashboard with the status of the various CI results. That doesn't get PR testing, though. |
|
@kategengler that would still be immensely useful; however, due to certain constraints around 3.4 LTS and internal branching for Obviously we would need to check if the tests failed due to our changes or those of the external app. One thought I have for reducing false failures is to run the external apps tests twice, once against it's own desired version and once against the new commit, and only fail if the first passes but the second doesn't. |
This seems like a good plan. Maybe I’ll finally figure our the intermittently-failing race-condition-y test 😳 Nice use of stages for this! I recently updated |
.travis.yml
Outdated
| script: yarn test-external:emberaddons.com | ||
| - env: NAME=ember-data-change-tracker | ||
| script: yarn test-external:ember-data-change-tracker | ||
| - env: NAME=ember-m3 |
There was a problem hiding this comment.
re all these NAME= workarounds, you can now explicitly name jobs using name: "Whatever", as with the “Max Transpilation” one earlier in the file.
lib/scripts/test-external.js
Outdated
| /* eslint-disable no-console, node/no-extraneous-require, node/no-unpublished-require */ | ||
| const fs = require('fs'); | ||
| const path = require('path'); | ||
| const { execSync } = require('child_process'); |
There was a problem hiding this comment.
lets use execa as it cleans up leaked children, if the script is exited early.
There was a problem hiding this comment.
runspired
force-pushed
the
add-test-infra
branch
from
e5a4764
to
4c7cab0
Compare
.travis.yml
Outdated
|
|
||
| # runs tests against various open-source projects for early-warning regression analysis | ||
| - stage: external partner tests | ||
| name: "Model Fragments" |
There was a problem hiding this comment.
It would be great to see Ember Data Factory Guy added to this list
runspired
dismissed
stefanpenner’s stale review
feedback has been addressed and a new review is needed
add names to external partner tests use shallow cloning, use bower if necessary, add more external partners nice names smoke tests prettier and storefront use execa fix bower command use node 10 in travis
runspired
force-pushed
the
add-test-infra
branch
from
a385e2b
to
cb9d740
Compare
Creates a script that enables us to run the tests of external apps and addons against commits in ember-data.