Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace semantic-release dependency with TravisCI build stages #1187

Merged
merged 5 commits into from
Sep 6, 2018

Conversation

orangejulius
Copy link
Member

@orangejulius orangejulius commented Aug 29, 2018

This change moves semantic-release out of dev-dependencies, but keeps its functionality by calling semantic-release as usual in a release TravisCI build stage. There are several advantages to this method:

1.) semantic-release is run only after all builds succeed. Our previous approach could have theoretically run semantic-release when some Node.js versions failed with the current code

2.) semantic-release (and it's many dependencies) are removed from node_modules. This increases the speed of npm install in all cases, and reduces the size of our Docker images by 20MB (from 284MB to 264MB)!
Since the only time semantic-release is needed is on TravisCI anyway, it seems pointless that every installation of Pelias should include it.

3.) Because semantic-release is not in package.json, Greenkeeper will not attempt to update it. Semantic release updates very frequently, and each update attempt seems to have a decent chance of experiencing a random TravisCI failure, causing unwanted notifications.

There are probably downsides to this approach. For example, we should consider pinning the major version of semantic release during install.

Additionally, and for obvious reasons, we can't fully test this change until it's merged to the production branch. We should consider testing it first on a lower priority repository.

If this change does work well, we should consider adopting it everywhere.

@missinglink
Copy link
Member

missinglink commented Aug 29, 2018

Sounds like a good idea to keep CI stuff in CI.

One issue you noted was managing the CI config, versioning etc.

Would it be a good idea to have a pelias/CI repo which contained all the CI scripts?

The advantage would be transparency and a familiar workflow for bug reports and issue fixes.

Ie. Travis clones the CI repo and runs all scripts inside before running tests.

@orangejulius orangejulius force-pushed the semantic-release-build-stage branch 2 times, most recently from 8b2a819 to 7e69c1b Compare August 30, 2018 21:04
@orangejulius
Copy link
Member Author

Great idea, I had been thinking about that as well. I've made https://github.com/pelias/ci-tools and added scripts to run semantic release (from a pinned version), and our Docker build. Both Travis and Circle are updated to use it. I'm using the curl url | bash - trick, which I think is safe given it's a URL we control. Any thoughts?

orangejulius added a commit to pelias/config that referenced this pull request Sep 1, 2018
This change moves semantic-release out of dev-dependencies, but keeps
its functionality by calling semantic-release as usual in a `release`
TravisCI build stage. There are several advantages to this method:

1.) semantic-release is run only after all builds succeed. Our previous
approach could have theoretically run semantic-release when some Node.js
versions failed with the current code
2.) semantic-release (and it's many dependencies) are removed from
`node_modules`. This increases the speed of `npm install` in all cases,
and reduces the size of our Docker images by 20MB (from 284MB to 264MB)!
Since the only time semantic-release is needed is on TravisCI anyway, it
seems pointless that every installation of Pelias should include it.
3.) Because semantic-release is not in `package.json`, Greenkeeper will
not attempt to update it. Semantic release updates _very_ frequently,
and each update attempt seems to have a decent chance of experiencing a
random TravisCI failure, causing unwanted notifications.

This change also lets us configure the script to run in one place, the
new pelias/ci-repo.

Connects pelias/api#1187, where this was first
tried.
orangejulius added a commit to pelias/wof-admin-lookup that referenced this pull request Sep 6, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
@orangejulius
Copy link
Member Author

Update: another benefit of this method has been discovered: it surfaces semantic-release failures better.

We have had many semantic-release runs fail, causing botched releases. This is especially relevant since NPM recently invalidated all credentials, and that is a common cause of failed runs.

This change moves semantic-release out of dev-dependencies, but keeps
its functionality by calling semantic-release as usual in a `release`
TravisCI build stage. There are several advantages to this method:

1.) semantic-release is run only after all builds succeed. Our previous
approach could have theoretically run semantic-release when some Node.js
versions failed with the current code
2.) semantic-release (and it's many dependencies) are removed from
`node_modules`. This increases the speed of `npm install` in all cases,
and reduces the size of our Docker images by 20MB (from 284MB to 264MB)!
Since the only time semantic-release is needed is on TravisCI anyway, it
seems pointless that every installation of Pelias should include it.
3.) Because semantic-release is not in `package.json`, Greenkeeper will
not attempt to update it. Semantic release updates _very_ frequently,
and each update attempt seems to have a decent chance of experiencing a
random TravisCI failure, causing unwanted notifications.

There are probably downsides to this approach. For example, we should
consider pinning the major version of semantic release during install.

Additionally, and for obvious reasons, we can't fully test this change
until it's merged to the `production` branch. We should consider testing
it first on a lower priority repository.

If this change _does_ work well, we should consider adopting it
everywhere.
This fixes an issue with the version of Alpine we're using:
@orangejulius orangejulius merged commit 66fb60c into master Sep 6, 2018
@orangejulius orangejulius deleted the semantic-release-build-stage branch September 6, 2018 16:56
orangejulius added a commit to pelias/openaddresses that referenced this pull request Sep 7, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/microservice-wrapper that referenced this pull request Sep 7, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/sorting that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/pbf2json that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/fuzzy-tester that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/polylines that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/labels that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/query that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias-deprecated/categories that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/model that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/polygon-lookup that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/mock-logger that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/dbclient that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/polylines that referenced this pull request Sep 8, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/openaddresses that referenced this pull request Sep 10, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/geonames that referenced this pull request Sep 10, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/openstreetmap that referenced this pull request Sep 10, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/transit that referenced this pull request Sep 10, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/csv-importer that referenced this pull request Sep 10, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/schema that referenced this pull request Sep 11, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/interpolation that referenced this pull request Sep 11, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/analysis that referenced this pull request Sep 11, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/pip-service that referenced this pull request Sep 11, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/placeholder that referenced this pull request Sep 11, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias-deprecated/document-service that referenced this pull request Sep 11, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
missinglink pushed a commit to pelias/schema that referenced this pull request Sep 28, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
orangejulius added a commit to pelias/polylines that referenced this pull request Nov 3, 2018
This makes CI quicker, and releases more stable.

Connects pelias/api#1187
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants