Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix database tests and enable running using GitHub workflows #5676

Merged
merged 38 commits into from
Nov 29, 2019

Conversation

koppor
Copy link
Member

@koppor koppor commented Nov 26, 2019

  • fix behavior when no entries have to be inserted during a synch
  • fix database tests
  • have dbms test test only one database
  • DBMS to test is passed via environment variable "DBMS", defaulting to PostgreSQL
  • add shared information on Canonical BibTeX entry
  • some code improvement
  • have checkstyle running on github workflows (not on Travis)

@koppor koppor force-pushed the migrate-database-tests-to-github branch 14 times, most recently from 61267b4 to 29a2c10 Compare November 27, 2019 05:56
@koppor koppor changed the title [WIP] Fix datbase tests and enable running using GitHub workflows Fix datbase tests and enable running using GitHub workflows Nov 27, 2019
@koppor koppor added the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label Nov 27, 2019
@koppor koppor marked this pull request as ready for review November 27, 2019 06:02
@Siedlerchr
Copy link
Member

In general: Why do you run the database tests on github and not on travis?
Travis should be faster as it has better caching.

Copy link
Member

@tobiasdiez tobiasdiez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! That looks really good.

I'm in favor of moving from travis to github. From what I've observed, the github servers are quicker than travis; and the integration here on github is better (you immediately see which jobs failed).

.github/workflows/checkstyle.yml Outdated Show resolved Hide resolved
docs/testing.md Show resolved Hide resolved
docs/testing.md Outdated Show resolved Hide resolved
@koppor
Copy link
Member Author

koppor commented Nov 28, 2019

OK, i merged the jobs.

Result: Increase build time by approx 15 minutes, because the build upload takes 12 minutes:

grafik

Before:

grafik

After:

grafik

@Siedlerchr
Copy link
Member

This is ridiculous. We always decoupled the build from the tests.

@koppor
Copy link
Member Author

koppor commented Nov 28, 2019

Time added via merged workflow: 16:15 min

  • Build upload: 11,47min
  • Build Download: 4,46min

Since workflows run in parallel (maybe even on different machines on Azure), the split workflows do add only the time required for "setup up job", which is 3s.

In all jobs, the source needs to be checked out (13s) and the JDK needs to be setup (8s)

@koppor
Copy link
Member Author

koppor commented Nov 28, 2019

GitHub workflow decouple the runtimes. Thus, it is understandable that transfer costs are high.

@koppor
Copy link
Member Author

koppor commented Nov 28, 2019

It also depends, where we want to decouple. Currently, we do it at the command line level. Meaning, we host a separate instance of the database system etc. The other possibility would be to spawn off different threads within one job and let gradle / bash do some paralellization. I dislike that option, because it is a very tight coupling and a working X11 server, PostgresDB should not be an issue for checkstyle or logic test.

@koppor
Copy link
Member Author

koppor commented Nov 28, 2019

(Just as reminder - we started 2015 with following test order)

grafik

@Siedlerchr
Copy link
Member

Travis is much more efficient running tests in parallel because it can better reuse stuff

@koppor koppor force-pushed the migrate-database-tests-to-github branch from aa9b108 to 29a2c10 Compare November 28, 2019 07:35
@koppor
Copy link
Member Author

koppor commented Nov 28, 2019

If we have different workflows, we also have more bubbles there.

Current state with one workflow:

grafik

In case more workflows are there, we get more bubbles:

grafik
grafik

@koppor
Copy link
Member Author

koppor commented Nov 28, 2019

Setting up the environment (databses) is very hard on travis. I spent many hours on this. With GitHub Workflow this was a matter of minutes for postgres and half an hour for MySQL. - I need to shut down services on travis because of memory usage. Needs much test tweaking.

Separate for workflows for different tests are IMHO not that hard to maintain.

${{ runner.OS }}-gradle-
${{ runner.OS }}-
- uses: actions/cache@v1
name: Cache gradle wrapper
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did the fetching from the gradle executable jar from the cache take longer than fetching from gradle? - I wanted to reduce the GET requests to the gradle server by caching the wrapper binaries with this task.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this check is now run weekly, there is no need to optimize anything.

@koppor
Copy link
Member Author

koppor commented Nov 29, 2019

I don't know why the fetcher tests are not executed. They are compiled, but no tests are run. Locally, they do run.

Copy link
Member

@tobiasdiez tobiasdiez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, everything seems to work now.

@koppor koppor mentioned this pull request Nov 29, 2019
@koppor
Copy link
Member Author

koppor commented Nov 29, 2019

LGTM now, I would merge and discuss the general things afterwards in the devcall.

@koppor koppor merged commit ea5e632 into master Nov 29, 2019
@koppor koppor deleted the migrate-database-tests-to-github branch November 29, 2019 15:58
@koppor koppor restored the migrate-database-tests-to-github branch November 29, 2019 16:25
@koppor koppor deleted the migrate-database-tests-to-github branch November 29, 2019 16:26
koppor pushed a commit that referenced this pull request Nov 1, 2021
12dbba5 Create tyndale-bulletin (#5673)
b0746db Create Engineered Regeneration (#5682)
e38b953 wikipedia citation template (#5662)
5e7f731 Create early-music-history.csl (#5679)
86443f3 Create zeitschrift-fur-politik.csl (#5676)
68f1996 Create annals-of-work-exposures-and-health.csl (#5666)
1ba9dc6 Create brazilian-journal-of-psychiatry.csl (#5672)
438f92c fix error for speech in ama styles (#5693)
7a0c2d3 set initialize-with-hyphen to false (#5689)
3bd2765 Update emu-austral-ornithology.csl (#5671)
31492b2 fix various errors in natura-croatica.csl (#5687)
94d6b23 Update iso690-author-date-cs.csl (#5677)
5d017da minor update on the "Haute école de gestion de Genève - ISO 690" style (#5665)
2cad8f6 add ibid/subsequent to comparative-politics.csl (#5669)
de0b116 Create taylor-and-francis-vancouver-national-library-of-medicine.csl (#5650)
ed87f99 Update bulletin-de-correspondance-hellenique.csl (#5663)

git-subtree-dir: buildres/csl/csl-styles
git-subtree-split: 12dbba5
koppor pushed a commit that referenced this pull request Nov 15, 2021
0654e16 Create scandinavian-journal-of-information-systems.csl (#5716)
ce2d537 Update journal-of-computer-applications-in-archaeology.csl (#5715)
755d3d3 Create human-rights-law-review.csl (#5626)
0feda94 Create journal-of-intercultural-studies.csl (#5709)
ae4756d Update acta-universitatis-agriculturae-sueciae.csl (#5713)
323d9ac Update mohr-siebeck-recht.csl (#5559)
15530a8 Bch corr (#5712)
094a1af Create forschungsjournal-soziale-bewegungen-fjsb.csl (#5699)
cb91566 initialize authors and editors (#5714)
2d5cfff Create cancer-biomarkers.csl (#5703)
5e264d5 Update multidisciplinary-digital-publishing-institute.csl (#5708)
46e961f Create klinische-padiatrie.csl (#5711)
e81e877 Create bulletin-archeologique-des-ecoles-francaises-a-l-etranger.csl (#5704)
0029c5a Create polar-research.csl 🧊 (#5702)
7db1361 Update vancouver-imperial-college-london.csl (#5641)
b953e9f Update iso690-author-date-fr-no-abstract.csl (#5706)
91eda8c Update thieme-german.csl (#5710)
ebe0787 Update harvard-imperial-college-london.csl (#5643)
2d4db76 Fix UNESCO IIEP in text
436cbf4 Create revue-archeologique-de-narbonnaise.csl (#5688)
5150bcf Create journal-of-computer-assisted-tomography.csl (#5690)
dd6f050 Create anti-trafficking-review.csl (#5658)
08e622f Create the-angle-orthodontist.csl (#5685)
c6a1907 journal-of-palm-oil-research.csl fix several errors (#5686)
6cbe29d Create bern-university-of-applied-sciences-school-of-agricultural-for… (#5684)
f590dc1 Update biomed-central.csl (#5701)
1efce81 Update turabian-author-date.csl (#5695)
12dbba5 Create tyndale-bulletin (#5673)
b0746db Create Engineered Regeneration (#5682)
e38b953 wikipedia citation template (#5662)
5e7f731 Create early-music-history.csl (#5679)
86443f3 Create zeitschrift-fur-politik.csl (#5676)
68f1996 Create annals-of-work-exposures-and-health.csl (#5666)
1ba9dc6 Create brazilian-journal-of-psychiatry.csl (#5672)
438f92c fix error for speech in ama styles (#5693)
7a0c2d3 set initialize-with-hyphen to false (#5689)
3bd2765 Update emu-austral-ornithology.csl (#5671)
31492b2 fix various errors in natura-croatica.csl (#5687)
94d6b23 Update iso690-author-date-cs.csl (#5677)
5d017da minor update on the "Haute école de gestion de Genève - ISO 690" style (#5665)
2cad8f6 add ibid/subsequent to comparative-politics.csl (#5669)
de0b116 Create taylor-and-francis-vancouver-national-library-of-medicine.csl (#5650)
ed87f99 Update bulletin-de-correspondance-hellenique.csl (#5663)

git-subtree-dir: buildres/csl/csl-styles
git-subtree-split: 0654e16
Siedlerchr added a commit that referenced this pull request Nov 16, 2021
* Squashed 'buildres/csl/csl-styles/' changes from 3a6a0a7..0654e16

0654e16 Create scandinavian-journal-of-information-systems.csl (#5716)
ce2d537 Update journal-of-computer-applications-in-archaeology.csl (#5715)
755d3d3 Create human-rights-law-review.csl (#5626)
0feda94 Create journal-of-intercultural-studies.csl (#5709)
ae4756d Update acta-universitatis-agriculturae-sueciae.csl (#5713)
323d9ac Update mohr-siebeck-recht.csl (#5559)
15530a8 Bch corr (#5712)
094a1af Create forschungsjournal-soziale-bewegungen-fjsb.csl (#5699)
cb91566 initialize authors and editors (#5714)
2d5cfff Create cancer-biomarkers.csl (#5703)
5e264d5 Update multidisciplinary-digital-publishing-institute.csl (#5708)
46e961f Create klinische-padiatrie.csl (#5711)
e81e877 Create bulletin-archeologique-des-ecoles-francaises-a-l-etranger.csl (#5704)
0029c5a Create polar-research.csl 🧊 (#5702)
7db1361 Update vancouver-imperial-college-london.csl (#5641)
b953e9f Update iso690-author-date-fr-no-abstract.csl (#5706)
91eda8c Update thieme-german.csl (#5710)
ebe0787 Update harvard-imperial-college-london.csl (#5643)
2d4db76 Fix UNESCO IIEP in text
436cbf4 Create revue-archeologique-de-narbonnaise.csl (#5688)
5150bcf Create journal-of-computer-assisted-tomography.csl (#5690)
dd6f050 Create anti-trafficking-review.csl (#5658)
08e622f Create the-angle-orthodontist.csl (#5685)
c6a1907 journal-of-palm-oil-research.csl fix several errors (#5686)
6cbe29d Create bern-university-of-applied-sciences-school-of-agricultural-for… (#5684)
f590dc1 Update biomed-central.csl (#5701)
1efce81 Update turabian-author-date.csl (#5695)
12dbba5 Create tyndale-bulletin (#5673)
b0746db Create Engineered Regeneration (#5682)
e38b953 wikipedia citation template (#5662)
5e7f731 Create early-music-history.csl (#5679)
86443f3 Create zeitschrift-fur-politik.csl (#5676)
68f1996 Create annals-of-work-exposures-and-health.csl (#5666)
1ba9dc6 Create brazilian-journal-of-psychiatry.csl (#5672)
438f92c fix error for speech in ama styles (#5693)
7a0c2d3 set initialize-with-hyphen to false (#5689)
3bd2765 Update emu-austral-ornithology.csl (#5671)
31492b2 fix various errors in natura-croatica.csl (#5687)
94d6b23 Update iso690-author-date-cs.csl (#5677)
5d017da minor update on the "Haute école de gestion de Genève - ISO 690" style (#5665)
2cad8f6 add ibid/subsequent to comparative-politics.csl (#5669)
de0b116 Create taylor-and-francis-vancouver-national-library-of-medicine.csl (#5650)
ed87f99 Update bulletin-de-correspondance-hellenique.csl (#5663)

git-subtree-dir: buildres/csl/csl-styles
git-subtree-split: 0654e16

* Squashed 'buildres/csl/csl-locales/' changes from 0cc3885f61..d5ee85de8e

d5ee85de8e Period after Übers. added (#241)

git-subtree-dir: buildres/csl/csl-locales
git-subtree-split: d5ee85de8e74d4109509014758b6f496a968ff03

* fix  merge error

Co-authored-by: github actions <jabrefmail+webfeedback@gmail.com>
Co-authored-by: Siedlerchr <siedlerkiller@gmail.com>
koppor pushed a commit that referenced this pull request Dec 1, 2021
3bb4b5f infoclio.ch styles for German: remove non-breaking space delimiters (#5754)
adf28db Create journal-of-health-care-for-the-poor-and-underserved.csl (#5752)
0713a8e Update chinese-gb7714-2005-numeric.csl (#5737)
1cd3754 Update china-national-standard-gb-t-7714-2015-author-date.csl (#5746)
c2536b7 Update china-national-standard-gb-t-7714-2015-numeric.csl (#5745)
f8c1392  Create steel-research-international.csl (#5720)
21fe1f5 Create asian-myrmecology.csl (#5718)
91e9e2b Update harvard-university-of-the-west-of-england.csl (#5734)
dd453d1 fix minor erros in polar-research.csl (#5730)
038a8f5 Remove group around no-date cluster (#5731)
0710b51 remove et-al from bibtex.csl (#5728)
bbd703d Add editorial-director to universite-laval-departement-des-sciences-historiques.csl (#5727)
58ea430 Create german-journal-of-agricultural-economics.csl (#5717)
0654e16 Create scandinavian-journal-of-information-systems.csl (#5716)
ce2d537 Update journal-of-computer-applications-in-archaeology.csl (#5715)
755d3d3 Create human-rights-law-review.csl (#5626)
0feda94 Create journal-of-intercultural-studies.csl (#5709)
ae4756d Update acta-universitatis-agriculturae-sueciae.csl (#5713)
323d9ac Update mohr-siebeck-recht.csl (#5559)
15530a8 Bch corr (#5712)
094a1af Create forschungsjournal-soziale-bewegungen-fjsb.csl (#5699)
cb91566 initialize authors and editors (#5714)
2d5cfff Create cancer-biomarkers.csl (#5703)
5e264d5 Update multidisciplinary-digital-publishing-institute.csl (#5708)
46e961f Create klinische-padiatrie.csl (#5711)
e81e877 Create bulletin-archeologique-des-ecoles-francaises-a-l-etranger.csl (#5704)
0029c5a Create polar-research.csl 🧊 (#5702)
7db1361 Update vancouver-imperial-college-london.csl (#5641)
b953e9f Update iso690-author-date-fr-no-abstract.csl (#5706)
91eda8c Update thieme-german.csl (#5710)
ebe0787 Update harvard-imperial-college-london.csl (#5643)
2d4db76 Fix UNESCO IIEP in text
436cbf4 Create revue-archeologique-de-narbonnaise.csl (#5688)
5150bcf Create journal-of-computer-assisted-tomography.csl (#5690)
dd6f050 Create anti-trafficking-review.csl (#5658)
08e622f Create the-angle-orthodontist.csl (#5685)
c6a1907 journal-of-palm-oil-research.csl fix several errors (#5686)
6cbe29d Create bern-university-of-applied-sciences-school-of-agricultural-for… (#5684)
f590dc1 Update biomed-central.csl (#5701)
1efce81 Update turabian-author-date.csl (#5695)
12dbba5 Create tyndale-bulletin (#5673)
b0746db Create Engineered Regeneration (#5682)
e38b953 wikipedia citation template (#5662)
5e7f731 Create early-music-history.csl (#5679)
86443f3 Create zeitschrift-fur-politik.csl (#5676)
68f1996 Create annals-of-work-exposures-and-health.csl (#5666)
1ba9dc6 Create brazilian-journal-of-psychiatry.csl (#5672)
438f92c fix error for speech in ama styles (#5693)
7a0c2d3 set initialize-with-hyphen to false (#5689)
3bd2765 Update emu-austral-ornithology.csl (#5671)
31492b2 fix various errors in natura-croatica.csl (#5687)
94d6b23 Update iso690-author-date-cs.csl (#5677)
5d017da minor update on the "Haute école de gestion de Genève - ISO 690" style (#5665)
2cad8f6 add ibid/subsequent to comparative-politics.csl (#5669)
de0b116 Create taylor-and-francis-vancouver-national-library-of-medicine.csl (#5650)
ed87f99 Update bulletin-de-correspondance-hellenique.csl (#5663)

git-subtree-dir: buildres/csl/csl-styles
git-subtree-split: 3bb4b5f
@koppor koppor mentioned this pull request Jan 9, 2024
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants