Speed up current development cycle with bypasses for some time-consuming tasks

[kltm]

The master (main development) build currently takes almost seven hours to complete--basically the length of time a full run used to take sometime last year. This severely slows down the development and testing cycle. We have tricks that that we'd like to formalize that we've used on branches to speed things up.

• Bypass ontology build phase (40m) by coping over the files from the last successful release to the local filesystem in the same pattern as the final step products. This would cause a time shift for the ontology, but would be contained in the dev branches.
• Add a control variable for the noctua-models repo and change the repository for noctua-models to be a branch the only has a handful of models (a few hours).

These options would be turned on/off with a REUSE_PREVIOUS_WORK_P = 'TRUE'. In additions to these, this would be also have

• Environment check that the flag cannot be 'TRUE' if the run is release or snapshot

An experiment indicated that we could trim 3-4hrs off of master build time, with an aggressive data trimming getting us down to about 1.5hr total time.

As we're already doing things like this as part of our normal development, this would just be to systematize these tricks. The next time we do something like this, we'd like to add them back in to master.

Tagging @dougli1sqrd @kltm

[kltm]

@pgaudet While I've added this as a TODO in the current sprint, it will likely be something that is incidentally done as part of other development and is a "freebie". Please let me know if you have concerns about its categorization.

[pgaudet]

Hi @kltm
I think this is fine to do this now - I am just not clear: snapshot will now have older versions of the ontology ?

Thanks, Pascale

[kltm]

This would have no effect on release or snapshot; it is only a formalization of some "tricks" to help speed development on master and other derivative branches.

[dougli1sqrd]

I'm starting a branch and going to start some initial work here. Branch is master_speedup_issue-126.

[kltm]

issue-126-master_speedup if you want to fit with the dominant pattern

[dougli1sqrd]

Here's the branch so far: https://github.com/geneontology/pipeline/tree/issue-126-master_speedup
I've made:

• REUSE_PREVIOUS_WORK_P
• CURRENT_NOCTUA_MODELS_BRANCH = "master"
• SMALL_NOCTUA_MODELS_BRANCH = "subset"

I check if we have REUSE_PREVIOUS_WORK_P set and if we're snapshot or release, in which case we bail. Then if we do have it set and we didn't bail, then we set the noctua branch to the smaller subset branch.

I have an unused variable in there at the moment that we can delete to set the standard noctua branch as well (if we wanted to switch back?? Not sure if we need that ultimately).

TODO: We will have to make a Makefile change I think in order for the noctua-models change to propagate into the Makefile step where we clone the repo ourselves. Perhaps that route of getting the noctua-models should be investigated in the Makefile at some point? But as it is the Makefile will only grab master noctua-models, and so will load everything into the blazegraph. If we want a faster run, we should make sure to get the small subset loaded.

[kltm]

Discussed with @dougli1sqrd on current branch work--looking good. TODO: generalized (single target branch) for noctua-models, ontology copyover, (bonus) noctua-models target branch into Makefile.

[dougli1sqrd]

geneontology/go-site#1183 For noctua models cloning in go-site Makefile

[dougli1sqrd]

#129 is the final set of things on the list above. We should run a test of this.

[dougli1sqrd]

@kltm I can make changes to the issue-126-master_speedup branch that allows it to be run on Jenkins, to actually run it, before it's merged (merging to be done after the release either way)

[kltm]

@dougli1sqrd Good to test and prep.

[kltm]

I think this one has timed out.