Generate rolling builds of the ontology

[pgaudet]

Hello,

We would like to have daily versions of GO for use in the curation tools (Noctua and P2GO).

@kltm

• is this a task for @balhoff ?
• should I create a separate project?
  Thanks, Pascale

[balhoff]

Oh, interesting, I was independently wondering to myself if we should disconnect ontology production from the pipeline runs, and do GitHub releases similar to other OBO ontologies. We should probably talk this through.

[kltm]

Is this a project? Likely, "yes", although possibly a short one if it went ahead. That said, I really want to be careful in the use cases and coordination here--there is a reason we have historically "locked" these things to annotation and it's an orthogonal (although very real) problem that we're having issues with the production pipelines.

Some important orienting notes on this:

• we already fully build the ontology 3 times a day (http://skyhook.berkeleybop.org/go-ontology-dev/), but the product is transient
  • archiving that so it is not volatile, to a public URL, is "easy"
• noctua/minerva are updated every two weeks and, as far as we can tell (noctua-models-migrations makes use of snapshot, rather than "NEO" pipeline#363), use the absolute latest, as it's self-contained

So the questions I have look like:

• exactly who is going to be using a high-frequency ontology and in what use cases? I'm assuming Alex here, but who else?
• where do PURLs go?
• assuming that we get snapshots back to where we want them to be, say daily-ish, where would this stand?
  • counting down getting the new machine [https://github.com/Buy and setup a new pipeline machine pipeline#351] and some fixes I think may help [https://github.com/Consider (and attempt) blessing snapshots runs to "release" status pipeline#352#issuecomment-2000772991]), what is the preferred use; does it change?

[pgaudet]

@kltm what do you mean by 'absolute latest' here?

noctua/minerva are updated every two weeks and, as far as we can tell (geneontology/pipeline#363), use the absolute latest, as it's self-contained

Is 'snapshot' considered the 'absolute latest'? Or do you use one of the 'volatile' daily products to build NEO?

[kltm]

There's nothing "volatile" per se, but it is "made-to"order" from GH--it is self-contained and has nothing to do with anything else. Ontology building is easy an occurs many places, depending on use.

[kltm]

Talking to @pgaudet this morning, we worked out the following points:

• a "rolling" ontology product would be useful right now
  • a given use case was to give other curation groups access to a built ontology product
• this would not be for "general consumption", as we want to generally steer people towards products that can be referenced
• this can be revisited when snapshots (and releases) are consistent again
• there would have to be good communication / documentation on this

[kltm]

Initial testing (just populating bucket go-data-product-ontology-build) on 4pm PT run.

[kltm]

This is a thing now: https://ontology-build.geneontology.org
If go-ontology-dev is successful, it will update. Currently, that means up to three times a day.

I'm now running through some tests and looking at maintenance.
As it stands now, if go-ontology-dev fails for two weeks, it will go dark (as the cleaning routines take passes through).

Assuming this is about right, we can turn over the conversation to documentation and communication about expectations.

[pgaudet]

Excellent!! Thanks !

[pgaudet]

Moved this to a new Project.

[kltm]

@pgaudet I updated the project metadata to remove mention of Noctua (this has nothing to do with noctua) and "daily".