Redo NEO pipeline

[kltm]

Project link

https://github.com/orgs/geneontology/projects/TBD

Project description

Redo NEO pipeline: more entities, scalable, and folded into main pipelines.

PI

TBD

Project owner (PO)

TBD

Technical lead (TL)

TBD

Other personnel (OP)

TBD

Technical specs

TBD

Other comments

See geneontology/pipeline#35

• get tests in
• auto-deploy to noctua-amigo
• fold into main pipeline
• Use ROBOT instead of owltools in NEO build neo#84
• Remove NEO build dependency on frozen datasets.json neo#58

[pgaudet]

Could we also add:

• ad hoc addition of TrEMBL entries (see Adding UniProt IDs? noctua#799)
• more frequent NEO updates (which possibly includes automation of the process)

[kltm]

@pgaudet
I'm not really sure of the scope of the first bullet. If it's another not-too-large list, it would be easy to add and could be bundled. If it was larger and approached scalability problems, it might be better added to an eventual (discussed elsewhere) rewrite of the very silly NEO infrastructure that we're using.
The second bullet is more of a Noctua and outage question than a technical or software question--we can produce NEO loads however past we want, but we specifically cut back to meet the cadence of the Noctua outages to keep messaging clear (tagging @vanaukenk). I was imagining this pre-project, lifted from a larger "super" ticket from a while ago, as more about cutting back on the proliferation of independent pipelines and the associated overhead.