nim CI broken: eth2_vectors.nim(17, 3) Error: cannot open file: yaml

[timotheecour]

/cc @mratsim

• your recent commit might've caused this f017051

• wouldn't nim-blscurve's own CI prevent such regressions? => might indicate a bug in this repo's CI, which said: 9 checks passed

https://dev.azure.com/nim-lang/255dfe86-e590-40bb-a8a2-3c0295ebdeb1/_apis/build/builds/2970/logs/68

2020-03-01T22:13:52.0997954Z PASS: https://github.com/bluenote10/nim-heap C                     ( 4.59764791 secs)
2020-03-01T22:13:52.0998413Z FAIL: https://github.com/status-im/nim-blscurve C
2020-03-01T22:13:52.0998905Z Test "https://github.com/status-im/nim-blscurve" in category "nimble-packages"
2020-03-01T22:13:52.0999336Z Failure: reBuildFailed
2020-03-01T22:13:52.0999628Z package test failed
2020-03-01T22:13:52.0999928Z $ nimble test
2020-03-01T22:13:52.1000511Z   Executing task test in D:\a\1\s\pkgstemp\blscurve\blscurve.nimble
2020-03-01T22:13:52.1000900Z 
2020-03-01T22:13:52.1001244Z [Suite] [Before IETF standard] BLS381-12 test suite (public interface)
2020-03-01T22:13:52.1001553Z 
2020-03-01T22:13:52.1002130Z [Suite] [v0.9.x] Ethereum2 specification BLS381-12 test vectors suite
2020-03-01T22:13:52.1002653Z D:\a\1\s\pkgstemp\blscurve\tests\eth2_vectors.nim(17, 3) Error: cannot open file: yaml
2020-03-01T22:13:52.1003159Z stack trace: (most recent call last)
2020-03-01T22:13:52.1003607Z C:\Users\VSSADM~1\AppData\Local\Temp\nimblecache\nimscriptapi.nim(165, 16)
2020-03-01T22:13:52.1004437Z D:\a\1\s\pkgstemp\blscurve\blscurve_6708.nims(40, 8) testTask
2020-03-01T22:13:52.1005523Z D:\a\1\s\pkgstemp\blscurve\blscurve_6708.nims(21, 8) test
2020-03-01T22:13:52.1006040Z D:\a\1\s\lib\system\nimscript.nim(252, 7) exec
2020-03-01T22:13:52.1006683Z D:\a\1\s\lib\system\nimscript.nim(252, 7) Error: unhandled exception: FAILED: nim c -d:BLS_USE_IETF_API=true --outdir:build -r --hints:off --warnings:off tests/eth2_vectors.nim [OSError]
2020-03-01T22:13:52.1007324Z        Tip: 1 messages have been suppressed, use --verbose to show them.
2020-03-01T22:13:52.1007766Z      Error: Exception raised during nimble script execution
2020-03-01T22:13:52.1008035Z 
2020-03-01T22:35:18.4345562Z PASS: https://github.com/status-im/nim-bncurve C                   (127.68649817 secs)
2020-03-01T22:35:18.4347310Z PASS: https://github.com/nim-lang/c2nim C                          ( 3.22719979 secs)

which breaks my recent PR's (eg nim-lang/Nim#13546 )

[mratsim]

1. For testing, blscurve now requires NimYAML, it's not a package dependency so I did not list it in the .nimble file
2. In the CI setup it has been added as via a nimble install https://github.com/status-im/NimYAML@#head preceding nimble test
3. To make matter worse, the original yaml package was broken by YAML upstream deleting a commit that prevents git submodule update to work in nimYAML "test/yaml-test-suite" submodule broken by upstream flyx/NimYAML#77. This means that requiring the yaml package will also trigger failures
4. AFAIK we can't do requires https://github.com/status-im/NimYAML@#head in a nimble file as a work-around
5. Nimble doesn't support task-level dependencies Feature: Task-level dependencies nim-lang/nimble#482

Plans forward:

• We can't wait for nimble features (i.e. requiring a repo path + branch and task-level dependencies)
• We can't wait for nimble install yaml to be fixed

So I recommend you disable blscurve and probably all packages that requires NimYAML in the Nim CI for the time being.

In the future (a week or so), once official test vectors are available (in json format) we will remove the dependency on the current YAML test vectors (cfrg/draft-irtf-cfrg-hash-to-curve#214)

[timotheecour]

• maybe send an issue to https://github.com/flyx/NimYAML/issues (and link here)?

AFAIK we can't do requires https://github.com/status-im/NimYAML@#head in a nimble file as a work-around

wouldn't that increase chances of mutually exclusive requirements, eg:

pkg1: 
requires https://github.com/status-im/NimYAML@#head
pkg2: 
requires https://github.com/status-im/NimYAML@#1234 # 
pkg3:
requires pkg1, pkg2

the alternative is:

• nimble develop https://github.com/status-im/NimYAML (or manually via git clone until nimble develop https://github.com/yglukhov/nimx clones to yglukhov/nimx instead of nimx nim-lang/nimble#577 is fixed)
• if needed checkout a specific hash
• proceed with installing other packages

in any case, that discussion should happen in https://github.com/nim-lang/nimble/issues otherwise will never be fixed; maybe open an issue there and link to here?

task-level dependencies
can then be achieved by (untested):
nimble install -d:with_foo mypkg

# mypkg.nimble:
when defined(with_foo):
  requires foo

[mratsim]

1. You probably missed "test/yaml-test-suite" submodule broken by upstream flyx/NimYAML#77 in my post.
2. For the mutually exclusive requirements, it should be discussed in the lock files issue: Locking of dependencies nim-lang/nimble#127. Being unable to install pkg3 due to conflicting requirements is not a bug.

• We can also requires that a minimal commit or a maximum commit in the package for more flexibitlity but targeting a specific commit seems like a nice start.
• Alternatively, Rust can somehow handle multiple conflicting requirements in the same codebase, including versions of the Rust compiler. But AFAIK that would require a lot of plumbing on Nim side to be able to do the same.

3. Using nimble install -d:with_foo mypkg is not really an improvement over nimble install yaml mypkg for testing. Both require to be aware of specific testing dependencies.