Remove StructuralVariant and StructuralVariantType, add names field to Variant

[heuermh]

Still left to do (in a later pull request) is to populate the somatic field from VCF INFO attributes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1425/

Build result: FAILURE

GitHub pull request #1131 of commit e8774e9 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1131/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains e3194f8 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/merge^{commit} # timeout=10Checking out Revision e3194f8 (origin/pr/1131/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f e3194f87b2dda6dc4757303228f43d1099d005d5First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[heuermh]

This won't pass CI until bigdatagenomics/bdg-formats#103 is merged and the new snapshot version of bdg-formats is available.

[heuermh]

Something is git-wrong with this branch.

It is supposed to have the commit from #1126 in it (bf886c9) but instead has the same commit with a different hash (ea53e2d).

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1430/

Build result: FAILURE

GitHub pull request #1131 of commit 5f55d86 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1131/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains a28e8fe # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/merge^{commit} # timeout=10Checking out Revision a28e8fe (origin/pr/1131/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f a28e8fe3a08c123682bd4ce66ea800741137fffbFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[fnothaft]

How'd you pull the commit over from #1126? Also, when I merged #1126 into the topic branch, I did github's squash-and-merge, which (unfortunately) triggers an update to the git commit hash.

[heuermh]

I cherry picked it over originally, then l did a rebase here against the topic branch.

[fnothaft]

If you cherry pick the commit over, I wouldn't expect the hash to be the same anyways.

[jpdna]

+1
I round tripped a VCF with vcf2adam then adam2vcf and the names (rs_ids) are present in the VCF as advertised.

[fnothaft]

Can you write some unit tests for this? Ideally, we'd have tests for just this function with a VC with no ID, one ID, more than one ID, and an end-to-end VCF-with-IDs test.

[jpdna]

Can you write some unit tests for this?

As in me Justin? I'd be happy to work on the unit tests - would be good for me. But if @heuermh wants to as his PR he can have first dibs. Just let me know what you prefer Michael - and if you want me to do it, should it be a separate PR or do you want me to PR against your PR Michael?

[fnothaft]

Oh, sorry for the confusion! My comment was directed at @heuermh, unless he'd like you to work on it, of course!

[heuermh]

No worries, I got it!

[fnothaft]

LGTM other than test nit!

[heuermh]

Added unit tests, rebased, and force pushed. Sorry about the conflicts, not sure how to resolve them.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1433/

Build result: FAILURE

GitHub pull request #1131 of commit ed3c420.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse ed3c420^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains ed3c420 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/head^{commit} # timeout=10Checking out Revision ed3c420 (origin/pr/1131/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f ed3c4205748cb4025d60657d1ed65c632ab5e432First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[heuermh]

Should I register mutable.ArrayBuffer or modify the unit tests (where I use Seq())?

- can write, then read in .vcf file *** FAILED ***
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 4, localhost): com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.collection.mutable.ArrayBuffer
Note: To register this class use: kryo.register(scala.collection.mutable.ArrayBuffer.class);

[fnothaft]

I'd probably register it.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1434/

Build result: FAILURE

GitHub pull request #1131 of commit 385cb9e.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse 385cb9e^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 385cb9e # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/head^{commit} # timeout=10Checking out Revision 385cb9e (origin/pr/1131/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 385cb9e > /home/jenkins/git2/bin/git rev-list ed3c420 # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1435/

Build result: FAILURE

GitHub pull request #1131 of commit 34877cf.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse 34877cf^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 34877cf # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/head^{commit} # timeout=10Checking out Revision 34877cf (origin/pr/1131/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 34877cf > /home/jenkins/git2/bin/git rev-list 385cb9e # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[heuermh]

Jenkins, retest this please.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1436/

Build result: FAILURE

GitHub pull request #1131 of commit 34877cf.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse 34877cf^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 34877cf # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/head^{commit} # timeout=10Checking out Revision 34877cf (origin/pr/1131/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 34877cf > /home/jenkins/git2/bin/git rev-list 34877cf # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1437/

Build result: FAILURE

GitHub pull request #1131 of commit b56b77c.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse b56b77c^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains b56b77c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1131/head^{commit} # timeout=10Checking out Revision b56b77c (origin/pr/1131/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b56b77c > /home/jenkins/git2/bin/git rev-list 34877cf # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[heuermh]

All right, this is starting to piss me off. Why can't I replicate this test failure locally?

[heuermh]

@fnothaft sorry, I'm not sure what to do to fix this one.

"spark.kryo.registrationRequired" -> "true" in ADAMFunSuite but I don't see these test failures locally.

[heuermh]

Jenkins, retest this please.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1439/
Test PASSed.

[heuermh]

WTF! Please merge this before Jenkins changes his mind.

[fnothaft]

Merged into upgrade-to-bdg-formats-0.10.0 as 97d9290. I squashed down a spurious cleanup commit and rebased upgrade-to-bdg-formats-0.10.0 on master. Thanks @heuermh!

[heuermh]

Great, thank you!