INDEL realigner cleanup #1412

fnothaft · 2017-03-02T09:01:24Z

Fixes #1402, and provides many additional improvements to the IR core. Still a WIP, testing on the cluster as we speak.

coveralls · 2017-03-02T09:10:15Z

Coverage increased (+0.7%) to 77.008% when pulling 3cf32f6 on fnothaft:issues/1402-indel-realigner-what into eb4aa6c on bigdatagenomics:master.

AmplabJenkins · 2017-03-02T09:16:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1823/
Test PASSed.

coveralls · 2017-03-02T23:55:21Z

Coverage increased (+1.0%) to 77.3% when pulling 31c68fa on fnothaft:issues/1402-indel-realigner-what into eb4aa6c on bigdatagenomics:master.

AmplabJenkins · 2017-03-02T23:58:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1825/
Test PASSed.

coveralls · 2017-03-03T18:00:51Z

Coverage increased (+0.9%) to 77.174% when pulling fe8a2f2 on fnothaft:issues/1402-indel-realigner-what into eb4aa6c on bigdatagenomics:master.

AmplabJenkins · 2017-03-03T18:04:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1829/
Test PASSed.

heuermh

Minor doc changes

heuermh · 2017-03-03T18:46:29Z

adam-cli/src/main/scala/org/bdgenomics/adam/cli/Transform.scala

@@ -77,6 +77,10 @@ class TransformArgs extends Args4jBase with ADAMSaveAnyArgs with ParquetArgs {
  var lodThreshold = 5.0
  @Args4jOption(required = false, name = "-max_target_size", usage = "The maximum length of a target region to attempt realigning. Default length is 3000.")
  var maxTargetSize = 3000
+  @Args4jOption(required = false, name = "-max_reads_per_target", usage = "The maximum number of reads attached to a target that we will attempt realigning. Default length is 20000.")


How about The maximum number of reads attached to a target considered for realignment. Default is 20000.

heuermh · 2017-03-03T19:05:03Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/read/realignment/RealignIndels.scala

@@ -421,20 +435,30 @@ private[read] class RealignIndels(
   * @param read Read to test.
   * @param reference Reference sequence to sweep across.
   * @param qualities Integer sequence of phred scaled base quality scores.
+   * @param originalQuality


needs doc comment

fnothaft · 2017-03-03T23:45:56Z

adam-core/src/main/scala/org/bdgenomics/adam/models/MdTag.scala

@@ -208,7 +208,9 @@ object MdTag {

          // dirty dancing to recalculate match sets
          for (i <- 0 until cigarElement.getLength) {
-            if (reference(referencePos) == sequence(readPos)) {
+            println(reference + " @ " + referencePos + ", " + sequence + " @ " + readPos)


AmplabJenkins · 2017-03-04T05:11:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1834/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1412/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains b8c4222ca4c78a234f74dfd72089d9343e62fa58 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1412/merge^{commit} # timeout=10Checking out Revision b8c4222ca4c78a234f74dfd72089d9343e62fa58 (origin/pr/1412/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b8c4222ca4c78a234f74dfd72089d9343e62fa58First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

coveralls · 2017-03-04T05:40:11Z

Coverage increased (+0.8%) to 77.192% when pulling 745df4f on fnothaft:issues/1402-indel-realigner-what into 07c1982 on bigdatagenomics:master.

AmplabJenkins · 2017-03-04T05:43:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1835/
Test PASSed.

AmplabJenkins · 2017-03-06T07:01:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1837/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1412/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 6d408c4 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1412/merge^{commit} # timeout=10Checking out Revision 6d408c4 (origin/pr/1412/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 6d408c471459434c79630e5724f1a67c964b88cbFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

coveralls · 2017-03-15T06:10:30Z

Coverage increased (+1.0%) to 77.466% when pulling 8a63091 on fnothaft:issues/1402-indel-realigner-what into cf39e6c on bigdatagenomics:master.

AmplabJenkins · 2017-03-15T06:13:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1877/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1412/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 0761d92 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1412/merge^{commit} # timeout=10Checking out Revision 0761d92 (origin/pr/1412/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 0761d9214029fa634f56bc4a72ea7a61e7d29d71First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2017-03-15T06:14:53Z

Jenkins, retest this please.

coveralls · 2017-03-15T06:32:05Z

Coverage increased (+1.0%) to 77.466% when pulling 8a63091 on fnothaft:issues/1402-indel-realigner-what into cf39e6c on bigdatagenomics:master.

AmplabJenkins · 2017-03-15T06:38:54Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1879/
Test PASSed.

coveralls · 2017-03-23T04:45:45Z

Coverage increased (+0.8%) to 81.35% when pulling 80ad790 on fnothaft:issues/1402-indel-realigner-what into 629b778 on bigdatagenomics:master.

AmplabJenkins · 2017-03-23T04:49:07Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1906/
Test PASSed.

fnothaft · 2017-03-26T04:07:49Z

Thanks @heuermh! Glad to have things cleaned up.

coveralls · 2017-03-26T04:10:52Z

Coverage increased (+0.6%) to 81.226% when pulling 54f328f on fnothaft:issues/1402-indel-realigner-what into 629b778 on bigdatagenomics:master.

AmplabJenkins · 2017-03-26T04:14:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1912/
Test PASSed.

coveralls · 2017-03-28T19:01:15Z

Coverage increased (+0.6%) to 81.439% when pulling e6376e8 on fnothaft:issues/1402-indel-realigner-what into 65d8c9e on bigdatagenomics:master.

AmplabJenkins · 2017-03-28T19:05:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1915/
Test PASSed.

coveralls · 2017-03-28T22:56:00Z

Coverage increased (+0.7%) to 81.541% when pulling 99f4673 on fnothaft:issues/1402-indel-realigner-what into 65d8c9e on bigdatagenomics:master.

AmplabJenkins · 2017-03-28T22:59:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1916/
Test PASSed.

coveralls · 2017-03-29T06:35:51Z

Coverage increased (+0.7%) to 81.523% when pulling f82fe65 on fnothaft:issues/1402-indel-realigner-what into 65d8c9e on bigdatagenomics:master.

AmplabJenkins · 2017-03-29T06:39:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1917/
Test PASSed.

heuermh · 2017-03-29T06:46:48Z

Don't unclip. Yes, or do I mean no?

fnothaft · 2017-03-30T06:47:18Z

@heuermh think this is good to go now; rerunning a final verification run. Can you give the second commit a looksee? That should be everything new since your last pass.

coveralls · 2017-03-30T06:55:56Z

Coverage increased (+0.8%) to 81.652% when pulling 2c868e1 on fnothaft:issues/1402-indel-realigner-what into 65d8c9e on bigdatagenomics:master.

AmplabJenkins · 2017-03-30T06:59:22Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1918/
Test PASSed.

heuermh · 2017-03-30T12:40:15Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/read/realignment/RealignIndels.scala

+    val startClipped = richCigar.softClippedBasesAtStart
+    val endClipped = richCigar.softClippedBasesAtEnd
+
+    if (unclipReads ||


Getting hard to follow the double negative here. Maybe I need more sleep. :)

heuermh · 2017-03-30T12:45:33Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/read/realignment/RealignIndels.scala

+                    builder.setOldPosition(r.getStart())
+                    builder.setOldCigar(r.getCigar())
+                    val rec = builder.build()
+                    if (r.getReadName == "H06JUADXX130110:1:1108:12070:36897") {


heuermh · 2017-03-30T12:47:54Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/read/realignment/RealignIndels.scala

                  } else {
                    r
                  }
                } catch {
                  case t: Throwable => {
-                    log.warn("Realigning read %s failed with %s.".format(r, t))
+                    log.warn("Realigning read %s failed with %s. At:".format(r, t))
+                    val stack = t.getStackTrace()


log.warn(String, Throwable)?

heuermh · 2017-03-30T12:49:32Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/read/realignment/RealignIndels.scala

+   * @param remappingIdx The location in the consensus sequence where this
+   *   read has been realigned.
+   * @param consensus The consensus variant to realign against.
+   * @peturn Returns a tuple with the (read start, read end, cigar).


@peturn → @return

heuermh · 2017-03-30T12:52:35Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/read/realignment/RealignIndels.scala

+    }
+
+    // if read is fully before or fully after the consensus, then we
+    // have a full match


thanks for the comments here

coveralls · 2017-03-30T16:00:55Z

Coverage increased (+0.9%) to 81.716% when pulling 106da85 on fnothaft:issues/1402-indel-realigner-what into 65d8c9e on bigdatagenomics:master.

AmplabJenkins · 2017-03-30T16:03:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1919/
Test PASSed.

coveralls · 2017-03-31T05:46:06Z

Coverage increased (+0.7%) to 81.586% when pulling 3efad26 on fnothaft:issues/1402-indel-realigner-what into 65d8c9e on bigdatagenomics:master.

AmplabJenkins · 2017-03-31T05:49:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1921/
Test PASSed.

Resolves bigdatagenomics#1402. Includes fixes to consensus generator and reference scorer. Improve INDEL realigner performance: * Exit early when realigning will not yield a better score. * Eliminate substring call in sweep over reference. * Change datastructures to be immutable wherever possible. * Add bound checking and other error checking. * Rewrite target association code to use array instead of set, and improve load balancing. * Delete high coverage targets with reduceByKey. Additionally: * Improve telemetry/logging to sort out load balance issue. * Support using reference file in INDEL realignment. * Log reads with negative alignment sizes. * Improved test coverage for insertion realignment. * Fix CIGARs on reads that partially overlap INDEL. * Soft clip reads that partially align to an insertion. * Eliminate non-determinism. * Fixed reference file. * Serialization fixes and debug. * Fix bad score. * Clean up clipping code? * Unclip clipped reads.

fnothaft · 2017-03-31T07:11:53Z

Squashed down; I say this is good to go from my end.

coveralls · 2017-03-31T07:20:54Z

Coverage increased (+0.7%) to 81.586% when pulling b6de9a5 on fnothaft:issues/1402-indel-realigner-what into 65d8c9e on bigdatagenomics:master.

AmplabJenkins · 2017-03-31T07:24:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1922/
Test PASSed.

heuermh · 2017-03-31T16:50:01Z

Thank you, @fnothaft!

fnothaft mentioned this pull request Mar 3, 2017

Remove reliance on MD tags #622

Closed

heuermh requested changes Mar 3, 2017

View reviewed changes

fnothaft mentioned this pull request Mar 3, 2017

Address classes with lower than ideal coverage #1405

Closed

fnothaft added this to the 0.21.1 milestone Mar 3, 2017

fnothaft commented Mar 3, 2017

View reviewed changes

fnothaft force-pushed the issues/1402-indel-realigner-what branch from e533c90 to 745df4f Compare March 4, 2017 05:32

heuermh approved these changes Mar 6, 2017

View reviewed changes

This was referenced Mar 8, 2017

Efficient Joins and (re)Partitioning #1324

Merged

Clean up replicated join code in INDEL realigner and BQSR #1429

Closed

Add test sample(verify number of reads in loadAlignments function) and ADAM SNAPSHOT document #1433

Closed

fnothaft force-pushed the issues/1402-indel-realigner-what branch from ef92e6c to 8a63091 Compare March 15, 2017 06:01

heuermh approved these changes Mar 22, 2017

View reviewed changes

fnothaft force-pushed the issues/1402-indel-realigner-what branch from 8a63091 to 80ad790 Compare March 23, 2017 04:34

fnothaft force-pushed the issues/1402-indel-realigner-what branch from 54f328f to e6376e8 Compare March 28, 2017 18:52

fnothaft force-pushed the issues/1402-indel-realigner-what branch from f82fe65 to 2c868e1 Compare March 30, 2017 06:44

heuermh requested changes Mar 30, 2017

View reviewed changes

fnothaft mentioned this pull request Mar 30, 2017

[ADAM-1453] Add hooks to publish ADAM docs from CI flow. #1461

Merged

heuermh approved these changes Mar 30, 2017

View reviewed changes

fnothaft force-pushed the issues/1402-indel-realigner-what branch from 3efad26 to b6de9a5 Compare March 31, 2017 07:11

heuermh merged commit d12fdfb into bigdatagenomics:master Mar 31, 2017

fnothaft mentioned this pull request Apr 1, 2017

Add new realigner arguments to docs #1465

Closed

INDEL realigner cleanup #1412

INDEL realigner cleanup #1412

Conversation

fnothaft commented Mar 2, 2017

coveralls commented Mar 2, 2017 • edited Loading

AmplabJenkins commented Mar 2, 2017

coveralls commented Mar 2, 2017 • edited Loading

AmplabJenkins commented Mar 2, 2017

coveralls commented Mar 3, 2017 • edited Loading

AmplabJenkins commented Mar 3, 2017

heuermh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Mar 4, 2017

Build result: FAILURE

coveralls commented Mar 4, 2017 • edited Loading

AmplabJenkins commented Mar 4, 2017

AmplabJenkins commented Mar 6, 2017

Build result: FAILURE

coveralls commented Mar 15, 2017 • edited Loading

AmplabJenkins commented Mar 15, 2017

Build result: FAILURE

fnothaft commented Mar 15, 2017

coveralls commented Mar 15, 2017 • edited Loading

AmplabJenkins commented Mar 15, 2017

coveralls commented Mar 23, 2017 • edited Loading

AmplabJenkins commented Mar 23, 2017

fnothaft commented Mar 26, 2017

coveralls commented Mar 26, 2017 • edited Loading

AmplabJenkins commented Mar 26, 2017

coveralls commented Mar 28, 2017 • edited Loading

AmplabJenkins commented Mar 28, 2017

coveralls commented Mar 28, 2017 • edited Loading

AmplabJenkins commented Mar 28, 2017

coveralls commented Mar 29, 2017 • edited Loading

AmplabJenkins commented Mar 29, 2017

heuermh commented Mar 29, 2017 • edited Loading

fnothaft commented Mar 30, 2017

coveralls commented Mar 30, 2017 • edited Loading

AmplabJenkins commented Mar 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Mar 30, 2017 • edited Loading

AmplabJenkins commented Mar 30, 2017

coveralls commented Mar 31, 2017 • edited Loading

AmplabJenkins commented Mar 31, 2017

fnothaft commented Mar 31, 2017

coveralls commented Mar 31, 2017 • edited Loading

AmplabJenkins commented Mar 31, 2017

heuermh commented Mar 31, 2017

coveralls commented Mar 2, 2017 •

edited

Loading

coveralls commented Mar 2, 2017 •

edited

Loading

coveralls commented Mar 3, 2017 •

edited

Loading

coveralls commented Mar 4, 2017 •

edited

Loading

coveralls commented Mar 15, 2017 •

edited

Loading

coveralls commented Mar 15, 2017 •

edited

Loading

coveralls commented Mar 23, 2017 •

edited

Loading

coveralls commented Mar 26, 2017 •

edited

Loading

coveralls commented Mar 28, 2017 •

edited

Loading

coveralls commented Mar 28, 2017 •

edited

Loading

coveralls commented Mar 29, 2017 •

edited

Loading

heuermh commented Mar 29, 2017 •

edited

Loading

coveralls commented Mar 30, 2017 •

edited

Loading

coveralls commented Mar 30, 2017 •

edited

Loading

coveralls commented Mar 31, 2017 •

edited

Loading

coveralls commented Mar 31, 2017 •

edited

Loading