HBase seperate module PR #1388

jpdna · 2017-02-13T12:29:43Z

Testing to see if tests succeed with Jenkins.
Not ready for review yet.

AmplabJenkins · 2017-02-13T13:16:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1783/
Test PASSed.

AmplabJenkins · 2017-02-13T14:28:07Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1784/
Test PASSed.

fnothaft

Looks like a good update! You'll want to update jenkins-test to run the HBase profile.

fnothaft · 2017-02-14T09:51:41Z

adam-hbase/pom.xml

+      <groupId>org.bdgenomics.utils</groupId>
+      <artifactId>utils-cli_${scala.version.prefix}</artifactId>
+    </dependency>
+    <dependency>


You should be able to trim out some of these dependencies, e.g., intervalrdd, fastutil, etc.

fnothaft · 2017-02-14T09:52:44Z

adam-hbase/src/main/resources/log4j.properties

@@ -0,0 +1,18 @@
+# Root logger option
+log4j.rootLogger=INFO, stderr, logfile


Do we need this in addition to the adam-core log4j property file? As an aside, I think that @ryan-williams was making a compelling argument recently for not committing log4j property files, but I can't quite remember it.

fnothaft · 2017-02-14T09:53:17Z

adam-hbase/src/main/scala/org/bdgenomics/adam/hbase/HBaseFunctions.scala

+package org.bdgenomics.adam.hbase
+
+import java.io.ByteArrayOutputStream
+


fnothaft · 2017-02-14T09:53:28Z

adam-hbase/src/main/scala/org/bdgenomics/adam/hbase/HBaseFunctions.scala

+import org.bdgenomics.adam.rich.RichVariant
+import org.bdgenomics.formats.avro._
+import org.bdgenomics.adam.models.{ ReferencePosition, ReferenceRegion, SequenceDictionary, VariantContext }
+


Nit: space.

fnothaft · 2017-02-14T09:53:59Z

adam-hbase/src/main/scala/org/bdgenomics/adam/hbase/HBaseFunctions.scala

+     *
+     * @param tableDescriptorMeta
+     */
+    private[hbase] def createTable(tableDescriptorMeta: HTableDescriptor) { admin.createTable(tableDescriptorMeta) }


Break long line here and in the next function.
Also, add docs for parameters.

fnothaft · 2017-02-14T09:57:33Z

adam-hbase/src/main/scala/org/bdgenomics/adam/hbase/HBaseFunctions.scala

+  private[hbase] object KeyStrategy1 extends KeyStrategy[KeyStrategy1rowKeyInfo, KeyStrategy1RangePrefixInfo] {
+    def getKey(rowKeyInfo: KeyStrategy1rowKeyInfo): Array[Byte] = {
+
+      println("rowkey: " + "%s_%010d_%s_%s_%d".format(rowKeyInfo.contigName,


Nix println.

fnothaft · 2017-02-14T10:00:02Z

adam-hbase/src/test/resources/small.vcf

@@ -0,0 +1,57 @@
+##fileformat=VCFv4.1
+##FILTER=<ID=IndelFS,Description="FS > 200.0">


You don't need to copy this file, you should be able to use the copyResource function in ADAMFunSuite.

fnothaft · 2017-02-14T10:00:13Z

adam-hbase/src/test/scala/org/bdgenomics/adam/hbase/HBaseSuite.scala

+import org.mockito.Mockito._
+
+/**
+ * Created by paschallj on 11/25/16.


Nix author comments.

fnothaft · 2017-02-14T10:00:51Z

pom.xml

@@ -565,6 +591,31 @@
        <artifactId>scala-guice_${scala.version.prefix}</artifactId>
        <version>4.0.1</version>
      </dependency>
+      <dependency>


I think you can actually move these dependencies into the hbase profile as well.

fnothaft · 2017-02-14T10:01:26Z

pom.xml

+        <!-- distribution should be last -->
+        <module>adam-hbase</module>
+      </modules>
+      <build>


You shouldn't need the build bits in the profile.

Added HBaseFunctions Added VCF genotype save and load Hbase functions Added saveHBaseGenotypesSingleSample packing multiple genotypes at same position and samle allele into same hbase value Added multisample capable loadHBaseGenotypes() Removed commented-out earlier versions of save and load genotypes removed more dead code Clean up formatting - limit line length Added saveHbaseSampleMetadata function Added save and load SequenceDictionary functions to HBaseFunctions Added createHbaseGenotypeTable Adding loadGenotypeRddFromHBase in progress updates to multi-sample hbase save multi sample VCF save and load now working Added repartitioning parameter tp hbase genotype load Added comments identifying the public api vs helper functions COB Aug 25 Added genomic start and stop parameters to loadGenotypesFromHBaseToGenotypeRDD Added boolean saveSequenceDictionary toggle parameter to saveVariantContextRddToHBase fixed start, stop null ptr exeception first steps in adding hbaseToVariantcontextRDD Changed region query to use ADAM ReferenceRegion Added custom HBaseEncoder1 save function Added custom Encoder1 Hbase loader Added Encoder1 hbase variant context loader Working - before rowkey int Changed end in key to be size, added data block encoding in create table Added create table splits Removed dead code of encoder2 Added option to repartion vcrdd before saving to HBase Added bulk load save to Hbase option changed to cdh hbase api depedencies in POM allow sample name file list as input to load functions made sample_ids lis parameter in load optional Added deleteSamplesFromHBase function Fixed bulk delete and made loadVariantContext work even when requested samplids are missing Removed code from failed version of sample delete function Moved delete function up with test of genotype code Fixed errors after rebase against master small formatting cleanup first pass hbase cli demo second pass hbase cli demo remove saveSeqDict check add seq dict id to cli import clean up removed undeeded demo and temp.js code Ran ./scripts/format-source due to build failure on Jenkins changed Hbase dependencies to provided addressed some review issues Changed hbases dependencies back to NOT being provided fixed None.get first step key Strategy pattern second step key Strategy pattern more cleanup from review made private functions package private for hbase Factored out hbase connection Factored out hbase connection - fix1 Factored out hbase connection - cleanup Removed the non-bulk load hbase function step one adding tests first step implement DAO step2 DAO step3 DAO Added first loadHBase Test Added some javadoc use dao everywhere added more javadoc step 1 in adding save hbase test step 2 in adding save hbase test clean up hbase doc add more assertions to hbase load test improved POM and imports in HBaseFunctions fixed more PR suggestions

AmplabJenkins · 2017-02-22T11:54:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1813/
Test PASSed.

jpdna · 2017-02-22T14:24:21Z

I find that to access the class (using the code from this PR)
org.bdgenomics.adam.hbase.HBaseFunctions
from adam-shell I must do:
./bin/adam-shell --jars adam-hbase/target/adam-hbase_2.10-0.20.1-SNAPSHOT.jar
otherwise without "--jars" the class cannot be found.

I'm not sure how the POM needs to modified so that adam-shell can see the hbase module, suggestions @heuermh ?

heuermh · 2017-02-22T16:00:26Z

I suppose we need a fix for #1349 and/or a conditional dependency on adam-hbase from adam-assembly.

fnothaft · 2017-02-22T16:26:07Z

I would personally go with:

a conditional dependency on adam-hbase from adam-assembly.

Since the hbase work comes in through a profile, I think we can just make the change (added dependency) there.

fnothaft · 2018-01-21T04:06:21Z

Closing this in favor of #1878.

fnothaft requested changes Feb 14, 2017

View reviewed changes

jpdna added 8 commits February 22, 2017 06:23

Fixes after rebase

250e703

changed mockito scope back to test

fe67b15

Address review comments

e351d97

Changed HBaseSuite for Jenkins

639b190

Trying to make adam-hbase module - still not working

dfa4dab

added adam-core dependency

985f26a

Fixed activation property for hbase and hbase artifcat name

8c6bae8

jpdna force-pushed the dev/b6 branch from 4d124a5 to 8c6bae8 Compare February 22, 2017 11:40

fnothaft mentioned this pull request Mar 3, 2017

HBase genotypes backend -revised #1335

Closed

heuermh added this to the 0.24.0 milestone Nov 29, 2017

fnothaft removed this from the 0.24.0 milestone Jan 21, 2018

fnothaft closed this Jan 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBase seperate module PR #1388

HBase seperate module PR #1388

jpdna commented Feb 13, 2017

AmplabJenkins commented Feb 13, 2017

AmplabJenkins commented Feb 13, 2017

fnothaft left a comment

fnothaft Feb 14, 2017

fnothaft Feb 14, 2017

fnothaft Feb 14, 2017

fnothaft Feb 14, 2017

fnothaft Feb 14, 2017

fnothaft Feb 14, 2017

fnothaft Feb 14, 2017

fnothaft Feb 14, 2017

fnothaft Feb 14, 2017

fnothaft Feb 14, 2017

AmplabJenkins commented Feb 22, 2017

jpdna commented Feb 22, 2017

heuermh commented Feb 22, 2017

fnothaft commented Feb 22, 2017

fnothaft commented Jan 21, 2018

		@@ -0,0 +1,18 @@
		# Root logger option
		log4j.rootLogger=INFO, stderr, logfile

		package org.bdgenomics.adam.hbase

		import java.io.ByteArrayOutputStream

		@@ -0,0 +1,57 @@
		##fileformat=VCFv4.1
		##FILTER=<ID=IndelFS,Description="FS > 200.0">

HBase seperate module PR #1388

HBase seperate module PR #1388

Conversation

jpdna commented Feb 13, 2017

AmplabJenkins commented Feb 13, 2017

AmplabJenkins commented Feb 13, 2017

fnothaft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Feb 22, 2017

jpdna commented Feb 22, 2017

heuermh commented Feb 22, 2017

fnothaft commented Feb 22, 2017

fnothaft commented Jan 21, 2018