add -merge command to Transform

[antonstamov]

Add ability to use as INPUT multiple splitted BAMs or SAMs separated by the comma and merge them to one RDD.
Example of INPUT: /path/to/1.bam,/path/to/2.bam,/path/to/3.bam,/path/to/4.bam

[antonstamov]

I'm sorry for old commits that were manually merged by @fnothaft

[AmplabJenkins]

Can one of the admins verify this patch?

[ryan-williams]

So, the impetus for this is your work on #689 and #733, where you have BAM/SAM file pieces laying around without headers, right?

Is there any motivation to be able to do this with ADAM files as well?

Do the reads in the merged files retain any provenance info about which file they came from? Do we care?

I guess my concern is that this makes it seem like we support merging BAM files that are logically actually multiple files, not just partitions of one; given that this is only about the latter, it feels like we're shipping some implementation details if we don't have a story for how this works in the general case.

At the very least maybe we should document the context that this is meant to be used in, and the contexts where it wouldn't make sense or would be lossy?

[antonstamov]

@ryan-williams no no no, this is just a small tweak to load splitted bam/sam. In my case it were output files from http://ngsutils.org/modules/bamutils/split/

[ryan-williams]

ah ok, cool. any reason to not allow it to work for e.g. ADAM files as well?

[ryan-williams]

in any case, this lgtm but i'll let @fnothaft or someone else have a look before merging

[massie]

Add to whitelist

[massie]

Jenkins, test this please.

[massie]

Jenkins, test this please.

[ryan-williams]

Seems like ./scripts/format-source is failing?

[massie]

@antonstamov Can you run the ./scripts/format-source script in the ADAM repo and then push any changes in the source? That's the cause of the test failure -- it's failing on the format of your source.

Thanks.

[massie]

@antonstamov I'd like to merge this PR as soon as possible. Can you run the following?

$ cd <adam repo>
$ ./scripts/format-source
$ git add .
$ git commit -m "Formatting source"
$ git push --force origin adam-merge-tool:adam-merge-tool

This will update this PR and fix the syntax errors. Tests are failing because of the format of the source.

[antonstamov]

@ryan-williams, @massie fixed, sorry for long response

[fnothaft]

@ryan-williams I think this is what you were looking for on #831, no?

[heuermh]

@fnothaft now that #831 has been merged, do you know what the status of this PR should be?

[fnothaft]

I believe this has been superseded by #831. We can reopen this later if needed.