Ability to filter records

[rbwendt]

It would be useful to be able apply a filter to records from the source data beyond simply pulling everything from the namespace.

In my use case, I have denormalized data in mongo with several different document types in the same schema. When I attempt to copy this to elastic search, no mapping can accommodate the different schemas correctly, so each document type in mongo (stored in the same mongo collection) needs to be sent to a different type in elasticsearch.

There are two ways I can conceive of doing this, neither of which this tool appears to support:

1. Filter by document type in mongo and set up a different pipeline for sending each into a separate type in elasticsearch.
2. Don't filter by document type, but alter the namespace being sent into save() per document.

[nstott]

I think that option number 2 is the right way to do this,
there's a PR that we should be merging today that deals with filtering namespaces, and will pass that information to the transformer
you can drop messages if you return 'false' from the transformation function, or change the namespace programatically

see: #101

[rbwendt]

Thanks for that false tip. I spent hours looking through your code, but didn't glean that one. I can just write one transformer for every type for now. It will be ugly and inefficient, but it should work.

[nstott]

the 'false' thing was just merged in the other day
see: #95

[jipperinbham]

fixed via #95