Skip to content
This repository has been archived by the owner on Nov 5, 2019. It is now read-only.

Latest commit

 

History

History
31 lines (19 loc) · 1.59 KB

README.md

File metadata and controls

31 lines (19 loc) · 1.59 KB
Raw

nifi-compose-bundle

An example NiFi Archive (.nar) with some custom Processors

Quick Build and Deploy

  1. git clone
  2. mvn package
  3. Copy nifi-compose-nar/target/nifi-compose-nar-1.0.0-BETA.nar to $NIFI_ROOT/lib assuming you took the defaults from the nifi.properties file and that you are running Apache Nifi version 1.0.0-Beta.

For details on these custom Processors please see the following blog (especially the end): What You Need to Know to Extend NiFi

For a general overview of Apache NiFi: Use NiFi to Lessen the Friction of Moving Data

Custom Processors in this bundle:

  • ComposeUniqueRocksDB provides the ability to locally check for uniqueness of an attribute. It is a persisted index of keys. Useful for local only unlike the de-duplication provided by NiFi via DetectDuplicate and its required Distributed Cache Service.

  • ComposeStreamingGetMongo provides immediate creation of FlowFiles for each Document prior to the query being finished. This is different than GetMongo which only calls session.commit() after the query is completed.

  • ComposeBatchPutMongo exposes Mongo's collection.insertMany(). It takes a FlowFile that has a JSON array of Documents to put into Mongo.

  • ComposeTailingGetMongo provides a snapshot of Mongo database plus a stream of oplog changes in the form of FlowFiles.

  • ComposeTailingPutMongo puts the dump from the TailingGet into another Mongo.