Pipelines

A Pipeline is a sequence of adaptors, defined in the configuration file, which connect a source adaptor, any number of transformers and any number of sink adaptors to create a path for messages to flow down. The Pipeline is created in a JavaScript file. The transporter init command creates this file as pipeline.js by default. Here is a typical generated pipeline:

Source({name:"source", namespace:"test./.*/"}).save({name:"sink", namespace:"test./.*/"})

There are three JavaScript functions, Source(), save() and transform() which are used to create a pipeline. All take a JavaScript object with named parameters as their argument. The two named arguments which they all need are name and namespace.

The name value should refer to the name of the node as defined in the configuration file.

The namespace value is a combination of string, dot and expression. Each message is labelled with a namespace and each adaptor has different ways using that namespace.

For example, the MongoDB adaptor as a source uses the namespace to determine what database and collections it reads, so in the above default pipeline, "test./.*/" would be the test database and any collection matching the regular expression between the // - the .* would match every collection.

The namespace of a sink acts as a filter for messages. The namespace of an incoming message has to match the sink's namespace setting to be processed by the sink. Note that the transformers further up the pipeline can modify a message's namespace allowing the pipeline to deliver messages to different sinks from one source.

Other named values in the parameters are overlaid on the configuration settings of the named node, allowing for ad hoc changes to be applied without altering the configuration file.