Skip to content
This repository has been archived by the owner on Feb 7, 2018. It is now read-only.

Installing a Simple Data Pipe Connector

Jump to bottom
Patrick Titzler edited this page Mar 3, 2016 · 11 revisions

Introduction

The Simple Data Pipe example application includes connectors for a few cloud data sources. However, additional connectors, like SugarCRM, reddit, and Flight Stats (both coming soon) are available as add-ons. If you need a different connector, you can create your own! Read how to create a custom connector for a new data source.

Once you've found or created the connector you want, you must install it into your Simple Data Pipe app.

Provision services required by the connector

Out of the box, the Simple Data Pipe application only utilizes the Cloudant NoSQL database service in Bluemix. Additional services may be required by connectors. Refer to the connector's README for details on how to provision them, before installing the connector.

A connector will not be loaded if its prerequisite services are not bound to the Simple Data Pipe application.

Depending on how you like to work and your Simple Data Pipe environment, install your connector in one of 2 ways. You can :

Install with Bluemix DevOps web interface

To follow these steps, you first should have deployed Simple Data Pipe to Bluemix, using the Deploy to Bluemix button. When you do so, Bluemix makes the app code available in the IBM Bluemix DevOps Services' Git repository.

To install a connector remotely on Bluemix DevOps:

  1. Log into IBM Bluemix.

  2. Go to your dashboard.

  3. Locate your the Simple Data Pipe application and click to open it.

  4. Go into the IBM Bluemix DevOps Services Web IDE in one of 2 ways:

    • If present on the upper right of the app's Overview page, click the EDIT CODE button.
    • Or, log in to IBM Bluemix DevOps Services, then find and open the Simple Data Pipe project.

  5. Once in IBM Bluemix DevOps Services, click on EDIT CODE to enable live edit.

  6. Click on package.json.

  7. Edit the package.json to add the connector (or connectors) to the dependencies list.

Example 1: To add the sample connector:

"dependencies": {   
"simple-data-pipe-connector-template": "https://github.com/ibm-cds-labs/simple-data-pipe-connector-template.git",
   "async": "^1.2.1",
   ...
},

Example 2: To add the Reddit connector:

"dependencies": {   
   "simple-data-pipe-connector-reddit": "https://github.com/ibm-cds-labs/simple-data-pipe-connector-reddit.git",
   "async": "^1.2.1",
   ...
},

Example 3: To add both the Reddit and SugarCRM connectors:

"dependencies": {   
    "simple-data-pipe-connector-reddit": "https://github.com/ibm-cds-labs/simple-data-pipe-connector-reddit.git",
    "simple-data-pipe-connector-sugarcrm": "https://github.com/ibm-cds-labs/simple-data-pipe-connector-sugarcrm.git",
    "async": "^1.2.1",
    ...
},

Note: Be sure to end the line above with a comma and follow proper JSON syntax.

  1. Save the changes. From the Bluemix DevOps menu, choose File > Save.
  2. Click on the Deploy the App button (right arrow). deploy button

The application redeploys. Once deployed, Simple Data Pipe should now include the new connector.

Install a Connector Locally with Cloud Foundry

If you prefer working on your local machine, or your Simple Data Pipe deployment doesn't include a Git repo in IBM Bluemix DevOps Services you probably cloned the Simple Data Pipe repository locally using theCloud Foundry command line interface

To install a connector locally via Cloud Foundry:

  1. In Cloud Foundry, open a command line.

  2. Switch into Simple Data Pipe repository directory: cd simple-data-pipe.

  3. Install the connector with npm.

    For example, with a connector in Github:

    npm install https://github.com/ibm-cds-labs/simple-data-pipe-connector-reddit.git

    or with a local connector (for local development):

    npm link [connectorPath]/simple-data-pipe-connector-reddit

    where [connectorPath] is the path to the connector

    Note: Using npm link adds a symlink in the pipes/nodes_modules directory. This allows for updates to the connector to be immediately available to your Simple Data Pipe. Alternatively, you can use npm install to install a copy of the connector in Simple Data Pipe. But, when the connector is updated, you need to re-run npm install to pick up the changes.

  4. Edit the package.json to add the connector (or connectors) to the dependencies list. Refer to the connector's README file for connector specific details.

Example 1: To add the sample connector:

"dependencies": {   
"simple-data-pipe-connector-template": "https://github.com/ibm-cds-labs/simple-data-pipe-connector-template.git",
   "async": "^1.2.1",
   ...
},

Example 2: To add the Reddit connector:

"dependencies": {   
   "simple-data-pipe-connector-reddit": "https://github.com/ibm-cds-labs/simple-data-pipe-connector-reddit.git",
   "async": "^1.2.1",
   ...
},

Example 3: To add both the Reddit and SugarCRM connectors:

"dependencies": {   
    "simple-data-pipe-connector-reddit": "https://github.com/ibm-cds-labs/simple-data-pipe-connector-reddit.git",
    "simple-data-pipe-connector-sugarcrm": "https://github.com/ibm-cds-labs/simple-data-pipe-connector-sugarcrm.git",
    "async": "^1.2.1",
    ...
},

Note: Be sure to end the line above with a comma and follow proper JSON syntax.

  1. Save the changes.

  2. Log into Bluemix inserting your [org] and [space] values in this command:

    cf login -a https://api.ng.bluemix.net -o [org] -s [space]

  3. Push the Simple Data Pipe application:

    cf push -f manifest.yml

The updated Simple Data Pipe is now available in Bluemix with the new connector.

Troubleshooting

Connector does not install
  • Check npm install output for errors or warnings
  • Confirm connector url is correct
  • Confirm the connector's package.json includes the parameter pipes-connector-name
Connector not available in Simple Data Pipes UI

  • Inspect the Simple Data Pipe log file (e.g. by running cf logs simple-data-pipe --recent) for an error message indicating that the connector was not loaded because a prerequisite service was missing.

    ERROR: Connector reddit in module simple-data-pipe-connector-reddit requires service { service_name: 'tone_analyzer',
service_plan: 'beta', instance_name: 'tone analyzer',  vcap_env_service_alias: 'WATSON_TONE_ANALYZER' }
ERROR: Connector reddit version 0.1.1 was not loaded due to missing service dependencies.

  • Check browser for JavaScript errors

  • Check development environment console for errors when starting Simple Data Pipe

  • Confirm connector JavaScript file name is index.js

  • Confirm connector code is present under /pipes/node_modules

More

Easily add additional connectors as needed to the Simple Data Pipe using either of the above approaches. If you'll create a custom connector, remember that for Simple Data Pipe to recognize and use a connector properly, the connector must use the simple-data-pipe-sdk Connector API.

More resources:

Clone this wiki locally