Stedi scheduled SFTP poller demo

This repo contains a demo for polling SFTP servers for files to process on a scheduled basis, via a GitHub workflow. When files are found during the polling operation, they are copied to a Stedi bucket and then deleted from the SFTP server. To automate the processing of these files, consider taking a look at the Stedi read EDI demo!

The SFTP polling is orchestrated via a Stedi function called sftp-external-poller, which is written in TypeScript. The function is invoked automatically according ot the scheduled defined in the GitHub workflow.

On each scheduled invocation, the sftp-external-poller performs several steps:

1. Calls Stash to retrieve a list of configured trading partners.

2. Filters the list of trading partners to only include trading partners with external SFTP polling configuration details.

3. For each trading partner in the filtered list, connect to the trading partner's SFTP server using the corresponding connection configuration.

4. Looks for files to process on the trading partner's SFTP server

5. For each file found, copies the file to the path associated with the trading partner in a Bucket.

6. Deletes the file from the trading partner's SFTP server.

7. After processing all files that were found on the trading partner's SFTP server, closes the connection.

Trading partner profiles

The SFP poller relies on trading partner profile data that is stored in Stash. The trading partner profiles are created based on a JSON configuration file (which you will need to create during setup). An example configuration file containing basic properties for a single trading partner is shown below:

{
  "items": [
    {
      "key": "ANOTHERMERCH",
      "value": {
        "name": "Another Merchant",
        "externalSftpConfig": {
          "hostname": "sftp.anothermerchant.com",
          "username": "sftpuser1",
          "password": "not-a-real-password"
        }
      }
    }
  ]
}

Trading partner profile configuration schema

The trading partner configuration file should conform to the TypeScript type shown below:

export interface TradingPartnerList {
  items: Item[];
}

export interface Item {
  key:   string;  // key used to fetch a specific trading partner config (my trading partner's ID)
  value: TradingPartnerConfig;
}

export interface TradingPartnerConfig {
  name:                string;
  myPartnershipId?:    string;            // optional identifier used by my trading partner to identify my messages
  externalSftpConfig?: ExternalSFTPConfig; // optional SFTP connection details when using my trading partner's SFTP
  resourceIds?:        ResourceIds[];      // optional IDs for Stedi guides and mappings specific to this trading partner
  additionalConfig?:   any;                // optional freeform attribute to hold any additional config required
}

export interface ExternalSFTPConfig {
  hostname:      string;
  username:      string;
  password:      string;
  port?:         number; // optional SFTP connection port (default is 22)
  inboundPath?:  string; // optional path used for reading inbound documents
  outboundPath?: string; // optional path used for writing outbound documents
}

export interface ResourceIds {
  key:          string; // key used to identify resource IDs specifc to this customer (for example, "x12-5010-855")
  value: {
     guideId?:   string;
     mappingId?: string;
  }
}

There is an example configuration file that includes trading partners with a variety of configurations that can be used as a starting point to create a configuration file for your trading partners.

Prerequisites

1. Node.js (npm version must be 7.x or greater)

2. Fork this repo to your account or organization and clone the forked repo (note: the clone command below needs to be updated to match your location of your forked repo):

   git clone https://github.com/<YOUR-USER-OR-ORG>/scheduled-sftp-poller.git

3. Install the necessary dependencies:

   cd scheduled-sftp-poller
   npm ci

4. This project uses dotenv to manage the environmental variables required. You must create a .env file in the root directory of this repo and add one environment variable:

   • STEDI_API_KEY: Your Stedi API Key - used to deploy the function and internally to interact with product APIs. If you don't already have one, you can generate an API Key here.

   Example .env file:

   STEDI_API_KEY=<REPLACE_ME>
   

5. Create a trading partner configuration file named tradingPartnerList.json in the tradingPartners resource directory. There is an example file that can be copied and renamed to tradingPartnerList.json and updated with details for your trading partners. Note: this file is intentionally excluded from git via the .gitignore file for the repo to avoid SFTP credentials from being stored in source control. The trading partner profile configuration must conform to the schema described above in the Trading partner profiles overview.

6. Configure your trading partners by running:

   npm run configure-parners

   This will create an empty trading-partner-configs Stash keyspace, and store the trading partner configuration data specified in the tradingPartnerList.json file that you created above:

   trading-partner-configs keyspace status: CREATING
   waiting for trading-partner-configs keyspace to become ACTIVE
   trading-partner-configs keyspace created successfully
   processing trading partner: ANOTHERMERCH (Another Merchant)
   processing trading partner: 111222333444 (Yet Another Merchant)
   processing trading partner: TP merchant ID (Seriously Another Merchant)
    
   Done.
   Populated configuration details for 3 trading partners

7. Configure the Buckets (one for storing the downloaded files and one for tracking function executions):

   npm run configure-buckets

   For each bucket, an environment variable entry will automatically be added to the .env file. The output of the script will include a list of the environment variables that have been added:

   Updated .env file with 2 bucket entries:
   
   SFTP_BUCKET_NAME=4c22f54a-9ecf-41c8-b404-6a1f20674953-sftp
   EXECUTIONS_BUCKET_NAME=4c22f54a-9ecf-41c8-b404-6a1f20674953-executions

Setup & Deployment

This repo includes a basic deployment script to bundle and deploy the sftp-external-poller function to Stedi. To deploy you must complete the following steps:

1. Confirm that your .env file contains the necessary environment variables:

   • STEDI_API_KEY
   • SFTP_BUCKET_NAME
   • EXECUTIONS_BUCKET_NAME

   It should look something like the following:

   STEDI_API_KEY=<YOUR_STEDI_API_KEY>
   SFTP_BUCKET_NAME=4c22f54a-9ecf-41c8-b404-6a1f20674953-sftp
   EXECUTIONS_BUCKET_NAME=4c22f54a-9ecf-41c8-b404-6a1f20674953-executions
   

2. To deploy the function:

   npm run deploy

   This should produce the following output:

   > stedi-sftp-poller@1.0.0 deploy
   > ts-node ./src/setup/deploy.ts
   
   Deploying sftp-external-poller
   Done sftp-external-poller
   Deploy completed at: 11/2/2022, 02:34:44 PM
   

Invoking the function

Invoking manually

Once deployed, you may invoke the function via the command line to verify functionality by running:

npm run invoke-sftp-poller

This will invoke the deployed sftp-external-poller Stedi function and poll the SFTP servers for any trading partners for which you specified externalSftpConfig details in your tradingPartnerList.json configuration file to look for new contents. The output of the script will include a summary of the polling operations:

> stedi-sftp-poller@1.0.0 invoke-sftp-poller
> ts-node-esm ./src/scripts/invokeSftpPoller.ts

Done.
Summary:
    polled 1 external SFTP server
    processed 0 files with 0 processing errors encountered
[
  {
    "name": "Another Merchant",
    "filteredItems": [],
    "processingErrors": [],
    "processedFiles": []
  }
]

Scheduled invocation

Configure your Stedi API Key

The function is configured to be invoked automatically via the scheduler GitHub action. In order for the workflow to be able to successfully invoke the sftp-external-poller Stedi function, the workflow needs to provide your STEDI_API_KEY, as an environment variable to the script that invokes the function. Create a new repository secret in your forked repo named STEDI_API_KEY and save the value of your API key as the secret value.

Change invocation schedule

To change the schedule for invoking the SFTP poller, you can modify the cron attribute of the schedule in accordance with the GitHub documentation for workflow schedules. After making changes to the workflow definition, be sure to commit the changes and push them to your forked repo.