NIFI-14336: Creating processor to list box folder contents

[ncover21]

Summary

NIFI-14336

A processor responsible for listing folder items for a Box Folder.

• different from ListBoxFIle because it is able to take in a flowFile containing a folder id in an attribute and use that instead of providing a static one. No state is maintained either

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using mvn clean install -P contrib-check
  • JDK 21

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[pvillard31]

Given the way this processor is likely going to be used, I think we should be prescriptive here and avoid generating one flow file per listed file in the configured folder. Instead I would generate a single FlowFile with JSON content with an array of records, each record containing the metadata information of the listed files.

Assuming a folder with 50k files, that will avoid generating 50k flowfiles in one execution of the processor. By generating one single flowfile, a user could then use a first SliptRecord processor configured with 1000 records split, then a second SplitRecord with 1 record split, and finally a ForkRecord with path(s) for the fields that should be moved into flowfile attributes. This way the backpressure would do its job.

Thoughts?

[ncover21]

Given the way this processor is likely going to be used, I think we should be prescriptive here and avoid generating one flow file per listed file in the configured folder. Instead I would generate a single FlowFile with JSON content with an array of records, each record containing the metadata information of the listed files.

Assuming a folder with 50k files, that will avoid generating 50k flowfiles in one execution of the processor. By generating one single flowfile, a user could then use a first SliptRecord processor configured with 1000 records split, then a second SplitRecord with 1 record split, and finally a ForkRecord with path(s) for the fields that should be moved into flowfile attributes. This way the backpressure would do its job.

Thoughts?

Thanks for the review, yes I think that would make more sense for folders with large amounts of files in them. I've adjusted the logic to add in a writer and write the contents to a record instead. I've also added a batch based writing system in case of large numbers of files.