SPARKNLP-823 Adding streaming functionality for seq2seq components

[danilojsl]

Description

This change adds a streamer argument to LightPipeline that will allow streaming data from both Scala and Python when using seq2seq features.

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Add a familiar functionality for users in Seq2Seq components

• Local Tests
• Google Colab notebook

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Code improvements with no or little impact
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING page.
• I have added tests to cover my changes.
• All new and existing tests passed.

[maziyarpanahi]

Thanks @danilojsl for making this possible. Just to make this similar to all the other streamers, can you please:

• Ignore/skip streaming for .fullAnnotate() 0 no need to stream Annotation
• Only support .annotate()
• Only support string input and not a list in .annotate()
• Since it's a string and not a list, no need to out put the list and name of the column

This

generation: [' My name is Leonardo. I am a man of letters. I have been a man for many years. I was born in the year 1776. I came to the United States in 1776, and I have lived in the United Kingdom since 1776']

can turn into

My name is Leonardo. I am a man of letters. I have been a man for many years. I was born in the year 1776. I came to the United States in 1776, and I have lived in the United Kingdom since 1776

This way it is similar to all the other streamers out there where they output only text (token after token)

[danilojsl]

Thanks @danilojsl for making this possible. Just to make this similar to all the other streamers, can you please:

• Ignore/skip streaming for .fullAnnotate() 0 no need to stream Annotation
• Only support .annotate()
• Only support string input and not a list in .annotate()
• Since it's a string and not a list, no need to out put the list and name of the column

This

generation: [' My name is Leonardo. I am a man of letters. I have been a man for many years. I was born in the year 1776. I came to the United States in 1776, and I have lived in the United Kingdom since 1776']

can turn into

My name is Leonardo. I am a man of letters. I have been a man for many years. I was born in the year 1776. I came to the United States in 1776, and I have lived in the United Kingdom since 1776

This way it is similar to all the other streamers out there where they output only text (token after token)

Done