Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialization features bc #78

Merged
merged 13 commits into from
Jan 8, 2018
Merged

Serialization features bc #78

merged 13 commits into from
Jan 8, 2018

Conversation

saif-ellafi
Copy link
Contributor

@saif-ellafi saif-ellafi commented Jan 8, 2018

Description

Improves memory consumption. Creates a new Param like concept called Feature which stores model information in a more scalable way.

Motivation and Context

Out of driver RAM situations when dealing with pipelines containing large training corpus

How Has This Been Tested?

Benchmarks made, comparing broadcasting vs non broadcasting versions. Serialization with KRYO as opposed to 1.2.4. Memory consumption reduced by many factors.

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Code improvements with no or little impact
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING page.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

aleksei-ai and others added 13 commits December 26, 2017 14:39
- Extended features to POS, Spell Checker and Lemmatizer
- Dataset conversion improvements
# Conflicts:
#	src/main/scala/com/johnsnowlabs/nlp/annotators/ner/crf/NerCrfModel.scala
# Conflicts:
#	src/main/scala/com/johnsnowlabs/nlp/annotators/ner/crf/NerCrfModel.scala
# Conflicts:
#	src/main/scala/com/johnsnowlabs/nlp/annotators/ner/crf/NerCrfModel.scala
@saif-ellafi saif-ellafi merged commit 3fd55f8 into master Jan 8, 2018
@maziyarpanahi maziyarpanahi deleted the serialization_features_bc branch March 29, 2021 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants