Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

540 Release Candidate #14247

Merged
merged 37 commits into from
Jul 1, 2024
Merged

540 Release Candidate #14247

merged 37 commits into from
Jul 1, 2024

Commits on May 14, 2024

  1. closed the connection (#14233)

    @mehmetbutgul
    mehmetbutgul authored May 14, 2024
    Configuration menu
    Copy the full SHA
    deb0a45 View commit details
    Browse the repository at this point in the history

  2. Fix missing java distribution for setup-java step

    @DevinTDHa
    DevinTDHa committed May 14, 2024
    Configuration menu
    Copy the full SHA
    b3d491b View commit details
    Browse the repository at this point in the history

  3. Lock macos version for runner

    @DevinTDHa
    DevinTDHa committed May 14, 2024
    Configuration menu
    Copy the full SHA
    d8a42c0 View commit details
    Browse the repository at this point in the history

  4. Add missing sbt setup

    @DevinTDHa
    DevinTDHa committed May 14, 2024
    Configuration menu
    Copy the full SHA
    5967577 View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. Add openvino dependency (#14255)

    @DevinTDHa
    DevinTDHa authored May 21, 2024
    Configuration menu
    Copy the full SHA
    fcd4e9c View commit details
    Browse the repository at this point in the history

  2. [SPARKNLP-1037] Adding addFile changes to to replace broadcast in all… 

    … ONNX based annotators (#14236)

* [SPARKNLP-1011] Adding changes to transfer ONNX files on executors through Spark files feature

* [SPARKNLP-1011] Adding missing copyright comment

* [SPARKNLP-1011] Adding changes to add prefix for models with onnx_data file

* [SPARKNLP-1037] Adding changes to transfer ONNX files on executors via addFile

* [SPARKNLP-1037] Adding unique suffix to avoid duplication in spark files
    @danilojsl
    danilojsl authored May 21, 2024
    Configuration menu
    Copy the full SHA
    4419a70 View commit details
    Browse the repository at this point in the history

  3. Integrating OpenVINO Runtime in Spark NLP (#14200) 

    * Use OpenVINO model engine for BertEmbeddings

Add OpenVINO model engine wrapper

Add default buffer size for reading weights file

Read OpenVINO IR format models

* Use Long Tensors with XlmRoberta

* Add OpenVINO support for RoBerta and XlmRoBerta embeddings

* Fix data type and formatting

* Add OpenVINO BERT test

- Resole merge conflicts

- Add test for openvino load model

* Update Python APIs to use OpenVINO

* Add param to enable OpenVINO through Python API
* Formatting changes

* Add OpenVINO support for E5 Embeddings

* Enable OpenVINO backend for E5 Embeddings
* Update Python APIs

* Resolve merge issues

* Add OpenVINO support for T5

* Read and write encoder-decoder models with OpenVINO

* OpenVINO Async Inference

* Refactor and cleanup

* Update comments

* Add config to set OpenVINO inference device

* Add OpenVINO support for BERT Sentence Embeddings

* Formatting

* Openvino synchronous inference

* Refactoring: OV Model Conversion

* BertSentenceEmbeddings Python API

* Enable OpenVINO support for Llama2

* Read/write Llama2 Transformer with OpenVINO

* Bugfix: Update saved model filename

* Replace broadcast with addFile for OpenVINO-based annotators

* Add OpenVINO Wrapper tests

* Add suffix to avoid duplication in Spark Files

* Set default OV inference device to CPU

* Use CPU by default for OpenVINO inference due to error loading device config in cluster envs

* Bugfix: Read serialized model from folder
    @rajatkrishna
    rajatkrishna authored May 21, 2024
    Configuration menu
    Copy the full SHA
    fabc4ab View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. Fixing colab notebook bugs (#14249) 

    * adding code for spp files and updating notebooks

* fixed code and notebook for xlnet embeddings

* fixing more notebooks

* adding notebook for bert sentence embeddings
    @ahmedlone127
    ahmedlone127 authored May 24, 2024
    Configuration menu
    Copy the full SHA
    9430402 View commit details
    Browse the repository at this point in the history

  2. adding model hub cards + updating readme + small typo fix on M2M100Te… 

    …stSpec (#14253)
    @ahmedlone127
    ahmedlone127 authored May 24, 2024
    Configuration menu
    Copy the full SHA
    3f59375 View commit details
    Browse the repository at this point in the history

  3. adding padded tokens (#14276)

    @ahmedlone127
    ahmedlone127 authored May 24, 2024
    Configuration menu
    Copy the full SHA
    d083420 View commit details
    Browse the repository at this point in the history

  4. Sparknlp 1035 test all notebooks to import tensor flow models to spar… 

    …k nlp (#14238)

* Updating file version in XlmRobertaForZeroShotClassification

* updating  transformer versions
    @ahmedlone127
    ahmedlone127 authored May 24, 2024
    Configuration menu
    Copy the full SHA
    e0e28e8 View commit details
    Browse the repository at this point in the history

  5. Adding caching to streamlit demos (#14232) 

    * Delete examples/demos/streamlit/CLIPForZeroShotClassification/inputs directory

* Add files via upload

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Update streamlit_app.py

* Delete examples/demos/streamlit/run_demos.ipynb

* Add files via upload
    @AbdullahMubeenAnwar
    AbdullahMubeenAnwar authored May 24, 2024
    Configuration menu
    Copy the full SHA
    262b802 View commit details
    Browse the repository at this point in the history

  6. Disable OpenVINO FastTest

    @DevinTDHa
    DevinTDHa committed May 24, 2024
    Configuration menu
    Copy the full SHA
    74f5151 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. Add openvino GPU dependency (#14309)

    @DevinTDHa
    DevinTDHa authored Jun 3, 2024
    Configuration menu
    Copy the full SHA
    c2048be View commit details
    Browse the repository at this point in the history

  2. Fix incorrect LLAMA2 position ID (#14308)

    @rajatkrishna
    rajatkrishna authored Jun 3, 2024
    Configuration menu
    Copy the full SHA
    adc193e View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. bump version to 5.4.0-rc1 [skip test]

    @maziyarpanahi
    maziyarpanahi committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    7274281 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. Sparknlp 1016 implement mp net for token classification (#14322) 

    * implementing mpnet for tokenclassification

* changing default pretrained model name

---------

Co-authored-by: Maziyar Panahi <maziyar.panahi@iscpif.fr>
    @ahmedlone127 @maziyarpanahi
    ahmedlone127 and maziyarpanahi authored Jun 10, 2024
    Configuration menu
    Copy the full SHA
    9c075f8 View commit details
    Browse the repository at this point in the history

  2. Uploading OpenVINO example notebooks (#14313)

    @rajatkrishna
    rajatkrishna authored Jun 10, 2024
    Configuration menu
    Copy the full SHA
    0ea5898 View commit details
    Browse the repository at this point in the history

  3. SparkNLP - 995 Introducing MistralAI LLMs (#14318) 

    * added mistral

* Mistral python API
    @prabod
    prabod authored Jun 10, 2024
    Configuration menu
    Copy the full SHA
    4583ccf View commit details
    Browse the repository at this point in the history

  4. SparkNLP 1043 integrate new casual lm annotators to use open vino (#1… 

    …4319)

* Phi2 scala api

* Phi2 python api

* Phi2 python and scala tests

* Phi2 python and scala tests

* added M2M100 openvino implementation

* added phi2  openvino implementation

* added openvino flag to python

---------

Co-authored-by: Maziyar Panahi <maziyar.panahi@iscpif.fr>
    @prabod @maziyarpanahi
    prabod and maziyarpanahi authored Jun 10, 2024
    Configuration menu
    Copy the full SHA
    cdb031a View commit details
    Browse the repository at this point in the history

  5. Fixed LLAMA generation bug (#14320) 

    * fixed LLAMA generation bug

* update params
    @prabod
    prabod authored Jun 10, 2024
    Configuration menu
    Copy the full SHA
    3054d4c View commit details
    Browse the repository at this point in the history

  6. Fix compilation error

    @maziyarpanahi
    maziyarpanahi committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    b4000d3 View commit details
    Browse the repository at this point in the history

  7. Bump to 5.4.0-rc2

    @maziyarpanahi
    maziyarpanahi committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    85c90dd View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Add Pooling Average to Broken XXXForSentenceEmbedding annotators (#14328 

    )

* SPARKNLP-1036: Onnx Example notebooks (#14234)

* SPARKNLP-1036: Fix dev python kernel names

* SPARKNLP-1036: Bump transformers version

* SPARKNLP-1036: Fix Colab buttons

* SPARKNLP-1036: Pin onnx version for compatibility

* SPARKNLP-1036: Upgrade Spark version

* SPARKNLP-1036: Minor Fixes

* SPARKNLP-1036: Clean Metadata

* SPARKNLP-1036: Add/Adjust Documentation

- Note for supported Spark Version of Annotators
- added missing Documentation for BGEEmbeddings

* Fixies (#14307)

* adding fix for broken annotators

---------

Co-authored-by: Devin Ha <33089471+DevinTDHa@users.noreply.github.com>
Co-authored-by: Lev <agsfer@gmail.com>
Co-authored-by: Maziyar Panahi <maziyar.panahi@iscpif.fr>
    @ahmedlone127 @DevinTDHa
    @agsfer @maziyarpanahi
    4 people authored Jun 12, 2024
    Configuration menu
    Copy the full SHA
    1cba7e3 View commit details
    Browse the repository at this point in the history

  2. Fix compilation error and formatting

    @maziyarpanahi
    maziyarpanahi committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    903e780 View commit details
    Browse the repository at this point in the history

  3. revert changes to BERT backend

    @maziyarpanahi
    maziyarpanahi committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    54027a4 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. Fix models link on FAQ (#14333) 

    * SPARKNLP-1036: Onnx Example notebooks (#14234)

* SPARKNLP-1036: Fix dev python kernel names

* SPARKNLP-1036: Bump transformers version

* SPARKNLP-1036: Fix Colab buttons

* SPARKNLP-1036: Pin onnx version for compatibility

* SPARKNLP-1036: Upgrade Spark version

* SPARKNLP-1036: Minor Fixes

* SPARKNLP-1036: Clean Metadata

* SPARKNLP-1036: Add/Adjust Documentation

- Note for supported Spark Version of Annotators
- added missing Documentation for BGEEmbeddings

* Fixies (#14307)

* Fix models link

---------

Co-authored-by: Devin Ha <33089471+DevinTDHa@users.noreply.github.com>
Co-authored-by: Lev <agsfer@gmail.com>
    @dcecchini @DevinTDHa @agsfer
    3 people authored Jun 21, 2024
    Configuration menu
    Copy the full SHA
    4356794 View commit details
    Browse the repository at this point in the history

  2. adding onnx support and average pooling (#14330) 

    * adding onnx support and average pooling

* Delete examples/python/transformers/HuggingFace_in_Spark_NLP_RoBertaSentenceEmbeddings.ipynb.txt

* adding openvino
    @ahmedlone127
    ahmedlone127 authored Jun 21, 2024
    Configuration menu
    Copy the full SHA
    5a86b70 View commit details
    Browse the repository at this point in the history

  3. uploading UAEEmbeddings notebook (#14324) 

    * SPARKNLP-1036: Onnx Example notebooks (#14234)

* SPARKNLP-1036: Fix dev python kernel names

* SPARKNLP-1036: Bump transformers version

* SPARKNLP-1036: Fix Colab buttons

* SPARKNLP-1036: Pin onnx version for compatibility

* SPARKNLP-1036: Upgrade Spark version

* SPARKNLP-1036: Minor Fixes

* SPARKNLP-1036: Clean Metadata

* SPARKNLP-1036: Add/Adjust Documentation

- Note for supported Spark Version of Annotators
- added missing Documentation for BGEEmbeddings

* Fixies (#14307)

* uploading UAEEmbeddings notebook

Adding UAEEmbeddings example notebook

---------

Co-authored-by: Devin Ha <33089471+DevinTDHa@users.noreply.github.com>
Co-authored-by: Lev <agsfer@gmail.com>
Co-authored-by: Maziyar Panahi <maziyar.panahi@iscpif.fr>
    @AbdullahMubeenAnwar @DevinTDHa
    @agsfer @maziyarpanahi
    4 people authored Jun 21, 2024
    Configuration menu
    Copy the full SHA
    ac9de09 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. Bump version to 5.4.0 [skip test]

    @maziyarpanahi
    maziyarpanahi committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    e88682c View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. Refactor OpenAIEmbeddings (#14334) 

    * SPARKNLP-1036: Onnx Example notebooks (#14234)

* SPARKNLP-1036: Fix dev python kernel names

* SPARKNLP-1036: Bump transformers version

* SPARKNLP-1036: Fix Colab buttons

* SPARKNLP-1036: Pin onnx version for compatibility

* SPARKNLP-1036: Upgrade Spark version

* SPARKNLP-1036: Minor Fixes

* SPARKNLP-1036: Clean Metadata

* SPARKNLP-1036: Add/Adjust Documentation

- Note for supported Spark Version of Annotators
- added missing Documentation for BGEEmbeddings

* Fixies (#14307)

* refactor OpenAIEmbeddings in Scala

* refactor OpenAIEmbeddings in Python

* add pytest.mark.slow and improve doc

---------

Co-authored-by: Devin Ha <33089471+DevinTDHa@users.noreply.github.com>
Co-authored-by: Lev <agsfer@gmail.com>
Co-authored-by: Maziyar Panahi <maziyar.panahi@iscpif.fr>
    @mehmetbutgul @DevinTDHa
    @agsfer @maziyarpanahi
    4 people authored Jun 28, 2024
    Configuration menu
    Copy the full SHA
    09dc500 View commit details
    Browse the repository at this point in the history

  2. Update CHANGELOG [run doc]

    @maziyarpanahi
    maziyarpanahi committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    9d235e0 View commit details
    Browse the repository at this point in the history

  3. Update Scala and Python APIs

    @actions-user
    actions-user committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    036fc50 View commit details
    Browse the repository at this point in the history

  4. Update ORT and Azure deps

    @maziyarpanahi
    maziyarpanahi committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    86e6725 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2024

  1. add the missing OpenVINO coordinates

    @maziyarpanahi
    maziyarpanahi committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    a5b88ad View commit details
    Browse the repository at this point in the history

  2. set ORT to 1.18.0 

    - ORT 1.18.1 has not been released yet
    @maziyarpanahi
    maziyarpanahi committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    1502757 View commit details
    Browse the repository at this point in the history

  3. Update jsl-openvino to GA

    @maziyarpanahi
    maziyarpanahi committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    595b8f4 View commit details
    Browse the repository at this point in the history