Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Adding Milvus demo to examples #4910

Merged
merged 17 commits into from
Jan 30, 2025
Merged

feat: Adding Milvus demo to examples #4910

merged 17 commits into from
Jan 30, 2025
+1,281 −33

Conversation

franciscojavierarceo
Copy link
Member

@franciscojavierarceo franciscojavierarceo commented Jan 10, 2025 •
edited
Loading

What this PR does / why we need it:

This pull request introduces several updates and new features to the Feast repository, particularly around the integration of Milvus as an online store. Below is a summary of the key changes:

Documentation Updates

  • Updated Overview of Online Stores: Added Milvus to the matrix of supported functionalities in the overview.md file under docs/reference/online-stores/.

New Example Files

  • RAG Example: Introduced a new example for RAG (Retrieval-Augmented Generation) under examples/rag/.
    • Added __init__.py.
    • Added README.md.
    • Added feature_repo/ directory containing:
      • __init__.py
      • example_repo.py
      • feature_store.yaml
      • test_workflow.py
    • Added milvus-quickstart.ipynb notebook demo containing documentation for data scientists to easily get started

Code Updates

  • Milvus Online Store Integration:
    • Renamed _get_collection method to _get_or_create_collection in milvus.py.
    • Updated method calls to use _get_or_create_collection instead of _get_collection.
    • Added logic to handle vector fields and search metrics.
    • Improved handling of Proto values and serialization in retrieve_online_documents_v2 method.

Detailed Changes

  1. Documentation:

    • Added Milvus to the list of supported online stores in the feature matrix.

  2. Examples:

    • Added new example files under examples/rag/.

  3. Milvus Integration:

    • Method Renaming: Renamed _get_collection to _get_or_create_collection.
    • Vector Field Handling: Added handling for vector fields and search metrics within the _get_or_create_collection method.
    • Method Updates: Updated the following methods to use _get_or_create_collection:
      • online_write_batch
      • update
      • retrieve_online_documents_v2
    • Proto Value Handling: Improved the handling of Proto values and serialization in retrieve_online_documents_v2.

Which issue(s) this PR fixes:

#4364

Misc

N/A

@franciscojavierarceo franciscojavierarceo force-pushed the milvus-demo branch 2 times, most recently from 3731e17 to b7c3426 Compare January 10, 2025 21:59
@franciscojavierarceo franciscojavierarceo force-pushed the milvus-demo branch 2 times, most recently from ce41480 to 8360bbb Compare January 29, 2025 01:02
@franciscojavierarceo franciscojavierarceo marked this pull request as ready for review January 29, 2025 02:32
@franciscojavierarceo franciscojavierarceo requested a review from a team as a code owner January 29, 2025 02:32
dmartinol
dmartinol reviewed Jan 29, 2025
View reviewed changes
examples/rag/README.md Outdated
@@ -0,0 +1,29 @@
# Feast Quickstart
If you haven't already, check out the quickstart guide on Feast's website (http://docs.feast.dev/quickstart), which
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This README seems a generic quickstart guide with no specific connections with the RAG function or the included example.
I assume that it will be more clear once we run it and see what happens, but I believe than a dev experience driven by a notebook may help, WDYT?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call! I'll update to include a notebook! Will update this README as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know what you think!

@franciscojavierarceo
checking in progress but this Pr still is not ready yet
468368a 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
feat: Adding new method to FeatureStore to allow more flexible retrie…
8f26494 
…val of features from vector similarity search

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
Adding requested_features back into online_store
ddc963b 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
feat: Adding RAG demo displaying Milvus usage for RAG
006e897 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
uploading sample data and updated yaml
a1a9408 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updating workflow
4fba66c 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updated example
c4736b7 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
removing modified files
f7393a7 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
reverting postgres change
d0a1e30 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updating test_workflow
0d4c351 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updated and fixed bug
7c006b1 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
fixing a bad merge/rebase
63aa43b 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updated linter because of latest nistall
aa81350 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
reverting feature store change
e2a1949 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
adding logging of local milvus back
017ff54 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
Updating readme and adding notebook
ac96e36 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updated readme
c629a42 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo franciscojavierarceo requested review from a team January 30, 2025 14:54
@franciscojavierarceo franciscojavierarceo requested review from shuchu and HaoXuAI and removed request for a team January 30, 2025 14:54
shuchu
shuchu approved these changes Jan 30, 2025
View reviewed changes
Copy link
Collaborator

@shuchu shuchu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@franciscojavierarceo franciscojavierarceo merged commit 2daf852 into master Jan 30, 2025
26 checks passed
franciscojavierarceo pushed a commit that referenced this pull request Feb 4, 2025
@feast-ci-bot
chore(release): release 0.44.0
aa7224a 
# [0.44.0](v0.43.0...v0.44.0) (2025-02-04)

### Bug Fixes

* Adding periodic check to fix the sporadic failures of the operator e2e tests.  ([#4952](#4952)) ([1d086be](1d086be))
* Adding the feast-operator/bin to the .gitignore directory. Somehow it… ([#5005](#5005)) ([1a027ee](1a027ee))
* Changed Env Vars for e2e tests ([#4975](#4975)) ([fa0084f](fa0084f))
* Fix GitHub Actions to pass authentication ([#4963](#4963)) ([22b9138](22b9138)), closes [#4937](#4937) [#4939](#4939) [#4941](#4941) [#4940](#4940) [#4943](#4943) [#4944](#4944) [#4945](#4945) [#4946](#4946) [#4947](#4947) [#4948](#4948) [#4951](#4951) [#4954](#4954) [#4957](#4957) [#4958](#4958) [#4959](#4959) [#4960](#4960) [#4962](#4962)
* Fix showing selected navigation item in UI sidebar ([#4969](#4969)) ([8ac6a85](8ac6a85))
* Invalid column names in get_historical_features when there are field mappings on join keys ([#4886](#4886)) ([c9aca2d](c9aca2d))
* Read project data from the 'projects' key while loading the registry state in the Feast UI ([#4772](#4772)) ([cb81939](cb81939))
* Remove grpcurl dependency from Operator ([#4972](#4972)) ([439e0b9](439e0b9))
* Removed the dry-run flag to test and we will add it back later. ([#5007](#5007)) ([d112b52](d112b52))
* Render UI navigation items as links instead of buttons ([#4970](#4970)) ([1267703](1267703))
* Resolve Operator CRD bloat due to long field descriptions ([#4985](#4985)) ([7593bb3](7593bb3))
* Update manifest to add feature server image for odh ([#4973](#4973)) ([6a1c102](6a1c102))
* Updating release workflows to refer to yml instead of yaml ([#4935](#4935)) ([02b0a68](02b0a68))
* Use locally built feast-ui package in dev feature-server image ([#4998](#4998)) ([0145e55](0145e55))

### Features

* Added OWNERS file for OpenshiftCI ([#4991](#4991)) ([86a2ee8](86a2ee8))
* Adding Milvus demo to examples ([#4910](#4910)) ([2daf852](2daf852))
* Adding retrieve_online_documents endpoint ([#5002](#5002)) ([6607d3d](6607d3d))
* Adding support to return additional features from vector retrieval for Milvus db ([#4971](#4971)) ([6ce08d3](6ce08d3))
* Creating/updating the stable branch after the release. ([#5003](#5003)) ([e9b53cc](e9b53cc))
* Implementing online_read for MilvusOnlineStore ([#4996](#4996)) ([92dde13](92dde13))
* Improve exception message for unsupported Snowflake data types ([#4779](#4779)) ([5992364](5992364))
* Operator add feast ui deployment ([#4930](#4930)) ([b026d0c](b026d0c))
* Updating documents to highlight v2 api for Vector Similarity Se… ([#5000](#5000)) ([32b82a4](32b82a4))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dmartinol dmartinol dmartinol left review comments

@shuchu shuchu shuchu approved these changes

@lokeshrangineni lokeshrangineni Awaiting requested review from lokeshrangineni

@HaoXuAI HaoXuAI Awaiting requested review from HaoXuAI HaoXuAI was automatically assigned from feast-dev/maintainers

Assignees
No one assigned
Labels
ok-to-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@franciscojavierarceo @shuchu @dmartinol