Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shreyas/kg runtime cfg #913

Merged
merged 10 commits into from
Aug 21, 2024
Merged

Shreyas/kg runtime cfg #913

merged 10 commits into from
Aug 21, 2024

Conversation

shreyaspimpalgaonkar
Copy link
Contributor

@shreyaspimpalgaonkar shreyaspimpalgaonkar commented Aug 21, 2024 •
edited by ellipsis-dev bot
Loading

🚀 This description was created by Ellipsis for commit 85ba0cb

Summary:

Introduced KGEnrichmentSettings for configurable knowledge graph enrichment, updating CLI, core abstractions, providers, services, and tests to support this enhancement.

Key points:

  • Renamed client.restructure() to client.enrich_graph() in py/cli/commands/restructure.py.
  • Added KGEnrichmentSettings class in py/core/base/abstractions/restructure.py for configuring knowledge graph enrichment.
  • Updated KGConfig in py/core/base/providers/kg.py to include kg_enrichment_settings.
  • Modified neo4j_kg.toml to include kg_search_config settings.
  • Enhanced RestructureRouter in py/core/main/api/routes/restructure/base.py to accept KGEnrichmentSettings.
  • Updated RestructureService.enrich_graph in py/core/main/services/restructure_service.py to use KGEnrichmentSettings.
  • Refactored KGClusteringPipe and KGTriplesExtractionPipe to utilize KGEnrichmentSettings for clustering and extraction processes.
  • Adjusted R2RChunkingProvider to log initialization details.
  • Modified UnstructuredParsingProvider to handle API and local parsing differently.
  • Updated tests in py/tests/test_kg.py to reflect changes in enrichment and clustering logic.

Generated with ❤️ by ellipsis.dev

ellipsis-dev[bot]
ellipsis-dev bot reviewed Aug 21, 2024
View reviewed changes
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Changes requested. Reviewed everything up to 85ba0cb in 39 seconds

More details
  • Looked at 485 lines of code in 16 files
  • Skipped 0 files when reviewing.
  • Skipped posting 2 drafted comments based on config settings.
1. py/core/main/api/routes/restructure/base.py:1
  • Draft comment:
    The import statement for KGEnrichmentSettings is not used in this file. Consider removing it to clean up the code.
  • Reason this comment was not posted:
    Confidence changes required: 10%
    The import statement for KGEnrichmentSettings in py/core/main/api/routes/restructure/base.py is not used in the file. This is unnecessary and should be removed to keep the code clean.
2. py/sdk/restructure.py:9
  • Draft comment:
    The parameter name KGEnrichmentSettings shadows the class name and is misleading. Consider renaming it to follow Python naming conventions, such as enrichment_settings.
  • Reason this comment was not posted:
    Marked as duplicate.

Workflow ID: wflow_DL5Iwg8RCFtlFBPa

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

py/core/main/api/routes/restructure/base.py
@@ -12,6 +13,10 @@ def setup_routes(self):
@self.router.post("/enrich_graph")
@self.base_endpoint
async def enrich_graph(
KGEnrichmentSettings: Union[dict, KGEnrichmentSettings] = Body(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The variable name KGEnrichmentSettings shadows the class name and is misleading. Consider renaming it to follow Python naming conventions, such as enrichment_settings.

py/core/main/services/restructure_service.py
@@ -30,7 +31,9 @@ def __init__(
logging_connection,
)

async def enrich_graph(self) -> Dict[str, Any]:
async def enrich_graph(
self, enrich_graph_settings: KGEnrichmentSettings = KGEnrichmentSettings()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The enrich_graph_settings parameter is not used in this method. Consider using it or removing it if it's unnecessary.

py/sdk/models.py
@@ -88,6 +88,17 @@ class KGSearchSettings(BaseModel):
}


class KGEnrichmentSettings(BaseModel):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The KGEnrichmentSettings class is defined twice in this file. Consider removing one of the definitions to avoid redundancy.

py/sdk/models.py

class KGEnrichementResponse(BaseModel):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The class name KGEnrichementResponse contains a typo. Consider renaming it to KGEnrichmentResponse.

@shreyaspimpalgaonkar shreyaspimpalgaonkar merged commit 4ea5b59 into dev Aug 21, 2024
1 check failed
emrgnt-cmplxty added a commit that referenced this pull request Aug 21, 2024
@emrgnt-cmplxty @NolanTrem @shreyaspimpalgaonkar
Feature/merge with dev (#916)
c0db4ec 
* Fix CLI Tests (#912)

Fix CLI tests

* Shreyas/kg runtime cfg (#913)

add kg runtime config

* rename kgenrichmentresponse (#914)

* revert change to chunking by_title

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
emrgnt-cmplxty added a commit that referenced this pull request Aug 23, 2024
@emrgnt-cmplxty @NolanTrem @shreyaspimpalgaonkar
Feature/merge dev and main (#968)
7cd3743 
* Feature/merge graphrag group mgmt (#876)

* add group ids to document abstraction, first steps

* extend group permissions

* up

* add tests for new group features

* up

* fixup auth

* onboard extensive regression tests

* adding regression tests

* finish tests

* rm selenium

* test observability

* uncomment tests

* checkin first set of group tests

* modify search, passing vector tests

* checkin work

* full delete logic

* update search to use new filters

* check in

* Clean up

* Check in

* add search

* tests/test_end_to_end.py::test_ingest_txt_document passing

* cleanup logging

* make schemas explicit

* move to run logger abstraction

* cleanup some test workflows

* revive tests

* tweak to pass tests

* tweak rrf

* finish hybrid search cleanup

* fixup on regr tests, regen payloads

* refresh payloads

* refactor api model

* Feature/refactor api model (#868)

* cleanup imports

* flake and cleanup

* coherent global import / export structure

* add ingestion response models

* add management response models

* cleanups

* checkin work on routes

* remove request models

* last fixes

* merge

* add user / group gating

* working test groups

* updating client

---------

Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com>

* Clean up API (#878)

* Get running

* fixes in sdk

* Add in more fixes

* Feature/merge dev owen changes (#880)

* add group ids to document abstraction, first steps

* extend group permissions

* up

* add tests for new group features

* up

* fixup auth

* onboard extensive regression tests

* adding regression tests

* finish tests

* rm selenium

* test observability

* uncomment tests

* checkin first set of group tests

* modify search, passing vector tests

* checkin work

* full delete logic

* update search to use new filters

* check in

* Clean up

* Check in

* add search

* tests/test_end_to_end.py::test_ingest_txt_document passing

* cleanup logging

* make schemas explicit

* move to run logger abstraction

* cleanup some test workflows

* revive tests

* tweak to pass tests

* tweak rrf

* finish hybrid search cleanup

* fixup on regr tests, regen payloads

* refresh payloads

* refactor api model

* Feature/refactor api model (#868)

* cleanup imports

* flake and cleanup

* coherent global import / export structure

* add ingestion response models

* add management response models

* cleanups

* checkin work on routes

* remove request models

* last fixes

* merge

* add user / group gating

* working test groups

* updating client

* rename service to restructure

* add get documents for group endpoint

* fix client bugs

* return delete format

* merge cleanups

* merge

* finalize

---------

Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com>

* Shreyas/graphrag test (#881)

* add group ids to document abstraction, first steps

* extend group permissions

* up

* add tests for new group features

* up

* fixup auth

* onboard extensive regression tests

* adding regression tests

* finish tests

* rm selenium

* test observability

* uncomment tests

* checkin first set of group tests

* modify search, passing vector tests

* checkin work

* full delete logic

* update search to use new filters

* check in

* Clean up

* Check in

* add search

* tests/test_end_to_end.py::test_ingest_txt_document passing

* cleanup logging

* make schemas explicit

* move to run logger abstraction

* cleanup some test workflows

* revive tests

* tweak to pass tests

* tweak rrf

* finish hybrid search cleanup

* fixup on regr tests, regen payloads

* refresh payloads

* refactor api model

* Feature/refactor api model (#868)

* cleanup imports

* flake and cleanup

* coherent global import / export structure

* add ingestion response models

* add management response models

* cleanups

* checkin work on routes

* remove request models

* last fixes

* merge

* add user / group gating

* sync

* enrich

* up

* fix global search

* rag

* remove client.py

* rm configs

* rm configs

---------

Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>

* Feature/fix embedding pipe (#882)

* up

* fixup concurrency

* fix ollama embeddings

* fix batching with ollama

* checkin all cleanups

* rm kg cruft (#884)

* rm kg cruft

* tweaks

* tweak 2 (#885)

* Feature/fix retrieval endpoint cruft (#887)

* tweak 2

* fix retrieval endpoint descriptions

* Python SDK (#886)

Clean up Python SDK and routes

* Separate out SDK, add js and go sdk to monorepo (#888)

* Add r2r-js sdk

* Add go sdk

* Pull out python sdk

* remove venv

* Update packages

* Check in fixes

* Remove alembic dependencies

* Feature/merge w nolan (#894)

* cleanup hybrid search

* cleanups in

* Fix structure

* Make graspologic optional

* fix rag stream (#895)

* add py r2r (#896)

* Clean up (#897)

* fix agent (#898)

* define `RAGAgentResponse` (#899)

* Shreyas/unstructured (#900)

* api + oss lib

* rm pdb

* rm poetry lock

* update version

* fixes

* Feature/cleanup client obj logic (#901)

* define `RAGAgentResponse`

* cleanup client logic

* Shreyas/tests (#889)

* init

* tests

* rename service

* api model

* add

* merge

* rm restructure router

* print descriptions

* Refactor CLI (#903)

* Rm files readded by git (#904)

* Remove Execution Wrapper (#905)

* Rm files readded by git

* Fix merge botch

* Feature/fix auth revive tests rebased (#906)

* adding the client touch ups

* fix auth, revive tests

* add back tests

* uncomment run auth workflow

* decruft

* refresh test kg

* fixup toml (#908)

* Feature/fix ingestion update (#909)

* fixup toml

* fix update

* Fix CLI Tests (#912)

Fix CLI tests

* Shreyas/kg runtime cfg (#913)

add kg runtime config

* rename kgenrichmentresponse (#914)

* Feature/add nltk hybrid expansion rebased (#917)

* expand hybrid search with nltk

* cleanups

* cleanup hybrid search

* format

* add setup.py

* update

* add script (#918)

* Fix bug in document chunks (#921)

* Fix bug in update files (#923)

* Shreyas/unstructured (#922)

* fix dockerfiles

* adding config

* fix paths

* mv unstructured dep to docker

* clean

* Update docker_utils.py

* Update unstructured_parsing.py

* Update r2r_chunking.py

* Update app_entry.py

* Feature/repair logging (#925)

* fixing logs

* fix

* rm double logging (#929)

* Configs (#926)

* Fix config logic

* Update config

* Clean up cli entry point

* Disable SSL when installing nltk wordnet (#930)

* Fix analytics endpoint

* Update OpenAI sdk calls (#933)

* Feature/revive advanced rag (#932)

* rm double logging

* revive advanced rag examples

* merge (#934)

* sync model (#935)

* Feature/remove version from ingestion end pt (#936)

* sync model

* remove ability to set version

* tweak versions impl

* fix version bug

* Move docker (#938)

* Move docker

* remove from root

* Clean up sdk/restructure.py

* Fix js tests, completion scoring (#939)

* Shreyas/unstructured docker image (#940)

unstructured docker image

* Update JS (#941)

* Update models (#942)

* Feature/complete group logic (#945)

* fix group logic

* up

* Fix Dockerbuild, Symlink Readme (#944)

* Add back tast prompt override and include title if availible

* Fix docker, sym link readme

* Fix compose file path

* Shreyas/KG Search Result model (#937)

* return type to kg_search_result

* add model

* local and global results

* modify config

* refresh should not be gated by auth (#946)

* Linting sync (#947)

* Remove email from refresh (#948)

* Fix link to image

* Feature/rm print cruft rebase (#953)

* refresh should not be gated by auth

* rm print cruft

* black and sort

* merge

* rm

* update api return type

* Update Actions (#954)

* Update Github Actions (#956)

* Update Actions

* Update actions

* Shreyas/kgsearchresult model (#957)

* return type to kg_search_result

* add model

* local and global results

* modify config

* add models

* up

* fix config path

* fix models

* Login and refresh token bug (#959)

* Update Actions

* Fix bug in login with refresh token

* Point pytest to linux (#960)

* collection docs (#955)

* Feature/merge dev to main (#962)

* merge dev and main

* git rm

* add back collection fix

* fix docker builds (#963)

* Running unstructured docker + code cleanups (#964)

* Small bugfixes on prompts, return types (#965)

* Fix failing CLI tests

* NPM publish action

* remove tarball

* Feature/fix dev tests (#966)

* update auth tests

* fix tests

* back and sort

* decruft

* revert back to gpt-4o

---------

Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
@emrgnt-cmplxty emrgnt-cmplxty deleted the shreyas/kg-runtime-cfg branch August 26, 2024 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ellipsis-dev ellipsis-dev[bot] ellipsis-dev[bot] left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@shreyaspimpalgaonkar