refactor: refactor source.py #351

kzamlynska · 2025-02-11T13:33:13Z

extract sources to separate files

github-actions · 2025-02-11T13:35:23Z

Trivy scanning results.

github-actions · 2025-02-11T14:01:51Z

Code Coverage Summary

Filename                                                                                                        Stmts    Miss  Cover    Missing
------------------------------------------------------------------------------------------------------------  -------  ------  -------  --------------------------------------------
packages/__init__.py                                                                                                0       0  100.00%
packages/ragbits-cli/src/ragbits/cli/__init__.py                                                                   31       4  87.10%   73-74, 81-82
packages/ragbits-cli/src/ragbits/cli/_utils.py                                                                     23       4  82.61%   45, 62-64
packages/ragbits-cli/src/ragbits/cli/state.py                                                                      79       3  96.20%   50-51, 61
packages/ragbits-cli/tests/unit/test_state.py                                                                      72       2  97.22%   103-104
packages/ragbits-conversations/src/ragbits/conversations/__init__.py                                                0       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/__init__.py                                        0       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/compressors/__init__.py                            3       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/compressors/base.py                               10       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/compressors/llm.py                                29       1  96.55%   79
packages/ragbits-conversations/src/ragbits/conversations/history/stores/__init__.py                                 3       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/stores/base.py                                    17       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/stores/sql.py                                     66       1  98.48%   128
packages/ragbits-conversations/tests/unit/history/test_llm_compressor.py                                           64       0  100.00%
packages/ragbits-conversations/tests/unit/history/test_sql_store.py                                                48       2  95.83%   29-30
packages/ragbits-core/src/ragbits/core/__init__.py                                                                  6       2  66.67%   8-9
packages/ragbits-core/src/ragbits/core/cli.py                                                                       6       0  100.00%
packages/ragbits-core/src/ragbits/core/config.py                                                                   17       0  100.00%
packages/ragbits-core/src/ragbits/core/options.py                                                                  17       0  100.00%
packages/ragbits-core/src/ragbits/core/types.py                                                                     9       0  100.00%
packages/ragbits-core/src/ragbits/core/audit/__init__.py                                                           74       5  93.24%   42-45, 52-54
packages/ragbits-core/src/ragbits/core/audit/base.py                                                               44       1  97.73%   119
packages/ragbits-core/src/ragbits/core/audit/cli.py                                                                69       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/__init__.py                                                       4       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/base.py                                                          20       2  90.00%   56, 69
packages/ragbits-core/src/ragbits/core/embeddings/exceptions.py                                                    17       7  58.82%   7-8, 17, 26-27, 36, 45
packages/ragbits-core/src/ragbits/core/embeddings/litellm.py                                                       38      18  52.63%   78-112
packages/ragbits-core/src/ragbits/core/embeddings/noop.py                                                           8       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/__init__.py                                                             4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/base.py                                                                78       3  96.15%   67, 79, 256
packages/ragbits-core/src/ragbits/core/llms/exceptions.py                                                          17       7  58.82%   7-8, 17, 26-27, 36, 45
packages/ragbits-core/src/ragbits/core/llms/factory.py                                                             12       3  75.00%   30, 41, 51
packages/ragbits-core/src/ragbits/core/llms/litellm.py                                                             95      29  69.47%   94, 100-121, 165, 175, 204-229, 251-256, 267
packages/ragbits-core/src/ragbits/core/llms/local.py                                                               56      27  51.79%   9-12, 66-74, 86-87, 108-119, 140-156
packages/ragbits-core/src/ragbits/core/llms/mock.py                                                                29       3  89.66%   69-72
packages/ragbits-core/src/ragbits/core/metadata_stores/__init__.py                                                  3       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/base.py                                                     11       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/exceptions.py                                                4       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/in_memory.py                                                17       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/__init__.py                                                           2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/_cli.py                                                              44      21  52.27%   25-33, 47-49, 63-65, 73-75, 89-97
packages/ragbits-core/src/ragbits/core/prompt/base.py                                                              28       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/parsers.py                                                           35       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/prompt.py                                                           126       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/discovery/__init__.py                                                 2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/discovery/prompt_discovery.py                                        33       2  93.94%   54-55
packages/ragbits-core/src/ragbits/core/utils/__init__.py                                                            0       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/_pyproject.py                                                         38       1  97.37%   113
packages/ragbits-core/src/ragbits/core/utils/config_handling.py                                                    72       8  88.89%   16, 54-55, 62-63, 152-154
packages/ragbits-core/src/ragbits/core/utils/decorators.py                                                         29       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/dict_transformations.py                                               72       3  95.83%   24, 27, 108
packages/ragbits-core/src/ragbits/core/vector_stores/__init__.py                                                    3       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/_cli.py                                                       51       4  92.16%   63, 85, 91, 121
packages/ragbits-core/src/ragbits/core/vector_stores/base.py                                                       40       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/chroma.py                                                     59       1  98.31%   77
packages/ragbits-core/src/ragbits/core/vector_stores/in_memory.py                                                  38       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/pgvector.py                                                  124      12  90.32%   173-181, 215-217, 289
packages/ragbits-core/src/ragbits/core/vector_stores/qdrant.py                                                     62       1  98.39%   80
packages/ragbits-core/tests/cli/__init__.py                                                                         0       0  100.00%
packages/ragbits-core/tests/cli/test_cli_trace_handler.py                                                          48       3  93.75%   30, 43, 56
packages/ragbits-core/tests/cli/test_vector_store.py                                                              105       0  100.00%
packages/ragbits-core/tests/integration/vector_stores/test_vector_store.py                                         31       0  100.00%
packages/ragbits-core/tests/unit/__init__.py                                                                        0       0  100.00%
packages/ragbits-core/tests/unit/test_options.py                                                                   21       0  100.00%
packages/ragbits-core/tests/unit/audit/__init__.py                                                                  0       0  100.00%
packages/ragbits-core/tests/unit/audit/test_cli.py                                                                102       0  100.00%
packages/ragbits-core/tests/unit/audit/test_trace.py                                                               93       3  96.77%   14, 17, 20
packages/ragbits-core/tests/unit/embeddings/test_from_config.py                                                    14       0  100.00%
packages/ragbits-core/tests/unit/llms/__init__.py                                                                   0       0  100.00%
packages/ragbits-core/tests/unit/llms/test_base.py                                                                 98       0  100.00%
packages/ragbits-core/tests/unit/llms/test_from_config.py                                                          16       0  100.00%
packages/ragbits-core/tests/unit/llms/test_litellm.py                                                              70       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/__init__.py                                                           0       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/test_get_default_llm.py                                              12       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/__init__.py                                                        0       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/test_from_config.py                                               11       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/test_in_memory.py                                                 22       0  100.00%
packages/ragbits-core/tests/unit/prompts/__init__.py                                                                0       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_parsers.py                                                           65       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_prompt.py                                                           165       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/__init__.py                                                      0       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/prompt_classes_for_tests.py                                     30       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/test_prompt_discovery.py                                        18       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/__init__.py                       2       1  50.00%   3
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/__init__.py               3       2  33.33%   2-4
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt1.py          14       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt2.py          14       0  100.00%
packages/ragbits-core/tests/unit/utils/__init__.py                                                                  0       0  100.00%
packages/ragbits-core/tests/unit/utils/test_config_handling.py                                                     64       2  96.88%   27-28
packages/ragbits-core/tests/unit/utils/test_decorators.py                                                          26       2  92.31%   17, 39
packages/ragbits-core/tests/unit/utils/test_dict_transformations.py                                                69       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_find.py                                                      13       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_config.py                                                 9       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_instace.py                                               37       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/__init__.py                                                          0       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_chroma.py                                                      61       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_from_config.py                                                 42       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_in_memory.py                                                   84       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_pgvector.py                                                   173       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_qdrant.py                                                      40       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/__init__.py                                            2       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/_main.py                                             119       4  96.64%   266-273, 276
packages/ragbits-document-search/src/ragbits/document_search/cli.py                                                39       2  94.87%   85, 104
packages/ragbits-document-search/src/ragbits/document_search/documents/__init__.py                                  0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/document.py                                 62       2  96.77%   100, 147
packages/ragbits-document-search/src/ragbits/document_search/documents/element.py                                  78      12  84.62%   87, 167-174, 183-185
packages/ragbits-document-search/src/ragbits/document_search/documents/exceptions.py                               11       5  54.55%   7-8, 17, 26-27
packages/ragbits-document-search/src/ragbits/document_search/documents/sources/__init__.py                          5       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/sources/base.py                             64       4  93.75%   166-167, 170-171
packages/ragbits-document-search/src/ragbits/document_search/documents/sources/gcs.py                              58       1  98.28%   42
packages/ragbits-document-search/src/ragbits/document_search/documents/sources/hf.py                               54      12  77.78%   52-55, 59-60, 90, 97-98, 112-114
packages/ragbits-document-search/src/ragbits/document_search/documents/sources/local.py                            36       2  94.44%   38, 76
packages/ragbits-document-search/src/ragbits/document_search/ingestion/__init__.py                                  0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/document_processor.py                       33       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/__init__.py             5       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/base.py                25       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/batched.py             18       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/distributed.py         30       8  73.33%   8-9, 36, 64-71
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/sequential.py          13       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/__init__.py                        3       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/base.py                           19       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/dummy.py                          20       7  65.00%   33, 54-60
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/__init__.py           4       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/default.py           46       4  91.30%   98, 103-104, 137
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/images.py            50      19  62.00%   73-80, 87-99, 111, 124
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/pdf.py               19       6  68.42%   23, 35-43
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/utils.py             38      11  71.05%   71, 82-83, 98-101, 110, 121-123
packages/ragbits-document-search/src/ragbits/document_search/retrieval/__init__.py                                  0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/__init__.py                       6       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/base.py                           9       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/llm.py                           25       4  84.00%   47-50
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/multi.py                         27       4  85.19%   51-54
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/noop.py                           6       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/prompts.py                       26       2  92.31%   65, 87
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/__init__.py                        3       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/base.py                           17       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py                        18       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/noop.py                           11       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/reciprocal_ranked_fusion.py       22       2  90.91%   50, 60
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/rerankers_answerdotai.py          21       0  100.00%
packages/ragbits-document-search/tests/__init__.py                                                                  0       0  100.00%
packages/ragbits-document-search/tests/helpers.py                                                                   3       0  100.00%
packages/ragbits-document-search/tests/cli/test_ingest.py                                                          21       0  100.00%
packages/ragbits-document-search/tests/cli/test_search.py                                                          71       0  100.00%
packages/ragbits-document-search/tests/integration/__init__.py                                                      0       0  100.00%
packages/ragbits-document-search/tests/integration/test_rerankers.py                                               24       6  75.00%   20-42
packages/ragbits-document-search/tests/integration/test_sources.py                                                 24      10  58.33%   23-33, 41-46
packages/ragbits-document-search/tests/integration/test_unstructured.py                                            48      10  79.17%   52-58, 71-77
packages/ragbits-document-search/tests/unit/test_config.py                                                         57       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_processor.py                                             17       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_search.py                                               233       1  99.57%   463
packages/ragbits-document-search/tests/unit/test_documents.py                                                      13       0  100.00%
packages/ragbits-document-search/tests/unit/test_elements.py                                                       19       0  100.00%
packages/ragbits-document-search/tests/unit/test_local_file_source.py                                              13       0  100.00%
packages/ragbits-document-search/tests/unit/test_processing_strategies.py                                          25       0  100.00%
packages/ragbits-document-search/tests/unit/test_providers.py                                                      41       0  100.00%
packages/ragbits-document-search/tests/unit/test_rephrasers.py                                                     40       0  100.00%
packages/ragbits-document-search/tests/unit/test_rerankers.py                                                      81       1  98.77%   25
packages/ragbits-document-search/tests/unit/test_source_discriminator.py                                           36       0  100.00%
packages/ragbits-document-search/tests/unit/test_sources.py                                                        53       8  84.91%   17-18, 53-58
packages/ragbits-document-search/tests/unit/testprojects/project_with_instance_factory/__init__.py                  0       0  100.00%
packages/ragbits-document-search/tests/unit/testprojects/project_with_instance_factory/factories.py                20       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/__init__.py                                                      0       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/base.py                                                         15       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/openai_moderation.py                                            19       5  73.68%   29-33
packages/ragbits-guardrails/tests/unit/test_openai_moderation.py                                                   35       0  100.00%
TOTAL                                                                                                            5480     342  93.76%

Diff against main

Filename                                                                                      Stmts    Miss  Cover
------------------------------------------------------------------------------------------  -------  ------  --------
packages/ragbits-document-search/src/ragbits/document_search/documents/document.py               +1       0  +0.05%
packages/ragbits-document-search/src/ragbits/document_search/documents/sources/__init__.py       +5       0  +100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/sources/base.py          +64      +4  +93.75%
packages/ragbits-document-search/src/ragbits/document_search/documents/sources/gcs.py           +58      +1  +98.28%
packages/ragbits-document-search/src/ragbits/document_search/documents/sources/hf.py            +54     +12  +77.78%
packages/ragbits-document-search/src/ragbits/document_search/documents/sources/local.py         +36      +2  +94.44%
packages/ragbits-document-search/tests/integration/test_sources.py                               +1       0  +1.81%
packages/ragbits-document-search/tests/unit/test_source_discriminator.py                         +1       0  +100.00%
packages/ragbits-document-search/tests/unit/test_sources.py                                      +1       0  +0.29%
TOTAL                                                                                          +221     +19  +0.04%

Results for commit: 172d0c4

Minimum allowed coverage is 60%

♻️ This comment has been updated with latest results

mhordynski

imo we should create package ragbits.document_search.documents.sources and there create separate modules per source type.

Preferably to not change public API let's expose all the sources in init.py making sure that correct extras are in place.

kzamlynska · 2025-02-12T16:37:08Z

imo we should create package ragbits.document_search.documents.sources and there create separate modules per source type.

Preferably to not change public API let's expose all the sources in init.py making sure that correct extras are in place.

please check my changes

micpst

looks better, but please keep naming consistent with the rest of the lib. should be:

sources/base.py (we can also put source resolver here)
sources/gcs.py
sources/hf.py
sources/local.py

kzamlynska · 2025-02-14T13:28:37Z

looks better, but please keep naming consistent with the rest of the lib. should be:

sources/base.py (we can also put source resolver here)

sources/gcs.py

sources/hf.py

sources/local.py

I've changed the names of files and I've moved the source resolver to the base.py file

packages/ragbits-document-search/src/ragbits/document_search/documents/sources/base.py

docs/how-to/document_search/search_documents.md

# Conflicts: # packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py

examples/evaluation/document-search/basic/optimize.py

examples/evaluation/document-search/basic/evaluate.py

mhordynski

lgtm

kzamlynska added 5 commits February 11, 2025 10:56

extract sources from source.py

6b79a17

extract sources to separate files

4de61ea

remormat file

af09d10

reformat file

d7db180

reformat file

b93af52

unit test correction

0689dab

mhordynski requested changes Feb 12, 2025

View reviewed changes

kzamlynska added 2 commits February 12, 2025 17:31

extract sources to separate package

2e44244

corrct init

dd16ecf

micpst reviewed Feb 14, 2025

View reviewed changes

micpst linked an issue Feb 14, 2025 that may be closed by this pull request

rafactor: refactor of source.py #340

Closed

kzamlynska self-assigned this Feb 14, 2025

rename source files to fit the convention

c0e3bca

kzamlynska requested a review from mhordynski February 14, 2025 13:27

Merge branch 'main' into kz/350-refactor-source

fbd2e06

micpst reviewed Feb 14, 2025

View reviewed changes

packages/ragbits-document-search/src/ragbits/document_search/documents/sources/base.py Outdated Show resolved Hide resolved

packages/ragbits-document-search/src/ragbits/document_search/documents/sources/base.py Outdated Show resolved Hide resolved

remove not needed suppress error

7887306

micpst approved these changes Feb 14, 2025

View reviewed changes

ludwiktrammer reviewed Feb 18, 2025

View reviewed changes

docs/how-to/document_search/search_documents.md Outdated Show resolved Hide resolved

kzamlynska added 2 commits February 19, 2025 14:26

Merge branch 'main' into kz/350-refactor-source

ca770f9

# Conflicts: # packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py

solve conflict with main

0896b59

mhordynski reviewed Feb 19, 2025

View reviewed changes

examples/evaluation/document-search/basic/optimize.py Outdated Show resolved Hide resolved

mhordynski mentioned this pull request Feb 19, 2025

feat: add azure storage source #329

Merged

kzamlynska added 3 commits February 19, 2025 20:41

change imports to document.sources

d190146

correct formatting

f549f56

change source type in config

0f20684

mhordynski reviewed Feb 19, 2025

View reviewed changes

examples/evaluation/document-search/basic/evaluate.py Outdated Show resolved Hide resolved

change source type in config

172d0c4

mhordynski approved these changes Feb 20, 2025

View reviewed changes

kzamlynska merged commit d21d999 into main Feb 20, 2025
5 checks passed

kzamlynska deleted the kz/350-refactor-source branch February 20, 2025 09:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: refactor source.py #351

refactor: refactor source.py #351

kzamlynska commented Feb 11, 2025

github-actions bot commented Feb 11, 2025

github-actions bot commented Feb 11, 2025 •

edited

Loading

mhordynski left a comment

kzamlynska commented Feb 12, 2025

micpst left a comment •

edited

Loading

kzamlynska commented Feb 14, 2025

mhordynski left a comment

refactor: refactor source.py #351

refactor: refactor source.py #351

Conversation

kzamlynska commented Feb 11, 2025

github-actions bot commented Feb 11, 2025

github-actions bot commented Feb 11, 2025 • edited Loading

Code Coverage Summary

Diff against main

mhordynski left a comment

Choose a reason for hiding this comment

kzamlynska commented Feb 12, 2025

micpst left a comment • edited Loading

Choose a reason for hiding this comment

kzamlynska commented Feb 14, 2025

mhordynski left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 11, 2025 •

edited

Loading

micpst left a comment •

edited

Loading