Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce instance mappings #132

Merged
merged 31 commits into from
Aug 9, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
8c08fed
introduce instance mappings
danyilq Jul 31, 2023
ead493b
Implement InstanceMappings and default and cloud implementations
farshidz Jul 31, 2023
97489cd
Remove index name from control url method
farshidz Jul 31, 2023
705082e
Update httprequests module to use InstanceMappings
farshidz Jul 31, 2023
24a06d1
Fix class name
farshidz Jul 31, 2023
e83f0fe
fix minimum version check and add cloud error
danyilq Aug 1, 2023
9fca70b
delete enrich and improve error message
danyilq Aug 1, 2023
661d440
fix existing and add new tests
danyilq Aug 1, 2023
8ee578f
Add docstring for InstanceMappings
farshidz Aug 2, 2023
c26885c
move version_check and refactor
danyilq Aug 2, 2023
a4d93e2
refactor tests to work with cloud
danyilq Aug 2, 2023
acc162e
add readiness check for index that we want to clean documents in
danyilq Aug 2, 2023
fb766fa
unsupported operation + change hardcoded url for refresh_urls
danyilq Aug 3, 2023
e7f514d
adapt tests behaviour for cloud and partially refactor code
danyilq Aug 3, 2023
7b2491c
post debug changes to tests and new run env in tox
danyilq Aug 6, 2023
c6f57cd
update logging for wait_for_index_status
danyilq Aug 7, 2023
76b7b8f
improve testing
danyilq Aug 7, 2023
e99c7af
use http instead of client
danyilq Aug 7, 2023
5940880
Merge branch 'mainline' into danyil/instance-mappings
danyilq Aug 7, 2023
6f47bc5
follow up on all of the comments
danyilq Aug 7, 2023
a24d93e
Merge remote-tracking branch 'origin/danyil/instance-mappings' into d…
danyilq Aug 7, 2023
4e4d91a
final changes
danyilq Aug 8, 2023
32fc98a
improve test
danyilq Aug 8, 2023
b56e53e
consistent hash
danyilq Aug 8, 2023
89c1fb8
consistent hash
danyilq Aug 8, 2023
0472dff
better namings
danyilq Aug 8, 2023
a711a95
bump pymarqo version
danyilq Aug 8, 2023
836b4b7
adapt to 32 characters index name limit
danyilq Aug 8, 2023
e7b381b
reduce test_demos.py index name length
danyilq Aug 8, 2023
53db2d3
raise error if index_name is too long
danyilq Aug 8, 2023
2be58c2
update version
danyilq Aug 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"tox"
],
name="marqo",
version="1.1.1",
version="1.1.2",
author="marqo org",
author_email="org@marqo.io",
description="Tensor search for humans",
Expand Down
36 changes: 24 additions & 12 deletions tests/marqo_test.py
danyilq marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -202,21 +202,28 @@ def warm_request(self, func, *args, **kwargs):

def create_cloud_index(self, index_name, settings_dict=None, **kwargs):
"""
Creates a cloud index with the given name and settings. If settings_dict or any kwargs are provided,
they will be used to create a unique index name by hashing the settings_dict and kwargs.
If settings_dict and index settings kwargs are not provided,
the index name will be the given index_name + the index_suffix.
Create a cloud index with the given name and settings.

Performs check for status of index and waits for index to be ready before returning.
If settings_dict or any kwargs are provided, a unique index name will be generated by hashing
the settings_dict or kwargs. If neither settings_dict nor index settings kwargs are provided,
the index name will be formed by combining the given index_name and the index_suffix.

Args:
index_name (str): Name of index to create
settings_dict (dict): Dictionary of settings to use when creating index,
same as settings_dict in create_index
**kwargs: Additional keyword arguments to pass to create_index e.g. model, treat_urls_and_pointers_as_image.
The index name must fit within 32 characters on the cloud and must not start with '-'.

Caveats:
- Please note that settings_dict override any kwargs during index creation
This function checks the status of the index and waits for the index to be ready before returning.

Args:
index_name (str): The name of the index to create.
settings_dict (dict, optional): Dictionary of settings to use when creating the index,
similar to settings_dict in create_index.
**kwargs: Additional keyword arguments to pass to create_index,
such as model, treat_urls_and_pointers_as_image.

Returns:
str: The name of the created index.

Caveats:
- Please note that settings_dict overrides any kwargs during index creation.
"""
danyilq marked this conversation as resolved.
Show resolved Hide resolved
client = marqo.Client(**self.client_settings)
index_name = f"{index_name}-{self.index_suffix}"
Expand All @@ -226,6 +233,11 @@ def create_cloud_index(self, index_name, settings_dict=None, **kwargs):
settings_dict.update({
"inference_type": "marqo.CPU", "storage_class": "marqo.basic"
})

if len(index_name) > 32:
index_name = index_name[len(index_name) - 32: len(index_name)]
if index_name[0] == "-":
index_name = index_name[1:]
pandu-k marked this conversation as resolved.
Show resolved Hide resolved
try:
status = client.http.get(f"indexes/{index_name}/status")["index_status"]
if status == "CREATING":
Expand Down
Loading