Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The test of reading the data from the blob storage located in Azurite container failed #1415

Closed
Anna050689 opened this issue May 31, 2023 · 3 comments
Labels
bug Something isn't working

Comments

@Anna050689
Copy link

Environment

Delta-rs version: 0.9.0

Binding:

Environment:

  • Cloud provider: Azurite
  • OS: Windows

Bug

What happened:
The unit test failed during the process of reading the data from the blob storage located in the Azurite container.

from deltalake import DeltaLake

AZURE_STORAGE_CONNECTION_STRING = "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;" \
                                  "AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsu" \
                                  "Fq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;" \
                                  "BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1"

def test_load_data_from_table_in_delta_format_located_in_azure_storage_with_valid_connection_string(rp_logger):
    rp_logger.info("Loading data from the table in Delta format located in Azure Storage "
                   "with valid 'AZURE_STORAGE_CONNECTION_STRING'")
    path = "azure://test-container/delta_tables/table_with_data"
    table = DeltaTable(
        path,
        storage_options={
            "AZURE_STORAGE_CONNECTION_STRING": AZURE_STORAGE_CONNECTION_STRING,
            "AZURE_STORAGE_ACCOUNT_NAME": "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsu",
            "AZURE_STORAGE_ACCOUNT_KEY": "devstoreaccount1",
            "allow_http": "true"}
    )
    df = table.to_pandas()

    assert assert_frame_equal(
        df, pd.DataFrame(
            {
                "gender": [0, 1, 0, 1],
                "height": [157.18518021548246, 166.7731072622863, 162.91821942384928, 173.51448996432848],
                "id": [925, 84, 821, 383]
            }
        )
    ) is None

    assert isinstance(df, pd.DataFrame)
    rp_logger.info(SUCCESSFUL_MESSAGE)

There is the following error:

deltalake.PyDeltaTableError: Failed to load checkpoint: Failed to read checkpoint content: Generic MicrosoftAzure error: Error performing get request delta_tables/table_with_
data/_delta_log/_last_checkpoint: response error "request error", after 0 retries: error sending request for url (https://eby8vdm02xnocqflquwjpllmetlcdxj1ouzft50usrz6ifsu.blob.core.w
indows.net/test-container/delta_tables/table_with_data/_delta_log/_last_checkpoint): error trying to connect: dns error: No such host is known. (os error 11001)
@Anna050689 Anna050689 added the bug Something isn't working label May 31, 2023
@roeap
Copy link
Collaborator

roeap commented May 31, 2023

account name and account key are switched :). for azurite you do have to pass the option "use_emulator": "true" though, since we have not implemented connection string parsing, and the url handling for azurite needs to be a bit different.

@Anna050689
Copy link
Author

Thank you a lot for your help) It works

@roeap
Copy link
Collaborator

roeap commented May 31, 2023

Great, I'll close the issue then :).

@roeap roeap closed this as completed May 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants