Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tables.automl.dataset_test: test_dataset_update failed #4397

Closed
flaky-bot bot opened this issue Jul 30, 2020 · 8 comments
Closed

tables.automl.dataset_test: test_dataset_update failed #4397

flaky-bot bot opened this issue Jul 30, 2020 · 8 comments
Assignees
Labels
priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@flaky-bot
Copy link

flaky-bot bot commented Jul 30, 2020

Note: #2941 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.


commit: 4a25177
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.FAILED_PRECONDITION
	details = "The values in weight column(4370278344944517120) must be less than 10,000."
	debug_error_string = "{"created":"@1596102474.724355100","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"The values in weight column(4370278344944517120) must be less than 10,000.","grpc_status":9}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/tables/automl/dataset_test.py", line 86, in test_dataset_update
dataset = ensure_dataset_ready()
File "/workspace/tables/automl/dataset_test.py", line 61, in ensure_dataset_ready
target_column_spec_name="Deposit",
File "/workspace/tables/automl/automl_tables_dataset.py", line 204, in update_dataset
column_spec_display_name=target_column_spec_name,
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/automl_v1beta1/tables/tables_client.py", line 1431, in set_target_column
return self.auto_ml_client.update_dataset(request, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/automl_v1beta1/gapic/auto_ml_client.py", line 445, in update_dataset
request, retry=retry, timeout=timeout, metadata=metadata
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 145, in call
return wrapped_func(*args, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "", line 3, in raise_from
google.api_core.exceptions.FailedPrecondition: 400 The values in weight column(4370278344944517120) must be less than 10,000.

@flaky-bot flaky-bot bot added buildcop: issue priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Jul 30, 2020
@flaky-bot
Copy link
Author

flaky-bot bot commented Jul 31, 2020

commit: f313fd4
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.FAILED_PRECONDITION
	details = "The values in weight column(4370278344944517120) must be less than 10,000."
	debug_error_string = "{"created":"@1596189113.676845221","description":"Error received from peer ipv4:74.125.197.95:443","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"The values in weight column(4370278344944517120) must be less than 10,000.","grpc_status":9}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/tables/automl/dataset_test.py", line 86, in test_dataset_update
dataset = ensure_dataset_ready()
File "/workspace/tables/automl/dataset_test.py", line 61, in ensure_dataset_ready
target_column_spec_name="Deposit",
File "/workspace/tables/automl/automl_tables_dataset.py", line 204, in update_dataset
column_spec_display_name=target_column_spec_name,
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/automl_v1beta1/tables/tables_client.py", line 1431, in set_target_column
return self.auto_ml_client.update_dataset(request, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/automl_v1beta1/gapic/auto_ml_client.py", line 445, in update_dataset
request, retry=retry, timeout=timeout, metadata=metadata
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 145, in call
return wrapped_func(*args, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "", line 3, in raise_from
google.api_core.exceptions.FailedPrecondition: 400 The values in weight column(4370278344944517120) must be less than 10,000.

@leahecole
Copy link
Collaborator

Hey! The error it's throwing has something to do with the weight - is this related to the API, the test, or the project config?

@tmatsuo
Copy link
Contributor

tmatsuo commented Jul 31, 2020

The service side changed the behavior a while ago, and it changed again

@sirtorry
Copy link
Contributor

yes, takashi fixed it earlier this month. i'm wondering if they reverted that.

@flaky-bot
Copy link
Author

flaky-bot bot commented Aug 1, 2020

commit: 32f96e4
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.FAILED_PRECONDITION
	details = "The values in weight column(4370278344944517120) must be less than 10,000."
	debug_error_string = "{"created":"@1596275534.091588234","description":"Error received from peer ipv4:74.125.195.95:443","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"The values in weight column(4370278344944517120) must be less than 10,000.","grpc_status":9}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/tables/automl/dataset_test.py", line 86, in test_dataset_update
dataset = ensure_dataset_ready()
File "/workspace/tables/automl/dataset_test.py", line 61, in ensure_dataset_ready
target_column_spec_name="Deposit",
File "/workspace/tables/automl/automl_tables_dataset.py", line 204, in update_dataset
column_spec_display_name=target_column_spec_name,
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/automl_v1beta1/tables/tables_client.py", line 1431, in set_target_column
return self.auto_ml_client.update_dataset(request, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/automl_v1beta1/gapic/auto_ml_client.py", line 445, in update_dataset
request, retry=retry, timeout=timeout, metadata=metadata
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 145, in call
return wrapped_func(*args, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "", line 3, in raise_from
google.api_core.exceptions.FailedPrecondition: 400 The values in weight column(4370278344944517120) must be less than 10,000.

@flaky-bot
Copy link
Author

flaky-bot bot commented Aug 2, 2020

commit: bae7b5a
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.FAILED_PRECONDITION
	details = "The values in weight column(4370278344944517120) must be less than 10,000."
	debug_error_string = "{"created":"@1596362331.449204503","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"The values in weight column(4370278344944517120) must be less than 10,000.","grpc_status":9}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/tables/automl/dataset_test.py", line 86, in test_dataset_update
dataset = ensure_dataset_ready()
File "/workspace/tables/automl/dataset_test.py", line 61, in ensure_dataset_ready
target_column_spec_name="Deposit",
File "/workspace/tables/automl/automl_tables_dataset.py", line 204, in update_dataset
column_spec_display_name=target_column_spec_name,
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/automl_v1beta1/tables/tables_client.py", line 1431, in set_target_column
return self.auto_ml_client.update_dataset(request, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/automl_v1beta1/gapic/auto_ml_client.py", line 445, in update_dataset
request, retry=retry, timeout=timeout, metadata=metadata
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 145, in call
return wrapped_func(*args, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/workspace/tables/automl/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "", line 3, in raise_from
google.api_core.exceptions.FailedPrecondition: 400 The values in weight column(4370278344944517120) must be less than 10,000.

@tmatsuo
Copy link
Contributor

tmatsuo commented Aug 2, 2020

I updated the csv with the following python script:

from tempfile import NamedTemporaryFile
import shutil
import csv

filename = 'bank-marketing.csv'
tempfile = NamedTemporaryFile(mode='w', delete=False)

fields = ['Age','Job','MaritalStatus','Education','Default','Balance','Housing','Loan','Contact','Day','Month','Duration','Campaign','PDays','Previous','POutcome','Deposit']

with open(filename, 'r') as csvfile, tempfile:
    reader = csv.DictReader(csvfile, fieldnames=fields)
    writer = csv.DictWriter(tempfile, fieldnames=fields)
    for row in reader:
        if row['Age'] != 'Age' and int(row['Balance']) < 10000:
            print('updating row')
            row['Balance'] = int(row['Balance']) + 10000
        writer.writerow(row)

shutil.move(tempfile.name, filename)

I hope that will fix the test.

@tmatsuo tmatsuo closed this as completed Aug 2, 2020
@tmatsuo
Copy link
Contributor

tmatsuo commented Aug 3, 2020

Oops, I misread the log:

google.api_core.exceptions.FailedPrecondition: 400 The values in weight column(4370278344944517120) must be less than 10,000.

I somehow made sure those values are guaranteed to be more than 10000.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

4 participants