Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests.system.aiplatform.test_model_monitoring.TestModelDeploymentMonitoring: test_mdm_two_models_invalid_configs_xai failed #1551

Closed
flaky-bot bot opened this issue Jul 29, 2022 · 3 comments
Assignees
Labels
api: vertex-ai Issues related to the googleapis/python-aiplatform API. flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@flaky-bot
Copy link

flaky-bot bot commented Jul 29, 2022

This test failed!

To configure my behavior, see the Flaky Bot documentation.

If I'm commenting on this issue too often, add the flakybot: quiet label and
I will stop commenting.


commit: 152563b
buildURL: Build Status, Sponge
status: failed

Test output
self = 
shared_state = {'resources': [ 
resource name: projects/580378083368/l...odel object at 0x7f112b3e2df0> 
resource name: projects/580378083368/locations/us-central1/models/4054308389935644672]}
def test_mdm_two_models_invalid_configs_xai(self, shared_state):
  temp_endpoint_with_two_models = self.temp_endpoint_with_two_models(shared_state)

tests/system/aiplatform/test_model_monitoring.py:365:


tests/system/aiplatform/test_model_monitoring.py:175: in temp_endpoint_with_two_models
endpoint.deploy(
google/cloud/aiplatform/models.py:815: in deploy
self._deploy(
google/cloud/aiplatform/base.py:793: in wrapper
return method(*args, **kwargs)
google/cloud/aiplatform/models.py:942: in _deploy
self._deploy_call(
google/cloud/aiplatform/models.py:1233: in _deploy_call
operation_future.result()


self = <google.api_core.operation.Operation object at 0x7f1128e3baf0>
timeout = None, retry = <google.api_core.retry.Retry object at 0x7f1144f0c8e0>

def result(self, timeout=None, retry=DEFAULT_RETRY):
    """Get the result of the operation, blocking if necessary.

    Args:
        timeout (int):
            How long (in seconds) to wait for the operation to complete.
            If None, wait indefinitely.

    Returns:
        google.protobuf.Message: The Operation's result.

    Raises:
        google.api_core.GoogleAPICallError: If the operation errors or if
            the timeout is reached before the operation completes.
    """
    kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry}
    self._blocking_poll(timeout=timeout, **kwargs)

    if self._exception is not None:
        # pylint: disable=raising-bad-type
        # Pylint doesn't recognize that this is valid in this case.
      raise self._exception

E google.api_core.exceptions.InternalServerError: 500 Model processing failed. If the error persists, please contact ai-platform-unified-feedback@google.com.

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:137: InternalServerError

@flaky-bot flaky-bot bot added flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Jul 29, 2022
@product-auto-label product-auto-label bot added the api: vertex-ai Issues related to the googleapis/python-aiplatform API. label Jul 29, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Jul 30, 2022

Test passed for commit da467d1 (Build Status, Sponge)! Closing this issue.

@flaky-bot flaky-bot bot closed this as completed Jul 30, 2022
@flaky-bot flaky-bot bot reopened this Aug 1, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Aug 1, 2022

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.


commit: da467d1
buildURL: Build Status, Sponge
status: failed

Test output
self = 
shared_state = {'resources': [ 
resource name: projects/580378083368/l...odel object at 0x7f0a5d0d94f0> 
resource name: projects/580378083368/locations/us-central1/models/5778061147311702016]}
def test_mdm_two_models_invalid_configs_xai(self, shared_state):
  temp_endpoint_with_two_models = self.temp_endpoint_with_two_models(shared_state)

tests/system/aiplatform/test_model_monitoring.py:365:


tests/system/aiplatform/test_model_monitoring.py:172: in temp_endpoint_with_two_models
endpoint.deploy(
google/cloud/aiplatform/models.py:815: in deploy
self._deploy(
google/cloud/aiplatform/base.py:793: in wrapper
return method(*args, **kwargs)
google/cloud/aiplatform/models.py:942: in _deploy
self._deploy_call(
google/cloud/aiplatform/models.py:1233: in _deploy_call
operation_future.result()


self = <google.api_core.operation.Operation object at 0x7f0a4d71fa90>
timeout = None, retry = <google.api_core.retry.Retry object at 0x7f0a61ae1c40>

def result(self, timeout=None, retry=DEFAULT_RETRY):
    """Get the result of the operation, blocking if necessary.

    Args:
        timeout (int):
            How long (in seconds) to wait for the operation to complete.
            If None, wait indefinitely.

    Returns:
        google.protobuf.Message: The Operation's result.

    Raises:
        google.api_core.GoogleAPICallError: If the operation errors or if
            the timeout is reached before the operation completes.
    """
    kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry}
    self._blocking_poll(timeout=timeout, **kwargs)

    if self._exception is not None:
        # pylint: disable=raising-bad-type
        # Pylint doesn't recognize that this is valid in this case.
      raise self._exception

E google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: DeployedCustomModelsPerProjectPerRegion

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:135: ResourceExhausted

@flaky-bot flaky-bot bot added the flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. label Aug 1, 2022
@rosiezou
Copy link
Contributor

Closing this issue for now. #1578 doesn't necessarily fix the root cause of the issue with resources not being deleted, but it should at least guarantee that model monitoring tests are cleaning up the resources created during test time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: vertex-ai Issues related to the googleapis/python-aiplatform API. flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

1 participant