Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

prom2teams marshmallow.exceptions.ValidationError #235

Closed
wadhahbenaissa opened this issue Nov 5, 2020 · 7 comments
Closed

prom2teams marshmallow.exceptions.ValidationError #235

wadhahbenaissa opened this issue Nov 5, 2020 · 7 comments
Assignees
@blalop
Labels
bug

Comments

@wadhahbenaissa
Copy link

Hello guys. I hope you're doing well in these times.
However, I think the issue is still persisting.
I have deployed prometheus using helm on a 3 nodes AKS, alongside prom2teams.
I configured my own rule on prometheus as shown below:

additionalPrometheusRules:
 - name: pod-pu.rules
   groups:
     - name: CPU-Usage
       rules:
       - alert: CPUSpikeWarning
         annotations:
           description: CPU Usage per namespace exceeded 85%
           summary: CPU spike caused WARNING
         expr: sum(rate(container_cpu_usage_seconds_total[5m])) by (pod) *100 > 1
         for: 1m
         labels:
           severity: warning

Furthermore, I configured a ms teams webhook :

  config:
    global:
      resolve_timeout: 5m
    
    route:
      group_by: ['alertname']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'prometheus-msteams'
      
      routes:
      - receiver: 'prometheus-msteams'
        continue: true
        match:
          alertname: CPUSpikeWarning
        
    receivers:
    - name: prometheus-msteams
      webhook_configs:
        - send_resolved: true
          url: 'http://prom2teams-server:8089' # Itried 'http://prom2teams-server:8089/v2/Connector' as well

Finally, please find below values.yaml file from prom2teams:

# Default values for prom2teams.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

image:
  repository: idealista/prom2teams
  tag:
  pullPolicy: IfNotPresent

resources:
 requests:
   cpu: 100m
   memory: 128Mi
 limits:
   cpu: 200m
   memory: 200Mi

service:
  type: ClusterIP
  port: 8089

prom2teams:
  host:
  port: 8089
  connector: https://outlook.office.com/webhook/xxxxx
  connectors: {}
  group_alerts_by:
  loglevel: DEBUG
  templatepath: /opt/prom2teams/helmconfig/teams.j2
  config: /opt/prom2teams/helmconfig/config.ini

# Security Context properties
securityContext:
  # enabled is a flag to enable Security Context
  enabled: true
  # runAsUser is the user ID used to run the container
  runAsUser: 65534
  # runAsGroup is the primary group ID used to run all processes within any container of the pod
  runAsGroup: 65534
  # fsGroup is the group ID associated with the container
  fsGroup: 65534
  # readOnlyRootFilesystem is a flag to enable readOnlyRootFilesystem for the Hazelcast security context
  readOnlyRootFilesystem: true

Logs from prom2teams pod show:

2020-11-05 10:28:17,330 - werkzeug - INFO - 10.244.0.2 - - [05/Nov/2020 10:28:17] "POST / HTTP/1.1" 500 -
2020-11-05 10:28:17,337 - werkzeug - ERROR - Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 325, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/resource.py", line 44, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/prom2teams-2.5.2-py3.5.egg/prom2teams/app/versions/v1/namespace.py", line 29, in post
    alerts = self.schema.load(request.get_json())
  File "/usr/local/lib/python3.5/site-packages/marshmallow-3.0.0rc6-py3.5.egg/marshmallow/schema.py", line 707, in load
    postprocess=True,
  File "/usr/local/lib/python3.5/site-packages/marshmallow-3.0.0rc6-py3.5.egg/marshmallow/schema.py", line 867, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'truncatedAlerts': ['Unknown field.']}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/Werkzeug-0.16.0-py3.5.egg/werkzeug/serving.py", line 304, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python3.5/site-packages/Werkzeug-0.16.0-py3.5.egg/werkzeug/serving.py", line 292, in execute
    application_iter = app(environ, start_response)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1748, in handle_exception
    return self.finalize_request(handler(e), from_error_handler=True)
  File "/usr/local/lib/python3.5/site-packages/prom2teams-2.5.2-py3.5.egg/prom2teams/app/api.py", line 26, in error_handler
    return str(e), e.code
AttributeError: 'ValidationError' object has no attribute 'code'

So I was wondering if you could help me on this.
Thanks
@vicsufer
Copy link
Collaborator

vicsufer commented Nov 5, 2020 •
edited
Loading

Hi there! Thanks for reporting this issue, we'll try to have a look soon.

Additional context: #105

Probably related to #155

@vicsufer vicsufer added the bug label Nov 5, 2020
@blalop
Copy link
Member

blalop commented Nov 5, 2020 •
edited
Loading

Hi @wadhahbenaissa, just to provide steps to reproduce the issue: you are using the 2.5.2 release, am I right? Are you able to reproduce this issue with the latest release? (3.1.0). And finally, which version of Alert Manager are you using?

@blalop blalop self-assigned this Nov 5, 2020
@wadhahbenaissa
Copy link
Author

Yep, I am using 2.5.2 release for prom2teams. As for alertmanager, it's v0.21.0.
With the latest release, 3.1.0, I get the same response (same error)

@blalop
Copy link
Member

blalop commented Nov 5, 2020 •
edited
Loading

Will try to reproduce it in my local machine. In the meantime, could you provide the logs of this bug in the lastest release? Thanks in advance!

In a first look, seems related to #189 and #210

@wadhahbenaissa
Copy link
Author

hey @blalop, here are the logs:

2020-11-05 18:38:20,136 - werkzeug - INFO - 10.244.0.17 - - [05/Nov/2020 18:38:20] "POST / HTTP/1.1" 500 -
2020-11-05 18:38:20,143 - werkzeug - ERROR - Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 325, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/resource.py", line 44, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/prom2teams-2.5.2-py3.5.egg/prom2teams/app/versions/v1/namespace.py", line 29, in post
    alerts = self.schema.load(request.get_json())
  File "/usr/local/lib/python3.5/site-packages/marshmallow-3.0.0rc6-py3.5.egg/marshmallow/schema.py", line 707, in load
    postprocess=True,
  File "/usr/local/lib/python3.5/site-packages/marshmallow-3.0.0rc6-py3.5.egg/marshmallow/schema.py", line 867, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'truncatedAlerts': ['Unknown field.']}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/Werkzeug-0.16.0-py3.5.egg/werkzeug/serving.py", line 304, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python3.5/site-packages/Werkzeug-0.16.0-py3.5.egg/werkzeug/serving.py", line 292, in execute
    application_iter = app(environ, start_response)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1748, in handle_exception
    return self.finalize_request(handler(e), from_error_handler=True)
  File "/usr/local/lib/python3.5/site-packages/prom2teams-2.5.2-py3.5.egg/prom2teams/app/api.py", line 26, in error_handler
    return str(e), e.code
AttributeError: 'ValidationError' object has no attribute 'code'

@wadhahbenaissa
Copy link
Author

Hello guys, it seems working now with the 3.1.0 version.
I tried to deploy prom2eams using this command:

helm upgrade --install prom2teams -n test-prometheus --version 3.1.0 prom2teams\helm

however, after checking the version with helm story prom2teams -n test-prometheus it was still showing 2.5.2.
So, I modified the "Chart.yaml" in helm folder and updated appVersion to 3.1.0

@vicsufer
Copy link
Collaborator

vicsufer commented Nov 6, 2020 •
edited
Loading

That's great to hear! Thank you for noticing the outdated appVersion value, we'll fix the default value as soon as we can.

If you notice something improvable in this new version or have a feature request, please do not hesitate to ask for it. 😁

@vicsufer vicsufer closed this as completed Nov 6, 2020
@vicsufer vicsufer mentioned this issue Nov 6, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@blalop blalop

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vicsufer @blalop @wadhahbenaissa