marshmallow.exceptions.ValidationError #105

shabeermm · 2019-01-14T08:46:04Z

Prerequisites

[ X] Put an X between the brackets on this line if you have done all of the following:
- Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
- Checked that there is not already provided the described functionality

Description

Getting marshmallow.exceptions.ValidationError when trying to post to the prom2teams V2 api on
curl http://localhost:8089/

Steps to Reproduce

curl http://localhost:8089/v2/prometheus-tool-dev -H "Content-Type: application/json" -v --data '

{
"@type": "Test Type",
"summary": "Test Summary",
"title": "Test Title"
}
'

Response

Hostname was NOT found in DNS cache
Trying localhost...
Connected to localhost (localhost) port 8089 (#0)

POST /v2/prometheus-tool-dev HTTP/1.1
User-Agent: curl/7.39.0
Host: localhost:8089
Accept: /
Content-Type: application/json
Content-Length: 83

upload completely sent off: 83 out of 83 bytes
HTTP 1.0, assume close after body
< HTTP/1.0 500 INTERNAL SERVER ERROR
< Content-Type: text/html
< Content-Length: 291
< Server: Werkzeug/0.14.1 Python/3.5.2
< Date: Mon, 14 Jan 2019 08:39:10 GMT
<

<title>500 Internal Server Error</title>

Internal Server Error

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

* Closing connection 0 -------------------------

Entries in the logs

2019-01-14 08:39:10,688 - flask.app - ERROR - Exception on /v2/prometheus-tool-dev [POST]
Traceback (most recent call last):
File "/home/azureuser/.local/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/home/azureuser/.local/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.5/dist-packages/flask_restplus/api.py", line 558, in error_router
return original_handler(e)
File "/usr/local/lib/python3.5/dist-packages/flask_restplus/api.py", line 558, in error_router
return original_handler(e)
File "/home/azureuser/.local/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/azureuser/.local/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/home/azureuser/.local/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/home/azureuser/.local/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/local/lib/python3.5/dist-packages/flask_restplus/api.py", line 314, in wrapper
resp = resource(*args, **kwargs)
File "/home/azureuser/.local/lib/python3.5/site-packages/flask/views.py", line 88, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/flask_restplus/resource.py", line 44, in dispatch_request
resp = meth(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/prom2teams/app/versions/v2/namespace.py", line 26, in post
alerts = self.schema.load(request.get_json())
File "/usr/local/lib/python3.5/dist-packages/marshmallow/schema.py", line 529, in load
postprocess=True,
File "/usr/local/lib/python3.5/dist-packages/marshmallow/schema.py", line 682, in _do_load
raise exc
marshmallow.exceptions.ValidationError: {'title': ['Unknown field.'], 'summary': ['Unknown field.'], '@type': ['Unknown field.']}
2019-01-14 08:39:10,689 - werkzeug - INFO - 172.18.37.14 - - [14/Jan/2019 08:39:10] "POST /v2/prometheus-tool-dev HTTP/1.1" 500 -

Expected behavior: I should be able to get 200 response

Actual behavior: Getting error response as 500

Reproduces how often: Everytime. New imolementation

Versions

Latest

Additional Information

Please see the above logs which i pasted for more information

shabeermm · 2019-01-16T04:34:52Z

Hi Team, Could you please advise on this ?

shabeermm · 2019-01-22T03:36:49Z

Hi Prom2teams,
We are stuck with this error and unable to use prom2teams. Tried on multiple Linux ubuntu flavors. Same error everywhere. Could you please advise on this ?
Thanks

jnogol · 2019-01-22T13:40:51Z

Hi @shabeermm, sorry for the late response. Could you please provide your .ini configuration (with the hosts anonymized, ofc)

NiasSt90 · 2019-01-24T21:50:37Z

i've got the same (unknown field) error

2019-01-23 23:33:21,770 - flask.app - ERROR - Exception on /v2/Connector [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.10.1-py3.5.egg/flask_restplus/api.py", line 557, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.10.1-py3.5.egg/flask_restplus/api.py", line 557, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.10.1-py3.5.egg/flask_restplus/api.py", line 313, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.10.1-py3.5.egg/flask_restplus/resource.py", line 44, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/prom2teams-2.2.3-py3.5.egg/prom2teams/app/versions/v2/namespace.py", line 26, in post
    alerts = self.schema.load(request.get_json())
  File "/usr/local/lib/python3.5/site-packages/marshmallow-3.0.0b16-py3.5.egg/marshmallow/schema.py", line 529, in load
    postprocess=True,
  File "/usr/local/lib/python3.5/site-packages/marshmallow-3.0.0b16-py3.5.egg/marshmallow/schema.py", line 682, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'commonLabels': ['Unknown field.'], 'groupKey': ['Unknown field.'], 'groupLabels': ['Unknown field.'], 'alerts': {0: {'labels': {'namespace': ['Unknown field.'], 'pod_name': ['Unknown field.'], 'prometheus': ['Unknown field.']}, 'annotations': {'runbook_url': ['Unknown field.'], 'message': ['Unknown field.']}}, 1: {'labels': {'container_name': ['Unknown field.'], 'pod_name': ['Unknown field.'], 'prometheus': ['Unknown field.'], 'namespace': ['Unknown field.']}, 'annotations': {'runbook_url': ['Unknown field.'], 'message': ['Unknown field.']}}}, 'commonAnnotations': ['Unknown field.']}
2019-01-23 23:33:21,771 - werkzeug - INFO - 192.168.2.68 - - [23/Jan/2019 23:33:21] "POST /v2/Connector HTTP/1.1" 500 -

i'm running the docker container (inside kubernetes cluster) idealista/prom2teams:2.2.3
without special config, only with env-variable
PROM2TEAMS_CONNECTOR = $teams-webhook

jnogol · 2019-01-25T08:05:28Z

Ok, I'll check it out

jnogol · 2019-01-25T08:23:56Z

Mmm, are you guys getting the JSONs from Prometheus? I used the ones in tests/data/jsons (got those from Prometheus and then modified the content) and worked like a charm. Could you try with those and see if they work? And please, provide me the JSON your Prometheus server is sending to Prom2teams (obviously without sensitive information) for debugging

NiasSt90 · 2019-01-28T20:40:04Z

{
 >  "receiver": "echo",
 >  "status": "resolved",
 >  "alerts": [
 >    {
 >      "status": "resolved",
 >      "labels": {
 >        "alertname": "CPUThrottlingHigh",
 >        "container_name": "config-reloader",
 >        "namespace": "monitoring",
 >        "pod_name": "alertmanager-ot-mon-prometheus-operator-alertmanager-0",
 >        "prometheus": "monitoring/ot-mon-prometheus-operator-prometheus",
 >        "severity": "warning"
 >      },
 >      "annotations": {
 >        "message": "27% throttling of CPU in namespace monitoring for container config-reloader in pod alertmanager-ot-mon-prometheus-operator-alertmanager-0.",
 >        "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-cputhrottlinghigh"
 >      },
 >      "startsAt": "2019-01-28T20:26:23.868893259Z",
 >      "endsAt": "2019-01-28T20:27:23.868893259Z",
 >      "generatorURL": "http://ot-mon-prometheus-operator-prometheus.monitoring:9090/graph?g0.expr=100+%2A+sum+by%28container_name%2C+pod_name%2C+namespace%29+%28increase%28container_cpu_cfs_throttled_periods_total%5B5m%5D%29%29+%2F+sum+by%28container_name%2C+pod_name%2C+namespace%29+%28increase%28container_cpu_cfs_periods_total%5B5m%5D%29%29+%3E+25\u0026g0.tab=1"
 >    }
 >  ],
 >  "groupLabels": {},
 >  "commonLabels": {
 >    "alertname": "CPUThrottlingHigh",
 >    "container_name": "config-reloader",
 >    "namespace": "monitoring",
 >    "pod_name": "alertmanager-ot-mon-prometheus-operator-alertmanager-0",
 >    "prometheus": "monitoring/ot-mon-prometheus-operator-prometheus",
 >    "severity": "warning"
 >  },
 >  "commonAnnotations": {
 >    "message": "27% throttling of CPU in namespace monitoring for container config-reloader in pod alertmanager-ot-mon-prometheus-operator-alertmanager-0.",
 >    "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-cputhrottlinghigh"
 >  },
 >  "externalURL": "http://ot-mon-prometheus-operator-alertmanager.monitoring:9093",
 >  "version": "4",
 >  "groupKey": "{}/{alertname=\"CPUThrottlingHigh\"}:{}"
 >}

the output was fetched from an echo.go webhook taken from here: https://github.com/prometheus/alertmanager/blob/1e01b2bdbae8edb393fcf555732304f34d192fc9/examples/webhook/echo.go

therefore you have to ignore these ">" chars.

ekarlso · 2019-02-14T21:33:26Z

I am getting the same error, any idea on this?

dortegau · 2019-02-14T22:26:07Z

Hi @ekarlso,

Could you provide more details?

Thanks in advance,
Daniel

ekarlso · 2019-02-15T07:45:21Z

marshmallow.exceptions.ValidationError: {'groupKey': ['Unknown field.'], 'commonLabels': ['Unknown field.'], 'alerts': {0: {'labels': {'prometheus': ['Unknown field.']}, 'annotations': {'message': ['Unknown field.'], 'runbook_url': ['Unknown field.']}}, 1: {'labels': {'prometheus': ['Unknown field.']}, 'annotations': {'message': ['Unknown field.'], 'runbook_url': ['Unknown field.']}}, 2: {'labels': {'namespace': ['Unknown field.'], 'prometheus': ['Unknown field.'], 'pod_name': ['Unknown field.'], 'container_name': ['Unknown field.']}, 'annotations': {'message': ['Unknown field.'], 'runbook_url': ['Unknown field.']}}, 3: {'labels': {'namespace': ['Unknown field.'], 'prometheus': ['Unknown field.'], 'pod_name': ['Unknown field.'], 'container_name': ['Unknown field.']}, 'annotations': {'message': ['Unknown field.'], 'runbook_url': ['Unknown field.']}}, 4: {'labels': {'namespace': ['Unknown field.'], 'prometheus': ['Unknown field.'], 'pod_name': ['Unknown field.'], 'container_name': ['Unknown field.']}, 'annotations': {'message': ['Unknown field.'], 'runbook_url': ['Unknown field.']}}, 5: {'labels': {'namespace': ['Unknown field.'], 'prometheus': ['Unknown field.'], 'pod_name': ['Unknown field.'], 'container_name': ['Unknown field.']}, 'annotations': {'message': ['Unknown field.'], 'runbook_url': ['Unknown field.']}}}, 'commonAnnotations': ['Unknown field.'], 'groupLabels': ['Unknown field.']}

apiVersion: v1
kind: ConfigMap
metadata:
name: prom2teams-config
namespace: monitoring
data:
config.ini: |
[Microsoft Teams]
Connector = https://outlook.office.com/webhook/REPLACED/IncomingWebhook/REPLACED/REPLACED

[HTTP Server]
Host = 0.0.0.0
Port = 8089

[Log]
Level = Debug

# [Template]

ekarlso · 2019-02-15T08:50:31Z

@dortegau any idea on this?

gjermy · 2019-02-15T13:50:22Z

I've got a local change to fix this. Have added unit test for the example from @NiasSt90.
Should I try submitting a PR?

jnogol · 2019-02-15T13:59:44Z

Sure, @gjermy

gjermy · 2019-02-15T14:31:02Z

Hi @jnogol - PR submitted. (But go gently - first github PR)

Fixes #105 marshmallow.exceptions.ValidationError

dortegau · 2019-02-19T10:13:11Z

Hi @gjermy,

Your PR has been merged. Thanks for your contribution 🙂

dortegau · 2019-02-19T11:50:02Z

Version 2.3.0 is now available in Pypi: https://pypi.org/project/prom2teams/2.3.0/

wadhahbenaissa · 2020-11-05T10:30:31Z

Hello guys. I hope you're doing well in these times.
However, I think the issue is still persisting.
I have deployed prometheus using helm on a 3 nodes AKS, alongside prom2teams.
I configured my own rule on prometheus as shown below:

additionalPrometheusRules:
 - name: pod-pu.rules
   groups:
     - name: CPU-Usage
       rules:
       - alert: CPUSpikeWarning
         annotations:
           description: CPU Usage per namespace exceeded 85%
           summary: CPU spike caused WARNING
         expr: sum(rate(container_cpu_usage_seconds_total[5m])) by (pod) *100 > 1
         for: 1m
         labels:
           severity: warning

Furthermore, I configured a ms teams webhook :

  config:
    global:
      resolve_timeout: 5m
    
    route:
      group_by: ['alertname']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'prometheus-msteams'
      
      routes:
      - receiver: 'prometheus-msteams'
        continue: true
        match:
          alertname: CPUSpikeWarning
        
    receivers:
    - name: prometheus-msteams
      webhook_configs:
        - send_resolved: true
          url: 'http://prom2teams-server:8089' # Itried 'http://prom2teams-server:8089/v2/Connector' as well

Finally, please find below values.yaml file from prom2teams:

# Default values for prom2teams.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

image:
  repository: idealista/prom2teams
  tag:
  pullPolicy: IfNotPresent

resources:
 requests:
   cpu: 100m
   memory: 128Mi
 limits:
   cpu: 200m
   memory: 200Mi

service:
  type: ClusterIP
  port: 8089

prom2teams:
  host:
  port: 8089
  connector: https://outlook.office.com/webhook/xxxxx
  connectors: {}
  group_alerts_by:
  loglevel: DEBUG
  templatepath: /opt/prom2teams/helmconfig/teams.j2
  config: /opt/prom2teams/helmconfig/config.ini

# Security Context properties
securityContext:
  # enabled is a flag to enable Security Context
  enabled: true
  # runAsUser is the user ID used to run the container
  runAsUser: 65534
  # runAsGroup is the primary group ID used to run all processes within any container of the pod
  runAsGroup: 65534
  # fsGroup is the group ID associated with the container
  fsGroup: 65534
  # readOnlyRootFilesystem is a flag to enable readOnlyRootFilesystem for the Hazelcast security context
  readOnlyRootFilesystem: true

Logs from prom2teams pod show:

2020-11-05 10:28:17,330 - werkzeug - INFO - 10.244.0.2 - - [05/Nov/2020 10:28:17] "POST / HTTP/1.1" 500 -
2020-11-05 10:28:17,337 - werkzeug - ERROR - Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 325, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/resource.py", line 44, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/prom2teams-2.5.2-py3.5.egg/prom2teams/app/versions/v1/namespace.py", line 29, in post
    alerts = self.schema.load(request.get_json())
  File "/usr/local/lib/python3.5/site-packages/marshmallow-3.0.0rc6-py3.5.egg/marshmallow/schema.py", line 707, in load
    postprocess=True,
  File "/usr/local/lib/python3.5/site-packages/marshmallow-3.0.0rc6-py3.5.egg/marshmallow/schema.py", line 867, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'truncatedAlerts': ['Unknown field.']}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/Werkzeug-0.16.0-py3.5.egg/werkzeug/serving.py", line 304, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python3.5/site-packages/Werkzeug-0.16.0-py3.5.egg/werkzeug/serving.py", line 292, in execute
    application_iter = app(environ, start_response)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/flask_restplus-0.12.1-py3.5.egg/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.5/site-packages/Flask-1.0.2-py3.5.egg/flask/app.py", line 1748, in handle_exception
    return self.finalize_request(handler(e), from_error_handler=True)
  File "/usr/local/lib/python3.5/site-packages/prom2teams-2.5.2-py3.5.egg/prom2teams/app/api.py", line 26, in error_handler
    return str(e), e.code
AttributeError: 'ValidationError' object has no attribute 'code'

So I was wondering if you could help me on this.
Thanks

gjermy pushed a commit to gjermy/prom2teams that referenced this issue Feb 15, 2019

Fix for bug idealista#105 marshmallow.exceptions.ValidationError

d5e8fd1

gjermy mentioned this issue Feb 15, 2019

Fixes #105 marshmallow.exceptions.ValidationError #106

Merged

gjermy added a commit to gjermy/prom2teams that referenced this issue Feb 18, 2019

Add changelog entry for fix idealista#105

de0dc19

dortegau added a commit that referenced this issue Feb 19, 2019

Merge pull request #106 from gjermy/develop

c52211e

Fixes #105 marshmallow.exceptions.ValidationError

dortegau closed this as completed Feb 19, 2019

vicsufer mentioned this issue Nov 5, 2020

prom2teams marshmallow.exceptions.ValidationError #235

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

marshmallow.exceptions.ValidationError #105

marshmallow.exceptions.ValidationError #105

shabeermm commented Jan 14, 2019

shabeermm commented Jan 16, 2019

shabeermm commented Jan 22, 2019

jnogol commented Jan 22, 2019

NiasSt90 commented Jan 24, 2019

jnogol commented Jan 25, 2019

jnogol commented Jan 25, 2019

NiasSt90 commented Jan 28, 2019

ekarlso commented Feb 14, 2019

dortegau commented Feb 14, 2019

ekarlso commented Feb 15, 2019

ekarlso commented Feb 15, 2019

gjermy commented Feb 15, 2019

jnogol commented Feb 15, 2019

gjermy commented Feb 15, 2019

dortegau commented Feb 19, 2019

dortegau commented Feb 19, 2019

wadhahbenaissa commented Nov 5, 2020

marshmallow.exceptions.ValidationError #105

marshmallow.exceptions.ValidationError #105

Comments

shabeermm commented Jan 14, 2019

Prerequisites

Description

Steps to Reproduce

Internal Server Error

Versions

Additional Information

shabeermm commented Jan 16, 2019

shabeermm commented Jan 22, 2019

jnogol commented Jan 22, 2019

NiasSt90 commented Jan 24, 2019

jnogol commented Jan 25, 2019

jnogol commented Jan 25, 2019

NiasSt90 commented Jan 28, 2019

ekarlso commented Feb 14, 2019

dortegau commented Feb 14, 2019

ekarlso commented Feb 15, 2019

ekarlso commented Feb 15, 2019

gjermy commented Feb 15, 2019

jnogol commented Feb 15, 2019

gjermy commented Feb 15, 2019

dortegau commented Feb 19, 2019

dortegau commented Feb 19, 2019

wadhahbenaissa commented Nov 5, 2020