Envoy crashing with NOT_REACHED_GCOVR_EXCL_LINE when loading YAML config using --config-yaml

[scaldwell-atlassian]

Bug Template

Title: One line description

Description:

Problem Statement:
When using —config-yaml , we encounter a panic in json_loader.cc at line 315 (NOT_REACHED_GCOVR_EXCL_LINE), which is defined as a panic. This panic is defined in assert.h at line 60 as an outcome we should never see unless code execution either (a) follows an unexpected path or (b) a “horrendous” bug exists.
Investigating the bug further, we find that the error is in a switch…case statement in json_loader.cc (which starts at line 246) during buildRapidJsonDocument(). This appears to be a type checking block, which tests for Arrays, Objects and Null values. The method has likely reached some input on
the YAML file which is not one of these data types, but the output does not tell us anything further.

Expected behavior:
Envoy should return an error message that guides the administrator in correcting the problem.

Repro steps:

Use minimum YAML config from documents with --config-yaml option and observe output to stdout/err when running envoy from commandline. At best there will be a silent error and the proxy will start. At worst case, the service will not start and an error will be seen on screen (seel below).

Config:

Issue was reproduced using google_com_proxy.yaml and google_com_proxy_v2.yaml included with software.

Logs:

Include the access logs and the Envoy logs.

root@89279f2e0b72:/# $(which envoy) --v2-config-only -l DEBUG --config-yaml /etc/envoy/envoy.yaml
[2018-07-30 21:40:04.499][243][info][main] source/server/server.cc:183] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312)
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:185] statically linked extensions:
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:187] access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:190] filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:193] filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:196] filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:198] stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:200] tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:203] transport_sockets.downstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-07-30 21:40:04.500][243][info][main] source/server/server.cc:206] transport_sockets.upstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-07-30 21:40:04.500][243][critical][assert] source/common/json/json_loader.cc:315] panic: not reached
[2018-07-30 21:40:04.500][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Aborted, suspect faulting address 0xf3
[2018-07-30 21:40:04.500][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<0> obj</lib/x86_64-linux-gnu/libc.so.6> (If unsymbolized, use tools/stack_decode.py):
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #0 0x7ff6dff4d428 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #1 0x7ff6dff4f029 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<0> obj</usr/local/bin/envoy>
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #2 0x9acc2c (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #3 0x9ad287 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #4 0x9b3ab0 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #5 0x9a2687 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #6 0x6a8f00 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #7 0x6a98a0 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #8 0x6ac655 (unknown)
[2018-07-30 21:40:04.501][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #9 0x477e66 (unknown)
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #10 0x4781e2 (unknown)
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #11 0x418015 (unknown)
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<0> obj</lib/x86_64-linux-gnu/libc.so.6>
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #12 0x7ff6dff3882f (unknown)
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<0> obj</usr/local/bin/envoy>
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #13 0x46e1c8 (unknown)
[2018-07-30 21:40:04.502][243][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 0
Aborted (core dumped)

Call Stack:

see above

[mattklein123]

@scaldwell-atlassian have you tried reproing this on master? If it still repros can you provide the config that causes the crash?

[tony-shi]

The same problem in OSX。I'm using https://github.com/envoyproxy/envoy/blob/master/configs/google_com_proxy.v2.yaml as my yaml config and run envoy with the command as path-to-bazel-bin/envoy-static --config-yaml path-to-my-config.yaml .

[dio]

@scaldwell-atlassian @tony-shi that --config-yaml expects the content of the .yaml config file (unless the intention was not this).

Here's a way to load full config.

cat configs/google_com_proxy.v2.yaml | ./bazel-bin/source/exe/envoy-static --config-yaml "$(</dev/stdin)"

However, the usage pattern for this is to override some portions of (bootstrap) config object that is loaded by --config-path. Ref: https://www.envoyproxy.io/docs/envoy/latest/operations/cli.html#cmdoption-config-yaml

[mattklein123]

@dio we still shouldn't crash here but cleanly exit if there is a config error. Do you know what the actual bug is?

[dio]

@mattklein123 I think because of the returned node.Type() is not something that can be converted to JSON. (a file path string is a valid YAML but it cannot be converted to JSON).

Submitted a PR here: #4110