This repository has been archived by the owner on Dec 16, 2020. It is now read-only.
Segfault while parsing CEL: proxy 1.6-alpha #497
Comments
|
Got the same error with the latest build. Caught Segmentation fault, suspect faulting address 0x8
Backtrace (use tools/stack_decode.py to get line numbers):
Envoy version: ff8d26a507d4e75863c9800c143a5b94b2a3fdd5/1.14.1/Clean/RELEASE/BoringSSL
#0: __restore_rt [0x7f2de1032890]
#1: antlr4::atn::ParserATNSimulator::execATN() [0x55ac6be8f16f]
#2: antlr4::atn::ParserATNSimulator::adaptivePredict() [0x55ac6be8eb50]
#3: cel_grammar::CelParser::exprList() [0x55ac6be5ae78]
#4: cel_grammar::CelParser::primary() [0x55ac6be5a2a2]
#5: cel_grammar::CelParser::member() [0x55ac6be581c5]
#6: cel_grammar::CelParser::unary() [0x55ac6be57681]
#7: cel_grammar::CelParser::calc() [0x55ac6be56166]
#8: cel_grammar::CelParser::relation() [0x55ac6be558a8]
#9: cel_grammar::CelParser::conditionalAnd() [0x55ac6be55188]
#10: cel_grammar::CelParser::conditionalOr() [0x55ac6be54946]
#11: cel_grammar::CelParser::expr() [0x55ac6be5436e]
#12: cel_grammar::CelParser::start() [0x55ac6be53f5d]
#13: google::api::expr::parser::ParseWithMacros() [0x55ac6be402a2]
#14: google::api::expr::parser::Parse() [0x55ac6be400d5]
#15: Envoy::Extensions::Common::Wasm::CreateExpressionFactory::create()::{lambda()#1}::operator()() [0x55ac6b3f02e7]
#16: std::__1::__invoke_void_return_wrapper<>::__call<>() [0x55ac6b3f0197]
#17: Envoy::Extensions::Common::Wasm::Exports::call_foreign_function() [0x55ac6b3e736c]
#18: Envoy::Extensions::Common::Wasm::Null::Plugin::Stats::PluginRootContext::addStringExpression() [0x55ac6ab09133]
#19: std::__1::__function::__func<>::operator()() [0x55ac6ab134e4]
#20: Wasm::Common::JsonArrayIterate() [0x55ac6ab1ad4b]
#21: Envoy::Extensions::Common::Wasm::Null::Plugin::Stats::PluginRootContext::initializeDimensions() [0x55ac6ab04e9d]
#22: Envoy::Extensions::Common::Wasm::Null::Plugin::Stats::PluginRootContext::onConfigure() [0x55ac6ab08306]
#23: std::__1::__function::__func<>::operator()() [0x55ac6abb17ab]
#24: Envoy::Extensions::Common::Wasm::Context::onConfigure() [0x55ac6b3d4b7e]
#25: Envoy::Extensions::Common::Wasm::getOrCreateThreadLocalWasm() [0x55ac6b3fbe84]
#26: std::__1::__function::__func<>::operator()() [0x55ac6af14067]
#27: std::__1::__function::__func<>::operator()() [0x55ac6c0ffcd8]
#28: std::__1::__function::__func<>::operator()() [0x55ac6c100f08]
#29: Envoy::Event::DispatcherImpl::runPostCallbacks() [0x55ac6c179266]
#30: event_process_active_single_queue [0x55ac6c5c2256]
#31: event_base_loop [0x55ac6c5c0dde]
#32: Envoy::Server::WorkerImpl::threadRoutine() [0x55ac6c16f474]
#33: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::$_0::__invoke() [0x55ac6c676ee3]
#34: start_thread [0x7f2de10276db] |
|
@TristonianJones Does the above segfault make sense to you? |
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/envoy -c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --dra'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f2dde429700 (LWP 28))]
(gdb) bt
#0 raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x000055ac6c63e0c5 in Envoy::SignalAction::sigHandler(int, siginfo_t*, void*) ()
#2 <signal handler called>
#3 0x000055ac6be90208 in antlr4::atn::ParserATNSimulator::getExistingTargetState(antlr4::dfa::DFAState*, unsigned long) ()
#4 0x000055ac6be8f16f in antlr4::atn::ParserATNSimulator::execATN(antlr4::dfa::DFA&, antlr4::dfa::DFAState*, antlr4::TokenStream*, unsigned long, antlr4::ParserRuleContext*) ()
#5 0x000055ac6be8eb50 in antlr4::atn::ParserATNSimulator::adaptivePredict(antlr4::TokenStream*, unsigned long, antlr4::ParserRuleContext*) ()
#6 0x000055ac6be5ae78 in cel_grammar::CelParser::exprList() ()
#7 0x000055ac6be5a2a2 in cel_grammar::CelParser::primary() ()
#8 0x000055ac6be581c5 in cel_grammar::CelParser::member(int) ()
#9 0x000055ac6be57681 in cel_grammar::CelParser::unary() ()
#10 0x000055ac6be56166 in cel_grammar::CelParser::calc(int) ()
#11 0x000055ac6be558a8 in cel_grammar::CelParser::relation(int) ()
#12 0x000055ac6be55188 in cel_grammar::CelParser::conditionalAnd() ()
#13 0x000055ac6be54946 in cel_grammar::CelParser::conditionalOr() ()
#14 0x000055ac6be5436e in cel_grammar::CelParser::expr() ()
#15 0x000055ac6be53f5d in cel_grammar::CelParser::start() ()
#16 0x000055ac6be402a2 in google::api::expr::parser::ParseWithMacros(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<google::api::expr::parser::Macro, std::__1::allocator<google::api::expr::parser::Macro> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ()
#17 0x000055ac6be400d5 in google::api::expr::parser::Parse(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ()
#18 0x000055ac6b3f02e7 in Envoy::Extensions::Common::Wasm::CreateExpressionFactory::create() const::{lambda(Envoy::Extensions::Common::Wasm::Wasm&, absl::string_view, std::__1::function<void* (unsigned long)>)#1}::operator()(Envoy::Extensions::Common::Wasm::Wasm&, absl::string_view, std::__1::function<void* (unsigned long)>) const ()
#19 0x000055ac6b3f0197 in Envoy::Extensions::Common::Wasm::WasmResult std::__1::__invoke_void_return_wrapper<Envoy::Extensions::Common::Wasm::WasmResult>::__call<Envoy::Extensions::Common::Wasm::CreateExpressionFactory::create() const::{lambda(Envoy::Extensions::Common::Wasm::Wasm&, absl::string_view, std::__1::function<void* (unsigned long)>)#1}&, Envoy::Extensions::Common::Wasm::Wasm&, absl::string_view, std::__1::function<void* (unsigned long)> >(Envoy::Extensions::Common::Wasm::CreateExpressionFactory::create() const::{lambda(Envoy::Extensions::Common::Wasm::Wasm&, absl::string_view, std::__1::function<void* (unsigned long)>)#1}&, Envoy::Extensions::Common::Wasm::Wasm&, absl::string_view&&, std::__1::function<void* (unsigned long)>&&) ()
#20 0x000055ac6b3e736c in Envoy::Extensions::Common::Wasm::Exports::call_foreign_function(void*, Envoy::Extensions::Common::Wasm::Word, Envoy::Extensions::Common::Wasm::Word, Envoy::Extensions::Common::Wasm::Word, Envoy::Extensions::Common::Wasm::Word, Envoy::Extensions::Common::Wasm::Word, Envoy::Extensions::Common::Wasm::Word) ()
#21 0x000055ac6ab09133 in Envoy::Extensions::Common::Wasm::Null::Plugin::Stats::PluginRootContext::addStringExpression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) () |
|
@mandarjog that error is coming from deep in the ANTLR stack. Which component of the config is the CEL expression? I wonder if this can be reproduced with just the CEL toolchain outside Envoy. I take it this is using the C++ stack, or is the Go one complied to WASM? |
|
@TristonianJones This is using the c++ cel stack, and (kuat is ooo). I will check which expression it was trying to compile. It does not always happen though. |
|
Great, thanks. I checked the ANLTR open / closed issues and haven't seen
anything in particular that relates to a segfault, but there have been some
updates between ANTLR 4.7.2 and ANTLR 4.8. I can try upgrading the
workspace and see if it fixes the issue.
Thoughts?
…-Tristan
On Wed, May 6, 2020 at 8:47 AM mandarjog ***@***.***> wrote:
@TristonianJones <https://github.com/TristonianJones> This is using the
c++ cel stack, and (kuat is ooo). I will check which expression it was
trying to compile. It does not always happen though.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#497 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADV7ZDHHT4ODWZ554DXSXDRQGBBZANCNFSM4MTAKGGQ>
.
|
|
Give it a go. Can't hurt. |
|
I think having an expression that causes parser to fail would help. I doubt it is an ANTLR issue (otherwise, you should be able to replicate it in google3). Seems like a memory corruption or lifecycle issue. |
|
expression: |
|
Thanks. That's helpful. Macro expansion usually involves AST manipulation.
The has macro is pretty trivial in its behavior though and I wouldn't
expect it to next with ANTLRs internal DFA storage at all.
Kuat's theory about object lifecycle is compelling. I'll poke at both and
see what I can find when I come back on shift.
…-Tristan
On Wed, May 6, 2020, 11:53 AM mandarjog ***@***.***> wrote:
expression: has(wasm.istio.operationId)?wasm.istio.operationId:'unknown'
#23 Envoy::Extensions::Common::Wasm::Exports::call_foreign_function (raw_context=<optimized out>, function_name=..., function_name_size=..., arguments=..., arguments_size=..., results=...,
results_size=...) at external/envoy/source/extensions/common/wasm/exports.cc:244
#24 0x000055ac6ab09133 in Envoy::Extensions::Common::Wasm::Null::Plugin::proxy_call_foreign_function (function_name_size=11,
arguments=0x55ac703b7a40 "has(wasm.istio.operationId)?wasm.istio.operationId:'unkown'", arguments_size=59, results=0x7f2dde414520, results_size=0x7f2dde414500, function_name=<optimized out>)
at bazel-out/k8-opt/bin/external/envoy/source/extensions/common/wasm/null/_virtual_includes/null_plugin_lib/extensions/common/wasm/null/wasm_api_impl.h:258
---Type <return> to continue, or q <return> to quit---
#25 Envoy::Extensions::Common::Wasm::Null::Plugin::createExpression (expr=..., token=<optimized out>) at external/envoy/api/wasm/cpp/contrib/proxy_expr.h:6
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#497 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADV7ZABHLUN76XTZX6FBETRQGW33ANCNFSM4MTAKGGQ>
.
|
|
Per offline discussion, Tristan will upgrade CEL-cpp Comments by Tim Niemueller
It would now be a good time to test envoy-wasm with the patched cel-cpp version if the problem persists or is resolved. |
|
I've just created cel-cpp v0.2.0. I can submit a PR to update envoy-wasm |
|
This has been merged. |
This was referenced Aug 14, 2020
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Istio-proxy: Version 1.6-alpha.f63181a5c5d886aa309f67ab098897e85f3b573c
With the following stats config:
The text was updated successfully, but these errors were encountered: