Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

POC wasm as upstream filter #33119

Closed
wants to merge 1 commit into from

Conversation

juanmolle
Copy link
Contributor

When the filter is trying to be created. it try to get the clusterManager reference from ServerFactoryContext but it seems it is not available at that moment the pointer is null. not initialized at that moment.

[2024-03-26 12:35:53.520][144854083][critical][assert] [source/server/server.cc:144] assert failure: config_.clusterManager() != nullptr.
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:104] Caught Abort trap: 6, suspect faulting address 0x182652a60
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 22e6a78/1.30.0-dev/Modified/DEBUG/BoringSSL
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x10c6dc9b8]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #1: _sigtramp [0x1826bb584]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #2: pthread_kill [0x18268ac20]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #3: abort [0x182597a20]
[2024-03-26 12:35:53.545][144854083][critical][backtrace] [./source/server/backtrace.h:96] #4: Envoy::Server::InstanceBase::clusterManager() [0x10a3f5bcc]
[2024-03-26 12:35:53.547][144854083][critical][backtrace] [./source/server/backtrace.h:96] #5: Envoy::Server::ServerFactoryContextImpl::clusterManager() [0x10a374710]
[2024-03-26 12:35:53.548][144854083][critical][backtrace] [./source/server/backtrace.h:96] #6: Envoy::Extensions::HttpFilters::Wasm::FilterConfig::FilterConfig()

To make it work I just mock the clusterManager and could create a policy to add a header as upstream filter.
I have to add a check to not make the clearCacheRoute, I guess this has no meaning as upstream. Probably other checks are needed in the code.
Due to mock the clusterManager, I understand some functionality is not available like making calls to any server. not sure if has sens in an upstream filter.

I don't know how we could solve the clusterManager issue.

More Context in the stack

[2024-03-26 12:35:53.509][144854083][debug][misc] [source/common/network/dns_resolver/dns_factory_util.cc:39] create Apple DNS resolver type: envoy.network.dns_resolver.apple in MacOS.
[2024-03-26 12:35:53.509][144854083][debug][misc] [source/common/network/dns_resolver/dns_factory_util.cc:75] create DNS resolver type: envoy.network.dns_resolver.apple
[2024-03-26 12:35:53.519][144854083][debug][config] [./source/common/http/filter_chain_helper.h:111] upstream http filter #0
[2024-03-26 12:35:53.520][144854083][critical][assert] [source/server/server.cc:144] assert failure: config_.clusterManager() != nullptr.
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:104] Caught Abort trap: 6, suspect faulting address 0x182652a60
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 22e6a78/1.30.0-dev/Modified/DEBUG/BoringSSL
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x10c6dc9b8]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #1: _sigtramp [0x1826bb584]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #2: pthread_kill [0x18268ac20]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #3: abort [0x182597a20]
[2024-03-26 12:35:53.545][144854083][critical][backtrace] [./source/server/backtrace.h:96] #4: Envoy::Server::InstanceBase::clusterManager() [0x10a3f5bcc]
[2024-03-26 12:35:53.547][144854083][critical][backtrace] [./source/server/backtrace.h:96] #5: Envoy::Server::ServerFactoryContextImpl::clusterManager() [0x10a374710]
[2024-03-26 12:35:53.548][144854083][critical][backtrace] [./source/server/backtrace.h:96] #6: Envoy::Extensions::HttpFilters::Wasm::FilterConfig::FilterConfig() [0x10553fe64]
[2024-03-26 12:35:53.550][144854083][critical][backtrace] [./source/server/backtrace.h:96] #7: Envoy::Extensions::HttpFilters::Wasm::FilterConfig::FilterConfig() [0x1055403d8]
[2024-03-26 12:35:53.551][144854083][critical][backtrace] [./source/server/backtrace.h:96] #8: std::__1::construct_at[abi:v160006]<>() [0x10553ac3c]
[2024-03-26 12:35:53.553][144854083][critical][backtrace] [./source/server/backtrace.h:96] #9: std::__1::allocator_traits<>::construct[abi:v160006]<>() [0x10553aa54]
[2024-03-26 12:35:53.555][144854083][critical][backtrace] [./source/server/backtrace.h:96] #10: std::__1::__shared_ptr_emplace<>::__shared_ptr_emplace[abi:v160006]<>() [0x10553a978]
[2024-03-26 12:35:53.557][144854083][critical][backtrace] [./source/server/backtrace.h:96] #11: std::__1::__shared_ptr_emplace<>::__shared_ptr_emplace[abi:v160006]<>() [0x10553a658]
[2024-03-26 12:35:53.558][144854083][critical][backtrace] [./source/server/backtrace.h:96] #12: std::__1::allocate_shared[abi:v160006]<>() [0x10553a558]
[2024-03-26 12:35:53.560][144854083][critical][backtrace] [./source/server/backtrace.h:96] #13: std::__1::make_shared[abi:v160006]<>() [0x1055397b8]
[2024-03-26 12:35:53.561][144854083][critical][backtrace] [./source/server/backtrace.h:96] #14: Envoy::Extensions::HttpFilters::Wasm::WasmFilterConfig::createFilterFactoryFromProtoTyped() [0x105539718]
[2024-03-26 12:35:53.563][144854083][critical][backtrace] [./source/server/backtrace.h:96] #15: Envoy::Extensions::HttpFilters::Common::DualFactoryBase<>::createFilterFactoryFromProto() [0x105539cdc]
[2024-03-26 12:35:53.565][144854083][critical][backtrace] [./source/server/backtrace.h:96] #16: Envoy::Http::FilterChainHelper<>::processFilter() [0x10b0bed64]
[2024-03-26 12:35:53.567][144854083][critical][backtrace] [./source/server/backtrace.h:96] #17: Envoy::Http::FilterChainHelper<>::processFilters() [0x10b054b38]
[2024-03-26 12:35:53.569][144854083][critical][backtrace] [./source/server/backtrace.h:96] #18: Envoy::Upstream::ClusterInfoImpl::ClusterInfoImpl() [0x10b0507a8]
[2024-03-26 12:35:53.571][144854083][critical][backtrace] [./source/server/backtrace.h:96] #19: Envoy::Upstream::ClusterInfoImpl::ClusterInfoImpl() [0x10b0550b8]
[2024-03-26 12:35:53.572][144854083][critical][backtrace] [./source/server/backtrace.h:96] #20: Envoy::Upstream::ClusterImplBase::ClusterImplBase() [0x10b056550]
[2024-03-26 12:35:53.574][144854083][critical][backtrace] [./source/server/backtrace.h:96] #21: Envoy::Upstream::BaseDynamicClusterImpl::?::ClusterImplBase() [0x1044ba19c]
[2024-03-26 12:35:53.576][144854083][critical][backtrace] [./source/server/backtrace.h:96] #22: Envoy::Upstream::StrictDnsClusterImpl::StrictDnsClusterImpl() [0x1045393cc]
[2024-03-26 12:35:53.577][144854083][critical][backtrace] [./source/server/backtrace.h:96] #23: Envoy::Upstream::StrictDnsClusterImpl::StrictDnsClusterImpl() [0x104539fac]
[2024-03-26 12:35:53.579][144854083][critical][backtrace] [./source/server/backtrace.h:96] #24: std::__1::construct_at[abi:v160006]<>() [0x104547a70]
[2024-03-26 12:35:53.580][144854083][critical][backtrace] [./source/server/backtrace.h:96] #25: std::__1::allocator_traits<>::construct[abi:v160006]<>() [0x10454785c]
[2024-03-26 12:35:53.582][144854083][critical][backtrace] [./source/server/backtrace.h:96] #26: std::__1::__shared_ptr_emplace<>::__shared_ptr_emplace[abi:v160006]<>() [0x104547778]
[2024-03-26 12:35:53.583][144854083][critical][backtrace] [./source/server/backtrace.h:96] #27: std::__1::__shared_ptr_emplace<>::__shared_ptr_emplace[abi:v160006]<>() [0x104547450]
[2024-03-26 12:35:53.585][144854083][critical][backtrace] [./source/server/backtrace.h:96] #28: std::__1::allocate_shared[abi:v160006]<>() [0x104547348]
[2024-03-26 12:35:53.586][144854083][critical][backtrace] [./source/server/backtrace.h:96] #29: std::__1::make_shared[abi:v160006]<>() [0x10453ace0]
[2024-03-26 12:35:53.588][144854083][critical][backtrace] [./source/server/backtrace.h:96] #30: Envoy::Upstream::StrictDnsClusterFactory::createClusterImpl() [0x10453abb4]
[2024-03-26 12:35:53.590][144854083][critical][backtrace] [./source/server/backtrace.h:96] #31: Envoy::Upstream::ClusterFactoryImplBase::create() [0x10b0f53d0]
[2024-03-26 12:35:53.591][144854083][critical][backtrace] [./source/server/backtrace.h:96] #32: Envoy::Upstream::ClusterFactoryImplBase::create() [0x10b0f4a38]
[2024-03-26 12:35:53.593][144854083][critical][backtrace] [./source/server/backtrace.h:96] #33: Envoy::Upstream::ProdClusterManagerFactory::clusterFromProto() [0x10a5a6e50]
[2024-03-26 12:35:53.595][144854083][critical][backtrace] [./source/server/backtrace.h:96] #34: Envoy::Upstream::ClusterManagerImpl::loadCluster() [0x10a58c7c0]
[2024-03-26 12:35:53.597][144854083][critical][backtrace] [./source/server/backtrace.h:96] #35: Envoy::Upstream::ClusterManagerImpl::init() [0x10a58a9ec]
[2024-03-26 12:35:53.599][144854083][critical][backtrace] [./source/server/backtrace.h:96] #36: Envoy::Upstream::ProdClusterManagerFactory::clusterManagerFromProto() [0x10a5a3a3c]
[2024-03-26 12:35:53.600][144854083][critical][backtrace] [./source/server/backtrace.h:96] #37: Envoy::Server::Configuration::MainImpl::initialize() [0x10bc37e9c]
[2024-03-26 12:35:53.602][144854083][critical][backtrace] [./source/server/backtrace.h:96] #38: Envoy::Server::InstanceBase::initializeOrThrow() [0x10a3fdae0]
[2024-03-26 12:35:53.604][144854083][critical][backtrace] [./source/server/backtrace.h:96] #39: Envoy::Server::InstanceBase::initialize() [0x10a3f886c]
[2024-03-26 12:35:53.606][144854083][critical][backtrace] [./source/server/backtrace.h:96] #40: Envoy::createFunction()::$_1::operator()() [0x10a3011a0]

Signed-off-by: Juan Manuel Ollé <jolle@mulesoft.com>
Copy link

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #33119 was opened by juanmolle.

see: more, trace.

@@ -23,7 +149,7 @@ FilterConfig::FilterConfig(const envoy::extensions::filters::http::wasm::v3::Was
});
};

if (!Common::Wasm::createWasm(plugin, context.scope().createScope(""), server.clusterManager(),
if (!Common::Wasm::createWasm(plugin, context.scope().createScope(""), mycluster /*server.clusterManager()*/,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here is the main issue I found. server.clusterManager() assert due to the manager is still not initialized (I guess) at this moment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this assert failing? was it a manual run of Envoy with upstream filter? If it's in a unit test, the unit test is probably just not set up correctly. if you add an e2e test with wasm upstream filters it will give me something Ican patch and run.

Usually I suggest folks doing upstream filter work parameterize the e2e test using prependFilter like we did for test/integration/filter_integration_test.cc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alyssawilk Just creating the filter in normal envoy work, it tried to get the cluster manager from the server factory and it is not initialized. I guess @yanjunxiang-google get the same issue while working to convert composite filter and propose a change to solve it #33221

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's the same issue: #33218, which is resolved now.

@@ -183,6 +183,30 @@ TEST_F(CredentialsFileCredentialsProviderTest, DefaultCredentialsFile) {
EXPECT_EQ("profile1_token", credentials.sessionToken().value());
}

TEST_F(CredentialsFileCredentialsProviderTest, CustomProfileFromConfigShouldBeHonored) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignore this, belong to other Patch i was working on

virtual Singleton::Manager& singletonManager() override { throw 1; }
};

struct MyClusterManager : public Upstream::ClusterManager {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this used?

@@ -23,7 +149,7 @@ FilterConfig::FilterConfig(const envoy::extensions::filters::http::wasm::v3::Was
});
};

if (!Common::Wasm::createWasm(plugin, context.scope().createScope(""), server.clusterManager(),
if (!Common::Wasm::createWasm(plugin, context.scope().createScope(""), mycluster /*server.clusterManager()*/,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this assert failing? was it a manual run of Envoy with upstream filter? If it's in a unit test, the unit test is probably just not set up correctly. if you add an e2e test with wasm upstream filters it will give me something Ican patch and run.

Usually I suggest folks doing upstream filter work parameterize the e2e test using prependFilter like we did for test/integration/filter_integration_test.cc

@alyssawilk
Copy link
Contributor

ah cool, let's wait on that then :-)

@juanmolle
Copy link
Contributor Author

juanmolle commented Apr 4, 2024

This is the issue to handle cluster_manager initialization order

#33218

@juanmolle juanmolle closed this Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants