RFC: Adding appconfig lambda layer support as appconfig backend.

[whardier]

Key information

• RFC PR: (leave this empty)
• Related issue(s), if known: None
• Area: Parameters
• Meet tenets: Yes
• Approved by: ''
• Reviewed by: ''

Summary

Utilize the AWS provided AppConfig Lambda Layer. This layer is used to provide a proxy request to AppConfig configurations. The layer has built-in caching and automatically refreshes based on a TTL. This is important when utilizing the "full power" of AppConfig to more accurately provide configuration deployments to live resources as well as asses how many client-ids are covered by the deployment stages.

Layer: https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-integration-lambda-extensions.html

Motivation

Preloading configurations (during cold start) automatically and utilizing AppConfig as intended - ongoing deployment of configurables. Having a background task handle resource management while persisting a client-id is very helpful and having the background task that acts as a keepalive for that client-id regardless of request requirements will assist with deployment fact gathering.

Proposal

Utilize urllib.request to read from a formated URL composed of several environment variables related to the AppConfig lambda layer. Should be a fairly straight forward update to existing appconfig code in parameters module. Some thought needs to go in to manual refresh and opting out for specific requests.

If AppConfig parameters helpers exist in the other languages powertools then it should be a fairly straight forward change.

User Experience

Enable the AWS AppConfig Lambda Layer

The only corner case for configuration would be providing sane defaults to opt-in/opt-out of using the layers HTTP endpoint rather than boto3 directly.

Drawbacks

This could muddy some thoughts on how TTLs are handled between powertools and the layer software.

No additional deps.

Rationale and alternatives

Low impact of not including this. If appconfig parameters are pulled per request the keepalive provided by the lambda layer would be redundant.

Unresolved questions

...

[heitorlessa]

Hi @whardier thanks for creating a RFC, much appreciated!

We do this today in the Parameters utility - by default we cache any parameter(s) for 5s: https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/parameters/#adjusting-cache-ttl

This ensures we cache a given parameter(s) in memory and only make a call to AppConfig in the next invocation if the TTL has expired. There are also cases where you might want to always fetch the latest available value for a given parameter whether it's in cache or not: https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/parameters/#always-fetching-the-latest

Until today, we were not exposing the TTL adjustment to higher level functions which might have contributed to believing this wasn't possible.

In the next release, you'll be able to go from this:

from aws_lambda_powertools.utilities import parameters
from botocore.config import Config

config = Config(region_name="us-west-1")
appconf_provider = parameters.AppConfigProvider(environment="my_env", application="my_app", config=config)

def handler(event, context):
    # Retrieve a single secret
    value: bytes = appconf_provider.get("my_conf", max_age=500)

To this:

from aws_lambda_powertools.utilities import parameters

def handler(event, context):
    # Retrieve a single configuration, latest version
    value: bytes = parameters.get_app_config(name="my_configuration", max_age=500, environment="my_env", application="my_app")

Does this help?

Look forward to hearing from you

[whardier]

Actually I do utilize the underlaying caching (and am leveraging it in a tool @ work). The RFC is meant to support the direct use of the AWS AppConfig Lambda Layer in order to pre-load and let it handle the client related aspects of that interface.

From AWS docs:

If you use AWS AppConfig to manage configurations for a Lambda function, then we recommend that you add the AWS AppConfig Lambda extension. This extension includes best practices that simplify using AWS AppConfig while reducing costs. Reduced costs result from fewer API calls to the AWS AppConfig service and, separately, reduced costs from shorter Lambda function processing times.

Powertools addresses the caching aspect for most use cases. There are some differences between the layer and powertools in this respect.

Using powertools along with the layer, as an option, would promote more frequent instantiations of the configuration as pattern where fetching the most recent configuration data is just a dip to a localhost daemon that is managing the configuration data.

I can appreciate that the daemon also communicates with appconfig as a daemon external to the runtime and that daemon has several configurable factors addressing retries and cache time as well. The daemon persists the client id interaction with AppConfig and deployments have a tighter view of the rollout.

Overall this is a simple thing to handle using the base parameters class as an add on.

[heitorlessa]

Understood - that's helpful, thank you!

My initial concern was supporting Extensions as part of the project as it's not trivial to debug and maintain yet.

I'll come back to this the week after next, so I concentrate on the next release with an important feature.

[heitorlessa]

Added labels to express we'd like to hear customer feedback before considering it. We had the same discussion on other Powertools and we didn't get much traction from other customers wanting AppConfig Extension support -- we're experiencing the other way around (start with Extension then move to Powertools/Custom code etc.)

[leandrodamascena]

Hi everyone! This issue has gone a long time without any user feedback and we are closing it.

As always, we love to hear from the community and if more customers are interested in this issue we can open it without problems and continue the discussion/implementation.

Thanks.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[Rogalek]

Hello. I wonder if is it possible to come back to this topic?

If I understand correctly this max_age parameter is only valid during lifecycle of lambda function and lambda layer allows you to cache that value between lifecycles. So new lambda do not need to fetch data from AppConfig because its fetched by layer. Is it correct?

In my application time of fetching data and return lambda is the essence. Flags will probably not change that often so caching during lifecycle its not that important for me.

Thanks!

[Rogalek]

@heitorlessa

I did run some tests for my application. I was trying to observe average request time with and without aws powertools of my lambda and its around 500 ms longer when you are using your lib. I think 500 ms is long time, that's why extension would be amazing here.

Tests without aws powertools feature-flags:

empty = []
for x in range(1000):
    response = requests.get(url, headers={"Authorization": f"Bearer {token}"})
    call_time = response.elapsed.total_seconds()
    empty.append(call_time)
    
min(empty)
0.541491
max(empty)
6.150538
sum(empty)/1000
0.8154316550000005

Tests with aws powertools feature-flags fetch inside lambda:

empty2 = []
for x in range(1000):
    response = requests.get(url, headers={"Authorization": f"Bearer {token}"})
    call_time = response.elapsed.total_seconds()
    empty2.append(call_time)
    
min(empty2)
0.928258
max(empty2)
11.054
sum(empty2)/1000
1.367485440999999

[heitorlessa]

Hey @Rogalek - could you share the code you've used for this simple test and what memory size you've used? Did you use two different functions and two different APIs?

500ms extra seems unreasonably long. We should look into it regardless.

I know feature flags needs a good refactor to improve overall performance but I'd only expect a AWS SDK call to be at worst 400ms for the first call (128M, Client initialize within the handler and not global scope, and maybe cross-region endpoint) -- there was a discussion on performance can't easily find it from mobile but it was largely memory size.

Setting status as triage so we don't miss it.

Thank you!

[Rogalek]

Hello - I cannot share my lambda code with you, but I can tell you how I was working with that stuff.

So I used the same function, deployed once with aws powertools and once without it. It was the same endpoint and the same API. Small difference between them was just a decorator above lambda which had call to AppConfig with powertools.

def validate_feature(name: str):
    def decorator(func):
        def wrapper(*args, **kwargs):
            settings = Settings()
            app_config = AppConfigStore(
                environment=settings.deployment_environment,
                application=settings.feature_application,
                name=name,
                sdk_config=Config(region_name=settings.aws_region),
            )
            feature_flags = FeatureFlags(store=app_config)
            check_feature_status = feature_flags.evaluate(name)
            if not check_feature_status(name, args[0]):
                request_id = args[0]["requestContext"]["requestId"]
                error_context = {
                    "id": request_id,
                    "code": 403,
                    "title": "Request forbidden",
                    "detail": f"Request forbidden - Feature [{name}] is not enabled for user",
                    "status": "403",
                }
                response = {
                    "statusCode": HTTPStatus.FORBIDDEN,
                    "headers": {"content_type": "application/vnd.api+json"},
                    "body": json.dumps({"errors": [error_context]}),
                }
                return response
            return func(*args, **kwargs)

        return wrapper

    return decorator

and later I used it on lambda or not use it.

@validate_feature("feature-name")
def lambda(event, context):
    pass

Lambda and AppConfig was inside the same region in AWS.

[leandrodamascena]

Hey @Rogalek! Thanks for explaining the scenario you're testing and being so helpful in getting us started on investigating this issue. I just wanted to give you a heads-up that I'll be diving into this starting on Thursday. It might take me a couple of days to have a position about this problem.

By the way, in the meantime, could you kindly let me know the memory configuration for your Lambda? It seems like you're using 128M, am I right?

Thank you

[Rogalek]

Lambda is 384 and API gateway is 128

[heitorlessa]

rescheduling this for mid next week. Depending on how p99 turns out, we might add temporary support for AppConfig Extensions. I say temporary, because this is a band-aid and an extension brings additional trade-offs on testing and operational overhead -- a long-term solution would be to have optimized SDK calls written in another language (e.g., Rust), since the bulk is coming from botocore session

[heitorlessa]

Happy with any PR to get temporary support for AppConfig Extensions - as expected it's AWS SDK during cold start (gunzip service model + runtime code generation + sigv4 + 2 distinct TLS handshake).

Next year, we will plan an optimized SDK for Powertools to address this and make other value add features more prevalent (e.g., correlation ID propagation, simplify AWS integrations).

Adding help wanted label for anyone who can help put a PR for this and we can get in the next release.

[walmsles]

@heitorlessa thought I would add another element to the discussion re: AWS SDK being "heavy". Over in the Node world, Begin has created aws-lite as a super-fast alternative to the AWS SDK.

[heitorlessa]

yeah, this is a bigger challenge in and of itself. Given how many billions of integrations we run a week, we'd need to be extremely careful on what we do here. Future wise, there's no doubt that we will be looking at Rust for Python and TS for bindings on AWS SDK (and some more).. but that requires a large investment we can't afford this year.

[adriantomas]

I believe this may now be possible after the v2.41.0 release.

Considering that the Lambda has the layer attached:

boto3_client = boto3.client("appconfigdata", endpoint_url="localhost:2772")

app_config = AppConfigStore(
    environment="dev",
    application="product-catalogue",
    name="features",
    boto3_client=boto3_client,
    max_age=0,  # The layer is already caching
)

I haven't tested it anyway