Environment variable in serverless.yml throws "Cannot resolve ..." error

[tforster]

The latest version of the serverless framework is no longer working for AWS Lambda deployments and is throwing the following error:

Cannot resolve serverless.yml: “provider.stage” property is not accessible (configured behind variables which cannot be resolved at this stage)


service: <redacted>
unresolvedVariablesNotificationMode: error
useDotenv: true
variablesResolutionMode: 20210326

provider:
  name: aws
  region: ${env:AWS_LAMBDA_REGION} 
  stage: ${env:AWS_STAGE}
  runtime: nodejs14.x
  lambdaHashingVersion: 20201221
  memorySize: 512
  timeout: 6
  profile: <redacted>
  
  httpApi:
    cors:
      allowedOrigins:
        - "*"
      allowedHeaders:
        - "*"
      allowedMethods:
        - GET
        - PUT 
        - POST 
        - DELETE
        - OPTIONS
      #allowCredentials: true
      exposedResponseHeaders:
        - "*"
      maxAge: 6000 # In seconds

  websocketsApiRouteSelectionExpression: $request.body.action

  iamRoleStatements:
    - Effect: 'Allow'
      Action:
        - dynamodb:PutItem 
        - dynamodb:BatchWriteItem
        - dynamodb:PartiQLInsert
        - dynamodb:PartiQLUpdate
        - dynamodb:PartiQLSelect
      Resource: 'arn:aws:dynamodb:ca-central-1:<redacted>:table/<redacted>-${env:AWS_STAGE}'
    - Effect: Allow
      Action:
        - "execute-api:ManageConnections"
      Resource:
        - "arn:aws:execute-api:*:*:**/@connections/*"


functions:
  # Read all users
  readUsers:
    handler: Handler.readUsers
    events:
      - httpApi:
          method: GET
          path: /users
          
  # Read individual user
  readUser:
    handler: Handler.readUser
    events:
      - httpApi:
          method: GET
          path: /users/{userId}

  # Exchange a Google issued JWT for a <redacted> JWT
  exchangeJWT:
    handler: Handler.exchangeJWT
    events:
      - httpApi:
          method: GET
          path: /users/jwt


  # Handle WebSocket (dis)connections
  connect:
    handler: Handler.connect
    events:
      - websocket:
          route: $connect
      - websocket:
          route: $disconnect

  # Close a class
  closeClass:
    handler: Handler.closeClass
    events:
      - websocket:
          route: closeClass

plugins:
  - serverless-offline
  - serverless-dotenv-plugin

package:
  include:
    - ../<redacted>.pem

AWS_PROFILE= npx serverless deploy output

 
 Serverless Error ----------------------------------------
 
  Cannot resolve serverless.yml: "provider.stage" property is not accessible (configured behind variables which cannot be resolved at this stage)
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              14.15.4
     Framework Version:         2.52.0 (local)
     Plugin Version:            5.4.3
     SDK Version:               4.2.3
     Components Version:        3.13.4
 

Installed version

Framework Core: 2.52.0 (local)
Plugin: 5.4.3
SDK: 4.2.3
Components: 3.13.4

[pgrzesik]

Hello @tforster 👋 Thanks for reporting, with the new resolver, such definition is not supported. In general, it is discouraged to configure stage behind env variables for example, as at the point where stage is going to be resolved, not whole env might be available (e.g. loading env vars from .env.{stage} needs to resolve stage first in order to properly load variables from file), which might introduce bugs that are hard to debug. Also, the provider.stage serves more as a "default" stage and --stage flag via CLI is the preferred way of setting it. Do you think you can safely switch to use CLI --stage flag for specifying stage?

[tforster]

What is "the new resolver"? And is there any documentation about it? Does it affect us in any other ways?

I don't understand why we are now discouraging the use of environment variables in serverless.yml. They've been fine for years.

While I can switch to --stage flag I would rather not. Since the inception of the Serverless Framework we have evolved an extensive set of scripts that rely on our stage environment variable to be able to function across dev, stage and prod. Rearchitecting this and porting it to dozens of mature projects is going to be a real PITA.

Is there an official document on this breaking change?

[pgrzesik]

Hello again @tforster 👋

In your configuration file you explicitly opt-in to use new resolver via variablesResolutionMode: 20210326 variable. It is mentioned here https://www.serverless.com/framework/docs/deprecations/#new-variables-resolver as well as here whenever it applies: https://www.serverless.com/framework/docs/providers/aws/guide/variables/

We are not discouraging the use of env variables - quite the contrary, we've been promoting them as a replacement for custom CLI options for example and it is generally a great practice to use them. As for env source for stage specifically, this has been introduced as a fix, as stage should be already resolved before we attempt env variables resolution, as loading .env files can depend on stage property. @medikoo I know we've talked about it today, do you think it could be safe to resolve stage from env source in specific circumstances (e.g. when dotenv is not used)?

[tforster]

Hi @pgrzesik Thanks for the ongoing explanation, much appreciated. Yes, I did add variablesResolutionMode: 20210326. It was part of my Google attempt at trying to resolve the broken deployments I started encountering. The explanation of what it does in the #new-variables-resolver link was terse, but when I came across it, I interpreted it to get more verbose error messages since "Cannot resolve serverless.yml: “provider.stage” property is not accessible" didn't really explain what the issue was.

If it's not intended to trigger verbose errors what is it for? Should I remove it?

[medikoo]

@medikoo I know we've talked about it today, do you think it could be safe to resolve stage from env source in specific circumstances (e.g. when dotenv is not used)?

Theoretically we can do that. Probably best way would be to require stage to be resolved at this stage if (1) dotenv is enabled and (2) there are some .env* files. The only problem I have with it, is performance concern, as we need to additionally scan dir to confirm on that.

If we agree to that, also then error message will have to be further tuned, to clearly indicate that stage requires to be resolved at that stage as .env* files are used. Otherwise we will open door for ugly UX issue, where we may have two users running same service, one seeing the error and other not. Upfront, if error message will not be verbose on that, it'll be quite hard for them to deduct why exactly it is the case.

---

@tforster old variables resolver didn't have error validation on that, and in such scenarios quite often internals silently worked with not resolved at given point crucial properties as stage. So when it may feel as limitation of new resolver it's actually result of it having the proper error handling, that prevents such situations from happening

[pgrzesik]

Thanks @medikoo - while theoretically, it seems like we could do that, it introduces an additional overhead and adds a bit to the complexity of variable resolution. I would rather avoid introducing such changes, but it's of course up for consideration.

[medikoo]

adds a bit to the complexity of variable resolution. I would rather avoid introducing such changes, but it's of course up for consideration.

+1

Our move should probably depend on demand. With strong demand I would do it, otherwise let's not do our logic more complex

[tforster]

Recognising the incredible effort that everyone already puts into this project, I would offer that an improved description of the resolution process would be extremely valuable. SLS is both powerful and flexible and it is clear to me now that I use and configure my projects differently than others and the resolver attempts to handle all our scenarios.

I would love to see a collection of official best-practices snippets for various aspects of SLS. I say official because if it is just a public repo it will be quickly flooded by conflicting patterns because everyone has an opinion.

Based on this thread I am considering rearchitecting our project scripts somewhat. They would still use a .env file but I am not sold on .env.dev, .env.stage, etc. Our boilerplate .gitignore includes .env since it contains secrets. But I am concerned about adding additional environment files with inconsistent extensions. To me, that feels as if there is potential to miss something and leak secrets. I am curious, was any consideration given to reversing the naming convention to .dev.env, .stage.env, etc?

Thank you @pgrzesik and @medikoo for your responses to this thread.

[medikoo]

I am curious, was any consideration given to reversing the naming convention to .dev.env, .stage.env, etc?

@tforster what would that solve exactly?

[tforster]

Without the .env extension, they are no longer recognised as .env files by the developer's IDE or code editor. It means that a simple *.env .gitignore pattern has to be replaced with multiple entries, increasing the risk if one is missed. Code prettiers, linters and documentation tools will not pick them up.

[medikoo]

@tforster that's an interesting suggestion, but I think we'd rather consider dropping support for .env.<stage> at all, as such practice is not recommended by most of the prolific sources.

[tforster]

@medikoo I personally would not use env.<stage>. I would ensure my devops scripts could work with and manage a single .env so dropping support for .env.<stage> definitely won't affect me.

[medikoo]

We will need a bigger discussion on that, as I think many users are used to have different env configs per stage (even though that doesn't seem to be best practice).

We can either rename this issue, or create a new one (feel free to open), so feedback is gathered there

[tforster]

I would say that this issue can be closed as it's not a bug, but behaving by design?

[medikoo]

I would say that this issue can be closed as it's not a bug, but behaving by design?

Yes, that's the case currently, so let me close it.

Still if you'd like use to drop support fro .env.<stage> variant, please open an issue with good reasoning, we will definitely put that for discussion,

[medikoo]

@tforster due to other requests, we've actually revisited that subject looked into it more deeply, and realized we can provide some sort of compelling solution without introducing either not optimal processing or changes to functionalities.

Please track #9880 for that, I believe it should be addressed soon

[medikoo]

@tforster good news! Starting with today's release we will support env variables in provider.stage.

[debragail]

@tforster good news! Starting with today's release we will support env variables in provider.stage.

Can you go into detail what this means? I am still struggling with the "configured behind variables which cannot be resolved at this stage" error even after adding the variablesResolutionMode: 20210326.

[medikoo]

@debragail ensure you're relying on latest Framework release. Otherwise provide a reproducible test case (that doesn't involve plugins, and is simplified to minimum) which exposes the issue

[DanielWFrancis]

I just updated and my deployment broke, what's the dealio

It feels like this is changed for no reason. I saw warnings for a long time but never thought I would have to update... never thought it would happen to me....

[dorman99]

I just updated and my deployment broke, what's the dealio

It feels like this is changed for no reason. I saw warnings for a long time but never thought I would have to update... never thought it would happen to me....

same as Daniel, still got the error tho

[nicemaker]

Do I understand this correctly that also custom plugins with custom variable resolver can't be used for provider.stage?
e.g.:

 provide:
    stage: ${opt:stage,"dev"}-${foo:versionSlug}

We wrote custom plugin that reads the microservice version from an external file, with some parsing (version property from node package.json).
Like this we could use the npm semantic versioning each time we want to spin a complete decoupled stack

[medikoo]

@nicemaker provider.stage plays a part in stage resolution, which has to be resolved at initialization stage, So yes you cannot alter stage value from a plugin (technically you can try, via hacks, but it's likely to put important logic parts out of sync).

Still, we've proposed a solution that allows to inject into Framework process before the initialization: #9311 and through that, it could be possible

[berni-mad]

Hello @tforster 👋 Thanks for reporting, with the new resolver, such definition is not supported. In general, it is discouraged to configure stage behind env variables for example, as at the point where stage is going to be resolved, not whole env might be available (e.g. loading env vars from .env.{stage} needs to resolve stage first in order to properly load variables from file), which might introduce bugs that are hard to debug. Also, the provider.stage serves more as a "default" stage and --stage flag via CLI is the preferred way of setting it. Do you think you can safely switch to use CLI --stage flag for specifying stage?

It should not be the case where a new release cause the app to break in various ways. Why couldn't you guys just deprecate it ? Upgrading serverless is a nightmare.

[medikoo]

@berni-mad what exactly got broken for you?

Note all we did in the Framework is we prevented broken configurations to be taken as valid (that created not always visible internal issues, which in many cases produced confusing outcomes)

[berni-mad]

@medikoo

Issues with the code:

• The environment variables are no longer found by serverless framework. For instance, most of the variables we defined using the syntax self: no longer work. Using sls: and adding a fallback value like null helps, but only in some cases.

• When executing yarn run serverless create-cert --stage $(STAGE) --region $(REGION) --conceal , the option --conceal does not work, although it still can be found in the official documentation.

We were having none of those issues with version 2.

---

Issues with documentation:

• All the examples provided in the serverless official documentation use serverless version <=2. For instance

  https://github.com/serverless/examples/blob/master/aws-dotnet-rest-api-with-dynamodb/src/DotNetServerless.Lambda/serverless.yml

• Serverless documentation does not have any kind of version switch where you can be sure which documentation version are you following. It seems to always be a mix and absolutely everything must be checked by trial and error.

Thanks.

[medikoo]

@berni-mad thanks for input

The environment variables are no longer found by serverless framework.

@berni-mad that's definitely not the case. Can you prepare a minimal test case demonstrating what case exactly is not working for you? (ideally if you open a new issue)

When executing yarn run serverless create-cert......

This is an issue of a plugin which introduces create-cert command, and which for some reason wasn't upgraded for v3. Simple fix is to configure command schema for it. Have you reported the issue at plugin repository?

In some cases, when defining an intrinsic value function like Fn::GetAtt, the following error is thrown..

Validation schema does not allow CF templates syntax for properties which are not submitted as is to CF template, or play part in internal resolution of other configuration parts.
If you feel this doesn't relate to your case, please open a dedicated issue, describing exactly which property is affected and presenting a reproducible test case.
We had a few cases where indeed configuration schema could be relaxed to allow CF intrinsic functions.

[medikoo]

All the examples provided in the serverless official documentation use serverless version <=2. For instance https://github.com/serverless/examples/blob/master/aws-dotnet-rest-api-with-dynamodb/src/DotNetServerless.Lambda/serverless.yml

Default branch for the examples is v3 -> https://github.com/serverless/examples (master contains v2 version of examples) - and over there I believe everything is up to date.

Serverless documentation does not have any kind of version switch where you can be sure which documentation version are you following. It seems to always be a mix and absolutely everything must be checked by trial and error.

That's indeed a shortcoming of a website. Still all documentation is hosted and versioned in a repository at https://github.com/serverless/serverless/tree/main/docs.
So to inspect documenation of specific version, you can inspect contents of this folder at given tag

[berni-mad]

@medikoo Thanks a lot for your replies!

-> I was able to solve all the problems related with environment variables, indeed sls: is working in every case.

-> For the other issues, we have a very big serverless project and your observation might indeed be true. The plugin logic gets really entangled with the serverless framework , so some errors might be due to a dependency issue.

-> Thanks for the examples.

[kenny1323]

#/mydir_py3_11/django_aws_lambda_002/django-aws-lambda/serverless.yaml

service: django-aws-lambda

plugins:
  - serverless-dotenv-plugin
  - serverless-prune-plugin
  - serverless-python-requirements
  - serverless-wsgi
useDotenv: true
variablesResolutionMode: 20210326

custom:
  dotenv:
    logging: false
  pythonRequirements:
    dockerizePip: non-linux
    zip: true
    fileName: requirements.txt
  stage: ${env:STAGE, 'default'}
  wsgi:
    app: django_aws_lambda.wsgi.application
    packRequirements: false
  prune:
    automatic: true
    number: 3

functions:
  - app:
      handler: wsgi_handler.handler
      events:
        - http: ANY /
        - http: ANY /{proxy+}
      timeout: 30

provider:
  name: aws
  role: arn:aws:iam::198784529794:role/exec_lambda
  profile: admin
  region: us-east-1
  runtime: python3.8
  versionFunctions: false
  stage: ${env:STAGE, 'default'}
  timeout: 60
  vpc:
    securityGroupIds:
      - sg-e2a9ddd5
      - sg-01d23aa5252981a2b
    subnetIds:
      - subnet-5751da76
      - subnet-36aa2369
      - subnet-65ea6603
      - subnet-3927f708
      - subnet-31623e7c
      - subnet-ff4f09f1
  deploymentBucket:
    name: ${env:DEPLOYMENT_BUCKET}
  apiGateway:
    shouldStartNameWithService: true
  lambdaHashingVersion: 20201221

package:
  individually:
    true
  exclude:
    - .cache/**
    - .env
    - .git/**
    - .github/**
    - .pytest_cache/**
    - .serverless/**
    - node_modules/**
    - static/**
    - db.sqlite3

---

#/mydir_py3_11/django_aws_lambda_002/django-aws-lambda/.envl

SLS_DEBUG=*
DB_HOST=
DB_USER=
DB_PASSWORD=
DB_NAME=
DJANGO_SECRET_KEY=
AWS_S3_CDN_DOMAIN=
AWS_S3_REGION_NAME=
AWS_STORAGE_BUCKET_NAME='bucket2066668n01'
DEPLOYMENT_BUCKET='bucket202308n01'
AWS_KEY_ID=
AWS_SECRET=
DJANGO_ADMIN_URL=
DJANGO_ALLOWED_HOSTS=

---

#bash shell 

cd #/mydir_py3_11/django_aws_lambda_002/django-aws-lambda/

serverless deploy -s production

[kailashj]

You can remove entries from provider section and pass that as an option while while deploying or offline. See the code snippet below how i achieved it without major changes.

provider:
name: aws
runtime: nodejs18.x
region: ${self:custom.config.appRegion}
deploymentBucket: ${self:custom.config.deploymentBucket}

custom:
config: ${file(config/${opt:stage}.yml)}

Created 3 files for dev, uat and prod.yml and provided --stage option as below

sls offline start --stage dev
sls deploy --stage uat
sls deploy --state prod