Ability to zip AWS Lambda function on the fly

[StyleT]

Hi there,

We're using small AWS Lambda functions to perform routine operations on our AWS infrastructure. We have separate aws_lambda module which contains configuration & source code of all our functions.
Currently the only way to upload new version of Lambda to AWS is to run some "packer" script before terraform launch to update zip file with function. I would like to do this without any pre-terraform scripts to keep TF usage simple.

Suggested steps

• add output variable result to the local-exec provisioner so it can zip lambda source code and calc value for source_code_hash property of aws_lambda_function
• add interpolation function which will recursively calc hash of folder content. This will allow us to trigger null_resource on folder change. Right now it's possible to track changes only in specific files.

Resulted TF config

resource "null_resource" "lambda" {
  provisioner "local-exec" {
    command = "${path.module}/source/build.sh"
  }

  triggers = {
    source_file = "${sha1Folder("${path.module}/source")}"
  }
}

resource "aws_lambda_function" "lambda" {
  filename = "${path.module}/build/lambda_source.zip" //file ignored in .gitignore
  function_name = "sample"
  role = "lambda_role"
  runtime = "nodejs4.3"
  handler = "index.index"
  source_code_hash = "${null_resource.lambda.result}"

  depends_on = ["null_resource.lambda"]
}

Terraform Version

0.7.0

Affected Resource(s)

Please list the resources as a list, for example:

• aws_lambda_function
• null_resource
• local-exec

Important Factoids

Also addition of result variable to local-exec provider will add a huge amount of new ways to use terraform

References

• aws_lambda_function: source_code_hash argument expects a zip file to be present #6513
• Proposal: "external" provider #8144

[StyleT]

Closed if favour to GH-8144

[dkniffin]

For anyone else who finds this issue, I found a good solution to this, using the archive_file data source:

data "archive_file" "lambda_zip" {
    type        = "zip"
    source_dir  = "source"
    output_path = "lambda.zip"
}

resource "aws_lambda_function" "my_lambda" {
  filename = "lambda.zip"
  source_code_hash = "${data.archive_file.lambda_zip.output_base64sha256}"
  function_name = "my_lambda"
  role = "${aws_iam_role.lambda.arn}"
  description = "Some AWS lambda"
  handler = "index.handler"
  runtime = "nodejs4.3"
}

This will zip up the source directory, creating a lambda.zip file in the process, and then you can use data.archive_file.lambda_zip.output_base64sha256 to get a sha of the zip to tell the resource when to update.

Edit: I wanted to add one more note here. I thought about this, and realized terraform is not necessarily the ideal way to do this. Essentially, this is deploying code, and most of the time, you wouldn't want to do that via a configuration management system like terraform. So, for my project, I've decided to deploy the lambda using apex, and then reference it in terraform using a static ARN.

[awilkins]

D'oh! There's an archive_file task... I was doing this with an external data source...

Edit : why doesn't the archive_file task have output_path as an exported attribute? How do you establish dependency when using the outputted file?

Edit : ah, ok, with output_sha

[hSATAC]

Thanks @dkniffin

This should be added into the official document

https://www.terraform.io/docs/providers/aws/r/lambda_function.html

[xoen]

Few notes on this (for the benefit of who's trying to achieve this):

• Yes the archive_file resource is very useful to zip the lambda package
• The local-exec provisioner is still useful if the build.sh is installing 3rd party dependencies (e.g. npm install / pip install)
• I found a big caveat of using the triggers to trigger the rebuild: If the source code doesn't change and another member of the team run plan/apply, the local-exec provisioner will not run => dependencies will not be installed => they'll not be part of the lambda zip. This is because (as far as I can tell, but feel free to correct me if I'm wrong) the calculated attribute values are compared against the version on the remote state.

I don't know of any way of running the local-exec provisioner only when necessary. The non-ideal solution is to have a force_rebuild = "${timestamp()}" trigger...

[pecigonzalo]

Some other ideas here:

resource "null_resource" "pip" {
  triggers {
    main         = "${base64sha256(file("source/main.py"))}"
    requirements = "${base64sha256(file("source/requirements.txt"))}"
  }

  provisioner "local-exec" {
    command = "./ci/pip.sh ${path.module}/source"
  }
}

data "archive_file" "source" {
  type        = "zip"
  source_dir  = "${path.module}/source"
  output_path = "${path.module}/source.zip"

  depends_on = ["null_resource.pip"]
}

resource "aws_lambda_function" "source" {
  filename         = "source.zip"
  source_code_hash = "${data.archive_file.source.output_base64sha256}"
  function_name    = "lamda"
  role             = "${aws_iam_role.lambda.arn}"
  handler          = "main.handler"
  runtime          = "python2.7"
  timeout          = 120

  environment {
    variables = {
      HASH             = "${base64sha256(file("source/main.py"))}-${base64sha256(file("source/requirements.txt"))}"
    }
  }

  lifecycle {
    ignore_changes = ["source_code_hash"]
  }
}

This prevents unnecessary deployments unless hash of the sources has changed, only works for simple configurations unless you add a couple more steps.
Its ugly but it gets the job done for now.

🤷‍♂️

[phuonghuynh]

@pecigonzalo whats this use for?

environment {
    variables = {
      HASH             = "${base64sha256(file("source/main.py"))}-${base64sha256(file("source/requirements.txt"))}"
    }
  }

[pecigonzalo]

Just tracking the hashes.

[digitalkaoz]

thats nearly what i have in mind...works but its a bit ugly:

sha1Folder would be awesome!

[apparentlymart]

Hi all,

Thanks for sharing approaches with archive_file here!

When AWS Lambda was first released, there was no means for varying settings between multiple deployments of the same function code (e.g. between staging and production environments) and so a common pattern was to construct the necessary zip file "just in time" in Terraform so that per-environment settings could be embedded in the generated zip file.

With the addition of environment variables we now recommend adopting a more conventional strategy of building a single, environment-agnostic artifact zip file and uploading it to S3 as a separate step prior to running Terraform, and then use Terraform only to update the function to use the newly-created artifact.

This is analogous to building immutable AMIs using packer and then deploying them across many envirnoments with Terraform, or building environment-agnostic Docker images and then passing in environment variables with Terraform when launching containers.

---

A common pattern I have seen is for the build process (ideally running in a CI system) to produce an S3 object within a well-known artifact bucket using a systematic naming convention for each build:

my-application/v1.0.0/batch-function.zip
my-application/v1.0.0/api-function.zip
my-application/v1.0.1/batch-function.zip
my-application/v1.0.1/api-function.zip
my-application/v1.1.0/batch-function.zip
my-application/v1.1.0/api-function.zip
# ... etc

Then pass a version number into the Terraform configuration somehow (e.g. via an input variable, via Consul, etc) and have the aws_lambda_function resource construct the path using the expected convention:

resource "aws_lambda_function" "api" {
  function_name = "MyApplicationAPI"

  s3_bucket = "mycompany-build-artifacts"
  s3_key    = "my-application/${var.app_version}/api-function.zip"

  role    = "${aws_iam_role.lambda.arn}"
  handler = "main.handler"
  runtime = "python2.7"
  timeout = 120

  environment {
    variables = {
      # For example, pass the ARN of a per-environment SNS topic created
      # elsewhere in this config.
      SNS_TOPIC_ARN = "${aws_sns_topic.example.arn}"
    }
  }
}

By using a distinct S3 object path for each new build and treating existing artifacts as immutable, we avoid the need to track sha256 hashes of the builds: s3_key changes each time the version number changes, and thus triggers a deployment.

This gives the usual benefits of immutable build artifacts:

• You can roll back to a previous version if a bug is found shortly after deployment, since the previous immutable artifact is still present in the S3 bucket.
• You can deploy an artifact to your staging environment, test it, and then deploy an identical artifact to your production environment, and thus have confidence that the only difference will be in the config that is passed to the code via the environment.

---

I'm glad that some of you have managed to achieve your goals with Terraform, but please note that Terraform is not intended to be a build tool and so this sort of use-case will always be a little clunky and limiting when implemented in Terraform. For most common situations I would strongly recommend following the conventional "build once, deploy many times" best-practice for Lambda code artifacts, and use Terraform only for deployment.

[bradleyayers]

Using environment works well for Lambda, but unfortunately Lambda@Edge doesn't support environment variables so we're back to square one for those. It's definitely awkward trying to combine a build tool and terraform, when you want parameterised builds based on terraform state.

Lambda@Edge support for environment variables can't come soon enough.

[ozbillwang]

I just realised the content in ./ci/pip.sh is so simple.

$ cat pip.sh
#!/bin/sh

cd $1
pip install -r requirements.txt -t .

For mac user, you may need create below file under folder source

$ cat source/setup.cfg
[install]
prefix=

Refer: https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

[fishpen0]

With the addition of environment variables we now recommend adopting a more conventional strategy of building a single, environment-agnostic artifact zip file and uploading it to S3 as a separate step prior to running Terraform, and then use Terraform only to update the function to use the newly-created artifact.

This solution unfortunately results in a chicken-egg issue where I cannot create a bucket in the same terraform configuration that I create my lambda in since the bucket wouldn't yet have the lambda file in it.

Choosing to upload the zip file directly from terraform lets me solve for this problem by no longer needing to rely on a bucket already existing outside of my current terraform apply context.

[zioalex]

Great it worked perfectly also with go:

resource "null_resource" "build_lambda_exec" {
  triggers = {
    source_code_hash = "${filebase64sha256("${path.module}/lambda_sns_slack_integration/main.go")}"
  }
  provisioner "local-exec" {
    command     = "${path.module}/lambda_sns_slack_integration/build.sh"
    working_dir = "${path.module}/lambda_sns_slack_integration/"
  }
}

data "archive_file" "main_go" {
  type        = "zip"
  source_file  = "${path.module}/lambda_sns_slack_integration/main"
  output_path = "${path.module}/lambda_sns_slack_integration/deployment.zip"

  depends_on = ["null_resource.build_lambda_exec"]
}

resource "aws_lambda_function" "lambda_to_slack" {

  filename      = "${path.module}/lambda_sns_slack_integration/deployment.zip"
  function_name = "sns_to_slack"
  role          = "${aws_iam_role.iam_for_lambda.arn}"
  handler       = "main"

  source_code_hash = "${data.archive_file.main_go.output_base64sha256}"
  runtime = "go1.x"
  # depends_on = ["null_resource.build_lambda_exec"]

  environment {
    variables = {
      SLACK_WEBHOOK = "https://hooks.slack.com/services/XXXXXXXXXXXXXXXXXXXXXXXXXXX"
    }
  }
}

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.