From d3cab3bd2c65afe9af8d4e8bcd100ac3b0facc76 Mon Sep 17 00:00:00 2001 From: Chao Li Date: Wed, 20 Nov 2019 16:28:40 +0800 Subject: [PATCH 1/2] RFC: input_vars Signed-off-by: Chao Li --- 042-input-vars/proposal.md | 181 +++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 042-input-vars/proposal.md diff --git a/042-input-vars/proposal.md b/042-input-vars/proposal.md new file mode 100644 index 00000000..7ab3193b --- /dev/null +++ b/042-input-vars/proposal.md @@ -0,0 +1,181 @@ +# Summary + +This proposal outlines a method that a task takes vars from a previous steps. + + +# Motivation + +The only way to pass data from a step to following steps is via files. Because +of that, resource types and tasks always need extra code to read files; also +resource types always need to define duplicate params like `xxxx` and `xxxx_file`. +One example is, the email resource type's `put` takes a parameter `subject` and +a duplicate parameter of `subject_file`. + + +# Proposal + +This proposal is introducing a general configuration, named `input_vars` to +steps `get`, `put` and `task`, where a `input_vars` make an input file from +a previous step to be a var source. + +A sample usage: + +```yaml +jobs: +- name: demo + plan: + - task: generate-email-data + config: + platform: linux + image_resource: + type: registry-image + source: {repository: busybix} + outputs: + - name: data + run: + # generate a json file named e.json, like: + # { "subject": "something", "body": "some body"} + +- put: email + input_vars: + - name: email_data + file: data/e.json + params: + subject: ((email_data:subject)) + body: ((email_data:body)) +``` + +In the above sample, the task no longer needs to generate multiple files for +email "subject" and "body", maybe even "to", "cc", e.g.. The resource `email` +no longer needs to read files by itself. Also, the pipeline code looks more +clear. + +## Syntax + +A `input_vars` is an array, each element contains two attributes: + +* `name` is used by following steps to refer to this `input_vars`. +* `file` points to a file that is generated by a previous step. +* `file` can be in `json` or `yaml` format. The file will be used as same way +as `-l` option of `fly set-pipeline`. +* `file` can also be a plain text file. Many existing resource types generate +plan text files. For example, the `git-resource` store a commit SHA in a file. +If `file` is not suffixed by `.yml`, `.yaml` or `json`, it should be treated +as a plain text file. Then following steps should refer to content of the file +as `((varname:value))`, where `value` is a reserved keyword. + + +`input_vars` should not be defined to a resource type and a resource. The +following usages are invalid: + +```yaml +resource_types: +- name: rt-tiny + type: registry-image + input_vars: # INVALID !!!! + - name: some-name + file: some-file + source: + repository: evanli/tiny2 + tag: v2 + +resources: +- name: tiny + type: rt-tiny + check_every: 2m + input_vars: # INVALID !!!! + - name: some-name + file: some-file + source: + key: ((dev-vault:hello)) +``` + + +## `get` step + +For a `get` step, `input_vars` should be defined at same level as `params`. The +input vars will only be used to interpolate `params` of the `get`. + +```yaml +- get: some-resource + input_vars: + - name: some-name + file: a-file-generated-by-a-previous-step + params: + p1: ((some-name:var)) +``` + +## `put` step + +For a `put` step, `input_vars` should be defined at same level as `params`. +Similar to `get_params`, `get_input_vars` should also be supported for implied +`get`. The input vars will only be used to interpolate `params` of `put` and +the implied `get`. + +```yaml +- put: some-resource + input_vars: + - name: name1 + file: a-file-generated-by-a-previous-step + params: + p1: ((name1:var1)) + get_input_vars: + - name: name2 + file: a-file-generated-by-a-previous-step + get_params: + p2: ((name2:var2)) +``` + +## `task` step + +For a `task` step, `input_vars` should be defined at same level as `input` and +`outputs`. The input vars will only be used to interpolate task config. + +```yaml +- task: some-task + config: + platform: linux + image_resource: + type: registry-image + source: {reposity: some-image} + inputs: + - some-input + output: + - some-output + input_vars: + - name: invar1 + file: a-file-generated-by-a-previous-step + - name: invar2 + file: a-file-generated-by-a-previous-step + run: + path: sh + args: + - -exc + - | + echo ((invar1:msg1)) + echo ((invar2:msg2)) +``` + +## `set_pipeline` step + +I'm not seeing an use case, `set_pipeline` may not support `input_vars`. + +## `check` step + +`input_vars` is designed to support workflow, and `check` is not a step in +workflow, so `check` step should not take input vars. + + +# Open Questions + + + +# Answered Questions + + + +# New Implications + +* Once `input_vars` is supported, most of resource types will no longer need to +duplicate defining `xxx_file` parameters, which would make interface of resource +types neater. From 80df70975e5fe85746aab5d39f4852b7fce26989 Mon Sep 17 00:00:00 2001 From: Chao Li Date: Mon, 16 Dec 2019 21:20:42 +0800 Subject: [PATCH 2/2] Add section "Restriction on names". Signed-off-by: Chao Li --- 042-input-vars/proposal.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/042-input-vars/proposal.md b/042-input-vars/proposal.md index 7ab3193b..ebf24c5f 100644 --- a/042-input-vars/proposal.md +++ b/042-input-vars/proposal.md @@ -50,6 +50,15 @@ email "subject" and "body", maybe even "to", "cc", e.g.. The resource `email` no longer needs to read files by itself. Also, the pipeline code looks more clear. +## Restriction on names + +Each `input_vars` in a pipeline should have an unique name, and name should not +duplicate to any `var_source` name either. Because the syntax of a var, +`((source_name:var_name))`, is not able to tell if a `source_name` is +`var_source` or `input_vars`. So that duplicate names between `var_source` and +`input_vars` will make implementation to not be able decide from which source +to fetch value for the var. + ## Syntax A `input_vars` is an array, each element contains two attributes: