RFC: input_vars

042-input-vars/proposal.md

@@ -0,0 +1,190 @@
+# Summary
+
+This proposal outlines a method that a task takes vars from a previous steps.
+
+
+# Motivation
+
+The only way to pass data from a step to following steps is via files. Because
+of that, resource types and tasks always need extra code to read files; also 
+resource types always need to define duplicate params like `xxxx` and `xxxx_file`.
+One example is, the email resource type's `put` takes a parameter `subject` and
+a duplicate parameter of `subject_file`.
+
+
+# Proposal
+
+This proposal is introducing a general configuration, named `input_vars` to 
+steps `get`, `put` and `task`, where a `input_vars` make an input file from
+a previous step to be a var source.
+
+A sample usage:
+
+```yaml
+jobs:
+- name: demo
+  plan:
+  - task: generate-email-data
+    config:
+      platform: linux
+      image_resource:
+        type: registry-image
+        source: {repository: busybix}
+      outputs:
+      - name: data
+      run:
+        # generate a json file named e.json, like:
+        # { "subject": "something", "body": "some body"}
+
+- put: email
+  input_vars:
+  - name: email_data
+    file: data/e.json
+  params:
+    subject: ((email_data:subject))
+    body: ((email_data:body))
+```
+
+In the above sample, the task no longer needs to generate multiple files for
+email "subject" and "body", maybe even "to", "cc", e.g.. The resource `email`
+no longer needs to read files by itself. Also, the pipeline code looks more
+clear.
+
+## Restriction on names
+
+Each `input_vars` in a pipeline should have an unique name, and name should not
+duplicate to any `var_source` name either. Because the syntax of a var, 
+`((source_name:var_name))`, is not able to tell if a `source_name` is 
+`var_source` or `input_vars`. So that duplicate names between `var_source` and
+`input_vars` will make implementation to not be able decide from which source
+to fetch value for the var.
+
+## Syntax
+
+A `input_vars` is an array, each element contains two attributes: 
+
+* `name` is used by following steps to refer to this `input_vars`.
+* `file` points to a file that is generated by a previous step. 
+* `file` can be in `json` or `yaml` format. The file will be used as same way
+as `-l` option of `fly set-pipeline`.
+* `file` can also be a plain text file. Many existing resource types generate
+plan text files. For example, the `git-resource` store a commit SHA in a file.
+If `file` is not suffixed by `.yml`, `.yaml` or `json`, it should be treated
+as a plain text file. Then following steps should refer to content of the file
+as `((varname:value))`, where `value` is a reserved keyword.
+
+
+`input_vars` should not be defined to a resource type and a resource. The 
+following usages are invalid:
+
+```yaml
+resource_types:
+- name: rt-tiny
+  type: registry-image
+  input_vars:  # INVALID !!!!
+  - name: some-name
+    file: some-file
+  source:
+    repository: evanli/tiny2
+    tag: v2
+
+resources:
+- name: tiny
+  type: rt-tiny
+  check_every: 2m
+  input_vars:  # INVALID !!!!
+  - name: some-name
+    file: some-file
+  source:
+    key: ((dev-vault:hello))
+```
+
+
+## `get` step
+
+For a `get` step, `input_vars` should be defined at same level as `params`. The
+input vars will only be used to interpolate `params` of the `get`.
+
+```yaml
+- get: some-resource
+  input_vars:
+  - name: some-name
+    file: a-file-generated-by-a-previous-step
+  params:
+    p1: ((some-name:var))
+```
+
+## `put` step
+
+For a `put` step, `input_vars` should be defined at same level as `params`. 
+Similar to `get_params`, `get_input_vars` should also be supported for implied
+`get`. The input vars will only be used to interpolate `params` of `put` and 
+the implied `get`.
+
+```yaml
+- put: some-resource
+  input_vars:
+  - name: name1
+    file: a-file-generated-by-a-previous-step
+  params:
+    p1: ((name1:var1))
+  get_input_vars:
+  - name: name2
+    file: a-file-generated-by-a-previous-step
+  get_params:
+    p2: ((name2:var2))
+```
+
+## `task` step
+
+For a `task` step, `input_vars` should be defined at same level as `input` and 
+`outputs`. The input vars will only be used to interpolate task config.
+
+```yaml
+- task: some-task
+  config:
+    platform: linux
+    image_resource:
+      type: registry-image
+      source: {reposity: some-image}
+    inputs:
+    - some-input
+    output:
+    - some-output
+    input_vars:
+    - name: invar1
+      file: a-file-generated-by-a-previous-step
+    - name: invar2
+      file: a-file-generated-by-a-previous-step
+    run:
+      path: sh
+      args:
+        - -exc
+        - |
+          echo ((invar1:msg1))
+          echo ((invar2:msg2))
+```
+
+## `set_pipeline` step
+
+I'm not seeing an use case, `set_pipeline` may not support `input_vars`.
+
+## `check` step
+
+`input_vars` is designed to support workflow, and `check` is not a step in
+workflow, so `check` step should not take input vars.
+
+
+# Open Questions
+
+
+
+# Answered Questions
+
+
+
+# New Implications
+
+* Once `input_vars` is supported, most of resource types will no longer need to
+duplicate defining `xxx_file` parameters, which would make interface of resource
+types neater.