Version 1.1 Spec

CHANGELOG.md

@@ -12,21 +12,60 @@ What should be mentioned (in order):
 + Credit where credit is due by mentioning the github account.
 
 Keep the changelog pleasant to read in the text editor:
-+ Max 80 characters per line
 + Empty line between changes.
 + Newline between summary and link+credit.
 + Properly indent blocks.
 -->
 
-version 2.0.0
+version development
 ---------------------------
+
++ Runtime section no longer accepts arbitrary keys. Added new hints section for arbitrary runtime hints.
+  [PR 315](https://github.com/openwdl/wdl/pull/315) by @patmagee
+
++ Removes string interpolator options. 
+  [PR 229](https://github.com/openwdl/wdl/pull/229) and [PR 368](https://github.com/openwdl/wdl/pull/366) 
+  by @EvanTheB and @illusional. 
+
++ `object` has been removed from WDL. `struct` can be used to achieve the same
+  type of functionality in a more explicit way.
+  [PR 283](https://github.com/openwdl/wdl/pull/283) by @patmagee.
+
++ Added a new directory type to make it easier when working with inputs that
+  consist of multiple files.
+  [PR 241](https://github.com/openwdl/wdl/pull/241) by @cjllanwarne.
+
+version 1.1.0
+---------------------------
+
++ Clarified that the `sub` function requires a POSIX Extended Regular Expression (ERE).
+  [PR 243](https://github.com/openwdl/wdl/pull/243) by @rhpvorderman
+
++ Added syntax for struct literals.
+  [PR 297](https://github.com/openwdl/wdl/pull/297) by @patmagee
+
++ Added engine functions for `min` and `max`.
+  [PR 304](https://github.com/openwdl/wdl/pull/304) by @pshapiro
+
++ Added section on file outputs including optional outputs.
+  [PR 310](https://github.com/openwdl/wdl/pull/310) by @jtratner
+
++ Added reserved keys, explicit formats, and default values for runtime attributes and hints.
+  [PR 315](https://github.com/openwdl/wdl/pull/315) by @patmagee
+
++ Namespacing has been clarified.
+  [PR 340](https://github.com/openwdl/wdl/pull/340) by @DavyCats
+
 + Abbreviated syntax for call inputs bound to workflow-scoped values by name:
   `{input: x, y=b, z}` is shorthand for `{input: x=x, y=b, z=z}`
   [PR 365](https://github.com/openwdl/wdl/pull/365) by @mlin
 
-+ Removes string interpolator options and adds an engine function for joining arrays of strings. 
++ Write a specification for unsatisfied task inputs and nested optional inputs.
+  [PR 359](https://github.com/openwdl/wdl/pull/359) by @rhpvorderman
+
++ Adds an engine function for joining arrays of strings. 
   [PR 229](https://github.com/openwdl/wdl/pull/229) and [PR 368](https://github.com/openwdl/wdl/pull/366) 
-  by @EvanTheB and @illusional. 
+  by @EvanTheB and @illusional.
 
 + Added an engine function for adding a suffix to an array of primitives as well
   as well as `quote` and `squote` engine functions.
@@ -47,10 +86,6 @@ version 2.0.0
 + Optional and non-empty type constraints have been clarified.
   [PR 290](https://github.com/openwdl/wdl/pull/290) by @mlin.
 
-+ `object` has been removed from WDL. `struct` can be used to achieve the same
-  type of functionality in a more explicit way.
-  [PR 283](https://github.com/openwdl/wdl/pull/283) by @patmagee.
-
 + The way comments work has been clarified.
   [PR 277](https://github.com/openwdl/wdl/pull/277) by @patmagee.
 
@@ -75,19 +110,12 @@ version 2.0.0
 + Added a `keys` function to get an array of keys from a map.
   [PR 244](https://github.com/openwdl/wdl/pull/244) by @ffinfo.
 
-+ Added a new directory type to make it easier when working with inputs that
-  consist of multiple files.
-  [PR 241](https://github.com/openwdl/wdl/pull/241) by @cjllanwarne.
-
 + Several bugs in the grammar have been fixed.
   [PR 238](https://github.com/openwdl/wdl/pull/238) and
   [PR 240](https://github.com/openwdl/wdl/pull/240) by @cjllanwarne.
 
-<!---
-This is not implemented yet.
 + Type conversions and meanings have been clarified.
   [PR 235](https://github.com/openwdl/wdl/pull/235) by @EvanTheB.
--->
 
 + **Backported to 1.0**: Imports are now relative to their current location.
   [PR 220](https://github.com/openwdl/wdl/pull/220) by @geoffjentry.
@@ -99,7 +127,6 @@ This is not implemented yet.
 + Add an `after` keyword to run a task after other tasks.
   [PR 162](https://github.com/openwdl/wdl/pull/162) by @cjllanwarne.
 
-
 version 1.0.0
 ---------------------------
 + Rename lexer to `WdlV1Lexer`

versions/1.1/ImplementationGuide.md

@@ -0,0 +1,45 @@
+
+# WDL Implementation Guide
+
+## Task Execution
+
+The execution engine is responsible for implementing all the necessary logic to execute a task's command given a set of inputs. Task execution is performed in these logical stages, although an execution engine is free to implement these stages however it wishes, including combining or reording stages, so long as the contract of each stage is upheld:
+
+1. WDL validation: The WDL document is validated as correct, meaning that:
+    * it can be parsed according to the WDL grammar rules, and
+    * all expressions will evaluate to the type expected by the declaration or call input parameter.
+2. Input validation: Task inputs are validated to ensure that:
+    * there are no duplicates,
+    * each one matches a valid input parameter name,
+    * the value can be deserialized to the declared type, and
+    * there are no missing required inputs.
+3. Declaration ordering: Any input and private declarations that are initialized with expressions and are not overridden by user-specified values have their expressions examined to determine their dependencies. Declarations are then ordered such that they can be evaluated unambiguously. For example, in the following task, the expressions need to be evaluated in the following order: `a`, `c`, `b`.
+    ```wdl
+    task out_of_order {
+      input {
+        File a
+      }
+      String b = "The input file is size: ${c} GB"
+      Int c = size(a, "GB")
+    }
+    ```
+4. Declaration evaluation: Input and private declarations are evaluated in order, with the values of all previously evaluated declarations being available to evaluate the next declaration's initialization expression.
+5. Runtime evaluation:
+    * The runtime attributes are evaluated in the context of all input and private declaration values.
+    * An appropriate runtime instance type is determined from the resource requirements. If no instance type can satisfy all the requirements, task execution is terminated with an error.
+    * The task may be executed in the current environment (where the evaluation is occurring) if that is acceptable, otherwise the runtime instance is provisioned and the task exection is "moved" to the runtime instance, where moving may involve a direct transfer of state or a relaunching of the task (starting over with stage 1).
+6. Input localization: Each `File` typed value - including compound values with nested `File`s - that references a non-local resource is localized, which involves:
+    * Creating a local file (or file-like object, such as a `fifo`), according to the [input localization](#task-input-localization) rules.
+    * Making the contents of the remote resource readable from that file, either by downloading the remote resource or by streaming its contents.
+    * Replacing the value of the `File` typed declaration with the path to the local file.
+7. Command instantiation:
+    * The command template is evaluated in the context of all the (localized) input and private declaration values, and all expression placeholders are replaced with their (stringified) values. 
+    * The instantiated command is written to local disk and made executable.
+8. Container resolution:
+    * If alternative container images are specified, the ["best"](SPEC.md#container) one is selected.
+    * The image is "pulled" to the local environment - this may involve a literal `docker pull`, downloading a tarball and calling `docker load` or `docker import`, or some other mechanism.
+9. Command execution: The instantiated command is staged within the container (along with any other necessary volumes) and executed there. All outputs should be created relative to a staged output directory. If the command exits with any return code not specified in `runtime.returnCodes`, task execution exits with an error.
+10. Output ordering: Any output declarations that are initialized with expressions have their expressions examined to determine their dependencies. Declarations are then ordered such that they can be evaluated unambiguously.
+11. Output evaluation: Output declarations are evaluated in order, with the values of all previously evaluated declarations being available to evaluate the next declaration's initialization expression.
+12. Output delocalization: The exection engine may choose to "delocalize" outputs - i.e. to move them from the execution environment to a permanent storage location. If so, the values of any `File` typed output parameters - including compound values with nested `File`s - are updated to replace the local path with the permanent storage location.
+13. Cleanup: The execution engine performs any "cleanup" tasks, such as terminating the cloud worker instance.