add user docs

docs/dev/devguide.md

@@ -90,4 +90,73 @@ INFO [job test.cwl] completed success
     ]
 ```
 
-As can be seen from the output json blob the order returned is not the same as input order.
+As can be seen from the output json blob the order returned is not the same as input order.
+
+
+## Partial Failures
+When the partial failures feature is enabled although the subprocess for the workflow step itself will pass, the post-processing javascript can potentially crash as seen below. The Sophios compiler only semantically understands Sophios/CWL. It is theoretically impossible to correct mistakes in the embedded JS of any arbitrary workflow. The corresponding cwl snippet is also shown.
+```
+outputs:
+
+  topology_changed:
+    type: boolean
+    outputBinding:
+      glob: valid.txt
+      loadContents: true
+      outputEval: |
+        ${
+          // Read the contents of the file
+          const lines = self[0].contents.split("\n");
+          // Read boolean value from the first line
+          const valid = lines[0].trim() === "True";
+          return valid;
+        }
+```
+```
+stdout was: ''
+stderr was: 'evalmachine.<anonymous>:45
+  const lines = self[0].contents.split("\n");
+                        ^
+TypeError: Cannot read properties of undefined (reading 'contents')
+```
+To fix this the developer needs to add a javascript snippet to check if the self object being globbed exists, shown below.
+```
+outputs:
+
+  topology_changed:
+    type: boolean
+    outputBinding:
+      glob: valid.txt
+      loadContents: true
+      outputEval: |
+        ${
+          // check if self[0] exists
+          if (!self[0]) {
+            return null;
+          }
+          // Read the contents of the file
+          const lines = self[0].contents.split("\n");
+          // Read boolean value from the first line
+          const valid = lines[0].trim() === "True";
+          return valid;
+        }
+```
+
+## Workflow Development
+When adding new .cwl or .wic files its best to remove the .wic folder containing paths to .cwl and .yml files
+```
+rm -r ~/wic
+```
+
+## Singularity
+When building images with Singularity its best to clean the cache to avoid potential errors with cwltool or cwl-docker-extract.
+```
+singularity cache clean
+```
+
+## Toil
+When working with toil be sure to clean the working state as well as the configuration file, otherwise if you change input flags the configuration file will not be updated.
+```
+toil clean
+rm -r ~/.toil
+```

docs/userguide.md

@@ -285,4 +285,14 @@ if __name__ == '__main__':
     when_wic.run()  # .run() here inside main
 ```
 Similar to `scatter`, `when` is a **special (and optional)** attribute to any step object in the Python API.
-The `when` attribute of a step object exposes the exact same js embedded syntax of `when` tag of the YAML/CWL syntax. One has to be careful about appropriate escaping in the string input of `when` in Python API. In the above case the comparison is between two strings so "" is around the literal 27 (i.e. value after `toString` step).
+The `when` attribute of a step object exposes the exact same js embedded syntax of `when` tag of the YAML/CWL syntax. One has to be careful about appropriate escaping in the string input of `when` in Python API. In the above case the comparison is between two strings so "" is around the literal 27 (i.e. value after `toString` step).
+## Partial Failures
+
+In running workflows at scale, sometimes it is the case that one of the workflow steps may crash due to a bug causing the entire workflow to crash. In this case can use `--partial_failure_enable` flag. For special cases when the exit status of a workflow step isn't 1, and a different error code is returned (for example 142), then the user can supply the error code to wic as a success code to prevent workflow from crashing with `--partial_failure_success_codes 0 1 142`. By default partial failure flag will consider only 0 and 1 as success codes. An example line snippet of the error code being printed is shown below.
+```
+[1;30mWARNING[0m [33m[job compare_extract_protein_pdbbind__step__4__topology_check] exited with status: 139[0m
+```
+
+## Parallelization
+
+In order to utilize scattering features in cwl, the user needs to provide the flag `--parallel`. Additionally cwltool has various issues regarding scattering features such as deadlocks and thus it is preferred to use toil in this case `--cwl_runner toil-cwl-runner`.