feat(SeqeraPlatform): seqerakit can write JSON to stdout for detailed records

[adamrtalbot]

We needed a method of storing run info from the seqerakit CLI. After this change, you can use the '--json' flag to write the JSON return from the CLI to STDOUT. This can be captured and used in a different process or piped. This is enabled by a new command line flag, --json but should be possible when instantiating the SeqeraPlatform class in Python. The to_json kwarg is removed and replaced with this method so this might break some uses. The JSON is an attribute of the class and can be referenced whenever you use the SeqeraPlatform class.

BREAKING CHANGE: This could break anyone using the Python implementation with to_json=True. This is mainly me.

To use, try running seqerakit with the flag -j and piping:

seqerakit -j launch/*.yml | jq --slurp > launched-pipelines.json

(update): Because the function is overlapping with to_json, we just use either will create JSON output. If to_json is used it will print to stderr, if -j is used it will write to stdout suitable for piping.

[adamrtalbot]

Appears to double up the -o json part sometimes:

ERROR:root:Resource creation failed: 'option '--output' (<output>) should be specified only once`

[ejseqera]

@adamrtalbot I like this implementation, it might break things but it's a feature worth adding. What's left to do?

[adamrtalbot]

@adamrtalbot I like this implementation, it might break things but it's a feature worth adding. What's left to do?

Fix the doubling of --output json twice is the main issue.

Ideally, we might cache the output and write a valid JSON as a list although piping to jq seems acceptable for a first draft.

[adamrtalbot]

Trying to finish this off and I'm confused about how I should handle the to_json flag in sp._execute_command():

seqera-kit/seqerakit/seqeraplatform.py

Lines 103 to 127 in 0271c38

	# Executes a 'tw' command in a subprocess and returns the output.
	def _execute_command(self, full_cmd, to_json=False, print_stdout=True):
	logging.info(f" Running command: {full_cmd}")
	process = subprocess.Popen(
	full_cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True
	)
	stdout, _ = process.communicate()
	stdout = stdout.decode("utf-8").strip()
	should_print = (
	print_stdout if print_stdout is not None else self.print_stdout
	) and not self._suppress_output
	if should_print:
	logging.info(f" Command output: {stdout}")
	if "ERROR: " in stdout or process.returncode != 0:
	self._handle_command_errors(stdout)
	if self.json:
	out = json.loads(stdout)
	print(json.dumps(out))
	else:
	out = stdout
	print(stdout)

It seems like these have overlapping purposes so I've added an or statement which fixes the tests: 1131da8, but not sure if this is the right thing to do.

[adamrtalbot]

Yo @ejseqera this one is good to go and I think it would be useful with our current projects. Let me know what else it needs.

[ejseqera]

Thanks Adam 🙏

[adamrtalbot]

Ah heck, this still doubles up if you use overwrite: true:

INFO:root: Running command: tw -o json -o json pipelines list -w $ORGANIZATION_NAME/$WORKSPACE_NAME
ERROR:root:Resource creation failed: 'option '--output' (<output>) should be specified only once