Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add json output to info command (both console and file) #4359

Merged
merged 12 commits into from
Jan 29, 2019
16 changes: 11 additions & 5 deletions conans/client/command.py
Original file line number Diff line number Diff line change
Expand Up @@ -475,8 +475,7 @@ def info(self, *args):
"specify both install-folder and any setting/option "
"it will raise an error.")
parser.add_argument("-j", "--json", nargs='?', const="1", type=str,
help='Only with --build-order option, return the information in a json.'
' e.g --json=/path/to/filename.json or --json to output the json')
help='Path to a json file where the information will be written')
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've intentionally removed the mention to --json (without value) will output to console, I think it is something we don't want to promote. Shall I revert it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not promote? isn't such text interface used as a base for unix command-line pipelining, e.g. conan --json | some_tool | some_other_tool?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We cannot make this commitment right now, there are some commands that are outputting a lot of information to the console and then generating the json data (like the info one with the profiles information), so you cannot pipe the output to the next tool.

It would be a nice-to-have, it will require to silent all the output and then just print the JSON data. Something to do, not so hard, but not yet.

More issues related to output: #4225 (what we are talking here), and two more issues also worth reading #4215, #4067

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure we can just remove this despite the fact that I don't like it as it is either. We will need to discuss it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm talking about removing it just from the --help output.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, it is fine for me

jgsogo marked this conversation as resolved.
Show resolved Hide resolved
parser.add_argument("-n", "--only", nargs=1, action=Extender,
help="Show only the specified fields: %s. '--paths' information can "
"also be filtered with options %s. Use '--only None' to show only "
Expand All @@ -485,8 +484,8 @@ def info(self, *args):
help='Print information only for packages that match the filter pattern'
' e.g., MyPackage/1.2@user/channel or MyPackage*')

dry_build_help = ("Apply the --build argument to output the information, as it would be done "
"by the install command")
dry_build_help = ("Apply the --build argument to output the information, as it would be done"
" by the install command")
parser.add_argument("-db", "--dry-build", action=Extender, nargs="?", help=dry_build_help)
build_help = ("Given a build policy, return an ordered list of packages that would be built"
" from sources during the install command")
Expand Down Expand Up @@ -526,7 +525,11 @@ def info(self, *args):
remote_name=args.remote,
check_updates=args.update,
install_folder=args.install_folder)
self._outputer.nodes_to_build(nodes)
if args.json:
json_arg = True if args.json == "1" else args.json
self._outputer.json_nodes_to_build(nodes, json_arg, get_cwd())
else:
self._outputer.nodes_to_build(nodes)

# INFO ABOUT DEPS OF CURRENT PROJECT OR REFERENCE
else:
Expand All @@ -553,6 +556,9 @@ def info(self, *args):

if args.graph:
self._outputer.info_graph(args.graph, deps_graph, get_cwd())
elif args.json:
jgsogo marked this conversation as resolved.
Show resolved Hide resolved
json_arg = True if args.json == "1" else args.json
self._outputer.json_info(deps_graph, json_arg, get_cwd(), show_paths=args.paths)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could output always the paths when json output and forbid the args.paths and args.json together. It simplifies a bit everything and I would expect a json to be always complete.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For packages installed as editable access to paths raises. There are two options:

  • keep the argument, if the user asks for paths, it will raise.
  • always True and try/except the paths section, paths won't be returned and the user won't know why.

else:
self._outputer.info(deps_graph, only, args.package_filter, args.paths)

Expand Down
129 changes: 122 additions & 7 deletions conans/client/conan_command_output.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
import json
import os
from collections import OrderedDict

from conans.client.graph.graph import RECIPE_CONSUMER, RECIPE_VIRTUAL
from conans.client.graph.graph import RECIPE_EDITABLE
from conans.client.installer import build_id
from conans.client.printer import Printer
from conans.model.ref import ConanFileReference, PackageReference
from conans.paths.simple_paths import SimplePaths
from conans.search.binary_html_table import html_binary_graph
from conans.unicode import get_cwd
from conans.util.env_reader import get_env
from conans.util.files import save
from conans.client.graph.graph import RECIPE_EDITABLE


class CommandOutputer(object):
Expand Down Expand Up @@ -82,13 +86,120 @@ def _read_dates(self, deps_graph):
def nodes_to_build(self, nodes_to_build):
self.user_io.out.info(", ".join(str(n) for n in nodes_to_build))

def _handle_json_output(self, data, json_output, cwd):
json_str = json.dumps(data)

if json_output is True:
self.user_io.out.write(json_str)
else:
if not os.path.isabs(json_output):
json_output = os.path.join(cwd, json_output)
save(json_output, json.dumps(data))
self.user_io.out.writeln("")
self.user_io.out.info("JSON file created at '%s'" % json_output)

def json_nodes_to_build(self, nodes_to_build, json_output, cwd):
data = [str(n) for n in nodes_to_build]
self._handle_json_output(data, json_output, cwd)

def _grab_info_data(self, deps_graph, grab_paths):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this "conversion function" doesn't belong to this module. A to_json in the graph? Not very fan either...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is created and called from outside the conan_api, the graph shouldn't have left the api. I'm sure that this json should be the output of the conan_api::info function, but this would be a refactor (I vote for it) and not the objective of this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. Open the engineering issue, please.

""" Convert 'deps_graph' into consumible information for json and cli """
compact_nodes = OrderedDict()
for node in sorted(deps_graph.nodes):
compact_nodes.setdefault((node.ref, node.conanfile.info.package_id()), []).append(node)

ret = []
for (ref, package_id), list_nodes in compact_nodes.items():
node = list_nodes[0]
if node.recipe == RECIPE_VIRTUAL:
continue

item_data = {}
conanfile = node.conanfile
if node.recipe == RECIPE_CONSUMER:
ref = str(conanfile)

item_data["reference"] = str(ref)
item_data["is_ref"] = isinstance(ref, ConanFileReference)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need this field in the JSON to perform some logic in the console output: will output Remote: None if it is a ConanFileReference, but it won't if it isn't... same with Binary remote:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... if we do not consider breaking to remove this Remote: None I will be very pleased to get rid of them

item_data["display_name"] = conanfile.display_name
item_data["id"] = package_id
item_data["build_id"] = build_id(conanfile)

# Paths
if isinstance(ref, ConanFileReference) and grab_paths:
jgsogo marked this conversation as resolved.
Show resolved Hide resolved
item_data["export_folder"] = self.cache.export(ref)
item_data["source_folder"] = self.cache.source(ref, conanfile.short_paths)
if isinstance(self.cache, SimplePaths):
# @todo: check if this is correct or if it must always be package_id()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have the package_id already in the main for iteration. Why to do all this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I can remove one line, the package_id variable is already available in the for; but the block related to the build_folder and the @todo section come from #1032, it should require a deeper rationale.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two calls to conanfile.info.package_id() yes. I understand the rest is necessary.

bid = build_id(conanfile)
if not bid:
bid = conanfile.info.package_id()
pref = PackageReference(ref, bid)
item_data["build_folder"] = self.cache.build(pref, conanfile.short_paths)

package_id = conanfile.info.package_id()
pref = PackageReference(ref, package_id)
item_data["package_folder"] = self.cache.package(pref, conanfile.short_paths)

try:
reg_remote = self.cache.registry.refs.get(ref)
if reg_remote:
item_data["remote"] = {"name": reg_remote.name, "url": reg_remote.url}
except:
pass

def _add_if_exists(attrib, as_list=False):
value = getattr(conanfile, attrib, None)
if value:
if not as_list:
item_data[attrib] = value
else:
item_data[attrib] = list(value) if isinstance(value, (list, tuple, set)) \
else [value, ]

_add_if_exists("url")
_add_if_exists("homepage")
_add_if_exists("license", as_list=True)
_add_if_exists("author")
_add_if_exists("topics", as_list=True)

if isinstance(ref, ConanFileReference):
item_data["recipe"] = node.recipe

if get_env("CONAN_CLIENT_REVISIONS_ENABLED", False) and node.ref.revision:
item_data["revision"] = node.ref.revision

item_data["binary"] = node.binary
if node.binary_remote:
item_data["binary_remote"] = node.binary_remote.name

node_times = self._read_dates(deps_graph)
if node_times and node_times.get(ref, None):
item_data["creation_date"] = node_times.get(ref, None)

if isinstance(ref, ConanFileReference):
dependants = [n for node in list_nodes for n in node.inverse_neighbors()]
required = [d.conanfile for d in dependants if d.recipe != RECIPE_VIRTUAL]
if required:
item_data["required_by"] = [d.display_name for d in required]

depends = node.neighbors()
requires = [d for d in depends if not d.build_require]
build_requires = [d for d in depends if d.build_require]

if requires:
item_data["requires"] = [repr(d.ref) for d in requires]

if build_requires:
item_data["build_requires"] = [repr(d.ref) for d in build_requires]

ret.append(item_data)

return ret

def info(self, deps_graph, only, package_filter, show_paths):
registry = self.cache.registry
Printer(self.user_io.out).print_info(deps_graph,
only, registry,
node_times=self._read_dates(deps_graph),
path_resolver=self.cache,
package_filter=package_filter,
data = self._grab_info_data(deps_graph, grab_paths=show_paths)
Printer(self.user_io.out).print_info(data, only, package_filter=package_filter,
show_paths=show_paths)

def info_graph(self, graph_filename, deps_graph, cwd):
Expand All @@ -104,6 +215,10 @@ def info_graph(self, graph_filename, deps_graph, cwd):
graph_filename = os.path.join(cwd, graph_filename)
grapher.graph_file(graph_filename)

def json_info(self, deps_graph, json_output, cwd, show_paths):
data = self._grab_info_data(deps_graph, grab_paths=show_paths)
self._handle_json_output(data, json_output, cwd)

def print_search_references(self, search_info, pattern, raw, all_remotes_search):
printer = Printer(self.user_io.out)
printer.print_search_recipes(search_info, pattern, raw, all_remotes_search)
Expand Down
Loading