Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature] ability to get build order that doesn't include already existing binaries (Download/Cache) #15566

Closed
1 task
jcar87 opened this issue Jan 30, 2024 · 2 comments · Fixed by #15573
Closed
1 task
Milestone

Comments

@jcar87
Copy link
Contributor

jcar87 commented Jan 30, 2024

What is your suggestion?

The build order returned by conan graph build-order contains the entire dependency graph, including packages whose binaries are already available and don't need to be built.

In particular in conjunction with the new --order=configuration flag, we may get something like this:

conan graph build-order --require=capnproto/1.0.1 --order=configuration --profile:build=windows-msvc193 --profile:host=windows-msvc193  --format=json
[
    [
        {
            "ref": "zlib/1.3.1#af8f2ef3d8e0fddf1975281d7fe43bf6",
            "pref": "zlib/1.3.1#af8f2ef3d8e0fddf1975281d7fe43bf6:7bfde258ff4f62f75668d0896dbddedaa7480a0f#6c7e76f69e587ea5ef21eef7e7596bb1",
            "package_id": "7bfde258ff4f62f75668d0896dbddedaa7480a0f",
            "prev": "6c7e76f69e587ea5ef21eef7e7596bb1",
            "context": "host",
            "binary": "Download",
            "options": [],
            "filenames": [],
            "depends": [],
            "overrides": {},
            "build_args": null
        }
    ],
    [
        {
            "ref": "openssl/3.2.0#30b857fa0927b5917fe3463c8c0ba38f",
            "pref": "openssl/3.2.0#30b857fa0927b5917fe3463c8c0ba38f:1bbe7e8571fa3803593d14cc7676329626a2b8da#88a7b19999d2f8f4d91b2a662e63d753",
            "package_id": "1bbe7e8571fa3803593d14cc7676329626a2b8da",
            "prev": "88a7b19999d2f8f4d91b2a662e63d753",
            "context": "host",
            "binary": "Download",
            "options": [],
            "filenames": [],
            "depends": [
                "zlib/1.3.1#af8f2ef3d8e0fddf1975281d7fe43bf6:7bfde258ff4f62f75668d0896dbddedaa7480a0f#6c7e76f69e587ea5ef21eef7e7596bb1"
            ],
            "overrides": {},
            "build_args": null
        }
    ],
    [
        {
            "ref": "capnproto/1.0.1#69dcb448ab12eb33b564f78bdc1ff907",
            "pref": "capnproto/1.0.1#69dcb448ab12eb33b564f78bdc1ff907:12c7fa3c9c8efb48ce0284dffc4740e5a18681c0",
            "package_id": "12c7fa3c9c8efb48ce0284dffc4740e5a18681c0",
            "prev": null,
            "context": "host",
            "binary": "Missing",
            "options": [],
            "filenames": [],
            "depends": [
                "openssl/3.2.0#30b857fa0927b5917fe3463c8c0ba38f:1bbe7e8571fa3803593d14cc7676329626a2b8da#88a7b19999d2f8f4d91b2a662e63d753",
                "zlib/1.3.1#af8f2ef3d8e0fddf1975281d7fe43bf6:7bfde258ff4f62f75668d0896dbddedaa7480a0f#6c7e76f69e587ea5ef21eef7e7596bb1"
            ],
            "overrides": {},
            "build_args": null
        }
    ]
]

In a scenario where we know binaries already exist binary: Download or Cache - there is really only one package to be built. It would be good if it was possible to get a build order that is "pruned" or "optimized" for only what needs to be built.

I understand that the dependencies and full graph is needed for a subsequent build-order-merge to compute a build order from multiple dependency graphs. It's only in the latter case build-order-merge where I'd like to remove all references that do not need to be built (because a package already exist), and reorder it in such a way that all that can be built in parallel are in the same group.

So perhaps a workflow could be:

conan graph build-order --require=xxx --build=xxx --order=configuration --format=json > build_order_1.json
conan graph build-order --require=yyy --build=yyy --order=configuration --format=json > build_order_2.json
conan graph build-order --require=zzz --build=zzz --order=configuration --format=json > build_order_3.json

conan graph build-order-merge --file=build_order_1.json --file=build_order_2.json --file=build_order_3.json --new-flag

Where build-order still operates as it currently does, but we could probably have a new flag in build-order-merge that gives us an "optimized/reordered" where only the missing (or references marked with build) are present and ordered correctly.

Have you read the CONTRIBUTING guide?

  • I've read the CONTRIBUTING guide
@jcar87 jcar87 added this to the 2.1 milestone Jan 30, 2024
@memsharded
Copy link
Member

There are 2 different things here, filtering the non "build" ones is easy. But the re-ordering is a different story. It would be great to write here some scenarios of the initial un-optimized graph and the possible optimized one.

@jcar87
Copy link
Contributor Author

jcar87 commented Jan 31, 2024

Added an example here:
https://github.com/jcar87/conan-build-order-configuration

(run build.sh to generate merged_build_order.json)

What we get:

  • 3 groups
  • only 2 packages have 'build' - they are in different groups, but they can actually be built in parallel
[
    [
        {
            "ref": "bar/1.0.0#f14f708fa931a09d123809d913824567",
            "pref": "bar/1.0.0#f14f708fa931a09d123809d913824567:24085be3dae734011e3c44cb8bf47131e967dbac#3ebf2f5853c7bf9b762e08ffb60a4949",
            "package_id": "24085be3dae734011e3c44cb8bf47131e967dbac",
            "prev": "3ebf2f5853c7bf9b762e08ffb60a4949",
            "context": "host",
            "binary": "Cache",
            "options": [],
            "filenames": [
                "build_order_linux"
            ],
            "depends": [],
            "overrides": {},
            "build_args": null
        },
        {
            "ref": "bar/1.0.0#f14f708fa931a09d123809d913824567",
            "pref": "bar/1.0.0#f14f708fa931a09d123809d913824567:811d8f01c05e3e9d1e550b35fb4fd576b29aa7e5#9128a4566fdda107b112d38c108b63a6",
            "package_id": "811d8f01c05e3e9d1e550b35fb4fd576b29aa7e5",
            "prev": "9128a4566fdda107b112d38c108b63a6",
            "context": "host",
            "binary": "Cache",
            "options": [],
            "filenames": [
                "build_order_macos"
            ],
            "depends": [],
            "overrides": {},
            "build_args": null
        }
    ],
    [
        {
            "ref": "foo/1.0.0#66760017d033d9c7e17936011fbad3fb",
            "pref": "foo/1.0.0#66760017d033d9c7e17936011fbad3fb:51459a569114334c42c7bedbd42f5a1699acf8c3",
            "package_id": "51459a569114334c42c7bedbd42f5a1699acf8c3",
            "prev": null,
            "context": "host",
            "binary": "Build",
            "options": [],
            "filenames": [
                "build_order_linux"
            ],
            "depends": [
                "bar/1.0.0#f14f708fa931a09d123809d913824567:24085be3dae734011e3c44cb8bf47131e967dbac#3ebf2f5853c7bf9b762e08ffb60a4949"
            ],
            "overrides": {},
            "build_args": "--require=foo/1.0.0 --build=foo/1.0.0"
        },
        {
            "ref": "qux/1.0.0#7d6fa33b269d8355dbe8d07685ac0282",
            "pref": "qux/1.0.0#7d6fa33b269d8355dbe8d07685ac0282:af1b71646e5385c5bfd6434204e3f4ac32ea47f2#2d77dd66d8e3e003cffcfbe2bf35c698",
            "package_id": "af1b71646e5385c5bfd6434204e3f4ac32ea47f2",
            "prev": "2d77dd66d8e3e003cffcfbe2bf35c698",
            "context": "host",
            "binary": "Cache",
            "options": [],
            "filenames": [
                "build_order_macos"
            ],
            "depends": [
                "bar/1.0.0#f14f708fa931a09d123809d913824567:811d8f01c05e3e9d1e550b35fb4fd576b29aa7e5#9128a4566fdda107b112d38c108b63a6"
            ],
            "overrides": {},
            "build_args": null
        }
    ],
    [
        {
            "ref": "foo/1.0.0#66760017d033d9c7e17936011fbad3fb",
            "pref": "foo/1.0.0#66760017d033d9c7e17936011fbad3fb:1f93c65511d5f26923e61b194880e3300b4b4883",
            "package_id": "1f93c65511d5f26923e61b194880e3300b4b4883",
            "prev": null,
            "context": "host",
            "binary": "Build",
            "options": [],
            "filenames": [
                "build_order_macos"
            ],
            "depends": [
                "bar/1.0.0#f14f708fa931a09d123809d913824567:811d8f01c05e3e9d1e550b35fb4fd576b29aa7e5#9128a4566fdda107b112d38c108b63a6",
                "qux/1.0.0#7d6fa33b269d8355dbe8d07685ac0282:af1b71646e5385c5bfd6434204e3f4ac32ea47f2#2d77dd66d8e3e003cffcfbe2bf35c698"
            ],
            "overrides": {},
            "build_args": "--require=foo/1.0.0 --build=foo/1.0.0"
        }
    ]
]

what I'd like to be able to get:

  • only the things that need to be built
  • in the same group if they can be built in parallel
[
    [
        {
            "ref": "foo/1.0.0#66760017d033d9c7e17936011fbad3fb",
            "pref": "foo/1.0.0#66760017d033d9c7e17936011fbad3fb:51459a569114334c42c7bedbd42f5a1699acf8c3",
            "package_id": "51459a569114334c42c7bedbd42f5a1699acf8c3",
            "prev": null,
            "context": "host",
            "binary": "Build",
            "options": [],
            "filenames": [
                "build_order_linux"
            ],
            "depends": [
                "bar/1.0.0#f14f708fa931a09d123809d913824567:24085be3dae734011e3c44cb8bf47131e967dbac#3ebf2f5853c7bf9b762e08ffb60a4949"
            ],
            "overrides": {},
            "build_args": "--require=foo/1.0.0 --build=foo/1.0.0"
        },
        {
            "ref": "foo/1.0.0#66760017d033d9c7e17936011fbad3fb",
            "pref": "foo/1.0.0#66760017d033d9c7e17936011fbad3fb:1f93c65511d5f26923e61b194880e3300b4b4883",
            "package_id": "1f93c65511d5f26923e61b194880e3300b4b4883",
            "prev": null,
            "context": "host",
            "binary": "Build",
            "options": [],
            "filenames": [
                "build_order_macos"
            ],
            "depends": [
                "bar/1.0.0#f14f708fa931a09d123809d913824567:811d8f01c05e3e9d1e550b35fb4fd576b29aa7e5#9128a4566fdda107b112d38c108b63a6",
                "qux/1.0.0#7d6fa33b269d8355dbe8d07685ac0282:af1b71646e5385c5bfd6434204e3f4ac32ea47f2#2d77dd66d8e3e003cffcfbe2bf35c698"
            ],
            "overrides": {},
            "build_args": "--require=foo/1.0.0 --build=foo/1.0.0"
        }

    ]
]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants