Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add zstd support #148

Merged
merged 12 commits into from
May 31, 2022
Merged

feat: Add zstd support #148

merged 12 commits into from
May 31, 2022

Conversation

NobodyXu
Copy link
Member

Signed-off-by: Jiahao XU Jiahao_XU@outlook.com

NobodyXu added 3 commits May 31, 2022 16:43
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
@passcod
Copy link
Member

passcod commented May 31, 2022

Awesome!

@NobodyXu
Copy link
Member Author

@passcod I am not very familiar with crate tar/zstd, so please make sure to review my code thoroughly!

@passcod
Copy link
Member

passcod commented May 31, 2022

Can you also add/check off the relevant box in the status section of the readme please :)

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
@NobodyXu
Copy link
Member Author

Can you also add/check off the relevant box in the status section of the readme please :)

Done!

@NobodyXu
Copy link
Member Author

Seems like cross does not have libclang installed.

Copy link
Member

@passcod passcod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@passcod passcod added the Blocked: upstream Fix or feature is needed to be implemented upstream (in a dependency) label May 31, 2022
@passcod
Copy link
Member

passcod commented May 31, 2022

Code here is approved, but let's not merge until the CI/cross situation is resolved.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Cross.toml Outdated Show resolved Hide resolved
NobodyXu added 3 commits May 31, 2022 19:09
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
musl targets work fine, no workaound is required.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Cross.toml Outdated Show resolved Hide resolved
NobodyXu and others added 2 commits May 31, 2022 19:31
Co-authored-by: Emil Gardström <emil.gardstrom@gmail.com>
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Cross.toml Outdated Show resolved Hide resolved
Co-authored-by: Emil Gardström <emil.gardstrom@gmail.com>
#
# Enable feature bindgen to generate C bindings.
# Enable feature zstdmt to enable multithreading in libzstd.
zstd = { version = "0.10.0", features = [ "bindgen", "zstdmt" ], default-features = false }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
zstd = { version = "0.10.0", features = [ "bindgen", "zstdmt" ], default-features = false }
zstd = { version = "0.10.0", features = [ "bindgen", "zstdmt", "pkg-config" ], default-features = false }

do note that this will still not work, you'll need something like I mentioned in cross-rs/cross#731 (comment)

which is a custom Dockerfile to add the missing dependencies

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

However, I still don't think it is a good idea to have a runtime dependency here, so I will explore other options to fix this problem, like abandoning the glibc and goes for musl-only for these two targets.

Copy link
Contributor

@Emilgardis Emilgardis May 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would not be a runtime dependency :) (afaik, if I interpret what zstd-coresys does)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry but what is zstd-core?

The zstd crate we use here depends on zstd-sys for compiling, linking and generating bindings for libzstd.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mb! I meant zstd-sys

Copy link
Member Author

@NobodyXu NobodyXu May 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@Emilgardis Emilgardis May 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, and to solve that/have it work, you need to have libzstd-dev installed in the builders environment, which is done with supplying a custom docker image to cross

Copy link
Member Author

@NobodyXu NobodyXu May 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It skips building zstd if pkg-config is enabled, which means that it will dynamically linked with libzstd provided by the system.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but it would statically link it with the installed zstd, or am I missing something?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but it would statically link it with the installed zstd, or am I missing something?

I'm not sure about that.

I wasn't able to find the println!("cargo:") statement that specifies the linkage for the system-wide libzstd.

@NobodyXu
Copy link
Member Author

@passcod Seems like only aarch64-unknown-linux-gnu and armv7-unknown-linux-gnueabihf are not working.

Maybe we can stop supporting these two targets and instead use the musl ones?

@passcod
Copy link
Member

passcod commented May 31, 2022

stop supporting these two targets and instead use the musl ones?

That sounds okay.

Since they are the only two targets that crate `zstd-sys` failed to
build.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
@NobodyXu
Copy link
Member Author

stop supporting these two targets and instead use the musl ones?

That sounds okay.

Done!

@passcod passcod merged commit db3f12c into cargo-bins:main May 31, 2022
@NobodyXu NobodyXu deleted the feature/zstd branch May 31, 2022 11:02
@passcod passcod mentioned this pull request May 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Blocked: upstream Fix or feature is needed to be implemented upstream (in a dependency)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants