Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AIX: bootstrap investigate efficient way of reading archive #133268

Closed
mustartt opened this issue Nov 20, 2024 · 1 comment · Fixed by #133462
Closed

AIX: bootstrap investigate efficient way of reading archive #133268

mustartt opened this issue Nov 20, 2024 · 1 comment · Fixed by #133462
Labels
C-cleanup Category: PRs that clean code up or issues documenting cleanup. O-aix OS: Big Blue's Advanced Interactive eXecutive.. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Comments

@mustartt
Copy link
Contributor

Currently in the bootstrap we are reading the entire archive into memory inorder to read the members:

 fn is_aix_shared_archive(path: &Path) -> bool {
      // reading the entire file as &[u8] into memory seems excessive
      // look into either mmap it or use the &CacheReader
      let data = match fs::read(path) {
          Ok(data) => data,
          Err(_) => return false,
      };
      let file = match ArchiveFile::parse(&*data) {
          Ok(file) => file,
          Err(_) => return false,
      };
  
      file.members()
          .filter_map(Result::ok)
          .any(|entry| String::from_utf8_lossy(entry.name()).contains(".so"))
 }

Find an easy way to either memory map the file or something else. Because currently this can take up to a second or two to read the larger dylibs into memory.

@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Nov 20, 2024
@mustartt
Copy link
Contributor Author

@rustbot label: +C-cleanup

@rustbot rustbot added the C-cleanup Category: PRs that clean code up or issues documenting cleanup. label Nov 20, 2024
@jieyouxu jieyouxu added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) O-aix OS: Big Blue's Advanced Interactive eXecutive.. and removed needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. labels Nov 21, 2024
jhpratt added a commit to jhpratt/rust that referenced this issue Nov 26, 2024
…ding, r=jieyouxu

Use ReadCache for archive reading in bootstrap

Address expensive archive reading in bootstrap. This fixes rust-lang#133268

Enable the `std` feature of `object` to use `ReadCache` instead of reading the entire archive file into memory to check for headers. This takes minimal extra time to compile compared to introducing other expensive dependencies to `bootstrap`.

r? jieyouxu
@bors bors closed this as completed in 7e3422f Nov 26, 2024
rust-timer added a commit to rust-lang-ci/rust that referenced this issue Nov 26, 2024
Rollup merge of rust-lang#133462 - mustartt:aix-improve-bootstrap-loading, r=jieyouxu

Use ReadCache for archive reading in bootstrap

Address expensive archive reading in bootstrap. This fixes rust-lang#133268

Enable the `std` feature of `object` to use `ReadCache` instead of reading the entire archive file into memory to check for headers. This takes minimal extra time to compile compared to introducing other expensive dependencies to `bootstrap`.

r? jieyouxu
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-cleanup Category: PRs that clean code up or issues documenting cleanup. O-aix OS: Big Blue's Advanced Interactive eXecutive.. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants