Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

revisit memory fix for fastmultigather - use into_par_iter properly #482

Open
ctb opened this issue Oct 17, 2024 · 1 comment
Open

revisit memory fix for fastmultigather - use into_par_iter properly #482

ctb opened this issue Oct 17, 2024 · 1 comment

Comments

@ctb
Copy link
Collaborator

ctb commented Oct 17, 2024

this commit in #430 deallocates the duplicate memory used for against_collection here. Unfortunately, the RSS does briefly spike to 2x because of this. We should switch to using into_par_iter() per luiz. From slack:

luizirber you want into_par_iter and into_iter
13h
luizirber
par_iter and iter return references, into_* returns values
13h
luizirber

   pub fn into_par_iter(&self) -> impl IndexedParallelIterator<Item = (&Collection, Idx, &Record)> {
        // first create a Vec of all triples (Collection, Idx, Record)
        self
            .collections
            .into_par_iter() // CTB: are we loading things into memory here? No...
            .flat_map(|c| c.into_iter().map(move |(_idx, record)| (c, _idx, record)))
    }

13h
luizirber
or something close to that?

@ctb ctb changed the title revisit memory fix for fastmultigather revisit memory fix for fastmultigather - use into_par_iter properly Oct 30, 2024
@ctb
Copy link
Collaborator Author

ctb commented Nov 4, 2024

After playing with things a bit, I'm pretty sure that the need to keep the Collection and specifically the ZipStorage around while loading things means that there's no practical way to do what I want here. More when I have time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant