Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

uploadDirectory(files) should ensure ordered files that produce deterministic merkling #1172

Closed
gobengo opened this issue Nov 20, 2023 · 1 comment
Closed

uploadDirectory(files) should ensure ordered files that produce deterministic merkling #1172

gobengo opened this issue Nov 20, 2023 · 1 comment
Assignees
@gobengo
Labels
P0 Critical: Tackled by core team ASAP

Comments

@gobengo
Copy link
Contributor

gobengo commented Nov 20, 2023 •
edited
Loading

Context

  • w3up-client has a Client#uploadDirectory function that accepts an iterable of files
  • if you pass the same underlying files in a different order, it creates (I think?) a different set of underlying blocks in the underlying IPLD. I believe this is due to the way the underlying UnixFSDirectoryBuilder is order-dependent
  • Right now, some users pass a files with no particular sort order. When this is called multiple times, it can lead to new byte-by-byte blocks being update each time, even though all the blocks together are for the same files.

Desirable

  • if a user naively passes unsorted files to uploadDirectory, we should sort them for them
  • a user should be able to call uploadDirectory with an option indicating they have already ordered the files and uploadDirectory must not change the custom order
### Tasks
- [ ] https://github.com/web3-storage/w3up/pull/1173
- [ ] https://github.com/web3-storage/secrets/issues/40
@gobengo gobengo self-assigned this Nov 20, 2023
@gobengo gobengo moved this to Todo in bengo.is/w3up Nov 20, 2023
@gobengo gobengo changed the title uploadDirectory(files) should produce store the same bytes regardless of files iteration order uploadDirectory(files) should error if files is too ambiguous to produce deterministic merkling Nov 21, 2023
@gobengo gobengo mentioned this issue Nov 21, 2023
Merged
@gobengo gobengo moved this from Todo to In Progress in bengo.is/w3up Nov 28, 2023
@gobengo gobengo changed the title uploadDirectory(files) should error if files is too ambiguous to produce deterministic merkling uploadDirectory(files) should ensure ordered files that produce deterministic merkling Nov 28, 2023
@gobengo gobengo added the P0 Critical: Tackled by core team ASAP label Nov 29, 2023
gobengo added a commit that referenced this issue Nov 29, 2023
@gobengo
feat: upload-client uploadDirectory, by default, sorts the provided f…
8cd2555 
…iles by file name to help the user call us in a way that is deterministic and minimizes cost (#1173)

Motivation:
* #1172
* "Stabilize file ordering in uploadDirectory" in
https://github.com/web3-storage/secrets/issues/28
* this stabilizes it by sorting the files param unless an option is
passed telling us not to
@gobengo gobengo closed this as completed Dec 4, 2023
@github-project-automation github-project-automation bot moved this from In Progress to Done in bengo.is/w3up Dec 4, 2023
@gobengo
Copy link
Contributor Author

gobengo commented Dec 4, 2023

this was released here #1202 in @web3-storage/upload-client@12.2.0 (should get pulled in via semver for users of @web3-storage/w3up-client latest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gobengo gobengo

Labels
P0 Critical: Tackled by core team ASAP
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gobengo