This repository has been archived by the owner on Feb 12, 2024. It is now read-only.
feat: jsipfs pin #1249
Merged
feat: jsipfs pin #1249
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ges but mostly minor.
… core/utils.parseIpfsPath
…me documentation.
…he unix-fs importer.
…re of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers
ghost
assigned
This was referenced Mar 6, 2018
Closed
JonKrone
commented
Mar 16, 2018
| * @param {String} ipfsPath An ipfs-path | ||
| * @return {Object} { root: base58 string, links: [string], ?err: Error } | ||
| */ | ||
| exports.parseIpfsPath = function parseIpfsPath (ipfsPath) { |
There was a problem hiding this comment.
Validating and decomposing a path into its components seems like a core utility. Do we already have something for this?
|
@JonKrone merging this PR into the other branch so that we only track work in one (makes reviews simpler). awesome stuff! |
|
@diasdavid Great. You had originally requested a separate PR so that my changes could be reviewed independently and I was going to suggest just comparing this against master after that had been okay'd but this is just as well. I'm happy that they're together again. I've got a batch of changes coming through after dinner tonight. It'll be ready for review tomorrow. |
JonKrone
added a commit
to JonKrone/js-ipfs
that referenced
this pull request
Mar 18, 2018
* initial sweep through to understand how pin works. Did make some changes but mostly minor. * refactor pb schema to it's own file * fix: don't pin files during files.add if opts.pin === false * feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath * feat: expand pin tests. \nFirst draft. still needs some further work. * feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation. * feat: add --pin to files.add, fix: improper pin option parsing in core. * feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer. * feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers * feat: use isIPFS to valiate a multihash.
JonKrone
added a commit
to JonKrone/js-ipfs
that referenced
this pull request
Mar 27, 2018
* initial sweep through to understand how pin works. Did make some changes but mostly minor. * refactor pb schema to it's own file * fix: don't pin files during files.add if opts.pin === false * feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath * feat: expand pin tests. \nFirst draft. still needs some further work. * feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation. * feat: add --pin to files.add, fix: improper pin option parsing in core. * feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer. * feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers * feat: use isIPFS to valiate a multihash.
JonKrone
added a commit
to JonKrone/js-ipfs
that referenced
this pull request
Apr 3, 2018
* initial sweep through to understand how pin works. Did make some changes but mostly minor. * refactor pb schema to it's own file * fix: don't pin files during files.add if opts.pin === false * feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath * feat: expand pin tests. \nFirst draft. still needs some further work. * feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation. * feat: add --pin to files.add, fix: improper pin option parsing in core. * feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer. * feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers * feat: use isIPFS to valiate a multihash.
JonKrone
added a commit
to JonKrone/js-ipfs
that referenced
this pull request
Apr 13, 2018
* initial sweep through to understand how pin works. Did make some changes but mostly minor. * refactor pb schema to it's own file * fix: don't pin files during files.add if opts.pin === false * feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath * feat: expand pin tests. \nFirst draft. still needs some further work. * feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation. * feat: add --pin to files.add, fix: improper pin option parsing in core. * feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer. * feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers * feat: use isIPFS to valiate a multihash.
JonKrone
added a commit
to JonKrone/js-ipfs
that referenced
this pull request
May 17, 2018
* initial sweep through to understand how pin works. Did make some changes but mostly minor. * refactor pb schema to it's own file * fix: don't pin files during files.add if opts.pin === false * feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath * feat: expand pin tests. \nFirst draft. still needs some further work. * feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation. * feat: add --pin to files.add, fix: improper pin option parsing in core. * feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer. * feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers * feat: use isIPFS to valiate a multihash.
JonKrone
added a commit
to JonKrone/js-ipfs
that referenced
this pull request
May 29, 2018
* initial sweep through to understand how pin works. Did make some changes but mostly minor. * refactor pb schema to it's own file * fix: don't pin files during files.add if opts.pin === false * feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath * feat: expand pin tests. \nFirst draft. still needs some further work. * feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation. * feat: add --pin to files.add, fix: improper pin option parsing in core. * feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer. * feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers * feat: use isIPFS to valiate a multihash.
JonKrone
added a commit
that referenced
this pull request
May 30, 2018
* initial sweep through to understand how pin works. Did make some changes but mostly minor. * refactor pb schema to it's own file * fix: don't pin files during files.add if opts.pin === false * feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath * feat: expand pin tests. \nFirst draft. still needs some further work. * feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation. * feat: add --pin to files.add, fix: improper pin option parsing in core. * feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer. * feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers * feat: use isIPFS to valiate a multihash.
JonKrone
added a commit
that referenced
this pull request
Jun 12, 2018
* initial sweep through to understand how pin works. Did make some changes but mostly minor. * refactor pb schema to it's own file * fix: don't pin files during files.add if opts.pin === false * feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath * feat: expand pin tests. \nFirst draft. still needs some further work. * feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation. * feat: add --pin to files.add, fix: improper pin option parsing in core. * feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer. * feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers * feat: use isIPFS to valiate a multihash.
JonKrone
added a commit
that referenced
this pull request
Jun 18, 2018
* initial sweep through to understand how pin works. Did make some changes but mostly minor. * refactor pb schema to it's own file * fix: don't pin files during files.add if opts.pin === false * feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath * feat: expand pin tests. \nFirst draft. still needs some further work. * feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation. * feat: add --pin to files.add, fix: improper pin option parsing in core. * feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer. * feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers * feat: use isIPFS to valiate a multihash.
alanshaw
pushed a commit
that referenced
this pull request
Jun 19, 2018
* revert: default assets are not added when running on a browser refactor: change pin.flush logging message * feat(test): add tests for failure cases of normalizeHashes fix: don't need to cast the object.get result with toJSON revert: use interface-datastore.Key for datastore pin storage The proper change would be that datastore-level automatically casts operations into Keys fix: do not invoke callback within a try/catch feat(test): make cli pin tests more robust By using files that aren't added on IPFS initialization. Still needs work on files.rm (direct) and ipfs ls (indirect). fix: remove commented code, traced test failures to pin-set Got to go for the night, though, so will checkpoint here and address tomorrow. feat: parseIpfsPath now throws errors for consistency feat: resolveIpfsPaths error message lists the relative path that failed feat: use follow.bind instead of mutating the links Also decided not show relative paths. Less human friendly but probably cleaner otherwise. refactor: resolveIpfsPaths -> resolvePaths feat: promisify resolvePaths test: change parseIpfsPath failure tests to use try/catch docs: edit resolvePath doc revert: accidentally deleted commands/pin.js * feat: jsipfs pin improvements (#1249) * initial sweep through to understand how pin works. Did make some changes but mostly minor. * refactor pb schema to it's own file * fix: don't pin files during files.add if opts.pin === false * feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath * feat: expand pin tests. \nFirst draft. still needs some further work. * feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation. * feat: add --pin to files.add, fix: improper pin option parsing in core. * feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer. * feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers * feat: use isIPFS to valiate a multihash. * fix: add some changes missed during rebase, syntax fixes, etc I think my original rebase for this branch 2 weeks ago might have changed history for the intervening commits, indirectly causing some of these missed changes. or I just rebase onto the wrong oldparent. fix: some onlyHash and pin tests broke after merging onlyHash and pin interact: shouldn't pin when --only-hash. fix: trim output for 'pin ls when no hash is passed' test: indirect pins supersede direct pins: turns out we had a bug feat: add expectTimeout test utility feat: promisify some additional pin utils * test: initial work testing the core/pin.js implementation I think I'll end up moving most tests here. test: add tests for pin.ls and pin.rm Based tests on other pin fixtures, need to migrate the isPinned* tests to them as well. fix: direct pins are now deleted by a default pin.rm(hash) test: prepare for pin.add tests 'indirect supersedes direct' test exposes a bug in pin.ls feat: switch away from multihashes for isPinned* tests test: impl pin.add tests fix: add fixture files only once test: add test for a potential bug, clean isPinned* tests refactor: remove a test that's no longer needed fix: pin.ls, indirect pins should supersede direct pins test: naive pin.load, pin.flush tests feat: remove most pin cli tests as functionality is tested in pin core tests refactor: rename solarSystem * refactor: move pin http-api tests to http-api/inject fix: attempt to find a way to use http-api/inject test structure for pin tests test: fix pin.rm http-api tests test: fix pin.add http-api tests docs: docs and cleanup of http-api pin tests refactor: renaming fix: lint errors fix: resolvePaths tests are failing on CI, it might be long ops, testing a timeout bump fix: add files explicitly before testing resolvePaths fix: remove mocha.only from resolvePaths. let's hope tests pass, they are passing CI now fix: rename test/core/utils.spec.js -> utils.js so it's not run during browser tests * test: first draft of pin-set tests Need to leave computer, this is a checkpoint. test: add sanity test for walkItems and hasChild, clean others These tests are more descriptive than really pushing the impl. I'd love others' thoughts on what else should be hit and how. I also need to compare go's pinset impl against ours fix: stop daemons feat: documentation and multihash buffer handling for dag.get fix: lint * feat: simplify root dagnode generation for storeItems base case * feat: rename vars, fix _depth default value, add docs fix: pinset.hasChild buffer check feat: hardcode expected length for flush/load tests * feat: parallelize pin.isPinnedWithType * refactor: refactor pinset.storeItems * fix: re-add pin interface tests I must have missed a commit during a rebase. * fix: lint * feat: docs, rename resolvePaths, pin.getIndirectKeys now uses eachLimit * chore: rebase a month of changes, resolve minor issues from that fix: yarg arugment naming for pin cli commands fix: convert file multihashes to a b58 string fix: another way of checking for CID-ness fix: lint fix: toB58String handles non-buffers fix: key-exchange core tests now shutdown daemon. * chore: update big.js version * revert: do not pin content added with a non-default hash algorithm * revert: internalKey recording * refactor: use lodash.flattenDeep refactor: pinset.hasChild -> pinset.hasDescendent fix: invoke someCb if we've seen the hash before refactor: async patterns in dag._getRecursive refactor: pinset.hasDescendant refactor: pinset.storeItems async patterns refactor: pinset.loadSet and pin.walkItem async patterns docs: add link to go-ipfs' fanout bin implementation refactor: async patterns of pin.load/flush refactor: lint refactor: privatize internal pin key storage refactor: change encapsulation of ipfs.pin, fix resulting issues fix: lint fix: 'files add --pin=false' test was giving a false positive refactor: use is-ipfs to check CID-ability of a string refactor: remove last instance of 'once' in the pin code * refactor: do not expose pinTypes They're simple enough, documented elsewhere, and not used by any exposed functionality. * fix: do not destructure node callback results
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR has proposed changes to #1045. It mostly includes tweaks with very little change to the core functionality in the previous PR.
Summary
Depends on:
js-ipfsd-ctldoes not signal that nodes spawned on repos are initialized: fix: daemon should recognize previously created repos. js-ipfsd-ctl#212. This is a dependency of the interop tests.ipld-dag-pb-dependentdag._getRecusivehere. I'll explore it when I can.This thread should be the primary place for discussion of pinning. When these changes have been validated to some extent, we can compare this branch to master and merge it.