Add IPFS Utilities application #1187

dobschal · 2022-09-27T08:42:21Z

Project Abstract

This application is about the utilities of Substrate within IPFS.
The existing offchain::ipfs will be extended and adjusted to have a proper handling of IPFS CIDs and a redirect feature to retrieve files from IPFS.

This is a follow up to an already closed application PR: #1127

Grant level

Level 1: Up to $10,000, 2 approvals
Level 2: Up to $30,000, 3 approvals
Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

The application template has been copied and aptly renamed (project_name.md).
I have read the application guidelines.
A BTC, Ethereum (USDT/USDC/DAI) or Polkadot/Kusama (aUSD) address for the payment of the milestones is provided inside the application.
The software delivered for this grant will be released under an open-source license specified in the application.
[] The initial PR contains only one commit (squash and force-push if needed).
The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

Noc2

Thanks for the updated grant application. I’m still willing to go ahead with it and will share it again with the rest of the committee.

SBalaguer · 2022-09-30T10:56:13Z

Hi @dobschal thanks for the updated proposal! It does look very interesting to me.

Would it be possible for you though to briefly explain how this proposal is improving current implementations like place_storage_order from Crust?

semuelle · 2022-10-13T14:25:16Z

Hi @dobschal. Thank you for the application. Are you building this out of an in-house need? Since we now have multiple unmaintained forks of offchain_ipfs, it would be good to know that it won't be dropped again right away.

semuelle · 2022-10-18T20:31:53Z

ping @dobschal

bbros-dev · 2022-10-19T00:48:32Z

@Noc2 this also seems to be relevant prior art/grant:

bbros-dev · 2022-10-19T02:30:25Z

@dobschal @SBalaguer @semuelle @laboon @Noc2 : Our understanding of the state of play is in the comment after this one.

With that background..... We wonder if something like the following isn't needed?

Objective:

including the Rust implementation IPFS in the native Substrate runtime

Milestones:

An example ...

pallet, which is essentially a Rust module that complies with the requirements to be included within a substrate runtime.

Merge into the native Substrate runtime Rust IPFS crate scaffolding sufficient for native runtime support for:
- Milestone 1,
- the the Wunderbar APAC submission (Part two),
- the JuniDB grant,
- the Iris grant use cases.

Or are all these substrate forks un-necessary, and these teams have missed a trick?

Or the Rust implementation of IPFS won't land in the native Substrate runtime for the foreseeable future/ever, and the Substrate forking can be expected to continue?

If the former, can someone say how these use cases should be implemented without using Substrate forks? can some detail be added to the IPFS section here

If the later, can that fork-your-own-substrate detail be added to the IPFS section here

bbros-dev · 2022-10-19T03:13:03Z

lacked points to post this here: https://substrate.stackexchange.com/questions/5406/storing-data-offchain-to-ipfs-pallet-vs-chain-extension

Subtrate runtime with native IPFS support

Objective:

including the Rust implementation IPFS in the native Substrate runtime

Benefit:

An example ...

pallet, which is essentially a Rust module that complies with the requirements to be included within a substrate runtime.

Status: 19 Oct 2022

8 Jul 2021: Web3 Foundation Grants Program awards JuniDB grant
1 Nov 2021: offchain_ipfs project can't handle the JuniDB contribution (PR to rs-ipfs:offchain_ipfs from JuniDB team)
3 Nov 2021: Web3 Foundation Grants Program awards Iris grant
1 Dec 2021: Open APAC submission (Part One): Add IPFS into the substrate runtime and use frame v3 to build pallets which interact with it!
9 Dec 2021: Open APAC submission (Part two) Example of creating a pallet which is using IPFS-core to talk to an IPFS ocw
6 Dec 2021: offchain-ipfs manual and the accompanying rs-ipfs/substrate fork halts
- offchain-ipfs manual discussion
30 Apr 2022: Iris grant completes
27 July 2022: WunderbarNetwork/substrate
- Polkadot APAC hackathon submission: Implementation of IPFS in Substrate Runtime! (Frame V3)
4 Oct 2022: JuniDB Milestone 1 update
19 Oct 2022: An open Web3 Foundation Grants Program application for Add IPFS Utilities

Activity in last 30 days:

Status Summary

Upstream changes to Substrate are required for native Substrate runtime support of IPFS (things have been tried).
Upstream is not resistant
Upstream changes won't be easy/quick
Web3 Foundation have made two grants related/using IPFS, but not focused on getting IPFS in the native Substrate runtime
The Iris grant project completed
The offchain_ipfs project review/merge of contributions stalled due to lack of resources
Substrate forks proliferate....

Status Conclusion

Until this resolves, however that is, you'll need to use a substrate fork closest to your use case:

Or start and maintain your own fork....

bbros-dev · 2022-10-19T10:40:56Z

Hmmmm.... rs-ipfs is in a state of flux, possibly being archived, with iroh being a new candidate. Performance is impressive (presentation) but it is early days.

Substrate forking seems unavoidable for the foreseeable future, say 12+ months

dobschal · 2022-10-21T06:31:13Z

Hi @dobschal. Thank you for the application. Are you building this out of an in-house need? Since we now have multiple unmaintained forks of offchain_ipfs, it would be good to know that it won't be dropped again right away.

Hey @semuelle
We faced this problem in our NiftyMarket project. We created a workaround but with this grant we want to solve it also for others. Once this implementation is available we plan to integrate this in our NiftyMarket blockchain implementation. Updates on this pallet during our development would be then contributed back to the community.

takahser

This looks interesting and could be useful to the community. However, I do think that the price is currently too high with 16.5k/month/FTE. In your past grant the rate was a bit more than 1/3 of that price: 5.5k/month/FTE. The other teams that worked on ipfs-related grants that you quoted also demanded much less for similar scopes.

takahser · 2022-10-24T13:47:53Z

applications/ipfs_utilities.md

+Provide a pallet implementation that allows clients to add files to IPFS and store the related CID on chain with one RPC. For instance when minting an NFT with a given artwork file. 
+
+In addition to that, the Substrate blockchain can offer an RPC API to read the file from IPFS too. Therefore a redirect feature needs to be implemented based on the PRG mechanism.
+As it takes a few moments until an added file is available on the IPFS network thru public gateways, the local IPFS node can used to retrieve the file.


Suggested change

As it takes a few moments until an added file is available on the IPFS network thru public gateways, the local IPFS node can used to retrieve the file.

As it takes a few moments until an added file is available on the IPFS network thru public gateways, the local IPFS node can be used to retrieve the file.

takahser · 2022-10-24T13:56:17Z

applications/ipfs_utilities.md

+### Ecosystem Fit
+
+The implementation helps people using IPFS with Substrate, this is a common scenario.
+Especially for NFT products this solution is helpful to avoid external middleware implemetations taking action. Having the whole logic for minting NFTs with related content inside Substrate makes it easier to maintain and more secure.


Suggested change

Especially for NFT products this solution is helpful to avoid external middleware implemetations taking action. Having the whole logic for minting NFTs with related content inside Substrate makes it easier to maintain and more secure.

Especially for NFT products, this solution is helpful to avoid external middleware implementations taking action. Having the whole logic for minting NFTs with related content inside Substrate makes it easier to maintain and more secure.

semuelle

Thanks. Happy to proceed.

alxs

@bbros-dev thanks for the in-depth summary. Regarding the wiki, feel free to open an issue in its repo.

@dobschal thank you for the updated application. I agree with @takahser regarding the rate, would be great if you could address this. But more importantly, as @bbros-dev pointed out, rust-upfs has now been archived, so it probably doesn't make a lot of sense to build a new integration with Substrate on top of it. Even with iroh being in its early days, I'd be more willing to support a project based on it. Such a project may be likely to break, but given the current state of things, relying on rust-ipfs would make it certain to break. Curious to hear your thoughts.

takahser

I still think it's too expensive, as already indicated and I haven't heard any feedback on that yet.

dobschal · 2022-10-27T12:21:59Z

Hey, thanks for the feedback.

As there are too many unsolved doubts we are considering other options.

Thanks!

dobschal · 2022-11-01T10:01:21Z

We would like to reopen this grant. There was a misunderstanding on our side and we would be very happy to continue with this grant. If you have any questions, please feel free to ask.

Noc2 · 2022-11-01T14:09:41Z

I shared your application again with the team.

github-actions · 2022-11-01T14:49:30Z

Congratulations and welcome to the Web3 Foundation Grants Program! Please refer to our Milestone Delivery repository for instructions on how to submit milestones and invoices, our FAQ for frequently asked questions and the support section of our README for more ways to find answers to your questions.

Before you start, take a moment to read through our announcement guidelines for all communications related to the grant or make them known to the right person in your organisation. In particular, please don't announce the grant publicly before at least the first milestone of your project has been approved. At that point or shortly before, you can get in touch with us at grantsPR@web3.foundation and we'll be happy to collaborate on an announcement about the work you’re doing.

Lastly, please remember to let us know in case you run into any delays or deviate from the deliverables in your application. You can either leave a comment here or directly request to amend your application via PR. We wish you luck with your project! 🚀

dobschal · 2022-11-03T08:39:51Z

Thanks for the approvals 🥳 Will check the next steps and are super happy to jump on that!

dominikdem · 2023-01-03T08:29:02Z

Hello W3F Team; we started the implementation phase.

keeganquigley · 2023-02-02T22:30:16Z

Thanks @dominikdem how is M1 coming along?

dominikdem · 2023-02-03T06:44:34Z

Hi @keeganquigley, thanks for reaching out. After some challenges, we have made good progress and are about to finish M1 by the end of next week.

dobschal · 2023-02-10T15:09:54Z

Hey 👋
we created the PR for the milestone 1 delivery.
w3f/Grant-Milestone-Delivery#743

keeganquigley · 2023-02-10T15:54:58Z

@dobschal awesome thanks for the submission! Someone will look at it shortly.

dobschal added 6 commits August 18, 2022 07:25

Add IPFS Utilities application

18bd174

remove Milestone 3

6dd0644

Merge branch 'master' into application/ipfs_utilities

c94c6e2

Update IPFS util application

e50b81f

Update IPFS utilities grant application

6b5db3f

Merge branch 'master' into application/ipfs_utilities

31fcebd

dobschal mentioned this pull request Sep 27, 2022

Add IPFS Utilities application #1127

Closed

10 tasks

Noc2 approved these changes Sep 28, 2022

View reviewed changes

Noc2 self-assigned this Sep 28, 2022

Noc2 added the ready for review The project is ready to be reviewed by the committee members. label Sep 28, 2022

gautamdhameja approved these changes Sep 30, 2022

View reviewed changes

laboon approved these changes Sep 30, 2022

View reviewed changes

Noc2 requested a review from semuelle October 24, 2022 09:49

takahser requested changes Oct 24, 2022

View reviewed changes

semuelle approved these changes Oct 25, 2022

View reviewed changes

Noc2 requested a review from takahser October 25, 2022 15:25

alxs suggested changes Oct 26, 2022

View reviewed changes

takahser requested changes Oct 27, 2022

View reviewed changes

dobschal closed this Oct 27, 2022

dobschal reopened this Nov 1, 2022

keeganquigley approved these changes Nov 1, 2022

View reviewed changes

Noc2 merged commit d3beeb9 into w3f:master Nov 1, 2022

taqtiqa-mark mentioned this pull request Dec 15, 2022

Grant: Substrate Tokenomics Survey #1348

Merged

9 tasks

dobschal mentioned this pull request Feb 10, 2023

Add delivery milestone 1, IPFS utilities w3f/Grant-Milestone-Delivery#743

Merged

5 tasks

dominikdem mentioned this pull request Mar 8, 2023

Application/ipfs utilities 2 #1598

Closed

10 tasks

dobschal mentioned this pull request Mar 30, 2023

IPFS utilities milestone 2 w3f/Grant-Milestone-Delivery#815

Merged

5 tasks

randombishop mentioned this pull request Apr 19, 2023

IPFS Utilities milestone 2 w3f/Grant-Milestone-Delivery#838

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add IPFS Utilities application #1187

Add IPFS Utilities application #1187

dobschal commented Sep 27, 2022

Noc2 left a comment

SBalaguer commented Sep 30, 2022

semuelle commented Oct 13, 2022

semuelle commented Oct 18, 2022

bbros-dev commented Oct 19, 2022 •

edited

Loading

bbros-dev commented Oct 19, 2022 •

edited

Loading

bbros-dev commented Oct 19, 2022 •

edited

Loading

bbros-dev commented Oct 19, 2022

dobschal commented Oct 21, 2022 •

edited by takahser

Loading

takahser left a comment

takahser Oct 24, 2022

takahser Oct 24, 2022

semuelle left a comment

alxs left a comment

takahser left a comment

dobschal commented Oct 27, 2022

dobschal commented Nov 1, 2022

Noc2 commented Nov 1, 2022

github-actions bot commented Nov 1, 2022

dobschal commented Nov 3, 2022

dominikdem commented Jan 3, 2023

keeganquigley commented Feb 2, 2023

dominikdem commented Feb 3, 2023

dobschal commented Feb 10, 2023

keeganquigley commented Feb 10, 2023

	As it takes a few moments until an added file is available on the IPFS network thru public gateways, the local IPFS node can used to retrieve the file.
	As it takes a few moments until an added file is available on the IPFS network thru public gateways, the local IPFS node can be used to retrieve the file.

	Especially for NFT products this solution is helpful to avoid external middleware implemetations taking action. Having the whole logic for minting NFTs with related content inside Substrate makes it easier to maintain and more secure.
	Especially for NFT products, this solution is helpful to avoid external middleware implementations taking action. Having the whole logic for minting NFTs with related content inside Substrate makes it easier to maintain and more secure.

Add IPFS Utilities application #1187

Add IPFS Utilities application #1187

Conversation

dobschal commented Sep 27, 2022

Project Abstract

Grant level

Application Checklist

Noc2 left a comment

Choose a reason for hiding this comment

SBalaguer commented Sep 30, 2022

semuelle commented Oct 13, 2022

semuelle commented Oct 18, 2022

bbros-dev commented Oct 19, 2022 • edited Loading

bbros-dev commented Oct 19, 2022 • edited Loading

bbros-dev commented Oct 19, 2022 • edited Loading

Subtrate runtime with native IPFS support

Status: 19 Oct 2022

Status Summary

Status Conclusion

bbros-dev commented Oct 19, 2022

dobschal commented Oct 21, 2022 • edited by takahser Loading

takahser left a comment

Choose a reason for hiding this comment

takahser Oct 24, 2022

Choose a reason for hiding this comment

takahser Oct 24, 2022

Choose a reason for hiding this comment

semuelle left a comment

Choose a reason for hiding this comment

alxs left a comment

Choose a reason for hiding this comment

takahser left a comment

Choose a reason for hiding this comment

dobschal commented Oct 27, 2022

dobschal commented Nov 1, 2022

Noc2 commented Nov 1, 2022

github-actions bot commented Nov 1, 2022

dobschal commented Nov 3, 2022

dominikdem commented Jan 3, 2023

keeganquigley commented Feb 2, 2023

dominikdem commented Feb 3, 2023

dobschal commented Feb 10, 2023

keeganquigley commented Feb 10, 2023

bbros-dev commented Oct 19, 2022 •

edited

Loading

bbros-dev commented Oct 19, 2022 •

edited

Loading

bbros-dev commented Oct 19, 2022 •

edited

Loading

dobschal commented Oct 21, 2022 •

edited by takahser

Loading