Reduce PublishStorageDeals cost

[MatrixStorage]

** Note: This discussion was created based on valid data storage. The cost of PublishStorageDeals range from 1FIL/TB to 2.5FIL/TB which is too high.

In the process of introducing valid data to Filecoin network, we found that many storage providers are unwilling to store valid data, mainly because of the high cost which consists of sealing cost and PublishStorageDeals.

Proposal:
1. The cost of PublishStorageDeals should depend on the size of the data.
Smaller data size should be implemented with less PublishStorageDeals fee. Otherwise, storage providers are likely to defer data uploading to the chain, which could end up with bad user experience for clients.

2. Reduce the PublishStorageDeals cost to near 0.
To deal with the negative effects of valid data storage costs, thus improving the storage service provider's willingness to store valid data.

[Fatman13]

2. Reduce the PublishStorageDeals cost to near 0.
   To deal with the negative effects of valid data storage costs, thus improving the storage service provider's willingness to store valid data.

This would make network vulnerable to DDOS attacks...

[MatrixStorage]

1. 将 PublishStorageDeals 成本降低到接近于 0。
   应对有效数据存储成本的负面影响，从而提高存储服务提供商存储有效数据的意愿。

这将使网络容易受到 DDOS 攻击......

Thank you for you reply!
How about setting up an account to make a part of PublishStorageDeals cost refundable. Only if the storage providers actually complete the service, the refund can be valid.

[jennijuju]

How about setting up an account to make a part of PublishStorageDeals cost refundable. Only if the storage providers actually complete the service, the refund can be valid.

Why do you think the network should pay for the PSD message? and how to prevent future proposal like "network should pay for precommit, prove commit and other messages too?" as those are the costs that SP are spending for onboarding sectors.

+1 to @Fatman13 's point, if PSD message is cheap but has high gas usage, how to prevent the network get DDOSed / mpool congested with it (wdpost get stuck in mpool potentially)?

Im super supportive for teams to investigate how to optimize the gas usage of PSD message in go-spec-actor tho. Even, do we need PSD to get those data into the states, can we do that along with later steps like PC (later PUR) and do market.ActivateDeals there directly?

[jennijuju]

1. The cost of PublishStorageDeals should depend on the size of the data.
   Smaller data size should be implemented with less PublishStorageDeals fee. Otherwise, storage providers are likely to defer data uploading to the chain, which could end up with bad user experience for clients.

@ZenGround0 please correct me if im wrong, but I think there is no gas usage that's dependent on the deal size? Im looking at PublishStorageDeals(rt Runtime, params *PublishStorageDealsParams) and dont seem to find any load/write/computation that's dependent on deal size.

[laurarenpanda]

@jennijuju mentioned this discussion earlier today.
filecoin-project/notary-governance#324 (comment)
Using secp256k1 wallet for deal-making could be a reliable way to reduce the PublishStorageDeals cost for now.

1. The cost of PublishStorageDeals should depend on the size of the data.

I think this idea has some feasibility.
Online deals are very often stuck on data transfer sessions. We usually need to decrease deal size into 1-8 GiB to make the online deals successful. However, the cost of PublishStorageDeals won't reduce at the same time.

[AthenaPoolOfficial]

Our company has focused on Filecoin storage and expected to store more valid data for a long time. However, we encounter the problem of time and high gas cost in this process, which has borthered us a lot. We sincerely hope to have more practical and available schemes to solve the dilemma.

[FroghubMan]

Similar to FIP-0009, eliminate base-fee burn of successful PublishStorageDeals message.

[MatrixStorage]

Why do you think the network should pay for the PSD message? and how to prevent future proposal like "network should pay for precommit, prove commit and other messages too?" as those are the costs that SP are spending for onboarding sectors.

Our intention is not to make the network to pay for it, but to optimize the onboarding process for valid data.

How to introduce more verified storage clients and truly store their valid data in the network is the most important thing in the early days of the ecosystem. From our point of view, PSD, as the crucial factor to onboarding valid data, has become the entry threshold for verified storage clients due to its high cost.

In other words, it is blocking the introduction of real storage. The cost of PSD should be reduced or optimized, or at the very least, given attention.

我们的初衷并不是让网络为其付费，而是为了优化onboarding流程。处在生态初期的当下，如何引入更多真实存储客户、并真正将他们的数据存到网络中是重中之重。在我们看来，PSD作为真实数据onboarding的必经之路，已经因为费用高昂成为了真实存储客户的进入门槛。换言之，它正在阻碍真实存储的引入。那么PSD的费用理应降低或进行优化。

Status Quo:

1. DataCap is possibly a short-term incentive, which has really strict requirements on data, such as open data, long duration of application and onboarding process.
2. It may result in same adj. power no matter storage providers store valid data or garbage data. However, the cost (time, Gas) can be very different.

From the business level, in pursuit of profit, storage providers are likey to stop at CC sector sealing. No one would like to see that.

As we all know, Filecoin's true value is only realized when real applications, real data, enter Filecoin network. In addition, the introduction of valid data will lead to a bigger Filecoin boom and more tokens for pledging.

As for the other costs you mentioned, it seems that they are the costs that storage providers have to pay for adj. power, which is a mature setting for the entire economic model to maintain stability of the network.

现状：
1、DataCap只是一个短期的激励措施，且对数据有非常严格的要求，例如数据需要开源、申请及onboarding流程冗长。
2、存储服务商存储有效数据或者垃圾数据都是产生一样的算力，成本（时间、Gas）却大不一样。

逐利为本，存储服务商从商务层面就只会停止于CC扇区封装。这将是谁也不乐见的。

我们都知道，只有当真实的应用、真实的数据进入Filecoin，才会使得Filecoin真正的价值得到体现。此外，随着真实数据的引入也会带来Filecoin的更大繁荣，以及更多的代币用于质押。至于您提到的其他消耗，在我看来主要是存储服务商获得算力奖励所必须的支出，是整个经济模型的用以维持稳定的成熟设定。无意冒犯。

Proposals:
Is it possible to design a "multiple deal merge submission verification" approach? One package only charges one time. Or as @FroghubMan mentioned, to exempt direct PSD messages that execute successfully from base-fee burn (i.e., don't burn baseFee*gasUsed).

By doing so, the storage providers can further reduce gas expenses on the cost of PSD and increase the willingness to accept orders to store valid data. We also hope that you can introduce more underlying technologies to us. We are looking forward to more communication opportunities.

具体做法的建议：
是否可以设计“多笔deal合并提交验证”的方式？该打包只收取一笔订单费用。或者如 @FroghubMan 所言，直接免除PSD消息，成功执行从基本费用燃烧(即，不燃烧baseFee*gasUsed)。
这样做的话，对存储提供商来说，即可进一步降低发送订单成本上的gas支出，提高接受存储有效数据订单的意愿。当然，也希望您能向我们介绍更多更优的底层技术步骤。期待更多交流。

+1 to @Fatman13 's point, if PSD message is cheap but has high gas usage, how to prevent the network get DDOSed / mpool congested with it (wdpost get stuck in mpool potentially)?

As for DDOS attacks, there are three main points:

1. PSD can only be launched by storage providers, and theoretically they have no incentive to launch DDOS attacks.
2. under the case of PSD，storage nodes are required to pay storage pledge. Therefore, even if Gas fee is reduced, the economic model will help the system resist DDoS attacks.
3. Any message type may have the risk of DDOS. As PSD is one of the core message types, we should improve the defense capability of DDOS from the technical level rather than simply increase Gas cost.

至于DDOS攻击，有三个要点：

1. PSD是存储提供商才能够发起的，理论上他们并无发起DDOS攻击的动机。
2. PSD逻辑下存储节点是需要付存储质押的，即使Gas费用降低，经济模型也会帮助系统抵御DDoS攻击。
3. 任何一种消息类型都可能具有DDOS的风险，PSD作为其中一种核心的消息类型，我们应从技术层面提升DDOS的防御能力而非简单提高Gas费用。

Im super supportive for teams to investigate how to optimize the gas usage of PSD message in go-spec-actor tho. Even, do we need PSD to get those data into the states, can we do that along with later steps like PC (later PUR) and do market.ActivateDeals there directly?

Thanks for your informing!

[darrellhendershot]

Agree with @MatrixStorage the CC sector is almost full with the sealed random data right now, the client can store the data first with a limited cost and the SP can put the Pledge after the storage is complete while enjoying the 10X power growth.

The PSD is too high to make the deal successful. Frankly speaking, such a huge space will be the best storage provider for enterprise not for the individual.

Also, you guys can follow the link to see this opinion #116, I also support this FIP. The 13.6EiB+ space can be severed for a lot of enterprises.

[jennijuju]

PSD can only be launched by storage providers, and theoretically they have no incentive to launch DDOS attacks.

Anyone can send any message, the same goes to PSD.

Any message type may have the risk of DDOS. As PSD is one of the core message types, we should improve the defense capability of DDOS from the technical level rather than simply increase Gas cost.

Both @Fatman13 and I mentioned DDOS is response of the original proposal "reduce the PSD cost near to 0", which is very different from "increasing gas cost to prevent DDOS"

Is it possible to design a "multiple deal merge submission verification" approach.

You can already publish multiple deals in one PSD message (not saying its most gas efficient, but it does exist).

Or as @FroghubMan mentioned, to exempt direct PSD messages that execute successfully from base-fee burn (i.e., don't burn baseFee*gasUsed).

How would you verify PSD messages? Blindly refunding all base fee burn to PSD may cause DDOS attack imho. For wdpost, we have a dispute system for bad wdposts.

To re-iterate, above comments were my opinions wrt the two proposals being mentioned so far only. the problem is clear - data onboarding needs to be cheaper for network usability.
@MatrixStorage However, I think more explanation on why "The cost of PublishStorageDeals range from 1FIL/TB to 2.5FIL/TB which is too high." What is it compare to? How much is reasonable and why? I think these answers will be crucial for coming up an acceptable solution.

[IreneYoung]

Well, as some progress has been made in the ecosystem, the Filecoin network has yet to become a mass market for real and useful data storage. I learned from the last governance meeting that It seems more than a few clients encountered a number of difficulties when cooperating with some storage providers. There must be some peoblems. However, I don't know the technical details on sealing the useful data through DataCap. If it would benefit the whole network and make a breakthrough out of the current status once we optimize the cost of storage providers, then let's discuss further more

[MatrixStorage]

@jennijuju Thank you for your reply.

What is it compare to?

- Storage fee comparison -

We compare the prices of centralized and decentralized storage in the real storage market:

According to the 90 days' records of PSD, we find out the average cost of storage is around 1.5 FIL/T. Meaning that 1T storage costs about 367 CNY, which is nothing competitive to the centralized storage mode in the storage market in China, even with the low price of FIL recently.
Data source: Athena browser (https://www.atpool.com/zh/dataCap)

How much is reasonable？

- Status Quo of the market -

We want to push the establishment of decentralized storage commercialization and compliance model. This leads to high costs in development, operation, sales, promotion and etc. in the actual business scenario. In order to introduce more important customers to our platform, we adopted the strategy of free storage. The benefits of decentralized storage products do not currently cover our costs. Indeed, DataCap is motivating a lot of storage providers. But we don't think DataCap can work as a long term solution.

Secondly, factors such as strict policy, market trend, tiring process of valid data storage(which takes too much time), occasional bugs of underlying development technology and etc, have brought negative impacts on introducing more costumers(storage clients).

- Proposals -

To deal with the problem of cost, we have two suggestions:

1. Replacement of CC sector will incur additional PSD and sealing costs, and we suggest these costs to be exempted.
2. Exempt the cost of PSD for new storage.
   Ps: The logic of pledge remains the same.

Convincing traditional Internet customers to adopt decentralized storage needs a long education period. We want to be competitive on price.

---

和什么比它太高了？

- 使用成本对比 -

我们将真实存储市场中的中心化存储和分布式存储价格进行了对比：

通过雅典娜浏览器（https://www.atpool.com/zh/dataCap ） 近90天的PSD消息发单成本展示，单T的平均成本大约在1.5FIL左右，根据最新FIL价格（CNY245），单T存储费用约为CNY367。即便币价低迷，分布式存储的价格在存储市场中仍然不具备任何竞争力。

多少是合理的？为什么？

- 市场现状 -

我们想要推动分布式存储商业化与合规化模式的建立，在实际的业务过程中，会产生大量的开发、运营、销售、推广等成本。为了将部分重要客户引入平台我们采用了免费存储的策略。分布式存储产品的收益目前不能覆盖这些成本。尽管使用DataCap的逻辑短期内可以起到激励存储服务商的目的，但我们认为不是长久之计。
其次，政策管控、市场走势、有效数据存储时间长、底层偶尔出现Bug等因素影响，这些都给我们引入更多的用户带来了影响。

- 费用建议 -

1.CC扇区替换会额外产生订单发送成本和封装成本，我们建议免去这部分费用。
2.对于新建存储这部分订单的发送成本降低为0。
Ps：质押逻辑不变。

说服传统互联网客户采用分布式存储需要相当长市场教育时间，我们希望至少能在价格上具备竞争优势。

[anorth]

We are painfully aware that doing on-chain deals is expensive, and presents a significant friction to client adoption. We are exploring both short- and long-run solutions to this. However, we can't just arbitrarily change the gas fees of PSD to be decoupled from the computation and state cost imposed on every validator in the network. We need to design marketplace solutions that are more efficient. And pending such protocol changes, direct clients to services like Estuary and Web3.Storage to act as brokers/aggregators.

exempt direct PSD messages that execute successfully from base-fee burn

Unfortunately PSD is not comparable to Window PoSt. SubmitWindowPoSt is a permissioned method that may only be called by storage providers, on their own miner actor. Further we know exactly how many of these messages can be accepted so can bound the network cost. A provider cannot cause arbitrary computation and state growth with Window PoSt messages, but they can with PSD. We can't adopt the same policy for PSD without opening up an easy DOS attack.