Thank you for joining the conversation and feedback. I do understand the point being made by yourself and Murch, which is that data carriers are exploiting permanance. However, i don’t believe this is necessarily the case and have tried to make this clear in the BIP.
Firstly, the incentive to use it is now established economically, as i posted the rationale above. The economic incentive is what has drawn data into witness, undercutting op_return. This is a side effect of the discount. On this basis it can be asserted that carriers are economically rational and respond to economic incentives.
Secondly, Bitcoin gives no guarantees of redundancy, whereas it does in theory (re: pruning) promise availability. SegData gives a degree less redundancy, that is conceded, but availability is also in theory promised. This nuance presents an opportunity to data carriers to declare their data prunable for a discoutn, which they are not able to do at the moment. The upside of this is that data continues to compete within the weight budget, and reduces burden on smaller resources constrained nodes, relocating data and contributing to decentralization e.g. a full SegData block would be ~1KB for an opt-out node.
I’m not pushing the proposal as a must-have, i’m just simply trying to establish it as a technical option which follows clearly in the design lineage of witness. No consensus requirement on data + script isolation enables pruning at the node by entry or type.
Posting the relevant parts of my mini-review in the BIPs repo:
The newer proposal would make data optional at every depth and keep consensus validation identical, fixing the reorg risk AJ Towns described. Nodes would only check a commitment plus the declared size against the weight limit; everything else would be policy.
It would be a soft fork for something that doesn’t need consensus rules. One can already commit to arbitrary data hashes today. The data still consumes block weight even though no one is required to keep or serve it. Most nodes would ignore it, making availability unreliable. Use cases that care about permanence would stick to normal witness or OP_RETURN data.
It would be adding protocol complexity for a feature most nodes will turn off and that has weak incentives to be used at all.
I agree with @garlonicon above. It would be nice if it could work, but incentives.
OP_RETURN is suitable under a small size threshold but above that it is not economically rational and hence why data is being carried in vectors otherwise not intended for it.
Yes, IIUC large arbitrary blobs would be that data that makes the most sense to use SegData for, with the caveats of (a soft fork, increased protocol complexity, and) whether enough nodes are willing to host and serve the data.
hey, i did respond earlier in brief to your review but i’ll repeat here ICYMI.
The newer proposal would make data optional at every depth and keep consensus validation identical, fixing the reorg risk AJ Towns described. Nodes would only check a commitment plus the declared size against the weight limit; everything else would be policy.
This is done, it’s in the PR bip pages and in the later links here. Maybe you saw the peer services BIP which has a 288 block best effort policy. I also added the following section:
Uniform Validation Consensus validation is identical at every depth. Every node applies the rules above from the base serialisation, the transactions and the coinbase commitment, for a block at the tip and for one buried under the deepest reorg alike, whether or not it holds the segdata region. There are no depth-scoped validation modes and no burial trust. A block’s validity is fixed by its base serialisation when it is first seen and never changes, so two nodes cannot reach different verdicts on the same block by having encountered it at different depths.
I agree with @garlonicon above. It would be nice if it could work, but incentives.
I think this is the common criticism aside from any initial weirdness or technical peculiarities with the whole concept (although it is basically a simplified SegWit design + script-isolation → prunability). I’ve tried to address the incentives questions but i’ll try again. There are two sides and we are maybe conflating the two.
Carriers.
Economic incentive: done, cheaper than witness (side effects a separate question)
Permanancy incentive: addressed via the redundancy vs availability argument. Bitcoin guarantees both although existing prune degrades both. SegData is in the same pattern but no doubt deeper, less redundancy → higher availability risk offset by economic incentive. A single copy of data provides availability, full archival nodes are highly likely to exist because they are explicitly the full node position, that’s the definition. In any case, if data “expires” then it served its purpose and is forgotten.
Exploit incentive: unaddressable by any mechanism, out of scope.
Operators.
Economic incentive: none, exactly the same as existing carriage. Everybody carries the full history unless they prune. However, coverage tiers and prunability allow an operator to reduce their data burden without compromising their money burden. This is better than the existing.
Decentralization incentive: offering something unique here. Resource contrained nodes do not receive or carry data payloads. This is a net win for decentralization and an inversion of the data burden into a benefit.
Am i missing any specific incentive question? It’s easy to wave it away as idealistic or not probable, but we should be able to rationalise about outcomes given the design, and if there is a flaw it should be revealed specifically.
Full nodes carry the entire consensus-relevant history: they can fully validate that the entire history is consensus valid and arrive at the current UTXO set by themselves. As proposed, a full node does not require the proposed segregated data, and therefore I cannot imagine full node implementations defaulting to retaining SegData, simply because it uses additional resources to facilitate an optional feature.
I don’t see the case for this manifesting as an opt-out feature, it seems obvious that it would be an opt-in feature like txindex or client-side block filters. I assume this is why almost everyone here thinks that a vanishing portion of nodes would retain the data.
I’ll respond with bullets so specific points can be addressed and referenced, as with the incentives points earlier.
Although SegData entries are not, block weight is consensus relevant so there is an argument for validation by policy.
The status quo is that data would be carried forever for free in existing vectors with no choice at all. This new idea exists to offer that choice and dissolve the argument.
Opt-in mode would of course be less compelling to carriers than the default, but now we are discussing the default relay mode, the effective setting for the whole concept, so arguing its weak form is like arguing prune’s weak form.
Coverage tiers give operators a choice in retention size, it’s not just all or nothing.
This proposal is default SegData enabled. Arguing against the default off is a different proposal which i have not proposed.
Building a smaller than permitted block is always valid.
Even if it were on by default, almost every node setup guide would recommend to turn it off for a smaller data footprint. Sorry for repeating myself, but this seems completely uninteresting because I just don’t see the incentives work out.
The apparent position from a few later comments here is “maybe nice idea, not realistic because incentives”. Outstanding concerns seem to be no longer about whether SegData is technically functional, but about whether it will be adopted.
SegData mirrors the BIP-159 precedent as closely as possible so an availability argument against pruning by type also weighs against pruning by depth, which the network already accepted and built BIP-159 to manage. SegData retention defaults to archival, coverage tiers provide depth control, and the recent-block floor policy keeps recent entries universally served, so redundancy stays broad unless operators actively opt out.
SegData is being judged against a uniform-redundancy ideal that does not exist. Pruning is already here, uniform redundancy is not real.
If SegData is not implemented, operators are forced into carrying arbitrary data which is inseparable from block storage. SegData separates it, makes it consensus-irrelevant and independently prunable, and gives the operators a choice they wouldn’t otherwise have. That choice improves the decentralization of the network because it allows resource-constrained nodes to carry less burden. That decentralization effect is a central claim, and it is the one that has not been engaged with, and has important incentive implications.