[BIP Draft] Segregated Data: a prunable, script-isolated block region for data carriage

Protocol Design
23

Thank you for the reply. I think there are a few points I’d like to address before the proposal is overlooked.

Within the retention window (288 blocks) full nodes require the data. Absent data cannot be accepted. Existence is enforced network-wide at inclusion, exactly as for witness. The optional part is re-validation after burial, the same trust model as assumevalid, except a SegData node can drop the entries rather than have to keep and skip them. They gate no spend, so are never needed for validation again. Consensus not needing the data is the point, and what makes the download-skip safe.

Therefore you’re right that there’s no guarantee of perpetual retrievability, though under default archival that is a question of redundancy rather than availability. What is enforced in full is proof of publication at inclusion, which is exactly what witness gives you. SegData keeps that while removing only the obligation to retain.

Carriage is already happening under the witness discount and will continue. The weight you call purposeless is weight a small node on a metered link must download during IBD and keep on a constrained disk today. This proposal (which i believe is less complex than segwit and follows in its design lineage) offers an alternative which lets this or any other node skip historical entries during sync and retain only the window, while still fully validating the chain and serving the monetary history. Pruning doesn’t offer this. So a question arises, is whether requiring every node, including the smallest, to retain abritrary data forever is better than letting it drop it?

I appreciate your time.

1 Like
25

For the record, some erroneous assumptions about the SegData design have been made. The inferrence from these errors has lead to a strange logical conclusion about free carriage. Perhaps it is my mistake for not making it clearer although i don’t know how it can be clearer than “Within the retention window every node enforces all rules” and “it conforms to existing block size and tx weight”. In any case, this proposal stands for the time being awaiting further discussion.

26

The question that is unclear is, if the SegData is not consensus relevant, how would the retention window be enforced?

1 Like
28

Apologies for being unclear earlier, let me state it more accurately.

Entry content (the payload) is not consensus relevant since no opcode can read it, it gates no spend, and the UTXO set never depends on it. This is what allows entries to be discardable by design.

Entry presence is subject to consensus within the retention window. A block is invalid unless its committed entries are present and match the commitment. This guarantees the committed data was correct at inclusion. No incorrect payloads are possible. Beyond the retention window that presence check is no longer required, so it’s trusted to burial.

The design is borrowing heavily from the witness+assumevalid pattern.

One more clarification. When referring to prunability, i’ve been overloading the term. SegData extends pruning from whole blocks to any segdata part of blocks. The default is to not prune, it just offers a granular choice to the operator.

1 Like
29

Suppose that you did this soft fork and it was successful, but some people decided they didn’t like it and refuse to run it, instead either disabling the new rules or just running a fork of 31.x or 29.x or whatever. In that case, I think you’re risking ~1000 block reorgs if miners don’t really care about the rule.

Consider the case where a miner mines a block with missing segdata (red). Then the most-work block will be rejected by enforcing nodes, who’ll keep their tip (green) pointing at the parent:

graph LR;

x[...] --> 2000 --> 2001a*;

classDef miss fill:red;
class 2001a* miss;

classDef tip fill:green;
class 2000 tip;

If miners mess around, they may spend more work building on this rejected block, despite most nodes following the less work chain that’s not missing data:

graph LR;

x[...] --> 2000 --> 2001a* --> 2002a --> 2003a --> y[...] --> 2010a;

2000 --> 2001b --> 2002b --> 2003b;

classDef miss fill:red;
class 2001a* miss;

classDef tip fill:green;
class 2003b tip;

However, what happens once the retention window passes? Consider a 100 block retention window, with 2x the work being applied to the missing-data branch. At block 2100a/2051b, as an enforcing node you’re still following the less-work chain, because the missing-data block is still in the retention window (non-yellow):

graph LR;

x[...] --> 2000 --> 2001a* --> 2002a --> y[...] --> 2099a --> 2100a;
2000 --> 2001b --> z[...] --> 2051b;

classDef oow fill:yellow;
class x,2000 oow;

classDef miss fill:red;
class 2001a* miss;

classDef tip fill:green;
class 2051b tip;

But a block later, the missing data no longer matters, and the block is now completely valid, so to stay in consensus with new nodes just coming online, you’ll do a big reorg, backing out 51 blocks on the shorter chain and filling in 101 blocks on the more work chain:

graph LR;

x[...] --> 2000 --> 2001a* --> 2002a --> y[...] --> 2100a --> 2101a;
2000 --> 2001b --> z[...] --> 2051b;

classDef oow fill:yellow;
class x,2000,2001a* oow;

classDef tip fill:green;
class 2101a tip;

That will cost the miners on the reorged-out chain all their rewards, which goes back to justifying why more hashrate would build on the “invalid” chain; bitcoin’s model is eventual consensus, so if the most work chain will eventually be valid, extending that chain is the thing to do. As far as I can see that devolves to miners don’t enforce the retention rules at all, so missing data blocks (eg from buggy or malicious low hashrate miners) aren’t necessarily rare, and because they don’t want to see rare but large reorgs or stalls as a result of such blocks, nodes don’t enforce availability either, and the soft fork can’t be maintained.

You can add extra rules/assumptions to try to avoid this – eg, “but as long as a majority of hashrate enforces availability, it’s fine: the data-available chain will have more work”. But the idea behind node enforcement is that it works even if a majority of hashrate violates the rules: their work just gets judged as invalid and ignored, with the most-work valid chain winning.

2 Likes
31

Thanks for the reply and explanation. I understand the problem.

Consensus rules need to be deterministic, and the entry presence rules i defined aren’t since their predicate can flip once a block buries. The pivot is the one Adam Back suggested (on X), reduce the consensus rule to the commitment weight check only. That predicate can’t flip, so there’s no reorg risk.

That moves entry validation out of consensus and into node policy. The coinbase commits the entry hash and length L, so every node knows both and will accept at most L bytes and reject anything that doesn’t match the committed root. Byte quantity is still bounded by weight, bandwidth, and storage, even against a hostile majority since the commitment is authoritative and the weight check is still consensus.

The only gap is availability, whether the committed data is ever provided. A miner could withhold data, but not even a majority can corrupt it, since all nodes can verify and integrity is replayable and preserved. Witholding data would simply be a pointless expense. Anyway nothing relies on the data so witholding it is not a risk, and that’s also the point of the whole thing and what makes it prunable.

Can you see a problem with this? Appreciate your time.

32

A miner including the transactions but withholding the data would 1) collect the fees, 2) see the block propagate fine among unupgraded nodes, 3) use less bandwidth. Other nodes might propagate the data for their transaction seeing the block, but the block itself would not propagate with SegData, since nobody has the entire SegData. This would probably lead to reduced visibility of the SegData and reduced data availability. So, data embedders would be be incentivized to use witness data instead of SegData for the same cost which guarantees data availability.

I just don’t see how this proposal solves any problem for any party.

33

I see you are challenging the incentives and that’s fair but I’d like to counter the initial premise.

Miners could only withhold their own out of band entries. All other entries are relayed between peers as part of transaction serialization and so are available for validation and reconstruction from compact blocks. In withholding their own data (or any other attack) block propagation would be slowed and so they lose any race. It’s a pointless and expensive attack with no upside.

Where segdata nodes are pruned in full, let’s say 20% of the network doesn’t serve, availability is still guaranteed, it’s redundancy of the entries only that is reduced. Since pruning already exists, redundancy of blocks is already reduced.

Coming to incentives, making SegData cheaper is an option but I don’t think it’s right. I’m also leaning on the op_return precedent as a carriage vector that drew data out of fake addresses. I don’t think there was any real incentive to use it but people moved because its designated.

34

In withholding their own data (or any other attack) block propagation would be slowed and so they lose any race.

For someone to lose a race, there has to be have to be competing chaintips in the first place, and the effect of propagation delay is often overestimated.

I don’t think there was any real incentive to use [OP_RETURN] but people moved because its designated.

OP_RETURN was cheaper than embedding data into payment outputs. E.g., a P2PKH output takes 34 bytes of which the 20-byte hash could be used for data embedding. For a payload of n an OP_RETURN output takes 11+n bytes with the original policy limiting n to 80 bytes. OP_RETURN has less overhead for any payload length.

35

I should point out after AJ’s scenario i’m proposing the reduced consensus rule version of the proposal, although i have to update the BIP, assuming there’s no deal breakers.

Right, racing is just one scenario. However the key point holds which is that withholding data isn’t an attack, it’s more of a miner footgun.

OP_RETURN was cheaper than embedding data into payment outputs

Thanks for the correction. If it’s purely economic incentivisation that is required then it is possible to discount deeper, it can be an open question. I discuss the scenarios somewhat in the bip.

This is an honest attempt to propose a structural solution to reduce node/operator burden and decollectivse bytes discrimination, without being coercive. I admit that through my own analysis, i’m making the assumption that it is beneficial to draw data out of op_return and witness into something like SegData which offers data prunability. Are there any other proposals i should look at which are addressing this at the moment, or am i completely off track?

36

Important follow up regarding the economic incentive, i ran the numbers and this is what i get:

Payload SegData Witness Commit+Reveal Saving Saving %
256 B ~865 WU (216 vB) ~1,210 WU (303 vB) ~345 WU (86 vB) ~28%
1 KB ~1,633 WU (408 vB) ~1,996 WU (499 vB) ~363 WU (91 vB) ~18%
10 KB ~10,849 WU (2,712 vB) ~11,266 WU (2,817 vB) ~417 WU (104 vB) ~4%
1 Like
37

Thanks for the reminder that the inscription envelope requires a commit-reveal output-input pair, and your proposal would be smaller because an output directly commits to the data.

There is still something I don’t understand. If the output only commits to the data amount and its hash, but the data is not consensus relevant and it’s optional for nodes to keep, why should it count toward the block weight at all? I am much more pessimistic than you regarding the number of node operators that would participate in relay of this data. In that case, why change consensus at all? Instead, you could have the data be shared on an overlay network of participating nodes: the client would be a wrapper on Bitcoin Core that interprets OP_RETURN outputs with a specific prefix as data transaction announcements that the overlay nodes would retrieve, store, and share. You could cost it by requiring those OP_RETURN outputs to burn some amount of sats per byte (e.g., tied to the feerate of the transaction that created it?). How is your system an improvement over such an overlay network?

1 Like
38

The SegData proposal is intended to decollectivise resource control on the declared intent of the carrier. It has to have a weight check otherwise people would be able to carry data for free, which is unacceptable. An overlay design is a different proposal, and such things exist already (OP_RETURN+IPFS, RGB, OpenTimestamps), and maybe they draw some traffic off-chain. Burns aren’t adopted well in the market because they destroy coins and compensate no one, whereas fees are transfers and reusable. Even at zero participation SegData yields a priced, bounded, committed record, and an overlay would cost a similar OP_RETURN but yield limited availability.

At low participation SegData might seem like an overlay but the question about participation isn’t settled. SegData retention is the default, the majority of Bitcoin node operators are not pruning so it’s not right to expect they would prune data in bulk either. The resource constrained or ideological can opt-out, but mostly the ideological grievance is based on values not resources. SegData just allows the protocol to treat data as a specific type which brings the pruning decision to the node, the only legitimate place for it. The other argument is, if everyone pruned blocks because of resource limits then block availability dies with much worse consequences, blocks are needed to sync, where SegData isn’t, so the accepted case is actually the more dangerous one.

If pruning is permissable then it is possible that pruning policies centralize, which could lead to redundancy attrition. Therefore prunability has to be priced appropriately and the risk of crowding out money uses addressed. SegData would probably need to be cheaper and so a region size cap may be required. I’d need to derive some numbers here, but the upside to that is it would be an effective block size reduction for opting-out nodes :face_with_raised_eyebrow:

39

Having the whole blockchain enables bootstrapping other nodes and is useful for privately servicing your own light client’s needs. If node runners could selectively prune just data embedding without falling out of consensus, I firmly believe that the vast majority of node runners would do so, and that this may even be the default node behavior for implementations. Just because people realize that you can’t prevent data embedding in Bitcoin shouldn’t be misconstrued as people supporting or condoning it. In the end, just the people using SegData would keep it available, if those.

Anyway, I don’t see this proposal having any runway.

40

Thanks Murch appreciate the feedback

The goal here isn’t to condone or support data carriage in Bitcoin, although OP_RETURN does that despite being undercut by witness, but to simply acknowledge data exists and treat it accordingly as a structurally addressable region which transfers resource control to node operators as we have discussed. I understand your doubts that this will take off, and that’s fine. If the debate continues, which i presume it will, this is available for further discussion until it is superceded by a better proposal, or the whole issue disappears forever.

As for next steps, i will update the BIP with the changes as discussed here, and then make a PR to the bips repo? or widen the conversation to the mailing list?

41

This discussion satisfies the “present and discuss your idea” criteria, so you could open a PR to the BIPs repository next, but I think it might be interesting to post the BIP draft also to the mailing list as the audience is somewhat different even if it has big overlaps.

I feel like I haven’t been able to fully get my point across, but I’ll try again on the repository, if nobody else manages to state it more comprehensibly.

1 Like