There’s a halflife of 12 hours specified, which decreases to 6 hours if the mempool is less than half full, or 3 hours if the mempool is less than a quarter full; provided there’s been a block since the last time the minfee was bumped due to tx eviction, the minfee will decay towards 0 at that rate. So at 20sat/vb, when the minfee is raised by 1sat/vb it’ll take ~53min to decay back; at 50sat/vb, about 21 minutes; at 100sat/vb about 10 minutes; at 5sat/vb about 4 hours.
1 Like
Predicting fees is Too Hard, and over-paying increasingly Too Expensive. So you really need to BYO fees at spend time (rather than construction/signature exchange time), and if you can do that you can ride out fee spikes without needing to renegotiate with the other party, too, making your protocol more robust.
With current tech, we can only do this for SIGHASH-SINGLE/ANYONE_CAN_PAY (limiting your protocol to single in and out, which doesn’t usually work for multi-party stuff), OR using CPFP which requires one output which is spendable immediately.
i.e. CPFP is the only game in town today. v3 helps here today!
But I think the end game of Bitcoin is high fees (which will justify a great deal of engineering and effort to mitigate), and introspection which allows us more room to optimize. The medium case is bringing your own fee input/output at spend time, but really optimal is paying someone (over lightning or anything but onchain Bitcoin) to gather multiple unrelated txs together from everyone, with one fee input and output covering the entire thing.
Perhaps we would end up using introspection to limit the tx size anyway, so we just need the no-unconfirmed-children rule. Or perhaps the diminishing returns from bundling mean the tx size doesn’t have to be very large.
Now I write this out, I think I understand @instagibbs point about the generality of the v3 proposal, which would still serve here?
(End of a long day here, I will digest the rest of your post before the weekend, I hope!)
3 Likes
Thanks, indeed, this makes the “free relay at bottom of the mempool” game less easy to play, depending on how fast that decays. From AJ’s answer, I’d guesstimate 10-100 blocks might be typical, but it’s not a far cry from 2 weeks…
Ah yes, unconfirmed children strike again 
(Please correct me if I made any numerical errors here.)
Even if you did, the order of magnitude is correct.
this is not something that we should permit to happen, for both anti-DoS and incentive compatibility reasons.
I agree with anti-DoS, but it’s not even clear that it’s not incentive compatible. If you expect other miners to do the replacement, it depends on your discount rate. Which is annoyingly circular reasoning, but I think we can do better.
Let’s assume v3 rules, so children need not apply. All the pre-signed transactions I know of can trivially enforce this condition, so it seems fair, and cuts the problem down a fair way.
What does it fundamentally cost to drop a random transaction from the mempool? If we assume the Mempool Is Always Full, then it seems reasonable that it will be eventually replaced by a transaction at feerate minfee - epsilon. And if we assume a tx of < 101k, and some continuous looking feerate distribution around and below the bottom of the mempool, I’m happy to ignore epsilon.
In this model, a miner would consider an RBF tx fundamentally incentive compatible be comparing the total fees above the minfee rate. To give some numbers:
My node’s mempool is oversized, but a quick grep/cut/awk hack gives me the threshold for block 78 as 7.7432 sat/vb (and block 1 as 14.0673 sat/vb). Under this model, a 100k tx paying 774,320 sats is worth epsilon (~0), so a miner would want to accept a replacement 1k tx paying any higher feerate, say 7744 sats.
Now, does this actually help our pinning attack? Not as much as the previous proposal, at least on these numbers. Halfway up the mempool (block 39) is 12.0378 sat/vb (higher in the mempool means it’s more likely to get mined and just fail, so this is a reasonable worst-case sketch). Mallory puts a 100k tx there, be paying 1,203,780 sats, thus a fee-above-minimum of 429,460 sats. Alice’s 1k tx would have to beat that fee-above-minimum, so 437 sat/vb feerate (it could otherwise get in the top block for 14 sat/vb, so this does feel like extortion).
But I think it does imply that there’s a significant conflict between miner incentives and the current RBF DoS rules (under which Alice’s 1k tx would have to pay a feerate of 1204 sats/vb).
1 Like
While I think that Suhas’s example above is somewhat similar, I recently elaborated an example how introducing a replace-by-feerate scheme in addition to our current RBF rules would cause new substantial DOS vectors. While my example is based on a concrete proposal, I believe that it illustrates a general issue afflicting all schemes that do away with the absolute fee increase in replacements: mempool - What is the problem with the recent "One-Shot Replace-by-Fee-Rate" Proposal? - Bitcoin Stack Exchange
My example entails a negligible cost to the attacker while permitting continuous resubmission of essentially the same transaction cycle. The collection of transaction has a substantially larger total weight than the data that is ever up for inclusion in the blocks and hence severely underpays for relay to the detriment of the entire network’s bandwith-usage. I hope studying my write-up would help substantiate why transaction replacements must increase the total fees in the mempool.
1 Like
I think this technical definition is overspecialized (or perhaps implied by the question?).
The problem is: you want to spend some shared input in a timely manner, and are prepared to pay competitive rates to do so vs other transactions. Another transaction is preventing this. You don’t really care why, or which tx gets mined, but you have some deadline.
I think we’ve demonstrated that there are cases where the miners are incentivized to fix this, and cases they are not. In the latter case, there’s a Schelling point where, even if it’s not directly incentive compatible for a miner to replace, it becomes so if the other miners do. Such a Schelling point might come into existence if (1) miners agree it’s generally better for the network, (2) the gains of not doing so are minor anyway, and (3) it’s the default.
We still have to deal with the DoS problem, but perhaps that’s as simple as “if a tx has been in the mempool for over 12 blocks and you’re offered feerate to get the replacement into the top of mempool, and they’ve both v3, allow RBF ignoring the total fee rule”.
This was the link I was looking for earlier, thanks!!
We need to consider three things here:
- Bandwidth consumption
- Higher layer protocol efficacy
- Miner incentives
I think we’ve demonstrated a conflict between them (i.e. we need to compromise, and we implicitly are already).
I don’t think your example works for lightning today, where the first transaction is of fixed size, so games can only be played with the children, and v3 restricts those.
But it does effect a general stackable-txs case, which is where we’re headed eventually, so must be considered. Sigh, ok
The real problem in your example seems to be step 7: by RBF not considering the package feerate, but a single tx feerate, it discards an imminent tx! This is not incentive compatible if you assume any reasonable discount rate for future blocks?
The stackable case is interesting in that an attacker can easily coordinate with itself to pin 100 channels, but collective action to outbid it might be prohibitive in terms of interactivity. So even if a miner “knows” it would pairwise not take the pin-evicting txs, it might err on the side of taking it anyways, excluding DoS concerns.
My ideal is if we had some future discounting, and we applied it to diagram checks, we could make all diagrams comparable during these checks. With that in hand, incentives and DoS checks would be two completely separate concerns. From there we can think about alternative anti-DoS measures with a bit more clarity.
2 Likes
Yeah, I am not concerned with benign users of a “RBF bypass” but with a malicious adversary using such a new rule creatively beyond the intended application.
Hmm, poking at this some more, I think my previous post went too far into the miners-all-cooperate edge case.
If you consider some pinning tx that won’t be worth mining for 100 blocks, and a stratumv2 mining pool with only 98% hashrate that keeps the pinning tx hoping for its high fees, that’s still leaves an 86% chance that the remaining 2% hashrate mines a block a block in the meantime with the high feerate tx, invalidating the pinning tx. In that case, the pinning tx needs to pay h^{1-n} times as much as the pinned tx, or the pool would have had a higher expected value for just trying to mine the higher feerate/lower fee pinned tx instead. For a pool with 100% hashrate, that devolves to always requiring a higher fee independent of n; but for a pool with 90% hashrate, it’s already 100x fee at about 45 blocks; at 50% hashrate, it’s 500x fee at 10 blocks. For the 98% pool at 100 blocks, it’s a factor of about 7.4.
That model may give a strong incentive to defect, if you get some direct benefit from fees, other than just “the pool has more funds to distribute”. For example, if a sv2 pool finds that some percent of hashpower is just building working on empty blocks, perhaps it will decide to issue rewards rated not just by number of shares found, but also by the potential reward that those blocks would return to the pool. In that case, defecting and attempting to mine the replacement transaction would increase your share of rewards in the short term, and, provided the pool doesn’t check for that sort of defection and punish it, and without having done the maths, I think that would end up being attractive, at least, for small miners within the pool.
I think that still adds up to “the replacement tx needs to pay a fee proportional to the total fee of the tx being replaced” though – along with having a threshold along the lines of “don’t replace txs you expect will confirm in the next 50 blocks” – but at least it allows that proportion to be substantially less than 1.
I think in this view, you can conclude that as long as you don’t have a mining pool/cartel with over 95% hashrate, than replacing a tx that won’t be mined for 135 or more blocks with one that would be mined immediately is fine unless the total fee of the existing tx is more than 1000x that of the new one (pretty implausible as it would mean even a 100vb tx replacement would be paying a lower feerate than a max size 100kvb tx), as is replacing a tx whose total fee is under 10x and won’t be mined for 50 or more blocks.
Those numbers seem much more reasonable, though I’m not sure how you’d make any of this practical to implement – both the “this tx won’t be mined for n blocks” and “h percent of hashpower will wait to try to collect fees from the pinning tx” aren’t really things you can measure very accurately… Dealing with a 50 or 135 block delay might not be very practical for many applications either.
Indeed, and restricting the carve-out to some reduced size would work for lightning, but not general protocols.
But to revisit your example, RBF rule 2 (no other unconfirmed inputs) seems to want to prevent the requirement to consider entire packages, but by playing on top of an unconfirmed tx you work around this? If the RBF bypass rule were only to apply to v3 transactions, I think this could not be done. Otherwise, we could consider replacing Rule 2 with a more sophisticated requirement that the new package outbid the one being replaced.
Thanks for the numbers, they definitely shed light! You’re assuming the ~100% cartel is too polite to reorg out for dominance, of course.
Given the widespread awareness of a 51% attack, I think it’s uncontroversial to write software optimized for miners who control less than that.
(Complete disclosure: I’m with Peter Todd (IIRC) that small miners are more valuable than large miners, so I’m biased to optimize the system for them with my volunteer efforts!).
But even without this bias, 50% discount degrades so fast, it’s not far from 100% really, which is far easier to implement. After 7 blocks the error for a 50% miner is < 1%. A 50% miner has resources to change this if they wish, but the expected return would be minimal.
Agreed! Despite discussing both here (partially in the hope of producing some magic solution which addresses both!) they are best considered separately.
While stackable txs are vulnerable to this problem, that’s actually OK I think? Because if we have the introspection tech required to stack txs, I assume we can also use that to restrict total tx size. If there’s a healthy marketplace of people making stacked txs with reasonable fees with some frequency, you don’t need this of course.
I’m assuming that it’s a stratumv2 (or similar) pool, that’s only about sharing profits, and that the pool has no control over the actual block itself, both in selecting the parent block and the transactions included in the block, but that members are choosing txs to maximise the pool’s income and hence their own. That might not be plausible (a pool might not want members who are competing with its other members, or who are selecting transactions “badly” and then whatever they do to prevent that might be used for more nefarious means?), but it seems like the steelman model of the worst case scenario here – ie, if there’s a solution for that scenario, the same solution should also work for any more realistic scenario.
The whole point of a mining pool is that it’s in the interests of small miners to band together to form a pool, because having big variance between your revenue and your expenses is hard to deal with, and reducing that is a huge win: so large pools are what you expect even when you are optimising for small miners…
I think the other aspect is whether you’re designing for miners who are making a short term investment (“I’m here for a week, I want to make as much money as I can, then cash out”), or a long term investment (“I’m here for years, I’m willing to make decisions that will only pay out over the long term, I’m going to establish a good reputation, and built up trusted relationships with other people over the long term”). Over the long term, with suitably skilled players and assuming the entire industry doesn’t collapse, the second approach generates more profit, so thinking through what happens when a majority of the remaining players in the space have adopted that approach seems like it makes sense. I think there’s a fair argument that having too many miners with a short term view becomes a security risk (eg, they may be willing to rent out their hashpower to attackers trying to do reorgs).
If you have 50% of miners say “anything that won’t confirm in the next 7 blocks can be instantly replaced by a higher feerate tx that will go in the next block”, then the remaining 50% of hashrate adopting a policy of “ditto, provided it pays at least 0.8% as much in total fees” will make more profit over the (very?) long term. But if the pool makes more profit, it’ll also get more hashrate as miners switch pools (or other pools will adopt the same approach giving a similar but more complicated result), and once it gets to 60% the constraint becomes 2.8% as much, 70% gives 8%, 80% gives 21%, 90% gives 48% and 95% gives a requirement of paying 70% as much in fees. I think that generalises like so:
| pool hashrate | 7 blocks | 10 | 20 | 50 | 100 | 135 |
|---|---|---|---|---|---|---|
| 50% | 0.78% | 0.10% | 0.00% | 0.00% | 0.00% | 0.00% |
| 60% | 2.80% | 0.60% | 0.00% | 0.00% | 0.00% | 0.00% |
| 70% | 8.24% | 2.82% | 0.08% | 0.00% | 0.00% | 0.00% |
| 80% | 20.97% | 10.74% | 1.15% | 0.00% | 0.00% | 0.00% |
| 90% | 47.83% | 34.87% | 12.16% | 0.52% | 0.00% | 0.00% |
| 95% | 69.83% | 59.87% | 35.85% | 7.69% | 0.59% | 0.10% |
| 100% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
(Formula is just x \ge h^n which satisfies f \cdot h^n \cdot h \le x f \cdot h where the left hand side gives you the expected value of eventually trying to mine the high-fee tx which you can only do if you avoid the alternative being mined by someone else first, and the right hand side gives you the expected value of mining the high feerate tx immediately, same as everyone else is attempting to do. That essentially assumes the mempool will be otherwise empty at the time you would mine the existing tx, which might be an unfair bias against the replacement)
Having to pay 1.15% as much in total fees means that if you’re replacing a 100kvB tx (max standard size) with a 100vB tx (smallest you get without just giving your money to miners), you’d need to pay 11.5 times the feerate for the smaller tx to make sense, so the 0.1% figure is appealing for making total-fee pinning largely irrelevant. However, if the original tx is expected to be confirmed in 20 blocks anyway in this scenario, maybe you’re just happy to wait that long anyway.
(Note that this a block count, not a mempool depth – a tx that is 5MvB deep in the mempool may not be mined for 20 blocks or 100 blocks or even longer, depending on how many new txs get proposed in the meantime)
2 Likes
I’m pretty sure we’re now measuring 0.8% of “Never Happens” though. This doesn’t seem to be the kind of thing that happens much by accident, but only as a result of deliberate attack. And that won’t happen in the scenario where 50% of miners are doing this?
So I think you’ve convinced me that it will come down to whatever’s simplest to implement?
Ah, but we don’t have 50% of miners doing this now; we have ~100% of miners requiring a 100% increased absolute fee. Those miners are in different pools though, which is a different constraint. For the first pool to defect from requiring 100% absolute fee, I think the numbers look like:
| pool hash rate | 7 blocks | 10 | 20 | 50 | 100 | 135 |
|---|---|---|---|---|---|---|
| 0.1% | 12.54% | 9.14% | 4.81% | 2.01% | 1.04% | 0.79% |
| 1% | 12.94% | 9.55% | 5.26% | 2.49% | 1.57% | 1.34% |
| 5% | 14.86% | 11.60% | 7.58% | 5.39% | 5.03% | 5.00% |
| 10% | 17.56% | 14.57% | 11.23% | 10.05% | 10.00% | 10.00% |
| 20% | 24.03% | 21.88% | 20.19% | 20.00% | 20.00% | 20.00% |
| 30% | 31.84% | 30.61% | 30.02% | 30.00% | 30.00% | 30.00% |
| 50% | 50.20% | 50.02% | 50.00% | 50.00% | 50.00% | 50.00% |
| 70% | 70.00% | 70.00% | 70.00% | 70.00% | 70.00% | 70.00% |
| 100% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
(Formula here is x \ge \frac{h}{1-(1-h)^{n+1}} – if you keep doing what everyone else is doing, you’re expected revenue is h \cdot f where f is the original fee, if you accept a replacement then either you’ll mine it if you get a block, or the original will be mined by someone else at block n+1, so your expected revenue is (1-(1-h)^{n+1}) \cdot xf where the replacement is paying a total fee of xf. Accepting the replacement has higher expected value when x satisfies the formula above. I think that table keeps working for the second defector and so on, provided all the defectors stick to their own optimal settings)
The question is “I’m writing an app that wants to urgently replace some transaction, and I have to choose a fee for that. Obviously I’m going to choose a high feerate because otherwise it definitely won’t get mined soon, but do I have to boost that feerate even higher than the bottom-of-the-top-block feerate, and if so, by how much?”
At the moment, the answer is very much yes – you need to make sure your total fee is strictly greater than what you’re replacing.
If we change that rule, we need to figure out a new answer to the conflict between what miners would like (more fees better!) and what users want (less fees better!). But miners are getting the final choice as to whether to use the replacement or not, so it’s likely that their decision matters most.
Simpler is always better, but the simplest solution is always to stick with what we’ve already got.
The good thing is that, going by the tables above and previously, what we’ve already got isn’t optimal for either miners (they’d be better off to accept some replacements paying a lower absolute fee both today with competing pools, and even in a stratum v2 world with a ~100% pool) or users (less fees good!), so there’s some concrete justification for change.
I think the above table implies that for any tx that’s 5MvB deep in the mempool or more, and is being replaced by a tx with a mining score that will put it in the next block, then we could reduce the total fee requirement to only need to pay 40% of the total fees of the replaced txs, and that would be a win for any pool with less than 40% hashrate. Probably still requires cluster mempool first in order to efficiently estimate mempool depth and next block feerate, but otherwise might be reasonably simple.
3 Likes
I think an adversary can game this by just replacing their “pinning” transaction right before the 12 block deadline passes with a minimal fee bump, to maintain the pin for another 12 blocks? (Not sure if you have some other implementation detail in mind to deal with such a case.)
1 Like
In my unfinished “V4-pool”(a very small mempool of txns that must enter top block) thinking, forcing the replacement to always top block fixes this corner case. After a tx “times out” we consider the tx to already be free relay, and allow a replacement, provided it’s not another bottom of mempool.
Didn’t Todd fix that infinite cycle issue? I’m trying out Libre Relay and my node doesn’t seem to be getting attacked. The fix seems pretty simple: [bitcoin-dev] One-Shot Replace-By-Fee-Rate
How do you reconcile that with pools like Mara and F2Pool making lots of money and staying in business? Why shouldn’t I just mine the high feerate tx now?
(I’m involved in mining)