Thinking more about how precisely this would be implemented as I think this can be made simpler, but posting my draft to give the idea:
Ok everything I said was wrong in last post; I think it all works. Let me try again with diagrams which I should have been making earlier:
When an ephemeral anchor spend is evaluated against TxA:
flowchart TD
A[TxA: fee 0\nsize 300\n1000 sats in anchor] --> B[TxB: fee 1100\nsize 200]
We simulate/induce an RBF against a 65byte OP_RETURN tx:
flowchart TD
A[TxA: fee 0\nsize 300\n1000 sats in anchor] --> C[TxC: fee 1000\nsize 65\npure burn]
which results in a diagram check:
As you can see, even though the total fees are higher, it doesn’t dominate the pure burn, so it would be rejected. The “pure burn” case would of course be accepted, since we would not be doing an incremental relay check.
If instead the proposed spend looked like:
flowchart TD
A[TxA: fee 0\nsize 300\n1000 sats in anchor] --> B[TxB: fee 1400\nsize 200]
we get:
and this would be accepted.