Thanks for putting this together!
I have a few comments on some of the sections.
Withholding signatures
So Bob’s UTXOs will remain locked until he pays transaction fees to double spend one of his inputs.
There is one important subtlety that you don’t mention here: the non-initiator will only contribute some funds if the initiator is paying them (via liquidity ads). In the liquidity ads protocol, there is a mechanism to ensure that it is the buyer who pays for the on-chain fees of the seller’s inputs and outputs.
Let’s suppose that Mallory buys liquidity from Bob and then withholds signatures (or does something else to prevent confirmation). When Alice wants to buy liquidity from Bob, it will be Alice who pays for double-spending Bob’s inputs from that previous transaction.
The way I’m currently implementing this for eclair is that node operators will use a dedicated bitcoin wallet for the funds they want to lease. This wallet will be used exclusively for liquidity ads, and only the funds in that wallet will be exposed to such griefing issues. Any double-spend on that wallet will be made by other liquidity ads transactions, which ensures the seller doesn’t pay those on-chain fees.
Preventing transaction confirmation
they generally cannot afford to leave UTXOs unlocked until transaction confirmation because that would cause lots of accidental double spends. The participant will usually need to lock UTXOs soon after the joint transaction is successfully broadcast.
I agree that this should be the default behavior, to avoid double-spending honest attempts that just take a bit of time to confirm. But that doesn’t prevent the non-initiator from unlocking utxos after broadcast, if they detect something weird. If the transaction looks like it has the right feerate but is still not confirming after some blocks, something is fishy and the utxos should probably be unlocked.
That’s not a very satisfying solution, but it’s something ¯\__(ツ)_/¯
Replacing ancestors
This one is really hard to detect (you can never be entirely sure that this is what is happening), but if you unlock utxos the double-spend will then be free, since the original transaction is gone anyway.
Require confirmed inputs
Downsides
Another downside is that this is very inefficient from a liquidity perspective. Every unconfirmed change output is effectively “locked liquidity” until it confirms. Ideally we’d really like to be able to use them.
Hopefully we can come up with some more ideas to address this fundamental problem in multi-party transaction protocols.
I think we’ve explored mitigations a lot, and I don’t think we’ll find new solutions at the lightning layer that wouldn’t have downsides.
The only real way to fix this is at the mempool layer: fortunately package relay, v3 transactions, ephemeral anchors and cluster mempool are seeing a lot of active research! Sure, it will still take years before that is widely available, but I’m confident that it will eventually be there.
Until then, there is additional risk when selling your liquidity: it’s hard to find the right balance between what should be put into the protocol to offer mitigations, and what should be in implementations’ heuristics/policies, but IMHO the current state of the proposals strikes a good balance between those.