If we can only add silent payment outputs if Inputs Modifiable is not set, then this point is moot, no?
I don’t think we should dismiss this optimization. Consider the common case of a wallet with 10 small utxos, and it wants to make a single payment with all of them to a silent payment address. Without consolidating all shares and proofs, the hardware wallet signer will need to compute 10 times more shares and proofs.