I think I see the confusion. What I meant was the silent payment aware signers must check for any ACP on any inputs they will sign for and fail if there are any silent payment outputs. If all signers are following the BIP spec, then there should not be any inputs signed with ACP anyways. Actually, the signer should probably fail if there is any sighash flag other than ALL if there are any silent payment outputs.
One other possibility that could simplify things. If a signer is able to sign for all inputs, they could add a global share and proof covering all inputs with the key data being the scan key. If a signer is unable to sign for all inputs, then they add a per input share and proof covering only that input as described by @achow101 above.
This keeps the key data clean and optimizes for the common scenario of a single signer wallet using a hardware signing device.