Yes that is an interesting/scary aspect, isn’t it. At a deep level, the whole concept of “xpub” could be argued to have been flawed from the start: an xpub does indeed count as being on the “pub” side in the classic public vs private distinction from asymmetric cryptography, but it’s a hybrid, and makes a convenience-safety tradeoff that was probably never acceptable (probably! that’s very debatable). I remember many years ago, even before anyone mentioned quantum computers, wallet software (good software anyway!) printing warnings to users to not share xpubs. It was a bad privacy violation, as well as a security threat (because you can not reasonably expect users to understand the cross-privkey-leakage).
Glad I could help with the analysis.