If you haven’t seen it before, you may want to check out Kalle Alm’s previous research on this subject, e.g. as presented at Scaling Bitcoin 2017:
One of his points I recall finding particularly insightful was that, for anyone in a position to RBF, we can simply choose a feerate that is min(confirmed_estimate,mempool_estimate) where confirmed_estimate is based on non-gameable data (like the existing Bitcoin Core estimatesmartfee) and mempool_estimate is based on mempool stats like you describe (which has the risk of potentially being gameable and might also break accidentally, such as after a soft fork).
Using the minimum of the two returned values allows the feerate to adapt downwards very quickly without introducing any new risk that feerates could be manipulated upwards by miners or other third parties. If people only use the enhanced estimation for cases where they can RBF, then they can always fee bump if they end up underpaying. (Though it’s worth noting that RBFing has non-financial costs: it typically leaks which output is the change output and it requires the fee bumper either be online to bump or create presigned incremental fee bumps.)