Thanks for the update and the resources, that all sounds very promising! I’ll focus on the assumevalid version with my reply for now, since I still haven’t fully processed the non-assumevalid one yet.
Nice idea! As already noted in private, I’m wondering though if only adding a salt is really enough to provide the same security guarantees as MuHash would do? Reducing the modulo field size from 3072 bits to 256 bits and using simple addition instead multiplication as basic operation seem to be quite drastic cuts. This is more “gut-feeling” and far away from a mathematical sound argument, but it feels to me that this was too good to be true if there aren’t any significant downsides to this approach. (That said, I hope I’m wrong, and even if there is a reduction in security, it’s maybe still good enough for the SwiftSync use-case, since the goals of MuHash and the aggregate hash here are different.)
Since it was relatively easy to do, I adapted the implementation above to take use of that suggestion (by abusing secp256k1 scalars for the aggregate hash type): GitHub - theStack/bitcoin at ibd_booster_v1_addsub_sha256 (commit [HACK] use modular arithmetic and SHA256 for aggregate hash · theStack/bitcoin@494692e · GitHub)
The results are pleasant, with this I’m now already observing a ~5x IBD speed-up compared to the assumevalid run (again, up to block 850900, and using -reindex-chainstate), and this is still without any parallel block validation involved:
| assumevalid only | SwiftSync (MuHash for aggregate hash) | SwiftSync (salted sha256 add/sub for aggregate hash) | |
|---|---|---|---|
| time | 2454m37.039s | 1094m31.100s | 464m37.234s |
| speed-up | - | ~2.24x | ~5.28x |
Doing parallel block validation as next proof-of-concept step would be very nice, but that needs of course much more invasive changes in the codebase.