I remember thinking about this when first hearing about IBD booster / SwiftSync and just had a quick and dirty look at using delta / differential encoding together with CompactSize and VarInt’s in this notebook. This seems to half the size of the uncompressed data. Though, as the hints will likely be distributed compressed and the xz compressed data is about the same size for all combinations, I’m not sure if it’s worth to differentially encode it.
| filename | size (bytes) |
|---|---|
| booster850900.bin | 364964716 |
| differential_compact_size.bin | 188209875 |
| differential_varint.bin | 190098381 |
| booster850900.bin.xz | 87882052 |
| differential_compact_size.bin.xz | 81697540 |
| differential_varint.bin.xz | 81321492 |
Note that most of the median deltas are below 253 and encodable with a 1 byte CompactSize:
