Probably want to specify the build options more completely if you want benchmark results to be comparable? (CMAKE_BUILD_TYPE as something other than Debug in particular)
(Also #ifdef DEBUG and #ifdef USE_GMP instead of #if in various val64 code)
Probably should provide a git repo to add the CSV’s to via PR?
I’m seeing errors, which look like they lead to useless data, fwiw:
Script error: OP_DIV or OP_MOD by zero
653/875: MOD_DUP_100Bx2 0.000 seconds ( 0 Schnorrs, 0.0% varops used)
May want to fix those first?
Capping at 100% varops budget seems to make the data less useful in evaluating whether the varops ratios are accurate, than if no capping was in place? eg these XOR tests:
511/875: DUP_XOR_DROP_DUP_10KBx2 0.514 seconds ( 12714 Schnorrs, 100.0% varops used)
509/875: DUP_XOR_DROP_DUP_100KBx2 0.446 seconds ( 11051 Schnorrs, 100.0% varops used)
514/875: DUP_XOR_DROP_DUP_1MBx2 0.477 seconds ( 11807 Schnorrs, 100.0% varops used)
516/875: DUP_XOR_DROP_DUP_2MBx2 0.475 seconds ( 11769 Schnorrs, 100.0% varops used)
don’t seem to give any useful information about relative performance at different input sizes, because in each case the varops limit is hit?
I would have expected to be able to use the data generated either to produce per-hardware calculations of the constant-and-per-byte factors for each opcode (and thus review the hardcoded varops factors) or alternatively to be able to judge what would cause the worst case blocks for my hardware (particularly for old hardware) – eg, “a block full of schnorr checks will take 5s to validate, but a block of X hash256 ops with Y bytes of data each will take 10s to validate”.
Looking at <100% varops hashing results, I think I can calculate:
- ripemd: constant = 0.322, per byte = 0.00251
- sha256: constant = 0.362, per byte = 0.00306
- hash160: constant = 0.549, per byte = 0.00299
Or normalized from “seconds per block over-filled with repeating script” to “schnorr ops per op”, perhaps:
- ripemd: constant = 0.006 schnorr ops per op, plus 0.000047 schnorr ops per byte
- sha256: constant = 0.007 plus 0.000057 per byte
- hash160: constant = 0.010 plus 0.000056 per byte
Or normalized from 80k schnorr ops per block to 20.8B compute units per block:
- ripemd: constant = 1554, per byte = 12.1
- sha256: constant = 1747, per byte= 14.8
- hash160: constant = 2650, per byte= 14.4
Those compare to current varops figures of constant=0, per byte=10, I believe. I’m surprised that ripemd seems to be faster than sha256 for me. Presumably either taking the DUP costs into account or setting up an absurdly large stack so that DUP is unnecessary would generate more accurate figures than I’ve done above though. FWIW DUP/DROP benchmarks seems to imply figures of 276 compute units per operation plus 0.05 compute units per byte for me (as opposed to 0 + 1 per byte).