[go: up one dir, main page]

lib_shell_benchmarks: add intercept benchmarks for DECODING_(INT|NAT)

Context

Resolves #5798 (closed)

The following commands were tested on the Snoop server.

$ ./octez-snoop list all benchmarks | grep -E "DECODING_(INT|NAT)"
encoding/DECODING_INT: Benchmarking encoding/DECODING_INT
encoding/DECODING_INT/intercept: Benchmarking encoding/DECODING_INT/intercept (intercept case)
encoding/DECODING_NAT: Benchmarking encoding/DECODING_NAT
encoding/DECODING_NAT/intercept: Benchmarking encoding/DECODING_NAT/intercept (intercept case)

$ ./octez-snoop generate code for benchmarks encoding/DECODING_INT encoding/DECODING_NAT --out-dir dir
...

$ cat dir/encoding.sol.txt 
dir/encoding.sol:
  inference_model_name: encoding
  free_variables:
    builtin/Timer_latency = 24
    encoding/DECODING_INT_coeff = 13.4035169156
    encoding/DECODING_INT_const = 18
    encoding/DECODING_NAT_coeff = 13.3660403858
    encoding/DECODING_NAT_const = 9
  scores:
    builtin/TIMER_LATENCY : R2: NA , RMSE: 1.000000 , T-Timer_latency: 1179.479153
    encoding/DECODING_INT : R2: 0.999924 , RMSE: 1073.454634 , T-DECODING_INT_coeff: 22241.809772
    encoding/DECODING_INT/intercept : R2: NA , RMSE: 1.000000 , T-DECODING_INT_const: 377.521230
    encoding/DECODING_NAT : R2: 0.999975 , RMSE: 628.440998 , T-DECODING_NAT_coeff: 62320.023344
    encoding/DECODING_NAT/intercept : R2: NA , RMSE: 1.000000 , T-DECODING_NAT_const: 232.486626

c.f: The reference values from the biweekly CSV

  • encoding/DECODING_INT_coeff := 13.3234013448
  • encoding/DECODING_NAT_coeff := 13.2910436478

Plots

DECODING_INT_encoding_validation-0.pdf DECODING_NAT_encoding_validation-0.pdf

Note

As experimented on the Snoop server, I observed variance on the _const parameters. (Both of the constants varied around 9 - 17.) This might raise false alerts at biweekly runs.

I think this problem is not specific to the encoding benchmarks, but is the same as the problem observed in some interpreter benchmarks (e.g. #5027). So I think fixing the variance is out of the scope of this MR, and need to be investigated in the issue 5027.

Checklist

  • Select suitable reviewers using the Reviewers field below.
  • Select as Assignee the next person who should take action on that MR
Edited by satos

Merge request reports

Loading