Snoop: Fix memory usage during Michelson generation
What
Memory usage during Michelson generation was rapidly increasing, to the point where a 16GB RAM machine couldn't generate 300 terms at once (the processes were sigkilled by the OS).
Why
The reason was because subterms during generation are hash consed, to increase performance during the brun-in step of the Markov chain generation. The issue is that this table is never cleared, so keeps on increasing forever (until the OS gracefully intervenes, when it doesn't crash completely).
How
The solution: clear the tables between each generated term. This does not impact performances much on generation (probably) because the set of hash consed subterms between two full terms is relatively small.
Manually testing the MR
./octez-snoop alpha michelson generate 10 terms of kind code in tmp and check with htop or something
Also reports using memtrace, for 10 terms with --seed 1234, without and with the table clear :
Also tested with dune exec tezt/snoop/main.exe -- --verbose, that before would crash (even on the benchmark machine). Now each process quickly stabilizes at 3% (~450MB) memory usage until the end of the run.
Checklist
-
Document the interface of any function added or modified (see the coding guidelines) -
Document any change to the user interface, including configuration parameters (see node configuration) -
Provide automatic testing (see the testing guide). -
For new features and bug fixes, add an item in the appropriate changelog ( docs/protocols/alpha.rstfor the protocol and the environment,CHANGES.rstat the root of the repository for everything else). -
Select suitable reviewers using the Reviewersfield below. -
Select as Assigneethe next person who should take action on that MR