Broader test suite for benchmarks #485

Other things that would be interesting to track:

• the proportion of files where we fail to parse one of the source files (useful metric to track when upgrading a parser to a new version)
• of course, performance metrics. Average time to merge? Or time to merge per kilobytes of input files? Or per bytes of conflicts? Not sure what's the most sensible thing.

Also, it would be good to be able to run the test suite only on certain file types (typically for parser updates or changes to specific language profiles).

I'm slowly getting there! Here is a table summarizing a benchmark on a small dataset of real merge scenarios extracted from merge commits:

Here's what the columns mean:

• Cases: the total number of test cases considered. Line-based merging returns conflicts for all of them (using git merge-file with Myers diff)
• Conflict: mergiraf also returns conflicts
• Exact: mergiraf returns exactly the merge that is recorded in the merge commit (up to blank lines)
• Format: mergiraf returns something slightly different, but commutatively isomorphic to the target file from the merge commit
• Differ: mergiraf returns a conflict-free merge output, that isn't commutatively isomorphic to the target file
• Parse: one of the source revisions failed to parse with the associated tree-sitter parser (so mergiraf fell back on line-based merging)
• Panic: mergiraf panicked during the merge process
• Time: the average duration of the mergiraf merge process, in seconds

So far the panics are all instances of #520, #521 or #333. It's interesting to see that almost half of the *.cpp files fail to parse (likely due to pre-processor usage). Also the *.js files take significantly longer - not sure if it's because they are often bigger, or if the parser is inefficient. Of course the numbers aren't so insightful on their own, the real deal will be observing the difference between two versions of mergiraf.

I'm still gathering a wider dataset of test cases (covering all file formats).

Very impressive!

I think for the "Conflicts" column, it could make sense to weight each merge result by the amount of conflicts created – maybe Mergiraf solves all but one? I guess one could argue for doing the same for "Format" using something like tree-edit-distance, but that might come too close to an attempt to white-wash Mergiraf 😅

I think for the "Conflicts" column, it could make sense to weight each merge result by the amount of conflicts created – maybe Mergiraf solves all but one?

What sort of weighting are you thinking about?
For me, the number of conflicts solved isn't super important… as long as there are conflicts in the file, I expect that some human will have a look at the merged results to solve the remaining conflicts, and so if there are issues with the resolution of any conflicts, they are likely to get noticed in the same go.
Another issue is that there can be more conflicts than in the line-based file, because they're narrower. So we could also track the conflict mass, but then again I'm not sure how to report that in the table.

Hm, my idea would be something like conflict_mass(structured_merge) / conflict_mass(line_based_merge), but I guess that's not guaranteed to be favorable to us, given that a fully structured merge can create just different conflicts, which can't really be compared to those created by a line-based merge

Yes… I guess one category we could still add is one for cases where mergiraf returns conflicts which are identical to the line-based merge output. This would let us distinguish cases where mergiraf really did something.

That does make sense I think. But now I'm wondering – how do you recognize the case where Mergiraf plain falls back to line-base merging (because of some side not parsing)? By analyzing the logs probably? Because this case and the case you mentioned could theoretically get conflated

For now I just manually check with mgf_dev if all revisions parse. It's true that in that case the line-based fallback also kicks in, one would need to make the definitions of the categories clear.

Just for fun I tried comparing the effect of #522 on benchmark results:

I'm still working on improving the rendering. Another useful piece of info would be the test cases that changed, to inspect that they went in the right direction (in particular the ones that land in Differ).

Here are some results on a big-ish dataset, for mergiraf 0.13.0.