tests: Utility to minimize test cases, continued #527

I think this is good enough as a start -- should we merge it?

Oh, wait, there are some things to fix up -- give me a sec

Ok should be good to go now

@wetneb wrote in #576 (comment):

The script should just call mergiraf on the example. You can then pass the expected status code (134, if I remember well?) to the minimizer.

Sorry, still not quite following you. Looking at the source code, I think the testing command invocation ends up looking like:

bash -c <script> testing_script <output directory>

Firstly, what does testing_script in the middle of the invocation achieve?.. I thought you'd want to pass to <script> the <output_directory>, not this literal string?

Secondly, what I understand "just call mergiraf on the example" to mean is to run mergiraf merge? But doesn't that require me to know what the paths to base, left, and right files are (most importantly, their extensions)? Knowing <output_directory> only brings me half-way.

The purpose of having a configurable script is for the cases where you want to ensure some additional properties on the input files or the merge output, such as the presence of a conflict or of a certain type of content.

I'm completely blanking on this one... I think having a couple of example mgf_dev minimize invocations would really go a long way

Sure, here's an attempt at documenting it:

First, compile mergiraf on the bundle-comments branch (say, in release mode, so that the minimizing goes faster). Let's call that binary faulty_mergiraf.

Then let's get the example:

wget https://codeberg.org/attachments/8f69e268-0c70-4bc7-8a12-ed2773800677 -O 97dbf903e1.tgz
tar -zxvf 97dbf903e1.tgz

Then, write a script like this:

#!/bin/bash
set -e
./faulty_mergiraf merge $1/Base.hs $1/Left.hs $1/Right.hs

Let's call this script run_faulty.sh.

Let's check that running the script on the example produces a return code indicating a panic (134):

chmod +x run_faulty.sh
./run_faulty.sh 97dbf903e1/
echo $?

The faulty script successfully fails on the example, yay! At each minimization step, the minimizer will check that the script has this status code and will reject any minimized version of the test case that doesn't pass this test. This makes it possible to minimize the test case under that particular constraint.

Let's do it then:

cargo run -p mgf_dev -- minimize -e 134 97dbf903e1/ ./run_faulty.sh

Note: running this on the current version of this branch seems to run into an infinite loop. From my original version it seems to work.

In such a simple case, writing a script is arguably a bit overkill, so you could just do instead:

cargo run -p mgf_dev -- minimize -e 134 97dbf903e1/ "./faulty_mergiraf merge $1/Base.hs $1/Left.hs $1/Right.hs"`

At the end you have a minimized test case at /tmp/minimized.

Ideas for improvement:

• make the minimizer run the provided script on the original example, without any change, to check that the expected return code is found. If not, print the error output of the program to help the user debug their testing script
• make the current version of this branch also work (I haven't looked into what goes wrong)

Funnily enough, in this particular case, the minimizer does an absolutely excellent job: it minimized the test case to the point that all three revisions (base, left and right) are completely empty, and the panic still happens! Not sure it really helps the debugging that much, because it's a bit degenerate, but hey, it's pretty good minimization :-D

Ah no nevermind I made a mistake - there is no panic for empty files.

Thank you very much, that was very helpful:)

Note: running this on the current version of this branch seems to run into an infinite loop. From my original version it seems to work.

Huh. Looking at the rebase diff, nothing really catches the eye.. I guess I'll try to bisect this

Okay, I now understood your explanation from the other thread. I guess if the result parses with treesitter we should be relatively alright? Succesfully compiling real world examples will be hard to impossible to automate. What we could try is running ghc on the files. GHC has multiple phases, first the parser, later renaming, then typechecking. We can probably check that ghc reaches the renamer (where name resolution happens and certainly will fail.) Sounds also a bit brittle. But feel free to ping me when you need more help in this regard.

Yeah I think that's as far as we can go with ghc -- but still, it sounds like a good additional step on top of the tree-sitter parsing, thanks for the tip @maralorn:)

We could maybe start collecting these language-specific testing scripts somewhere in the repo -- not sure whether we'll end up with a lot of them, but they could serve as a kind of documentation examples for future contributors.