-
Notifications
You must be signed in to change notification settings - Fork 1
Description
When running metacoder configured with multiple test cases, the eval_results.yaml
file is overwritten with each test case that is run, resulting in a results file that only contains the evaluation of the final test case in the list. The filename can be changed via the -o
or --output
parameter, but this still results in a single file with the same problem.
A second issue is that an output file could end up overwriting an existing file.
It seems that the expected behavior would be to write the individual result files to their corresponding working folder, instead of the current working directory. Unfortunately, that path information is not available to the function that opens that file (main
from metacoder.py
, and possibly also the main
from runner.py
).
So there are two options that do not involve making major changes to the runner:
- Increment a counter and append a number to the basename of the file, (e.g.,
file_0.yaml
,file_1.yaml
). This could result in multiple runs of test suites to get mixed up if these files are not cleaned up between runs. - Catenate results into a single file (easiest). Fail to run if the file already exists (addresses second issue above).