Train and evaluate without evalute_model initialization in config does not work

[kathyxchen]

In previous versions of Selene, we allowed users to automatically train and evaluate in 1 step without having to specify both the TrainModel and EvaluateModel classes. This functionality has somehow been removed in subsequent versions, so I'd like to look into this and figure out whether there was a reason it was removed and/or if we can put this back in.

[jeffhsu3]

The n_test_samplers parameter for evalute_model seems to be required in the config even though IntervalsSampler is specified in the config as well. The docstring for EvaluateModel makes it seem like it is optional. Not sure if it is related.

Throws this error:

File "evaluate.py", line 14, in <module>
    parse_configs_and_run(configs, lr=0.01)
  File "/media/jhsu/edata/repos/selene/selene_sdk/utils/config_utils.py", line 341, in parse_configs_and_run
    execute(operations, configs, current_run_output_dir)
  File "/media/jhsu/edata/repos/selene/selene_sdk/utils/config_utils.py", line 208, in execute
    evaluate_model = instantiate(evaluate_model_info)
  File "/media/jhsu/edata/repos/selene/selene_sdk/utils/config.py", line 239, in instantiate
    return _instantiate_proxy_tuple(proxy, bindings)
  File "/media/jhsu/edata/repos/selene/selene_sdk/utils/config.py", line 144, in _instantiate_proxy_tuple
    obj = proxy.callable(**kwargs)
  File "/media/jhsu/edata/repos/selene/selene_sdk/evaluate_model.py", line 154, in __init__
    self.sampler.get_data_and_targets(self.batch_size, n_test_samples)
  File "/media/jhsu/edata/repos/selene/selene_sdk/samplers/online_sampler.py", line 329, in get_data_and_targets
    n_batches = int(n_samples / batch_size)
TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'```

[kathyxchen]

Thanks @jeffhsu3! Yes, it seems like there's a bug in online_samplers that should be fixed. I'll look into that - we've got a couple of PRs to resolve right now but I wonder if I can include this fix in one of the PRs

[kathyxchen]

@jeffhsu3 I've been looking into this a bit recently and I was wondering if you could give me more information about your use case? i.e. what was the configuration file you used? I'd like to reproduce the error you were getting. Thanks!

[jeffhsu3]

I can paste the full config if you want, but it is the basic the Deeperdeepsea config with only the ops: [evaluate] set. The train_model and evaulate_model are both specified. Uncommenting the n_test_samples fixes the issue:

evaluate_model: !obj:selene_sdk.EvaluateModel {
features: !obj:selene_sdk.utils.load_features_list {
input_path: ./distinct_classes.txt,
},
batch_size: 128,
#n_test_samples: 120000,
trained_model_path: ./checkpoint.pth.tar,
use_cuda: True
}