Let split_ncvars split large Netcdf-4 files

[ceblanton]

resolves #311

[ilaflott]

The default behavior was to take a netCDF file input, and create 64-bit offset netCDF3. Perhaps that is not desired in the case of issue #311, but do we know where this is desired? do we know where people are expecting this tool to output specifically a netCDF3 file?

more to the point: why is the proposed solution to change the default behavior, rather than adding an extra flag to achieve the desired behavior?

[ilaflott]

Another thought- given that the desired behavior is input_file_type == output_file_type, if the input file is of the 64-bit offset variety enabling larger file output, then isn't removal of the ncks version checking logic re-introducing potential for failures when the version of ncks isn't quite what's expected? wouldn't we need that logic to preserve the offset?

[ceblanton]

Thanks Ian. Good questions. Your last question is the easiest. The NCO version checking is only needed for outputting the 64-bit NC3 format, so if we remove that option, the NCO version can be ignored. (That's a nice document about netcdf, there!)

I see your point about flexibility. If split-ncvars.pl were used for multiple purposes and some of the purposes were format dependent, then I would agree with an --format option. But FRE is the only known user of split-ncvars, and if there is an outcry we can add an option (But.. then we'd have to check the NCO version probably). Users can always use nccopy or even ncks to convert formats.

The main reason to not add an extra flag to get the desired format behavior is that FRE would then have to check the version of the input file, just to pass it to the split tool. That wouldn't be terrible but would be unnecessary, since the desired behavior should be to output the same format as the input. That is the convention used by other tools, such as mppnccombine, fregrid, and timavg.csh.

[ilaflott]

I see, using 64-bit offset DEMANDS netcdf3 format.

Looking through the development history of the script, it seems that netCDF3 with 64-bit offset was the initial choice. Given that points to a commit converting a c-shell implementation to a perl one, and the commit before that adds the c-shell script in for the first time, the original intention of specifying 64-bit is probably lost to time.

I'll speculate that 64-bit offset around a one or two decades ago was likely the most flexible choice back then. I'll say that justifies an update now.

If there is user demand, there shouldn't be any reason we can't pass a file-format argument through to ncks's fl_fmt field for greater control of output.