[Feature Request] Allow sz* to read new files that have not been downloaded previously

[Beepnblink]

Subject of the issue

I am using the OLA to log operational data for an industrial machine. A data set of 22 values is recorded every second via the serial port. On top of that an environment sensor is recorded from the Qwiic bus every 5s. This creates a fair amount of data. Unloading this data over the USB port via Z modem takes time and creates frustration for maintenance operators.
The options I can see are
sz* to read all files or
sz filename to read a specific file.
Currently we run a script to down load all data which leads to longer and longer download times.
Ideally I would want to only read new files that have not been downloaded previously to optimise the download time.
Would it make sense to add a feature that adds a flag to each data set if it has been downloaded? In combination with a command e.g. ‘sz new’ to only download the latest information?

Your workbench

• microSD card 16GB FAT32
• logging frequency: 1Hz serial, every 5 seconds analog
• Version of firmware Artemis OpenLog v2.6
• One BME280 on QWIIC bus
• External 3V3 power / 3V coin cell for RTC backup
• Z-Modem speed set to 115200Bd
• The datalogger is packaged in an enclosure, so access to the SD card requires dissassembly. Also handling microSD cards in industrial environments is not ideal (seen them being dropped and lost).

Steps to reproduce

download large data sets from SD via Z-Modem

Expected behavior

Feature to reduce download time be selectively downloading new files only

Actual behavior

Either download all data or deal with individual filenames for download.

Regards, Peter

[PaulZC]

Hi Peter ( @Beepnblink ),

I can think of two ways to tackle this: maintain a list of downloaded files; use the file attributes to indicate if the file has been downloaded.

For the list of files, the firmware would: create a file containing the list of downloaded files, initially empty; as files are downloaded, the filename (and directory?) is added to the list file; the sz* would only only download files not named in the list file, adding them after download. The list file could be deleted to allow all files to be downloaded again if needed.

For the file attributes, it might be possible to use the SdFat attrib methods to set (e.g.) the archive attribute on files which have been downloaded. sz* would ignore any files with the archive attribute bit set. A new command would need to be added to allow the archive attribute to be cleared on all (or individual) files so they can be downloaded with sz* again.

The file attribute option may be slightly easier to write and faster in operation.

I don't have time to look into this right now. If you have time to add this feature, please do and send us a Pull Request. Meanwhile, I'm very happy to leave this request open.

Best wishes,
Paul

[PaulZC]

Hi Peter,

I'm adding support for "rm *" / "del *". Not a great solution for you - not if you want to preserve your data - but this would at least let you avoid downloading all the files each time.

Keep an eye out for a v2.10 update later today.

Best wishes,
Paul