Cancelling a FileSystemIdentificationRequest / Droid execution reliably

[toby19800]

Hello,

we have integrated Droid in our java application, however we would like to know what is the appropriate way to abort a FileSystemIdentificationRequest if it takes too much time. Is calling close on the request / interrupting the thread running the code sufficient or what is the best way to get a reliable abort from the API? Code fragment in use as example:

            RequestMetaData metadata = new RequestMetaData(Files.size(filePath), Files.getLastModifiedTime(filePath).toMillis(), fileName);
            RequestIdentifier identifier = new RequestIdentifier(file.toURI());

            droidRequest = new FileSystemIdentificationRequest(metadata, identifier);
            droidRequest.open(filePath);
            IdentificationResultCollection results = binSigIdentifier.matchBinarySignatures(droidRequest);  // <= used here

            IdentificationResultCollection containerResults = identifyContainer(droidRequest, results);     // <= and here

Thank you

[MancunianSam]

I've had a look into this. There isn't really a way at the moment to cancel an identification request once its running. You can do something like this where you close the droidRequest on the interrupt.

FutureTask<IdentificationResultCollection> task = new FutureTask<>(() -> binarySignatureIdentifier.matchBinarySignatures(droidRequest)) {
            @Override
            public boolean cancel(boolean mayInterruptIfRunning) {
                try {
                    droidRequest.close();
                    return true;
                } catch (IOException e) {
                    throw new RuntimeException(e);
                }
            }
        };
        try (var executorFactory = new BlockingThreadPoolExecutorFactory().newInstance()) {
            executorFactory.submit(task);
            Thread.sleep(10); // Simulate waiting
            task.cancel(true);
        }

This won't stop file identification as such. DROID gets a list of signatures from the signature file and loops through these to try to identify the file. It reads the file in (usually 4kb) chunks based on where the signature says to check.
Once a section of the file has been read, DROID caches that part of the file. If it's needed again, it's retrieved from the cache rather than reading the file.
So closing the request will only have an effect if DROID has to read a chunk that it hasn't previously read. In that case, it will throw an exception and move onto the next chunk or the next signature.

We can have a look at adding an interruptible version of this method into DROID core and there's a few options. What sort of scenarios do you have where you would need to cancel a file identification request? If it's because file reads are slow then the solution above should help slightly. Or is it something else.

If you let me know what sort of problems you're seeing that would make you want to cancel the request then we can have a look at implementing something to fix it.

[toby19800]

Thanks for the reply @MancunianSam.

So far, we have been encapsulating Droid and other format identification tools within a dedicated Java CLI application. Due to differing Java versions and dependency collisions, we are now moving towards implementing them as individual microservices, with scalability as another goal.

We would like to be able to abort an identification process either after a user-defined timeout (e.g. 60 seconds) or on user demand. Ideally, the API would provide a mechanism to cancel a request within a reasonable waiting time or to react to interrupts, as we want to avoid the overhead of starting Droid as a CLI process in the service and reloading the signature files each time.

I cannot realistically estimate the exact execution times of Droid in our environment, but since we have a very large volume of data coming into the archive – including many files over 50 GB (pre-ingest also includes formats such as zip files) – my concern is that if requests are closed prematurely multiple times while processing “problematic” files, tasks might continue to run in the background and eventually degrade or crash the service. It would be extremely helpful if there was a way to interrupt the identification process cleanly. I assume this would also apply to the ContainerIdentifier, which is invoked like this:

ContainerIdentifier containerIdentifier = containerIdentifierFactory.getIdentifier(containerFormat);
containerIdentifier.setMaxBytesToScan(droidMaxBytes);
IdentificationResultCollection containerResults = containerIdentifier.submit(request);

We have many files that need to be processed, but at the same time, we are somewhat limited in terms of available hardware and scalability. I would be very interested to hear your perspective on this, and any alternative suggestions you might have would be greatly appreciated.

Thanks a lot in advance for your help.

[steve-daly]

Hi @toby19800 are you using DROID to generate checksums or just for characterising (identifying) file formats?

[toby19800]

Hi @steve-daly we only run format identification (binary and container signatures).

[steve-daly]

I don't know whether this is available from the mechanisms you use to call the DROID libraries, but in the GUI and CLI you can specify a maximum number of bytes to scan from each end of the file. It defaults to 65,536 on a fresh install, but many people change this to -1, which means unlimited. Ideally DROID will characterise a file within that range, but there are some formats that need to read much more of the file to find the appropriate signature sequence. You could do some testing on how large a file you could identify within your desired time constraints, and set an appropriate limit to automatically abort larger detections. This is better than adjusting based on source file size, as even very large files might only need a few bytes read from the header/footer to identify them, but this can set an effective limit on how long the DROID job could run for.