Real-time use

[SuperKogito]

Hello Jinay,
Thank you for this nice project.
Is it possible to use this in real-time applications? If possible, how should I go about changing the code in your opinion :)) ?

[jinay1991]

Hi @SuperKogito Yes you could use the NVIDIA CUDA Accelerations by converting the Model to the TensorRT and add new inference Engine to use NVIDIA Inference APIs and Model. If this is not an option, you could use the TFLite Model (Android/iOS compatible) to use this in real-time by leveraging the Neural Engines that modern devices offer.

Basically, for each type of acceleration, you can introduce a new Inference Engine in the source code and convert the model for that accelerator. This should work-out-of-the-box (may require little tweaks here-n-there)

[SuperKogito]

Thank you for this fast response <3
Did you try to convert the model using TensorRT? I took a look at the library and it has no maxpooling layer, which is as far as I know part of Spleeter architecture.

Leveraging the Neural Engines that modern devices offer is actually exactly what I am trying to do but with https://github.com/gvne/spleeterpp, Currently I am failing to use a Tensorflow built from source though. As you are more experienced with inference engines, do you think building Tensorflow from source helps accelerate the project or it is just a minor improvement?

Based on my analysis of your neat project, I noticed that this library mainly do inference, on the wav file read and passed as a CLI argument. Is there any handling of the buffering needed for RT, e.g. Ring buffers for writing and reading that makes the library integration into a VST easier?

I am sorry for these many questions but I got one more, in your experience, what's the expected latency?
I really appreciate you taking the time to answer me. Thank you so much :))

[jinay1991]

Did you try to convert the model using TensorRT?

No, I haven't try! You are welcome to try and please provide your feedback so that others can benefit.

do you think building Tensorflow from source helps accelerate the project or it is just a minor improvement?

I don't think building TF from source will help since it can still be optimised for CPU (uses Intel Acceleration instruction sets), for NVIDIA GPU it may use CUDA. Ideally, these binaries you could download readily and use. But since they don't offer libtensorflow_cc.so (monolithic) binaries from TF, unfortunately. You have to build it manually on your system. Instructions can be found here

Is there any handling of the buffering needed for RT, e.g. Ring buffers for writing and reading that makes the library integration into a VST easier?

Yes, you have to write complete RTSP like buffer to feed the stream in real-time. Use FFMPEG and extend the existing code to support RTSP (Real-Time Streaming Protocol). Google for more info, you will also get some good GitHub Repositories for this.

what's the expected latency?

I have not estimated/calculated for this Model so can't comment really. But since the model is quite small, it should be pretty fast if you have decent accelerated platform.