New feature: OnnxPredict algorithm #1488
Conversation
| OS=$(uname -s) | ||
| CONFIG=Release | ||
|
|
||
| if [ "$OS" = "Darwin" ]; then |
There was a problem hiding this comment.
@xaviliz, since we are inside debian_3rdparty, should we remove or move somewhere else the MacOS support?
There was a problem hiding this comment.
Yes, that's true. I kept it for testing purpouses. Let me clean it a bit.
There was a problem hiding this comment.
It has been tested on Linux.
| const char* OnnxPredict::name = "OnnxPredict"; | ||
| const char* OnnxPredict::category = "Machine Learning"; | ||
|
|
||
| const char* OnnxPredict::description = DOC("This algorithm runs a Onnx graph and stores the desired output tensors in a pool.\n" |
There was a problem hiding this comment.
it should be an ONNX model, there is no access to graphs in onnxruntime. It is fixed now.
|
|
||
| // Do not do anything if we did not get a non-empty model name. | ||
| if (_graphFilename.empty()) return; | ||
| cout << "after return" << endl; |
| _env = Ort::Env(ORT_LOGGING_LEVEL_WARNING, "multi_io_inference"); // {"default", "test", "multi_io_inference"} | ||
|
|
||
| // Set graph optimization level - check https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html | ||
| _sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED); |
There was a problem hiding this comment.
Since there are different optimization options, I'm wondering if there is a chance that extended optimization doesn't work or affects model performance in some cases. I think this should be turned into a parameter that defaults to extended.
There was a problem hiding this comment.
That's a good point, I am not sure how optimizations could affect the performance. Adding new parameter sounds good to me. So, do you propose to add boolean parameter for each optimization? or just an string to use one of them?
| // Set graph optimization level - check https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html | ||
| _sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED); | ||
| // To enable model serialization after graph optimization set this | ||
| _sessionOptions.SetOptimizedModelFilePath("optimized_file_path"); |
There was a problem hiding this comment.
I think this is mainly intended for debugging purposes. Can we skip saving the optimized graph for efficiency?
https://onnxruntime.ai/docs/api/c/struct_ort_api.html#ad238e424200c0f1682947a1f342c39ca
There was a problem hiding this comment.
yes, we don't need to store the optimized graph in a model.
| return out; | ||
| } | ||
|
|
||
| void OnnxPredict::reset() { |
There was a problem hiding this comment.
Shouldn't we reset the session and env too?
There was a problem hiding this comment.
That's a good point. I couldn't find a reset method for session and env in the CPP_API like in Tensorflow. But let me try it using std::unique_ptr maybe that could work. however, I am doubting if we should do that after compute(), because if we reset the session at the end of configure(), session.Run() will fail.
| const Pool& poolIn = _poolIn.get(); | ||
| Pool& poolOut = _poolOut.get(); | ||
|
|
||
| std::vector<std::vector<float>> input_datas; // <-- keeps inputs alive |
There was a problem hiding this comment.
input_datas -> input_data?
I think data is already plural
| // Step 2: Convert data to float32 | ||
| input_datas.emplace_back(inputData.size()); | ||
| for (size_t j = 0; j < inputData.size(); ++j) { | ||
| input_datas.back()[j] = static_cast<float>(inputData.data()[j]); |
There was a problem hiding this comment.
Instead of forcing casting data to float, shouldn't we keep it in Real format (that is actually float32 by default) and make sure that the model runs in whatever type Real points to?
| } | ||
|
|
||
| // Step 3: Create ONNX Runtime tensor | ||
| _memoryInfo = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault); |
There was a problem hiding this comment.
Would it be possible to run the models on GPU if available?
| def _create_essentia_class(name, moduleName = __name__): | ||
| essentia.log.debug(essentia.EPython, 'Creating essentia.standard class: %s' % name) | ||
|
|
||
| # print(f"name: {name}") |
…hen cuda, metal or open_ml are not compiled in onnxruntime library
…ETAL or OPEN_ML providers are found
2 participants
New feature: OnnxPredict algorithm
Feature
It makes additional changes in Essentia library to build ONNX Runtime Inferencing library from the source and to implement a new algorithm
OnnxPredictfor running ONNX models (.onnx) with multiple IO.Implementation
onnxruntimedynamic library.OnnxPredictto run ONNX models in Essentia.test_onnxpredict.pyPrerequisites
python >= 3.10cmake >= 3.28Testing
How to Test
Tested in
onnxruntime-v1.22.1:python 3.13.4andcmake 4.0.2python 3.10.18andcmake 4.1.0How to build ONNX Runtime
After installing Essentia dependencies in a virtual environment, install
cmakeThen we can run the building script:
cd packaging/debian_3rdparty bash build_onnx.shHow to build OnnxPredict
In MacOS:
In Linux:
python3 waf configure --fft=KISS --include-algos=OnnxPredict,Windowing,Spectrum,MelBands,UnaryOperator,TriangularBands,FFT,Magnitude,NoiseAdder,RealAccumulator,FileOutputProxy,FrameCutter --static-dependencies --with-onnx --lightweight= --with-python --pkg-config-path /usr/share/pkgconfig --std=c++14 python3 waf -v && python3 waf installHow to unittest