oneDNN

[Victor-Ch]

Have you ever considered to implement oneDNN technology (oneAPI Deep Neural Network Library) as the alternative to eigen and OpenCL for old windows PCs or PCs with no modern GPU?
Would be good to see a build with DNN for CPU.

https://github.com/uxlfoundation/oneDNN

[lightvector]

Are there relevant benchmarks between oneDNN and eigen on the same deep-learning-style matrix operations that would indicate that it would actually be better, and by how much? Adding a backend is a lot of work and forever thereafter has a cost of future changes harder because there's one more backend whose code needs to be perpetually updated if there are any changes to the network.

[Victor-Ch]

Here is the answer of an AI on your question above

Comparative performance tests of OneDNN and Eigen for neural networks (especially in games like Go and chess) show a clear distinction:

OneDNN (formerly MKL-DNN) is significantly more efficient and faster than Eigen for neural network inference on the CPU. This is especially critical for game engines where minimal latency is required for move prediction.

---

📊 Comparison in Key Aspects:

Parameter	OneDNN (Intel)	Eigen
CPU Optimization	✅ Deep: JIT code, AVX-512/VNNI, specific kernels	⚠️ General: vector operations (SSE/AVX)
Performance	Higher (2-5x in tests)	Lower, especially on small data sizes
Network Support	Convolutions, RNNs, LSTMs, fully connected layers	Basic linear algebra operations
Memory Usage	Optimized for CPU cache	Standard allocations
Ease of Integration	Medium (requires learning API)	✅ Very simple (header-only library)
Quantization Support	✅ INT8, BF16 (up to 4x speedup)	❌ FP32 only

---

🔍 Details for Game Engines (Chess, Go):

1. Network Architecture:

   • Usually fully connected networks or convolutions (e.g., AlphaZero-style architectures).
   • OneDNN is optimized specifically for such operations (GEMM, convolutions).

2. Data Sizes:

   • Inference for a single move is processing one board state (batch=1).
   • OneDNN wins even on small batch sizes due to hand-optimized assembly kernels.

3. Test Example (FC-layer Inference, AVX-512):

   Operation: FullyConnected, Size: 256x256x256
   - OneDNN: ~0.12 ms
   - Eigen:  ~0.45 ms
   

   A difference of 3.5-4x in favor of OneDNN.

---

🚀 Why is OneDNN Faster?

• Assembly Optimization: Hand-tuned kernels for specific Intel microarchitectures (Haswell, Skylake, Ice Lake).
• JIT Compilation: Generates optimal code for a specific operation and data size.
• Cache Optimization: Minimizes cache misses during memory operations.
• INT8 Support: Quantization provides up to 4x speedup with minimal accuracy loss (critical for fast search).

---

🧠 When to Choose Eigen?

• For Prototyping: Eigen is easier to integrate (header-only).
• For Cross-Platform: Eigen works anywhere C++ works.
• If AVX Support is Lacking: The difference is smaller on older CPUs.

---

✅ Practical Conclusion for Engines like Lc0 (chess), KataGo (Go):

• OneDNN is the de facto standard for production engines on CPU.
• Lc0 uses OneDNN for CPU mode (the onednn backend).
• KataGo also uses highly optimized code (written manually in C++/assembly) based on principles similar to OneDNN.

Recommendation:
Use OneDNN for maximum performance in games. Consider Eigen only as a temporary solution for rapid prototyping or if you are targeting weak CPUs without AVX support.

---

The result will always be in favor of OneDNN (50-300% faster depending on the CPU and network size).

[lightvector]

I don't find the AI answer very useful here, particularly as it gives no citation for its "test example" which appears to just be some made-up benchmark numbers.

I do have some trouble finding benchmarks online. One of the things I found was this which is advertising for yet a different library/product, but does have some side-by-side comparisons including eigen and onednn which suggest that it's pretty mixed and variable depending on hardware and the particular matrix sizes involved, but that eigen is sometimes significantly better too. Maybe it's different and more consistent with convolution, but this suggests a much more unclear picture than the AI answer above. Do you have other real benchmarks?
https://www.modular.com/blog/the-worlds-fastest-unified-matrix-multiplication

[Victor-Ch]

No, I don't have any real benchmarks, but from my own experience I can say that Intel SSE4.2 with 2 cores CPU with OpenDNN outperformed the old GPU Nvidia with CUDA 2.1 with OpenCL by about 30-50%, maybe even more (on 256x10 and 512x15 Lc0 networks). The advantage is especially noticeable on the 512 network.
Regarding the comparison with Eigen, I can’t say anything on my own.
Probably, DNN is not as good on other types of processors, since it is mainly designed for Intel.
Still it looks like DNN may be a better alternative to OpenCL for older GPUs or PCs without GPUs.

[MAOmao000]

I hope this helps...
Below is a fork of Leela zero that uses eigen, openblas, and onednn backends.
While it's not accurate due to limited comparison data, the onednn backend appears to be 10% to 20% faster than the eigen backend.
https://github.com/MAOmao000/leela-zero-ladder