Make ML/PyTorch native libraries compatible with external JDK for FIPS 140-2 compliance

[elastickent]

Description

Summary

Currently, Elasticsearch FIPS 140-2 compliance and Machine Learning (ML) features are mutually exclusive due to PyTorch native library incompatibility with external JDKs.

Problem

FIPS Documentation Requirement:

• FIPS compliance docs require:

  "The JVM bundled with Elasticsearch is not configured for FIPS 140-2. You must configure an external JDK with a FIPS 140-2 certified Java Security Provider."

• Specifically requires Oracle JDK 11/17+ for production FIPS compliance

ML Native Library Limitation:

• PyTorch native libraries (libtorch_cpu.so) are pre-compiled against the bundled OpenJDK
• Using an external JDK (Oracle JDK 17, Eclipse Temurin, etc.) causes ML processes to crash with SIGILL (Illegal Instruction)
• Error example:

Fatal error: 'si_signo 4, si_code: 2' (SIGILL - Illegal Instruction) library: /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libtorch_cpu.so

Impact:
Organizations requiring both FIPS 140-2 compliance AND ML features (ELSER, NLP models, anomaly detection) cannot use both simultaneously.

Current Workarounds

1. Disable ML entirely - Use external Oracle JDK for FIPS, lose all ML functionality
2. Use bundled OpenJDK with BouncyCastle FIPS - Maintain ML, but unclear regulatory compliance status (OpenJDK not FIPS-certified platform)
3. Choose between compliance OR features - Unacceptable for federal/regulated industries

Expected Behavior

Users should be able to:

1. Run Elasticsearch in FIPS 140-2 compliant mode (with Oracle JDK or documented alternative)
2. Use all ML features (ELSER, NLP, anomaly detection, etc.)
3. Have clear compliance documentation for regulatory audits

Actual Behavior

Users must choose between FIPS compliance OR ML features, but cannot have both.

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)