6.1.5

[DevinTDHa]

📢 Spark NLP 6.1.5: Smarter Readers and More Resilient Pipelines

Spark NLP 6.1.5 focuses on improving data ingestion reliability and pipeline flexibility. This release enhances reader components with better fault tolerance, broader input support, and introduces a new ReaderAssembler annotator for streamlined integration. Several key fixes also improve model loading and stability in distributed environments.

🔥 Highlights

• New ReaderAssembler Annotator: Unify multiple reader annotators into one configurable component for simpler and cleaner ingestion pipelines.

🚀 New Features & Enhancements

Reader Pipeline Enhancements

• ReaderAssembler Annotator
  A new meta-annotator that unifies Reader2X components (e.g., Reader2Doc, Reader2Image, Reader2Table) under a single interface.

  • Automatically selects the right reader(s) based on configuration.
  • Supports declarative assembly of reading stages.
  • Provides parameters for reader selection, fallback rules, and error handling.
    This simplifies pipeline construction and improves maintainability for multi-format ingestion workflows. (Link to notebook)

• Support for String Input Columns in Readers (SPARKNLP-1291)
  Spark NLP readers only supported inputs via file paths. That means if you already had a DataFrame with text content (say from another pipeline or a preliminary load), you had to write it to disk just to let the reader ingest it. This adds friction and overhead, especially in streaming or in-memory pipelines.

  With this change, you can:

  • Feed raw text stored in a DataFrame column directly into Spark NLP readers — zero I/O overhead when not needed.
  • Simplify workflows and pipelines (no need for temporary file staging just to “read” back data).
  • Improve performance and resource usage in scenarios where input is already available as strings (e.g. generated, preprocessed, or coming from another system).
  • Make the reader APIs more flexible and general-purpose.

• Fault-Tolerant XML Reader
  The XML reader now skips malformed XML fragments (e.g., mismatched tags, missing closures, invalid characters) instead of failing the job.
  Enhanced error handling ensures more resilient ingestion of imperfect real-world data.

🐛 Bug Fixes

• GGUF Model Loading Duplication
  Fixed an issue in FeaturesFallbackReader that caused duplicate loading or missing model files when calling .pretrained() on GGUF-based annotators such as AutoGGUFModel and rerankers, especially in Databricks environments.

❤️ Community Support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

💻 Installation

Python

pip install spark-nlp==6.1.5

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.5

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.5

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.5

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.5

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.1.5.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.1.5.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.1.5.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.1.5.jar

What's Changed

• [SPARKNLP-1291] Adding support fort input string column on readers #14665 by @danilojsl
• [SPARKNLP-1292] Adding fault-tolerance support for malformed XML #14666 by @danilojsl
• [SPARKNLP-1290] Introducing ReaderAssembler Annotator #14668 by @danilojsl
• fix duplicate loading in FeaturesFallbackReader #14667 by @C-K-Loan, @DevinTDHa

Full Changelog: 6.1.4...6.1.5

---

This discussion was created from the release 6.1.5.