🎉 Pure Go HDF5 Library - Production-Ready HDF5 2.0.0 Format Support

[kolkov]

Overview

We're pleased to announce that the HDF5 Go Library has reached a significant milestone with comprehensive HDF5 2.0.0 format support (current version: v0.13.1). This pure Go implementation provides production-ready HDF5 functionality without external C dependencies, making it ideal for modern Go applications requiring HDF5 file handling.

HDF5 2.0.0 Format Support

The library implements complete support for HDF5 Format Specification v4.0, handling all defined superblock formats:

Superblock Support:

• ✅ Version 0 (HDF5 1.0-1.6): Complete read/write
• ✅ Version 2 (HDF5 1.8+): Complete read/write with CRC32 validation
• ✅ Version 3 (HDF5 2.0.0): Complete read/write with CRC32 validation

Key Capabilities:

• Read/Write Operations: Full dataset and attribute support across all versions
• Checksum Validation: CRC32 verification for v2/v3 superblocks
• Backward Compatibility: Seamless handling of files from HDF5 1.0 through 2.0.0
• Format Compliance: Implements HDF5 Format Specification v4.0

Security & Reliability

Recent releases addressed 4 security vulnerabilities (HIGH to MEDIUM severity):

• CVE-2025-7067 (HIGH 7.8): Buffer overflow in chunk reading
• CVE-2025-6269 (MEDIUM 6.5): Heap overflow in attribute reading
• CVE-2025-2926 (MEDIUM 6.2): Stack overflow in string handling
• CVE-2025-44905 (MEDIUM 5.9): Integer overflow in hyperslab selection

Security Infrastructure:

• Overflow-safe arithmetic operations (internal/utils/overflow.go)
• 39 comprehensive security test cases
• Proactive validation across all I/O operations

Advanced Features

Large Dataset Support

64-bit chunk dimensions enable handling of massive scientific datasets:

• Chunks >4GB per dimension
• Auto-detection based on superblock version
• Seamless backward compatibility

AI/ML Datatypes

Native support for modern machine learning datatypes:

• FP8 E4M3: 8-bit float (4-bit exp, 3-bit mantissa) - Neural network training
• FP8 E5M2: 8-bit float (5-bit exp, 2-bit mantissa) - Gradient computation
• bfloat16: Brain floating point - Google TPU/GPU standard

Perfect for storing model weights, activations, and gradients in HDF5 format.

Production-Ready Rebalancing API

The B-tree rebalancing API is now stable and recommended for production use, offering automatic performance optimization for large datasets.

Quality Metrics

The library maintains scientific-grade software quality standards:

• ✅ 86.1% test coverage (exceeds 70% target)
• ✅ 0 linter issues (34+ linters via golangci-lint)
• ✅ 98.2% reference test pass rate (56/57 files from HDF Group)
• ✅ 433 test files from official HDF5 test suite
• ✅ Cross-platform: Linux, macOS, Windows
• ✅ Pure Go: No CGo dependencies

Development Phase Transition

With HDF5 2.0.0 format support now complete, the project is transitioning from rapid development to a stable maintenance phase. Future focus areas include:

• Bug fixes and stability improvements
• Performance optimization
• Community feedback integration
• Monitoring HDF Group reference library developments
• Minor feature enhancements

Installation & Requirements

go get github.com/scigolib/hdf5@latest

Requirements:

• Go 1.25+
• No external dependencies
• No CGo required

Compatibility: Files created by HDF5 C library (versions 1.0 through 2.0.0) are fully supported.

Quick Example

package main

import (
    "fmt"
    "log"
    "github.com/scigolib/hdf5"
)

func main() {
    // Open HDF5 file (any version: 1.0 through 2.0.0)
    file, err := hdf5.Open("data.h5")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    // Check superblock version (0, 2, or 3)
    version := file.SuperblockVersion()
    fmt.Printf("Superblock version: %d\n", version)

    // Read dataset
    dataset, err := file.Dataset("/measurements")
    if err != nil {
        log.Fatal(err)
    }

    var data []float32
    if err := dataset.Read(&data); err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Read %d values\n", len(data))
}

Use Cases

The library is particularly valuable for:

• Scientific Computing: Large-scale simulations requiring HDF5 I/O
• Machine Learning: Storing FP8/bfloat16 model weights and activations
• Data Security: Applications requiring validated, overflow-safe I/O
• Cloud-Native Applications: Pure Go deployment without C dependencies
• Cross-Platform Tools: Unified HDF5 support across Linux, macOS, Windows

Resources

• Repository: https://github.com/scigolib/hdf5
• Documentation: https://github.com/scigolib/hdf5/tree/main/docs
• Examples: https://github.com/scigolib/hdf5/tree/main/examples
• GoDoc: https://pkg.go.dev/github.com/scigolib/hdf5
• Latest Release: v0.13.1 (Documentation Correction Hotfix)
• Forum Discussion: https://forum.hdfgroup.org/t/pure-go-hdf5-library-hdf5-2-0-0-format-support-announcement/13584

Recent Release - v0.13.1

The latest hotfix corrects documentation to accurately reflect HDF5 specification. A previous release incorrectly referenced "Superblock Version 4," which does not exist in the HDF5 Format Specification. The specification defines superblock versions 0 through 3 only. This was a documentation-only issue with no functional impact.

License & Community

License: MIT (permissive open source)

We welcome feedback, bug reports, and contributions from the community:

• Issues: https://github.com/scigolib/hdf5/issues
• Discussions: https://github.com/scigolib/hdf5/discussions
• Contributing: See repository for guidelines

Acknowledgments

This project demonstrates the power of human-AI collaboration in scientific software development. Built with assistance from Claude Code, combining domain expertise with AI-accelerated development while maintaining rigorous quality standards.

Thank you to The HDF Group for maintaining the HDF5 specification and reference implementation, and to everyone in the community using the library, reporting issues, and providing feedback!

---

Current Version: v0.13.1
Status: Production-ready, stable maintenance phase
Installation: go get github.com/scigolib/hdf5@latest