Enhanced s5.js - Complete Grant Implementation (Milestones 1-8)

.gitignore

@@ -1 +1,74 @@
-node_modules
+# Dependencies
+node_modules
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# Build outputs
+dist/
+build/
+*.tgz
+out/
+.next/
+
+# Runtime data
+pids
+*.pid
+*.seed
+*.pid.lock
+
+# Coverage directory used by tools like istanbul
+coverage/
+*.lcov
+.nyc_output/
+
+# Environment variables
+.env
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+.env.*.local
+
+# IDE files
+.vscode/settings.json
+.idea/
+*.swp
+*.swo
+*~
+
+# OS generated files
+.DS_Store
+.DS_Store?
+._*
+.Spotlight-V100
+.Trashes
+ehthumbs.db
+Thumbs.db
+
+# Temporary files
+*.tmp
+*.temp
+*.backup
+debug_*.js
+.cache/
+
+# Logs
+logs
+*.log
+
+# Miscellaneous
+docs/design/
+docs/grant/
+screenshots/
+docs/development/
+
+# Docker
+.dockerignore
+docker-compose.override.yml
+docker-compose.dev.yml
+
+demos/media/baseline-performance.json
+demos/media/metadata-report.html
+tmp/
+*.gz

.nvmrc

@@ -0,0 +1 @@
+20

CHANGELOG.md

@@ -0,0 +1,184 @@
+# Changelog
+
+All notable changes to Enhanced s5.js will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+Post-grant releases are summarised below. For exhaustive per-version notes (including production-hardening fixes across beta.2–beta.44), see [`docs/POST_GRANT_UPDATE.md`](docs/POST_GRANT_UPDATE.md).
+
+## [0.9.0-beta.47] - 2026-04-19
+
+### Added
+
+- **Cross-Identity Directory Key Lookup** — `FS5.getPublicDirectoryKeyFrom(remotePubKey, subpath)` resolves the 32-byte Ed25519 registry pubkey for any sub-directory under another user's public tree. Returned pubkey is ready to pass to `api.registryListen(pk)` for push-based live subscriptions (no polling). No identity required on the caller side.
+- Empty subpath (`""` or `"/"`) returns the input `remotePubKey` unchanged (pass-through).
+- 8 new tests (total: 548 passing).
+
+### Notes
+
+- Purely additive; no breaking changes, no protocol changes.
+- Returns `undefined` for missing segments, file-as-final, `fixed_hash_blake3` links, or encrypted intermediates.
+- Throws on invalid `remotePubKey` length (must be exactly 32 bytes).
+
+## [0.9.0-beta.46] - 2026-04-09
+
+### Added
+
+- **Cross-Identity Public Directory Read** — two new `FS5` methods enable multi-user data sharing via a shared Ed25519 public key:
+  - `getPublicDirectoryKey(path)` — extract the 32-byte registry pubkey for one of your own directories (requires identity).
+  - `readFromPublicDirectory(remotePubKey, subpath)` — read file content from another user's unencrypted directory tree (no identity required for the reader).
+- Supports nested subpaths and both Map- and HAMT-backed directories.
+- 11 new tests (total: 540 passing).
+
+### Notes
+
+- Purely additive; no breaking changes. FS5 child directories were already stored unencrypted, so no encryption changes were needed.
+
+## [0.9.0-beta.45] - 2026-04-03
+
+### Changed
+
+- **Per-Directory Mutex** — concurrent `fs.put()` calls to the same directory now serialize via a keyed `AsyncMutex`, eliminating retry cascades (30–65s → 2–10s under contention). Different directories remain fully parallel. Zero external dependencies, automatic lock release on error.
+
+### Added
+
+- 123 concurrency tests and 108 mutex unit tests.
+
+## [0.9.0-beta.1] - 2025-10-31
+
+### Major Features - Sia Foundation Grant Implementation
+
+This release represents the culmination of an 8-month Sia Foundation grant to enhance s5.js with a comprehensive set of features for decentralized storage applications.
+
+#### Path-based API (Phases 2-3)
+- **Added** simplified filesystem API with `get()`, `put()`, `delete()`, `list()`, and `getMetadata()` operations
+- **Added** automatic path normalization and Unicode support
+- **Added** CBOR-based DirV1 directory format for deterministic serialization
+- **Added** DAG-CBOR encoding for cross-implementation compatibility
+- **Added** cursor-based pagination for efficient large directory iteration
+- **Added** directory creation and management utilities
+
+#### HAMT Sharding (Phase 3)
+- **Added** Hash Array Mapped Trie (HAMT) for scalable directory storage
+- **Added** automatic sharding at 1000+ entries per directory
+- **Added** 32-way branching with xxhash64 distribution
+- **Added** transparent fallback between flat and sharded directories
+- **Added** O(log n) performance for directories with millions of entries
+
+#### Directory Utilities (Phase 4)
+- **Added** `DirectoryWalker` class for recursive directory traversal
+- **Added** configurable depth limits and filtering options
+- **Added** resumable traversal with cursor support
+- **Added** `BatchOperations` class for high-level copy/delete operations
+- **Added** progress tracking and error handling for batch operations
+
+#### Media Processing (Phases 5-6)
+- **Added** `MediaProcessor` for image metadata extraction
+- **Added** WebAssembly (WASM) based image processing with Canvas fallback
+- **Added** automatic browser capability detection
+- **Added** support for JPEG, PNG, WebP formats
+- **Added** thumbnail generation with smart cropping
+- **Added** dominant color extraction and color palette generation
+- **Added** progressive image loading support
+- **Added** FS5 integration: `putImage()`, `getThumbnail()`, `getImageMetadata()`, `createImageGallery()`
+
+#### Advanced CID API (Phase 6)
+- **Added** `FS5Advanced` class for content-addressed operations
+- **Added** `pathToCID()` - convert filesystem paths to CIDs
+- **Added** `cidToPath()` - resolve CIDs to filesystem paths
+- **Added** `getByCID()` - retrieve data directly by CID
+- **Added** `putByCID()` - store data with explicit CID
+- **Added** CID utility functions: `formatCID()`, `parseCID()`, `verifyCID()`, `cidToString()`
+- **Added** 74 comprehensive tests for CID operations
+
+#### Bundle Optimization (Phase 6)
+- **Added** modular exports for code-splitting
+- **Added** `@s5-dev/s5js` - full bundle (61 KB brotli)
+- **Added** `@s5-dev/s5js/core` - core functionality without media (60 KB)
+- **Added** `@s5-dev/s5js/media` - media processing standalone (10 KB)
+- **Added** `@s5-dev/s5js/advanced` - core + CID utilities (61 KB)
+- **Achievement**: 61 KB compressed - **10× under the 700 KB grant requirement**
+
+#### Testing & Documentation (Phases 7-8)
+- **Added** 437 comprehensive tests across all features
+- **Added** real S5 portal integration testing (s5.vup.cx)
+- **Added** browser compatibility testing (Chrome, Firefox, Safari)
+- **Added** performance benchmarks for HAMT operations
+- **Added** comprehensive API documentation
+- **Added** getting-started tutorial and demo scripts
+- **Added** mdBook documentation for docs.sfive.net integration
+
+### Core Improvements
+
+#### Compatibility
+- **Fixed** browser bundling by removing Node.js-specific dependencies
+- **Fixed** replaced undici with native `globalThis.fetch` for universal compatibility
+- **Added** support for Node.js 18+ native fetch API
+- **Added** dual browser/Node.js environment support
+
+#### Architecture
+- **Added** dual MIT/Apache-2.0 licensing matching s5-rs ecosystem
+- **Improved** TypeScript type definitions and IDE support
+- **Improved** error handling and validation across all APIs
+- **Improved** test coverage to 437 tests passing
+
+#### Bundle Exports
+- **Fixed** export architecture to properly include all functionality
+- **Fixed** advanced bundle now correctly includes core features
+- **Fixed** media bundle can be used standalone or lazy-loaded
+
+### Breaking Changes
+
+- **Path API**: New primary interface for file operations (legacy CID-based API still available)
+- **Directory Format**: Uses DirV1 CBOR format (not compatible with old MessagePack format)
+- **Package Name**: Published as `@s5-dev/s5js` (replaces `s5-js`)
+- **Node.js**: Requires Node.js 20+ (for native fetch support)
+
+### Grant Context
+
+This release fulfills Milestones 2-8 of the Sia Foundation grant for Enhanced s5.js:
+- **Month 2-3**: Path-based API and HAMT integration
+- **Month 4**: Directory utilities (walker, batch operations)
+- **Month 5**: Media processing foundation
+- **Month 6**: Advanced media features and CID API
+- **Month 7**: Testing and performance validation
+- **Month 8**: Documentation and upstream integration
+
+**Total Grant Value**: $49,600 USD (8 months × $6,200/month)
+
+### Performance
+
+- **HAMT Sharding**: O(log n) operations on directories with millions of entries
+- **Bundle Size**: 61 KB (brotli) - 10× under budget
+- **Cursor Pagination**: Memory-efficient iteration over large directories
+- **Media Processing**: Thumbnail generation in ~50ms (WASM) or ~100ms (Canvas)
+
+### Known Limitations
+
+- Browser tests require Python 3 for local HTTP server
+- WebAssembly media processing requires modern browser support
+- HAMT sharding threshold set at 1000 entries (configurable)
+
+### Contributors
+
+- **Jules Lai (julesl23)** - Grant implementation
+- **redsolver** - Original s5.js architecture and guidance
+- **Lume Web** - S5 protocol development
+
+### Links
+
+- **Grant Proposal**: [Sia Foundation Grant - Enhanced s5.js](docs/grant/Sia%20Standard%20Grant%20-%20Enhanced%20s5_js.md)
+- **API Documentation**: [docs/API.md](docs/API.md)
+- **Design Documents**:
+  - [Enhanced S5.js - Revised Code Design](docs/design/Enhanced%20S5_js%20-%20Revised%20Code%20Design.md)
+  - [Enhanced S5.js - Revised Code Design - Part II](docs/design/Enhanced%20S5_js%20-%20Revised%20Code%20Design%20-%20part%20II.md)
+- **Testing Guide**: [docs/testing/MILESTONE5_TESTING_GUIDE.md](docs/testing/MILESTONE5_TESTING_GUIDE.md)
+- **Bundle Analysis**: [docs/BUNDLE_ANALYSIS.md](docs/BUNDLE_ANALYSIS.md)
+- **Benchmarks**: [docs/BENCHMARKS.md](docs/BENCHMARKS.md)
+
+---
+
+## Pre-Grant History
+
+For changes prior to the Enhanced s5.js grant project, see the original s5.js repository history.

Dockerfile

@@ -0,0 +1,59 @@
+FROM ubuntu:22.04
+
+# Set environment variables to prevent interactive prompts
+ENV DEBIAN_FRONTEND=noninteractive
+ENV TZ=UTC
+
+# Update and install essential packages (excluding nodejs/npm for now)
+RUN apt-get update && apt-get install -y \
+    curl \
+    wget \
+    git \
+    build-essential \
+    sudo \
+    python3 \
+    python3-pip \
+    vim \
+    nano \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Node.js 20.x (LTS) from NodeSource
+RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \
+    apt-get install -y nodejs && \
+    rm -rf /var/lib/apt/lists/*
+
+# Install global npm packages for TypeScript development
+RUN npm install -g \
+    typescript \
+    ts-node \
+    @types/node \
+    npm@latest
+
+# Create developer user with sudo privileges
+RUN useradd -m -s /bin/bash developer && \
+    echo "developer:developer" | chpasswd && \
+    usermod -aG sudo developer && \
+    echo "developer ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
+
+# Switch to developer user
+USER developer
+WORKDIR /home/developer
+
+# Create project directory
+RUN mkdir -p /home/developer/s5.js
+
+# Set up npm global directory for the developer user
+RUN mkdir -p /home/developer/.npm-global && \
+    npm config set prefix '/home/developer/.npm-global' && \
+    echo 'export PATH=/home/developer/.npm-global/bin:$PATH' >> /home/developer/.bashrc
+
+# Expose ports
+# 5522 for Enhanced s5.js
+# 5523 for external access
+EXPOSE 5522 5523
+
+# Set the working directory
+WORKDIR /home/developer/s5.js
+
+# Keep container running
+CMD ["/bin/bash"]

Dockerfile.mock

@@ -0,0 +1,9 @@
+FROM node:20-slim
+WORKDIR /app
+COPY package*.json ./
+COPY test/integration/test-server.js ./test/integration/
+COPY dist ./dist
+RUN npm install express
+EXPOSE 5524
+ENV PORT=5524
+CMD ["node", "test/integration/test-server.js"]

Dockerfile.prod

@@ -0,0 +1,49 @@
+# Production Dockerfile for S5.js Server
+FROM node:20-alpine
+
+# Install required system dependencies
+RUN apk add --no-cache \
+    python3 \
+    make \
+    g++ \
+    && rm -rf /var/cache/apk/*
+
+# Create app directory
+WORKDIR /app
+
+# Copy package files
+COPY package*.json ./
+
+# Install production dependencies only
+RUN npm ci --only=production && \
+    npm cache clean --force
+
+# Copy only necessary production files
+COPY dist/ ./dist/
+COPY server-real-s5.js ./
+
+# Create a non-root user to run the app
+RUN addgroup -g 1001 -S nodejs && \
+    adduser -S nodejs -u 1001
+
+# Create directory for seed file mount
+RUN mkdir -p /home/nodejs/.s5 && \
+    chown -R nodejs:nodejs /home/nodejs/.s5
+
+# Switch to non-root user
+USER nodejs
+
+# Expose the S5 server port
+EXPOSE 5522
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
+    CMD node -e "require('http').get('http://localhost:5522/health', (r) => process.exit(r.statusCode === 200 ? 0 : 1))"
+
+# Default to real mode, can be overridden
+ENV S5_MODE=real
+ENV PORT=5522
+ENV NODE_ENV=production
+
+# Start the server
+CMD ["node", "server-real-s5.js"]