Skip to content

feat(decoder): Add support for loading Zstandard-compressed text files (resolves #272). #368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

feat(decoder): Add support for loading Zstandard-compressed text files (resolves #272). #368

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
e2fbeee
snapshot
Henry8192 Aug 18, 2025
ef95eda
zstd files now can be opened
Henry8192 Aug 20, 2025
2ddcdae
add PlainTextDecoder & refactor Zstd Decoder logic
Henry8192 Aug 21, 2025
50aa871
address suggestions
Henry8192 Aug 21, 2025
680fb22
Merge branch 'main' into zstd-decompression
junhaoliao Aug 25, 2025
629b5e5
Merge remote-tracking branch 'yscope/main' into zstd-decompression
Henry8192 Aug 25, 2025
bf412c8
revert changes
Henry8192 Aug 26, 2025
6b669d8
address code review changes; switch to zstddec for decompression
Henry8192 Aug 26, 2025
6ef328f
add back break
Henry8192 Aug 26, 2025
928f5f7
bump zstddec
Henry8192 Aug 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions package-lock.json
View file
Open in desktop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems some dependencies are getting updated / removed unexpectedly. can we

git checkout 7c008d95de908cd73e5fc1090e24157a9d614a18 -- package-lock.json
npm i zstddec

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
},
"homepage": "https://github.com/y-scope/yscope-log-viewer#readme",
"dependencies": {
"@bokuweb/zstd-wasm": "^0.0.27",
"@emotion/react": "^11.14.0",
"@emotion/styled": "^11.14.0",
"@mui/icons-material": "^7.1.0",
Expand Down
1 change: 0 additions & 1 deletion src/services/LogFileManager/decodeUtils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ const tryCreateDecoderBySignature = async (
};
} catch (e) {
console.warn(`Magic number matches ${entry.name}, but decoder creation failed:`, e);
break;
}
}
}
Expand Down
21 changes: 21 additions & 0 deletions src/services/decoders/ZstdDecoder/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
import {
decompress,
init,
} from "@bokuweb/zstd-wasm";

import PlainTextDecoder from "../PlainTextDecoder";


class ZstdDecoder extends PlainTextDecoder {
private constructor (logArray: Uint8Array) {
super(logArray);
}

static override async create (dataArray: Uint8Array) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Add missing DecoderOptions parameter.

The method signature is missing the decoderOptions parameter that should match the interface expected by other decoders in the codebase.

Apply this diff to add the missing parameter:

-    static override async create (dataArray: Uint8Array) {
+    static override async create (dataArray: Uint8Array, _decoderOptions: DecoderOptions) {

You'll also need to import the DecoderOptions type:

+import {DecoderOptions} from "../../../typings/decoders";
+
 import PlainTextDecoder from "../PlainTextDecoder";
🤖 Prompt for AI Agents 
In src/services/decoders/ZstdDecoder/index.ts around line 7, the static create
method is missing the decoderOptions parameter expected by other decoders;
update the signature to accept a second parameter named decoderOptions typed as
DecoderOptions and add an import for the DecoderOptions type at the top of the
file; ensure the implementation forwards or uses decoderOptions consistently
with other decoder implementations in the codebase.

await init();
const logArrayBuffer = decompress(dataArray);
return new ZstdDecoder(logArrayBuffer);
}
}

export default ZstdDecoder;
11 changes: 10 additions & 1 deletion src/typings/file.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import ClpIrDecoder from "../services/decoders/ClpIrDecoder";
import {CLP_IR_STREAM_TYPE} from "../services/decoders/ClpIrDecoder/utils";
import JsonlDecoder from "../services/decoders/JsonlDecoder";
import PlainTextDecoder from "../services/decoders/PlainTextDecoder";
import ZstdDecoder from "../services/decoders/ZstdDecoder";
import {Decoder} from "./decoders";


Expand All @@ -21,7 +22,8 @@ interface FileTypeInfo {
* Represents a file type with its identifying properties and decoder.
*/
interface FileTypeDef {
DecoderFactory: typeof ClpIrDecoder | typeof JsonlDecoder | typeof PlainTextDecoder;
DecoderFactory: typeof ClpIrDecoder | typeof JsonlDecoder |
typeof ZstdDecoder | typeof PlainTextDecoder;
Comment on lines +25 to +26
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Avoid ever-growing unions for DecoderFactory; use a structural static type.

Every new decoder requires editing this union. Define a structural type for the static side (create()) and reference that here to improve maintainability.

Apply this diff within the selected lines:

-    DecoderFactory: typeof ClpIrDecoder | typeof JsonlDecoder |
-        typeof ZstdDecoder | typeof PlainTextDecoder;
+    DecoderFactory: DecoderStatic;

Add this type alias near the top of the file (e.g., above interface FileTypeDef):

type DecoderStatic = {
  // Matches the static factory shape used by decodeUtils
  create: (dataArray: Uint8Array) => Promise<Decoder>;
};
🤖 Prompt for AI Agents 
In src/typings/file.ts around lines 25 to 26, the DecoderFactory union grows
with every new decoder; define a structural static type instead and use it here:
add a type alias named DecoderStatic near the top of the file (e.g., above
interface FileTypeDef) that declares a create: (dataArray: Uint8Array) =>
Promise<Decoder> method to represent the decoder's static factory shape, then
replace the current DecoderFactory union with the new DecoderStatic type so new
decoders no longer require editing this union.


checkIsStructured: (decoder: Decoder) => FileTypeInfo["isStructured"];
extensions: FileTypeInfo["extension"][];
Expand All @@ -46,6 +48,13 @@ const FILE_TYPE_DEFINITIONS: FileTypeDef[] = [
name: "JSON Lines",
signature: ["{".charCodeAt(0)],
},
{
DecoderFactory: ZstdDecoder,
checkIsStructured: () => false,
extensions: [".zst", ".zstd"],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a good mindset to brainstorm different file extensions, though in the IANA registry only .zst is claimed: https://www.iana.org/assignments/media-types/application/zstd

Suggested change
extensions: [".zst", ".zstd"],
extensions: [".zst"],

name: "Zstandard",
signature: [0x28, 0xb5, 0x2f, 0xfd],
},
Comment on lines +51 to +57
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick (assertive)

DRY the Zstd magic bytes into a shared constant to prevent drift.

Both CLP IR and Zstandard rely on the same magic bytes. Centralise them to avoid subtle mismatches later.

Apply within the selected lines:

-        signature: [0x28, 0xb5, 0x2f, 0xfd],
+        signature: ZSTD_MAGIC,

Add this near the top of the file (close to other typings) and use it for the CLP IR entry as well:

const ZSTD_MAGIC = [0x28, 0xb5, 0x2f, 0xfd] as const;
🤖 Prompt for AI Agents 
In src/typings/file.ts around lines 51 to 57, the Zstandard magic bytes are
duplicated; DRY them by adding a shared constant near the top of the file with
the other typings (e.g., declare ZSTD_MAGIC as a readonly/const tuple of the
four bytes) and then replace the inline signature arrays for both the Zstandard
entry and the CLP IR entry to reference that constant instead of repeating the
literal bytes.

💡 Verification agent

🧩 Analysis chain

Guard against decompression bombs; confirm streaming/lazy WASM init.

Zstandard content can expand massively. Ensure ZstdDecoder enforces a sane max-out size or uses streaming, and lazily imports the WASM so non-Zstd users don’t pay the startup cost.

Run:

If limits/streaming aren’t present, I can draft a guard that rejects when the projected decompressed size exceeds a threshold and documents the UX.

🏁 Script executed:

#!/bin/bash
# ZstdDecoder should import the WASM lazily (dynamic import) inside create().
rg -nP -C3 '@bokuweb/zstd-wasm' src/services/decoders/ZstdDecoder
rg -nP -C3 'import\(\s*[\'"]@bokuweb/zstd-wasm[\'"]\s*\)' src/services/decoders/ZstdDecoder

# Check for size limits / streaming usage.
rg -nP -C3 '(max|limit|cap).*bytes' src/services/decoders/ZstdDecoder
rg -nP -C3 'decompress(Stream|ing)?' src/services/decoders/ZstdDecoder
rg -nP -C3 '(AbortSignal|controller|cancel|abort)' src/services/decoders/ZstdDecoder

# Confirm bundler config excludes the WASM from pre-bundling (as mentioned in the PR).
rg -nP -C3 'optimizeDeps\s*:\s*\{[^}]*exclude[^}]*@bokuweb/zstd-wasm' vite.config.ts

Length of output: 751

This will confirm whether the decoder still uses a static import, lacks size guards, and if the Vite config properly excludes the Zstd WASM from eager bundling.

🏁 Script executed:

#!/bin/bash
# Show static imports from @bokuweb/zstd-wasm in ZstdDecoder
rg -n "@bokuweb/zstd-wasm" -C3 src/services/decoders/ZstdDecoder/index.ts || true

# Check for any dynamic imports (should be none if static import present)
rg -n "import(" -C3 src/services/decoders/ZstdDecoder/index.ts || true

# Inspect the create() method to see where WASM is initialized
rg -n "create" -C3 src/services/decoders/ZstdDecoder/index.ts || true

# Look for any cap/limit/max guards in the decompression logic
rg -nE "(limit|max|cap)" -C3 src/services/decoders/ZstdDecoder/index.ts || true

# Search for streaming or chunked decompress calls
rg -n "stream" -C3 src/services/decoders/ZstdDecoder/index.ts || true

# Check for AbortSignal/controller usage in decoder (to allow cancellation)
rg -nE "(AbortSignal|controller|cancel|abort)" -C3 src/services/decoders/ZstdDecoder/index.ts || true

# Locate vite config file(s) to verify exclude of @bokuweb/zstd-wasm
rg -l "optimizeDeps" -g "vite.config.*" || echo "No vite.config.* found"
rg -n "optimizeDeps" -C3 vite.config.ts vite.config.js || true
rg -n "exclude" -C3 vite.config.ts vite.config.js || true

Length of output: 2080

ZstdDecoder eagerly loads WASM and lacks decompression safeguards

Our verification shows the decoder still statically imports @bokuweb/zstd-wasm and calls its APIs without any output‐size limits or streaming/abort support:

  • Static import at module top (src/services/decoders/ZstdDecoder/index.ts:1–4)
  • create() calls await init() and decompress(dataArray) directly (index.ts:14–17)
  • No checks for maximum decompressed size (no max/limit guards)
  • No streaming or chunked decompression use
  • No AbortSignal or cancellation mechanism
  • Vite config does exclude the WASM (optimizeDeps.exclude: ["@bokuweb/zstd-wasm"] in vite.config.ts:24–30), but that only prevents pre-bundling—it does not defer runtime loading for non-Zstd paths

Recommendations to address both performance and safety concerns:

  • Lazy-load the WASM: move the import into create(), e.g.:

    const { init, decompress } = await import('@bokuweb/zstd-wasm');
await init();

    This ensures non-Zstd consumers don’t pay the WASM startup cost.

  • Enforce a max-output threshold: after decompression, reject if the size exceeds a sane limit:

    const output = decompress(dataArray);
if (output.byteLength > MAX_DECOMPRESSED_BYTES) {
  throw new Error('Decompressed size exceeds safe limit');
}

    Document the user-facing error/UX for size rejections.

  • Consider streaming (if supported by the WASM API) to process chunks and cap memory growth.

  • Add cancellation support via an AbortSignal to allow callers to abort long-running decompressions.

These changes are critical to prevent decompression-bomb attacks and optimize startup performance for users who never decode Zstandard payloads.

🤖 Prompt for AI Agents 
In src/typings/file.ts around lines 51 to 57 the Zstd decoder entry references a
decoder that currently eagerly loads @bokuweb/zstd-wasm and lacks
safety/abort/streaming measures; update the decoder implementation (not this
index) so create() lazy-imports the WASM runtime at call-time (use dynamic
import inside create), initialize it there, enforce a configurable
MAX_DECOMPRESSED_BYTES check and throw/reject if exceeded, add optional
AbortSignal support so callers can cancel long-running decompression, and if the
WASM API allows, implement chunked/streaming decompression to bound memory
growth; ensure Vite exclusion remains but runtime load is deferred and document
the size-limit error for callers.

{
DecoderFactory: PlainTextDecoder,
checkIsStructured: () => false,
Expand Down
8 changes: 7 additions & 1 deletion vite.config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,13 @@ export default defineConfig({
optimizeDeps: {
// Exclude the worker or else `ClpFfiJs-worker.wasm` won't be found when served from the
// debug server (this option only affects debug builds).
exclude: ["clp-ffi-js/worker"],
exclude: [
"clp-ffi-js/worker",
"@bokuweb/zstd-wasm",
],
esbuildOptions: {
target: "es2020",
},
},
plugins: [react()],
server: {
Expand Down