Error: Error inserting data: SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON

[ShivanshJ]

This is my file db.ts:

import { EntityDB } from "@babycommando/entity-db";


let db: EntityDB | null = null;

// Initialize function to be called when needed
const initDB = async () => {
  if (!db) {
    db = new EntityDB({
      vectorPath: "spotter-db",
      model: "Xenova/all-MiniLM-L6-v2",
    });
  }
  return db;
};


const ensureDB = async () => {
  const database = await initDB();
  if (!database) {
    throw new Error("[SpotterDB] Database not initialized");
  }
  return database;
};



export const storage = {


  async savePage(url: string, content?: string): Promise<string> {
    const db = await ensureDB();

    const json_obj = {
      text: "check", 
      url,
      timestamp: new Date().toISOString(),
    };
    console.log("[SpotterDB] Saving page:", json_obj);

      if (!db){
        console.error("[SpotterDB] DB NOT init()");
      }
      const key = await db.insert(json_obj);
      console.log("[SpotterDB] Page saved successfully");
      return key;
    
    }
  },

I'm constantly getting this error:
Error: Error inserting data: SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON

The reason I have a complicated initialization is because the normal one was also giving the same error. So I wanted to ensure that my db is initialized before I do an insert. (to keep my bases covered)

But whenever I do storage.savePage(url, content) , iget the error :(

[babycommando]

Hi there ::) I suspect this to be a JS error, not really related to the database functionality.

Can you please share the data you tried to embed? The error suggests that an HTML response (maybea 404 or an error page?) might have ended up inside your JSON instead of actual text. Maybe the content you're trying to store contains an unexpected HTML document? If possible do a console.log over it and show it to me.

Additionally, I think you should not over complicate the database initialization.
Use it as:

  const db = new EntityDB({
    vectorPath: "embedding",
  });

(don't even need to tell the model if you are using Xenova/all-MiniLM-L6-v2, as it is the default one.)

[777Portal]

Im using vite and also had the same issue.
I tried making a separate test file

import { pipeline } from "@xenova/transformers";

// * **Example:** Calculating embeddings with `sentence-transformers` models.
// javascript
const extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
const output = await extractor('This is a simple test.', { pooling: 'mean', normalize: true });

Which threw the same error. After investigating I found This issue on transformers.js. Updating my test file to use

import { pipeline, env } from '@xenova/transformers';
env.allowLocalModels = false;

Worked on the test file, so I simply added env.allowLocalModels = false; to the src of the node module and it worked.

Its kinda hacky to just add it to the src of the module, it would be a good addition to have an argument in the creation function to disable using local models.

[MrMalfunction]

Same Issue

Im using vite and also had the same issue. I tried making a separate test file

import { pipeline } from "@xenova/transformers";

// * **Example:** Calculating embeddings with `sentence-transformers` models.
// javascript
const extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
const output = await extractor('This is a simple test.', { pooling: 'mean', normalize: true });

Which threw the same error. After investigating I found This issue on transformers.js. Updating my test file to use

import { pipeline, env } from '@xenova/transformers';
env.allowLocalModels = false;

Worked on the test file, so I simply added env.allowLocalModels = false; to the src of the node module and it worked.

Its kinda hacky to just add it to the src of the module, it would be a good addition to have an argument in the creation function to disable using local models.

This works for me, thanks a lot! (PS: had to comment this line too env.localModelPath = "/huggingface";)
Maybe Pipeline env should be exposed for user to configure ?