PreProcessor node should return the documents with same ID

[r4881t]

The PreProcessor node returns the documents with IDs changed. This is troublesome, when the ID guarantee is needed.

Ideally, I would expect the PreProcessor to return documents with same IDs.

A simple example to show this

import os, sys
from haystack.document_stores import OpenSearchDocumentStore
from haystack.schema import Document

document_store = OpenSearchDocumentStore(
      host=os.getenv("ES_HOST", "localhost"),
      port=os.getenv("ES_PORT", 9200),
      username=os.getenv("ES_USER", ""),
      password=os.getenv("ES_PASSWORD", ""),
      scheme='https',
      embedding_dim=384,
      index="test_1",
      verify_certs=False)

docs_data = [{
  "id": "my_id1",
  "content": "Something",
  "meta": {
    "dappid": "my_id1",
    }
}]

docs = [Document(id=x['id'].strip().lower(), content=x['content'], meta=x['meta'], id_hash_keys=['meta.dappid']) for x in docs_data]

print(docs[0].id) # Shows "my_id1"
document_store.write_documents(docs)


from haystack.nodes import PreProcessor
processor = PreProcessor(
      clean_empty_lines=True,
      clean_whitespace=True,
      clean_header_footer=True,
      remove_substrings=None,
      split_by="word",
      split_length=512,
      split_respect_sentence_boundary=True,
      split_overlap=32,
      max_chars_check= 10_000
    )

docs2 = processor.process(docs)

print(docs2[0].id) # 891f4dc8a56bf76bd7589675fb5e1004
document_store.write_documents(docs2)

[julian-risch]

Hello @r4881t Depending on the length of the input documents, the Preprocessor splits up a long document into multiple smaller documents. That's why when using the Preprocessor, we cannot guarantee that the document ids remain unchanged. Instead, you can provide id_hash_keys, which are the names of the fields of a document that will be used to calculate a hash. So when you set id_hash_keys=['meta.dappid'], we will calculate the hash of dappid here: https://github.com/deepset-ai/haystack/blob/v1.x/haystack/schema.py#L156