Specifying custom IDs for Document objects

[sanjayc2]

Hi,

I am trying to find a way to specify custom ids for Documents when they are created using a Converter. In particular, I am unable to do this for a JSON Converter. Neither the init nor the run methods take ids as input. Could someone please help?

Thanks

[anakin87]

Hello!

I agree with you that currently it is not possible to specify the ID when using a converter (specifically JSONConverter).

As a workaround, you can store this information in meta (which is accepted in the run method of converters) and then update the ID after document conversion.

If we see high demand for this feature, we may decide to implement it in the future.

[sanjayc2]

Hi Stephano,

Thanks. I will try that. btw, the reason for my question is that I ran into an issue with the ids generated by Haystack itself.. I had about 4000 queries, and it generated about 25-30 duplicates after I ran the JSONConverter. I need to do some preprocessing of the Documents before creating the indexing pipeline (with the document store). The documentation does say that there could be duplicates, but I did not expect an issue to arise with such few queries. If there is an alternative solution (for my situation) to the workaround you've proposed, please let me know.

Thanks for your help.
Sanjay